19:02:30 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:02:30 <zodbot> Meeting started Tue Jan 24 19:02:30 2023 UTC.
19:02:30 <zodbot> This meeting is logged and archived in a public location.
19:02:30 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
19:02:30 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:02:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:02:30 <phsmoura> #chair nirik aheath1992 smooge
19:02:30 <zodbot> Current chairs: aheath1992 nirik phsmoura smooge
19:02:30 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:02:30 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:02:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:02:30 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily
19:02:43 <phsmoura> hello
19:02:45 <nirik> ah yes, standup.
19:02:46 <nirik> morning
19:02:58 <phsmoura> #info reminder: speak up if you want to work on a ticket!
19:02:58 <phsmoura> #topic Tickets needing review
19:03:04 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:03:19 <phsmoura> .ticket 11101
19:03:20 <zodbot> phsmoura: Issue #11101: f38-rebuild merging into f38 overrode higher EVR pkgs already built (from side tag) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11101
19:03:36 <nirik> I think I fixed this, just waiting for confirmation
19:03:42 <nirik> med gain, low effort, ops?
19:03:43 <phsmoura> ok
19:04:31 <phsmoura> #info https://pagure.io/releng/issues?status=Open
19:04:48 <phsmoura> we dont have tickets in releng
19:05:00 <phsmoura> #topic Planning, Upcoming work and Open floor
19:05:45 <nirik> cool. ;)
19:06:34 <nirik> so, yesterday I did some rhel9 stuff I wanted to share. I reinstalled vmhost-x86-08.stg... the only thing I ran into was that you can no longer have /boot and /boot/efi on raid. So we will need to adjust out setup for that.
19:07:04 <nirik> I'm gonna look at doing some vm's today... probibly next week I'll do a tech talk on it
19:07:23 <nirik> The mass rebuild is all done, we are trying to get a rawhide compsoe working now
19:07:54 <darknao> are you going to use leapp for some rhel8->9 or only reinstall?
19:08:25 <nirik> reinstall. I guess leapp has gotten much better, but clean wipe seems more safe to me still.
19:08:42 <nirik> darknao: later today I can land that pr of yours in stg and we can see how it does.
19:08:56 <smooge> here
19:09:19 <nirik> hey smooge
19:09:25 <darknao> alright, Leapp is much better than with rhel7 to 8, but I agree that it should be used only as of last resort
19:10:29 <smooge> my only report was on power9 systems
19:11:00 <darknao> Leapp is even the preferred method now for some RH product upgrade like on Satellite and Ceph (but I still don't recommend it :p)
19:11:35 <darknao> and regarding the pr, sure, you can ping me when you're ready
19:12:03 <smooge> we had one give an alert early and david found it with a load of 140. It later reported nrpe was dead but none of us could get into at that point. He did a power cycle and the box got back
19:12:31 <smooge> my guess is it oom'd both nrpe and whatever else would allow ssh or login to work
19:12:35 <nirik> yeah, they have been... much more sluggust on recent kernels.
19:12:42 <nirik> sluggish
19:14:09 <smooge> i looked a little and it looked like its a lot of little things which add up to memory and io issues
19:14:27 <smooge> i don't know if it is qemu, kernel, both, etc.
19:14:53 <smooge> and I had not looked at this for 2 years so my guessing is spaghetti at a wall versus useful
19:15:11 <smooge> that was it
19:15:13 <nirik> It might be worth trying newer kernel..
19:15:18 <nirik> but we can see.
19:15:55 <nirik> phsmoura: whats on your list this week?
19:15:57 <smooge> yeah. i didn't want to try anything at the time beyond watching
19:18:19 <phsmoura> nirik: I really would like to deploy fedora planet asap. Im taking OCP course very quick to see how things works. About communishift where should be pushed images? and should I be able to create a project? cause I cant
19:18:23 <nirik> I didn't really have anything else I don't think
19:18:44 <nirik> phsmoura: it should have given you a 'communishift-planet' project
19:19:24 * nirik looks
19:19:48 <nirik> https://console-openshift-console.apps.fedora.cj14.p1.openshiftapps.com/topology/ns/communishift-planet
19:19:48 <phsmoura> so I should see a project there already? there isnt any project
19:20:32 <nirik> huh.
19:20:40 <nirik> It looks like it didn't add you as an admin...
19:21:13 <nirik> refresh it?
19:22:40 <phsmoura> now I can see it :)
19:23:05 <nirik> I added you to the group. Not sure why the playbook didn't. ;(
19:24:30 <phsmoura> ok only with that access I should be able to push the image to a registry and deploy it?
19:25:09 <nirik> yes, but ideally you would set it up to build the image there. ;)
19:25:29 <phsmoura> ok thanks
19:25:40 <phsmoura> changing the subject to another issue
19:25:49 <phsmoura> .ticket 10877
19:25:50 <zodbot> phsmoura: Issue #10877: Fedora Wiki image upload not working (fatal internal error) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10877
19:26:00 <nirik> ah yeah... that fun.
19:26:17 <smooge> i think at this point it is 'try to upgrade to f37 and see if it breaks things worse'
19:26:18 <nirik> So, is it possible to band-aid/fix that one issue for now?
19:26:26 <phsmoura> the issue persists for the reporter so planning to upgrade fedora and mediawiki pkg?
19:26:45 <nirik> upgrading the wiki is... not super easy.
19:26:47 <smooge> there are several issues going on
19:27:20 <smooge> 1. something is up with their specific file on the system. I don't know if it is a DB issue or with the file itself
19:27:24 <nirik> doing so needs: rebuilding a bunch of rpms in infra tags for the new release, reinstalling the vm to the new release, running the database upgrade.
19:27:56 <smooge> 2. something is wrong with the PHP and mediawiki in that it is using non-typed calls and PHP is set to expect types
19:27:59 <nirik> oh? it doesn't happen for all uploads?
19:28:22 <smooge> i don't think so. phsmoura said he was able to upload something
19:28:27 <smooge> but i might have misunderstood
19:28:39 <nirik> well, I guess the problem is uploading a new version of a file?
19:28:44 <phsmoura> yep, seems its affecting only the reporter now
19:29:13 <nirik> ie, upload File.jpg, then later upload a different one named File.jpg ?
19:29:37 <smooge> yeah. this person as a lot of archived images
19:29:48 <smooge> when I looked at the files in attachments all of the ones looked ok
19:29:59 <smooge> as in file says they are all jpeg etc :)
19:30:11 <smooge> with meta data sayin how big/small/etc
19:31:08 <smooge> i didn't try to copy and check them
19:31:52 <smooge> i was thinking of copying an older version of the jpeg into the spot and seeing if that 'fixed' it but I do not know what metadata the db says about said files
19:32:09 <smooge> and if doing so might blow up other things
19:33:11 <smooge> i am at the end of my limited rope
19:34:20 <nirik> well, copying things might be viable if it's the images that are broken
19:34:28 <nirik> but save off a copy of whats there first. ;)
19:35:01 <nirik> I'd suggest getting jwf on irc/matrix and perhaps trying to debug more interactively?
19:35:27 <nirik> phsmoura: sound possible to you? or what do you think?
19:37:29 <nirik> well, we can also take this out of meeting. ;)
19:37:39 <nirik> shall we end and continue in -noc?
19:37:45 <phsmoura> I dont know if I can help like that, not confident to work with php :/ but maybe I can go back to the issue after taking care of fedora planet?
19:39:25 <phsmoura> ok we should have closed already
19:39:38 <phsmoura> thanks all for attending :)
19:39:50 <phsmoura> #endmeeting