19:02:30 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:02:30 <zodbot> Meeting started Tue Jan 24 19:02:30 2023 UTC. 19:02:30 <zodbot> This meeting is logged and archived in a public location. 19:02:30 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 19:02:30 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:02:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:02:30 <phsmoura> #chair nirik aheath1992 smooge 19:02:30 <zodbot> Current chairs: aheath1992 nirik phsmoura smooge 19:02:30 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting 19:02:30 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops 19:02:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:02:30 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily 19:02:43 <phsmoura> hello 19:02:45 <nirik> ah yes, standup. 19:02:46 <nirik> morning 19:02:58 <phsmoura> #info reminder: speak up if you want to work on a ticket! 19:02:58 <phsmoura> #topic Tickets needing review 19:03:04 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 19:03:19 <phsmoura> .ticket 11101 19:03:20 <zodbot> phsmoura: Issue #11101: f38-rebuild merging into f38 overrode higher EVR pkgs already built (from side tag) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11101 19:03:36 <nirik> I think I fixed this, just waiting for confirmation 19:03:42 <nirik> med gain, low effort, ops? 19:03:43 <phsmoura> ok 19:04:31 <phsmoura> #info https://pagure.io/releng/issues?status=Open 19:04:48 <phsmoura> we dont have tickets in releng 19:05:00 <phsmoura> #topic Planning, Upcoming work and Open floor 19:05:45 <nirik> cool. ;) 19:06:34 <nirik> so, yesterday I did some rhel9 stuff I wanted to share. I reinstalled vmhost-x86-08.stg... the only thing I ran into was that you can no longer have /boot and /boot/efi on raid. So we will need to adjust out setup for that. 19:07:04 <nirik> I'm gonna look at doing some vm's today... probibly next week I'll do a tech talk on it 19:07:23 <nirik> The mass rebuild is all done, we are trying to get a rawhide compsoe working now 19:07:54 <darknao> are you going to use leapp for some rhel8->9 or only reinstall? 19:08:25 <nirik> reinstall. I guess leapp has gotten much better, but clean wipe seems more safe to me still. 19:08:42 <nirik> darknao: later today I can land that pr of yours in stg and we can see how it does. 19:08:56 <smooge> here 19:09:19 <nirik> hey smooge 19:09:25 <darknao> alright, Leapp is much better than with rhel7 to 8, but I agree that it should be used only as of last resort 19:10:29 <smooge> my only report was on power9 systems 19:11:00 <darknao> Leapp is even the preferred method now for some RH product upgrade like on Satellite and Ceph (but I still don't recommend it :p) 19:11:35 <darknao> and regarding the pr, sure, you can ping me when you're ready 19:12:03 <smooge> we had one give an alert early and david found it with a load of 140. It later reported nrpe was dead but none of us could get into at that point. He did a power cycle and the box got back 19:12:31 <smooge> my guess is it oom'd both nrpe and whatever else would allow ssh or login to work 19:12:35 <nirik> yeah, they have been... much more sluggust on recent kernels. 19:12:42 <nirik> sluggish 19:14:09 <smooge> i looked a little and it looked like its a lot of little things which add up to memory and io issues 19:14:27 <smooge> i don't know if it is qemu, kernel, both, etc. 19:14:53 <smooge> and I had not looked at this for 2 years so my guessing is spaghetti at a wall versus useful 19:15:11 <smooge> that was it 19:15:13 <nirik> It might be worth trying newer kernel.. 19:15:18 <nirik> but we can see. 19:15:55 <nirik> phsmoura: whats on your list this week? 19:15:57 <smooge> yeah. i didn't want to try anything at the time beyond watching 19:18:19 <phsmoura> nirik: I really would like to deploy fedora planet asap. Im taking OCP course very quick to see how things works. About communishift where should be pushed images? and should I be able to create a project? cause I cant 19:18:23 <nirik> I didn't really have anything else I don't think 19:18:44 <nirik> phsmoura: it should have given you a 'communishift-planet' project 19:19:24 * nirik looks 19:19:48 <nirik> https://console-openshift-console.apps.fedora.cj14.p1.openshiftapps.com/topology/ns/communishift-planet 19:19:48 <phsmoura> so I should see a project there already? there isnt any project 19:20:32 <nirik> huh. 19:20:40 <nirik> It looks like it didn't add you as an admin... 19:21:13 <nirik> refresh it? 19:22:40 <phsmoura> now I can see it :) 19:23:05 <nirik> I added you to the group. Not sure why the playbook didn't. ;( 19:24:30 <phsmoura> ok only with that access I should be able to push the image to a registry and deploy it? 19:25:09 <nirik> yes, but ideally you would set it up to build the image there. ;) 19:25:29 <phsmoura> ok thanks 19:25:40 <phsmoura> changing the subject to another issue 19:25:49 <phsmoura> .ticket 10877 19:25:50 <zodbot> phsmoura: Issue #10877: Fedora Wiki image upload not working (fatal internal error) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10877 19:26:00 <nirik> ah yeah... that fun. 19:26:17 <smooge> i think at this point it is 'try to upgrade to f37 and see if it breaks things worse' 19:26:18 <nirik> So, is it possible to band-aid/fix that one issue for now? 19:26:26 <phsmoura> the issue persists for the reporter so planning to upgrade fedora and mediawiki pkg? 19:26:45 <nirik> upgrading the wiki is... not super easy. 19:26:47 <smooge> there are several issues going on 19:27:20 <smooge> 1. something is up with their specific file on the system. I don't know if it is a DB issue or with the file itself 19:27:24 <nirik> doing so needs: rebuilding a bunch of rpms in infra tags for the new release, reinstalling the vm to the new release, running the database upgrade. 19:27:56 <smooge> 2. something is wrong with the PHP and mediawiki in that it is using non-typed calls and PHP is set to expect types 19:27:59 <nirik> oh? it doesn't happen for all uploads? 19:28:22 <smooge> i don't think so. phsmoura said he was able to upload something 19:28:27 <smooge> but i might have misunderstood 19:28:39 <nirik> well, I guess the problem is uploading a new version of a file? 19:28:44 <phsmoura> yep, seems its affecting only the reporter now 19:29:13 <nirik> ie, upload File.jpg, then later upload a different one named File.jpg ? 19:29:37 <smooge> yeah. this person as a lot of archived images 19:29:48 <smooge> when I looked at the files in attachments all of the ones looked ok 19:29:59 <smooge> as in file says they are all jpeg etc :) 19:30:11 <smooge> with meta data sayin how big/small/etc 19:31:08 <smooge> i didn't try to copy and check them 19:31:52 <smooge> i was thinking of copying an older version of the jpeg into the spot and seeing if that 'fixed' it but I do not know what metadata the db says about said files 19:32:09 <smooge> and if doing so might blow up other things 19:33:11 <smooge> i am at the end of my limited rope 19:34:20 <nirik> well, copying things might be viable if it's the images that are broken 19:34:28 <nirik> but save off a copy of whats there first. ;) 19:35:01 <nirik> I'd suggest getting jwf on irc/matrix and perhaps trying to debug more interactively? 19:35:27 <nirik> phsmoura: sound possible to you? or what do you think? 19:37:29 <nirik> well, we can also take this out of meeting. ;) 19:37:39 <nirik> shall we end and continue in -noc? 19:37:45 <phsmoura> I dont know if I can help like that, not confident to work with php :/ but maybe I can go back to the issue after taking care of fedora planet? 19:39:25 <phsmoura> ok we should have closed already 19:39:38 <phsmoura> thanks all for attending :) 19:39:50 <phsmoura> #endmeeting