19:04:18 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:04:18 <zodbot> Meeting started Mon Jan 30 19:04:18 2023 UTC.
19:04:18 <zodbot> This meeting is logged and archived in a public location.
19:04:18 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
19:04:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:04:18 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:04:43 <phsmoura> #chair nirik smooge
19:04:43 <zodbot> Current chairs: nirik phsmoura smooge
19:04:54 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:04:54 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:05:02 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:05:03 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily
19:05:08 <nirik[m]> morning everyone. Hope everyone had a good weekend...
19:05:22 <dherrera> morning :)
19:05:27 <phsmoura> morning
19:05:40 <nirik[m]> I'm trying to use matrix more these days (via nheko client currently). The bridge seems slow today, so I might have to go back to irc.
19:06:15 <nirik[m]> Hey Diego H (he/him)! welcome
19:06:39 <dherrera> I'm using the same matrix client xD
19:07:04 <phsmoura> #info reminder: speak up if you want to work on a ticket!
19:07:05 <phsmoura> #topic Tickets needing review
19:07:15 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:07:17 <nirik[m]> Nheko? it's gotten much better in the very latest version... does everything element does. :)
19:08:19 <nirik[m]> Pile of tickets from friday / over the weekend. :)
19:08:32 <phsmoura> .ticket 11105
19:08:33 <zodbot> phsmoura: Issue #11105: Planned Outage -pagure.io - 2023-01-31 13:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11105
19:08:58 <phsmoura> does it need more tags?
19:09:00 <nirik[m]> I just need someone to review this and make sure it looks right and I didn't make any typos and then I can send it out.
19:09:25 <nirik[m]> Yeah, med med ops ? and waiting on external
19:09:31 <nirik[m]> can someone review it for me?
19:10:42 <phsmoura> just the ticket nirik? seems there isnt any typos
19:11:07 <nirik[m]> Yeah, make sure I didn't typo or list the wrong date or anything. ;)
19:11:17 <nirik[m]> If it looks ok, I can send it after this standup.
19:12:24 <dherrera> it seems fine
19:12:25 <phsmoura> looks good :)
19:12:36 <nirik[m]> Great. Thanks!
19:13:00 <phsmoura> .ticket 11106
19:13:01 <zodbot> phsmoura: Issue #11106: Can't add people to FAS groups in FAS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11106
19:13:58 <nirik[m]> low pri, med gain, ops
19:14:08 <nirik[m]> we need to try and figure out why it's not working for them. ;(
19:14:56 <phsmoura> +1
19:15:03 <phsmoura> .ticket 11108
19:15:04 <zodbot> phsmoura: Issue #11108: [GitLab] Enable FAS group membership sync for fedora/council, fedora/dei, fedora/mindshare - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11108
19:15:45 <phsmoura> same?
19:16:02 <nirik[m]> Yep. Sounds fine.
19:16:11 <nirik[m]> I can do this after the meeting... can walk thru the process in noc.
19:16:28 <nirik[m]> but really I guess it's all just clicking in gitlab
19:16:51 <phsmoura> .ticket 11111
19:16:54 <zodbot> phsmoura: Issue #11111: fedora.is mirror slow download speed - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11111
19:17:19 <nirik[m]> asked the user to check a setting here...
19:17:26 <nirik[m]> but low pri / low gain / ops?
19:17:48 <phsmoura> +1
19:18:23 <phsmoura> #info https://pagure.io/releng/issues?status=Open
19:18:40 <phsmoura> we dont have tickets in releng :)
19:19:09 <phsmoura> #topic Planning / Upcoming work / Discussion
19:19:42 <phsmoura> nirik Im having trouble to run oc commands
19:20:04 <nirik[m]> I've got a few small things to finish up, then I hope to do more rhel9 reinstalls... \
19:20:08 <phsmoura> seems that I can login but it also throws an error and I cant "get pods"
19:20:09 <nirik[m]> Oh? whats it doing?
19:20:28 <nirik[m]> whats the error?
19:20:37 <phsmoura> panic: runtime error: invalid memory address or nil pointer dereference
19:20:51 <nirik[m]> ah yes. I know this one...
19:21:18 <nirik[m]> you are using the 'oc' command from fedora's origin-clients rpm... thats not compatible with openshift4. So, you need to download the openshift4 one.
19:22:26 <phsmoura> Ahh ok, so need to remove that pkg and download from oficial website?
19:22:51 <nirik[m]> https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz
19:23:12 <nirik[m]> yeah. ;( I don't know why origin-clients isn't updated...
19:24:25 <nirik[m]> it trips up pretty much everyone. :)
19:27:16 <phsmoura> worked
19:27:20 <phsmoura> thanks :)
19:28:21 <nirik[m]> Excellent. 😄
19:28:50 <phsmoura> Im having ErrImagePull error. will try to fix but if I spend too much time on this will ping you or darknao :)
19:28:55 <nirik[m]> Diego H (he/him): any progress on mbs in containers? or have you had a chance to look?
19:29:12 <nirik[m]> phsmoura: ok.
19:29:43 <dherrera> I was able to build it on copr, but i've been having some dependency problems on epel7-infra target
19:30:31 <nirik[m]> Ah... I thought we were going to look at moving it to openshift... although I suppose it still has to build for that too
19:31:08 <nirik[m]> The current vm is rhel7... so if we get that working it would unblock us there too.
19:31:32 <dherrera> I checked if it was posible to build it on rhel9
19:31:51 <dherrera> it's possible, but it needs lot's of dependencies XD https://copr.fedorainfracloud.org/coprs/dherrera/mbs-epel9/packages/
19:32:26 <nirik[m]> wow. Yeah, thats not great.
19:33:16 <dherrera> most of them build fine as they are, but some are incompatible upgrades and others most likely wont get ported (like python-mock)
19:33:19 <nirik[m]> So, yeah, if we can get it building/working in epel7-infra we can at least have more time to decide what to do with it down the road.
19:33:57 * nirik[m] wonders if epel8 would be worth looking at...
19:34:18 <dherrera> https://copr.fedorainfracloud.org/coprs/dherrera/mbs-epel8/packages/
19:34:21 <nirik[m]> but there we have all the python3 fun... so meh
19:34:46 <dherrera> but epel8 eol is not that far either
19:34:50 <nirik[m]> better than 9!
19:35:04 <smooge> sorry. was in meetings
19:35:34 <smooge> the only thing I saw this weekend was a couple of nrpe deaths on the ppc builder
19:35:49 <nirik[m]> It might be that down the road we want to deploy this like we do bodhi (partly) and noggin now... ie, in a container with pip/etc... not sure.
19:36:31 <nirik[m]> I mean really ideally we would just retire it. ;) But sadly people are still using it...
19:37:02 <smooge> there was something I read about differences in how fedora and rhel boot memory wise on page layouts.
19:37:13 <nirik[m]> smooge: yeah, I was gonna try some newer / older kernels there. It didn't use to have any problems.... recent kernels just are more memory / swappy
19:38:29 <smooge> the difference is that rhel uses THP=always and fedora uses THP=madvise . I don't know if the transparent huge pages are the cause or if there are other kernel issues
19:39:16 <smooge> I don't know when the boxes switched from having RHEL as the base OS so not sure how it might show up
19:39:28 <nirik[m]> smooge: I doubt that, because when these were back on 5.x fedora kernels they were fine. Only after 6.0/6.1 has it been doing this. But it's very hard to pin down for a bug report... so I have been hoping to try newer and hope that it goes away.
19:39:29 <smooge> [sorry trying to type while finishing a meeting]
19:39:51 <smooge> ok cool. I wasn't sure how long ago this had been happening
19:39:59 <smooge> I only became aware of it over winter break
19:40:03 <nirik[m]> They were switched quite a while back when I was told rhel server on ppc64le didn't handle fedora vm's fully well...
19:40:30 <nirik[m]> it was before then... mid last year? before? Just before break is when they got updated to 6.0/6.1
19:40:46 <smooge> ah ok. I had not realized I had been away for nearly 2 years now :)
19:41:13 <nirik[m]> Seems like you never left! 😀
19:41:26 <phsmoura> sorry to interrupt, but maybe we can close the meeting and continue the discussion on fedora-noc?
19:41:29 <smooge> or was it shorter.. Dec 2020?
19:41:34 <smooge> yeah
19:41:47 <smooge> sorry phsmoura
19:41:54 <nirik[m]> yep. +1 to close?
19:41:55 <dherrera> +1
19:42:01 <phsmoura> np
19:42:11 <phsmoura> thank you all for joining :)
19:42:13 <phsmoura> #endmeeting