2024-12-12 16:59:36 <@Zlopez:matrix.org> !startmeeting Infrastructure (2024-12-12)
2024-12-12 16:59:38 <@meetbot:fedora.im> Meeting started at 2024-12-12 16:59:36 UTC
2024-12-12 16:59:38 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-12-12)'
2024-12-12 16:59:48 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-12-12 16:59:48 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-12-12 16:59:48 <@Zlopez:matrix.org> !topic ahoy
2024-12-12 16:59:48 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-12-12 16:59:48 <@Zlopez:matrix.org> !meetingname infrastructure
2024-12-12 16:59:48 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-12-12 16:59:49 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-12-12 16:59:56 <@Zlopez:matrix.org> !hi
2024-12-12 16:59:57 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-12-12 17:00:07 <@lenkaseg:matrix.org> !hi
2024-12-12 17:00:08 <@Zlopez:matrix.org> Welcome everyone
2024-12-12 17:00:09 <@zodbot:fedora.im> No Fedora Accounts users have the @lenkaseg:matrix.org Matrix Account defined
2024-12-12 17:01:17 <@thisisyaash:fedora.im> !hi
2024-12-12 17:01:18 <@zodbot:fedora.im> Yashwanth Rathakrishnan (thisisyaash)
2024-12-12 17:01:22 <@james:fedora.im> !hi
2024-12-12 17:01:23 <@zodbot:fedora.im> James Antill (james)
2024-12-12 17:01:26 <@nirik:matrix.scrye.com> morning
2024-12-12 17:01:43 <@Zlopez:matrix.org> How is everyone?
2024-12-12 17:03:00 <@nirik:matrix.scrye.com> tired and annoyed that things are broken... otherwise fine.
2024-12-12 17:03:04 <@humaton:fedora.im> !hi
2024-12-12 17:03:05 <@zodbot:fedora.im> Tomáš Hrčka (humaton) - he / him / his
2024-12-12 17:03:12 <@thisisyaash:fedora.im> fantastic :D
2024-12-12 17:03:44 <@Zlopez:matrix.org> @nirik:matrix.scrye.com I noticed few of them :-/
2024-12-12 17:04:05 <@Zlopez:matrix.org> I'm trying to fix what I messed in the latest changes for release-monitoring
2024-12-12 17:04:36 <@Zlopez:matrix.org> As this is my last day in work for this year, I want to finish what I started
2024-12-12 17:05:04 <@zardian:matrix.org> !hi
2024-12-12 17:05:06 <@Zlopez:matrix.org> Let's continue to your first topic
2024-12-12 17:05:07 <@zodbot:fedora.im> Aman Singh (zardian)
2024-12-12 17:05:16 <@Zlopez:matrix.org> !topic New folks introductions
2024-12-12 17:05:16 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-12-12 17:05:16 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-12-12 17:05:51 <@Zlopez:matrix.org> Anybody new here?
2024-12-12 17:07:08 <@zardian:matrix.org> Not new here, but able to catch back after a long time.
2024-12-12 17:07:12 <@seddik:fedora.im> !hi
2024-12-12 17:07:13 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-12-12 17:07:22 <@Zlopez:matrix.org> @zardian:matrix.org Welcome back
2024-12-12 17:07:42 <@zardian:matrix.org> Thanks, Hello everyone!!
2024-12-12 17:08:14 <@Zlopez:matrix.org> Let's look at who will chair the next meeting :-)
2024-12-12 17:08:15 <@Zlopez:matrix.org> !info magic eight ball says:
2024-12-12 17:08:15 <@Zlopez:matrix.org> !info chair 2025-01-16 - ???
2024-12-12 17:08:15 <@Zlopez:matrix.org> !info chair 2025-01-09 - phsmoura
2024-12-12 17:08:15 <@Zlopez:matrix.org> !topic Next chair
2024-12-12 17:08:15 <@nirik:matrix.scrye.com> welcome
2024-12-12 17:08:52 <@seddik:fedora.im> Can take 16
2024-12-12 17:09:15 <@Zlopez:matrix.org> The next three weeks are cancelled as people are on holidays mostly
2024-12-12 17:09:22 <@Zlopez:matrix.org> @seddik:fedora.im It's yours
2024-12-12 17:09:31 <@Zlopez:matrix.org> !info chair 2025-01-16 - saibug
2024-12-12 17:10:02 <@Zlopez:matrix.org> !info chair 2025-01-23 - ???
2024-12-12 17:10:30 <@Zlopez:matrix.org> Does anybody wants to go for 23th January
2024-12-12 17:10:51 <@Zlopez:matrix.org> We don't need to fill it up now, just asking for volunteer in advance
2024-12-12 17:11:08 <@thisisyaash:fedora.im> i will take it
2024-12-12 17:11:46 <@Zlopez:matrix.org> And sold :-)
2024-12-12 17:11:59 <@Zlopez:matrix.org> !info chair 2025-01-23 - Yaash
2024-12-12 17:12:49 <@Zlopez:matrix.org> I think we are covered for now :-)
2024-12-12 17:13:05 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-12-12 17:13:05 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-12-12 17:13:05 <@Zlopez:matrix.org> !topic announcements and information
2024-12-12 17:13:15 <@Zlopez:matrix.org> Any other announcements?
2024-12-12 17:13:42 <@Zlopez:matrix.org> !info Holidays incoming, people could be unavailable for next few weeks
2024-12-12 17:14:19 <@Zlopez:matrix.org> !info Infrastructure meeting for 19th, 26th December and 2nd January is CANCELLED
2024-12-12 17:14:28 <@Zlopez:matrix.org> Did I miss something?
2024-12-12 17:14:43 <@nirik:matrix.scrye.com> !info happy holidays to everyone!
2024-12-12 17:14:52 <@seddik:fedora.im> LGTM
2024-12-12 17:16:10 <@Zlopez:matrix.org> Let's move to next topic
2024-12-12 17:16:20 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-12-12 17:16:20 <@Zlopez:matrix.org> !info nirik is on call from 2024-12-06 to 2024-12-12
2024-12-12 17:16:20 <@Zlopez:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-12-12 17:16:20 <@Zlopez:matrix.org> !info ??? is on call from 2024-12-13 to 2024-12-19
2024-12-12 17:16:20 <@Zlopez:matrix.org> !topic Oncall
2024-12-12 17:16:36 <@Zlopez:matrix.org> Do we want to have oncall during holidays?
2024-12-12 17:16:50 <@Zlopez:matrix.org> Or should we just ask people to fill tickets?
2024-12-12 17:17:18 <@nirik:matrix.scrye.com> right, lets just unset it and have people file tickets
2024-12-12 17:17:57 <@Zlopez:matrix.org> And set the next one at the next meeting
2024-12-12 17:17:59 <@nirik:matrix.scrye.com> Also, if anyone is working on something, do keep in mind that you want to be carefull not to pull others back from holidays
2024-12-12 17:18:34 <@thisisyaash:fedora.im> quick question, what does this 'on call' means? :)
2024-12-12 17:18:58 <@nirik:matrix.scrye.com> https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2024-12-12 17:18:58 <@Zlopez:matrix.org> !info NOBODY is on call from 2024-12-13 to 2025-01-09
2024-12-12 17:19:48 <@Zlopez:matrix.org> Should we remove the wiki page for oncall from info?
2024-12-12 17:19:59 <@offdutypirate:fedora.im> The link https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ in the above notes in not valid. 
2024-12-12 17:20:12 <@Zlopez:matrix.org> Let me fix that
2024-12-12 17:20:27 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2024-12-12 17:20:57 <@nirik:matrix.scrye.com> yeah, we should nuke that cpe one and wiki one. ;)
2024-12-12 17:21:00 <@Zlopez:matrix.org> Changed it in agenda document
2024-12-12 17:21:46 <@thisisyaash:fedora.im> can i make a PR updating it with current doc page?
2024-12-12 17:21:46 <@thisisyaash:fedora.im> actually, i found that infra/ansible repo README.md in pagure still uses wiki
2024-12-12 17:22:02 <@thisisyaash:fedora.im> actually, i found that infra/ansible repo's README.md in pagure still uses wiki
2024-12-12 17:22:02 <@thisisyaash:fedora.im> can i make a PR updating it with current doc page?
2024-12-12 17:22:09 <@Zlopez:matrix.org> Definitely
2024-12-12 17:22:48 <@thisisyaash:fedora.im> okay then
2024-12-12 17:23:10 <@Zlopez:matrix.org> We are usually not good at keeping READMEs and docs up to date
2024-12-12 17:24:16 <@nirik:matrix.scrye.com> it's hard when there's a flood of higher pri work. ;(
2024-12-12 17:24:18 <@thisisyaash:fedora.im> yup, but it gives more opportunity for new contributors tho XD
2024-12-12 17:24:26 <@Zlopez:matrix.org> As we have oncall "covered" let us look at the next topic
2024-12-12 17:24:35 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2024-12-12 17:24:59 <@Zlopez:matrix.org> @nirik:matrix.scrye.com That is true
2024-12-12 17:25:25 <@Zlopez:matrix.org> @nirik:matrix.scrye.com You were the one on oncall, so the floor is yours for now
2024-12-12 17:25:48 <@nirik:matrix.scrye.com> I don't know of any oncall alerts... or at least I don't recall any
2024-12-12 17:26:34 <@Zlopez:matrix.org> I didn't saw anything as well
2024-12-12 17:27:37 <@Zlopez:matrix.org> !info Go over existing items and fix them
2024-12-12 17:27:37 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2024-12-12 17:27:37 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-12-12 17:27:59 <@nirik:matrix.scrye.com> lets see...
2024-12-12 17:28:00 <@Zlopez:matrix.org> I see only the logdetective in nagios
2024-12-12 17:28:12 <@nirik:matrix.scrye.com> we are down to just that one alert. ;)
2024-12-12 17:28:33 <@nirik:matrix.scrye.com> we have been told we don't need to monitor that instance. It's often off.
2024-12-12 17:28:34 <@Zlopez:matrix.org> That is really good before the holidays :-)
2024-12-12 17:28:51 <@nirik:matrix.scrye.com> but I was unable so far to get nagios to stop doing so. ;( If someone wants to look into it, feel free.
2024-12-12 17:29:08 <@seddik:fedora.im> I can take a look 
2024-12-12 17:29:29 <@Zlopez:matrix.org> I thought it would be simple to remove a machine
2024-12-12 17:29:43 <@seddik:fedora.im> From monitoring?? 
2024-12-12 17:29:51 <@nirik:matrix.scrye.com> there's some variables to get it to not monitor, and I set them, but perhaps I missed one.
2024-12-12 17:30:21 <@nirik:matrix.scrye.com> nagios assembles it's config from inventory with ansible templates. So default is to monitor
2024-12-12 17:31:02 <@nirik:matrix.scrye.com> saibug: if you want to look that would be excellent. ;)
2024-12-12 17:33:18 <@Zlopez:matrix.org> Let's see what the agenda has next for us
2024-12-12 17:33:45 <@nirik:matrix.scrye.com> we could touch on current broken things? ;)
2024-12-12 17:33:55 <@Zlopez:matrix.org> That is a good idea
2024-12-12 17:34:29 <@Zlopez:matrix.org> I saw that @abompard:fedora.im already has some fix in mind for elections
2024-12-12 17:34:59 <@Zlopez:matrix.org> I'm not sure how is the situation about the koji-listener
2024-12-12 17:35:05 <@nirik:matrix.scrye.com> Yeah, I am sorry about breaking that. ;( I am not sure if we need to ask if current elections need to be extended given people cannot currently vote
2024-12-12 17:35:27 <@nirik:matrix.scrye.com> any idea when elections fix might land?
2024-12-12 17:35:32 <@Zlopez:matrix.org> I wanted to vote today as well :-/
2024-12-12 17:35:46 <@nirik:matrix.scrye.com> I thought another option might be pinning the thing that changed and broke it to an older version?
2024-12-12 17:36:06 <@Zlopez:matrix.org> I'm not sure if @abompard:fedora.im is here today and I only saw that in the ticket
2024-12-12 17:36:18 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issue/12330
2024-12-12 17:36:22 <@nirik:matrix.scrye.com> I talked with him eariler in noc...
2024-12-12 17:36:39 <@abompard:fedora.im> Just finishing the fixes! :-)
2024-12-12 17:36:45 <@nirik:matrix.scrye.com> yeah, so... hey!
2024-12-12 17:37:12 <@nirik:matrix.scrye.com> I was going to suggest pinning the flask-wtforms too... but if you have a fix thats probibly better
2024-12-12 17:37:16 <@abompard:fedora.im> I think I got it to work, I still have to commit those in elections, and then it'll need quite a bit of work to fix the unit tests
2024-12-12 17:37:50 <@nirik:matrix.scrye.com> I think if we can just bandaid it for now to get it working and fix it properly later that would be fine
2024-12-12 17:38:18 <@nirik:matrix.scrye.com> BTW, auth also broke on it, but I managed to stumble my way to a fix on that
2024-12-12 17:38:18 <@abompard:fedora.im> yeah that's the plan, the bandaid should be in place right now
2024-12-12 17:38:54 <@abompard:fedora.im> oh yeah when looking at the code I was wondering how auth could work at all 😁
2024-12-12 17:39:29 <@nirik:matrix.scrye.com> yeah... flask-oidc upgraded a bunch and it only was working with the old one...
2024-12-12 17:39:35 <@nirik:matrix.scrye.com> but anyhow... so people can vote again now
2024-12-12 17:40:05 <@Zlopez:matrix.org> @abompard:fedora.im I thought the same when I saw the ansible-lint job for ansible :-D
2024-12-12 17:40:26 <@Zlopez:matrix.org> And it didn't worked, it just silently passed without checking anything
2024-12-12 17:40:32 <@zodbot:fedora.im> kevin gave a cookie to abompard. They now have 118 cookies, 1 of which were obtained in the Fedora 41 release cycle
2024-12-12 17:41:13 <@nirik:matrix.scrye.com> so, that just leaves the koji_sync_listener problem. I recreated the queue, but need to have a message that should get to it happen to see if that helps any.
2024-12-12 17:41:25 <@zodbot:fedora.im> zlopez gave a cookie to abompard. They now have 119 cookies, 2 of which were obtained in the Fedora 41 release cycle
2024-12-12 17:41:30 <@nirik:matrix.scrye.com> and... it just did
2024-12-12 17:41:43 <@zodbot:fedora.im> zlopez gave a cookie to kevin. They now have 705 cookies, 13 of which were obtained in the Fedora 41 release cycle
2024-12-12 17:42:22 <@zodbot:fedora.im> abompard gave a cookie to kevin. They now have 706 cookies, 14 of which were obtained in the Fedora 41 release cycle
2024-12-12 17:42:26 <@nirik:matrix.scrye.com> so... hurray? but why do queues get in this state, and how can we detect them?
2024-12-12 17:42:49 <@abompard:fedora.im> yeah I don't know, maybe we should recreate all the bindings (without deleting the queues)
2024-12-12 17:42:55 <@Zlopez:matrix.org> I thought that we are monitoring the queues
2024-12-12 17:43:11 <@abompard:fedora.im> rabbitmq reported the correct bindings in the UI
2024-12-12 17:43:12 <@nirik:matrix.scrye.com> we are.. the queues exist fine and look right, but don't get any messages they should
2024-12-12 17:43:17 <@abompard:fedora.im> it was just not actually bound
2024-12-12 17:43:38 <@nirik:matrix.scrye.com> Aurélien B: oh, do you know what the batcave_test_watcher queue is for?
2024-12-12 17:43:55 <@abompard:fedora.im> hmm, nothing from the top of my mind
2024-12-12 17:44:00 <@abompard:fedora.im> sounds testish
2024-12-12 17:44:40 <@nirik:matrix.scrye.com> yeah, it's got no consumers and... 79,000 messages and growing
2024-12-12 17:45:11 <@Zlopez:matrix.org> A big one
2024-12-12 17:45:28 <@nirik:matrix.scrye.com> the biggest one.
2024-12-12 17:45:31 <@nirik:matrix.scrye.com> in prod anyhow
2024-12-12 17:45:40 <@Zlopez:matrix.org> Let me switch to open floor as we are talking about everything now :-)
2024-12-12 17:45:42 <@Zlopez:matrix.org> !topic Open Floor
2024-12-12 17:45:59 <@nirik:matrix.scrye.com> sure. ;)
2024-12-12 17:47:05 <@seddik:fedora.im>  Question hehe
2024-12-12 17:47:05 <@seddik:fedora.im> Who will be at FOSDEM ? 
2024-12-12 17:47:13 <@Zlopez:matrix.org> I forgot to mention that the staging instance of Anitya now supports OIDC https://stg.release-monitoring.org/login
2024-12-12 17:47:32 <@Zlopez:matrix.org> I will do a proper release when I'm back from holidays
2024-12-12 17:48:08 <@Zlopez:matrix.org> @seddik:fedora.im I don't plan to
2024-12-12 17:48:35 <@nirik:matrix.scrye.com> I'm not planning to either.
2024-12-12 17:49:03 <@nirik:matrix.scrye.com> However, it's sounding like flock might be around the time of devconf...
2024-12-12 17:49:04 <@Zlopez:matrix.org> But I didn't planned it last year as well and I was there :-D
2024-12-12 17:49:20 <@Zlopez:matrix.org> We will see
2024-12-12 17:49:29 <@seddik:fedora.im> See you there hhh
2024-12-12 17:49:58 <@humaton:fedora.im> I will be there
2024-12-12 17:49:58 <@Zlopez:matrix.org> The Flock is great :-)
2024-12-12 17:50:28 <@seddik:fedora.im> Some times I have trouble finding subjects to work on.. 
2024-12-12 17:50:28 <@seddik:fedora.im> Another question
2024-12-12 17:51:17 <@seddik:fedora.im> Lot of issues, but those issues need update from people.. 
2024-12-12 17:51:47 <@nirik:matrix.scrye.com> yeah, there's a lot of old things where people aren't answering us very well. ;(
2024-12-12 17:52:22 <@seddik:fedora.im> :(
2024-12-12 17:52:41 <@nirik:matrix.scrye.com> saibug: I'll try and finish those last 2 apps for memcached and then we can retire memcached01 and close that out
2024-12-12 17:52:54 <@nirik:matrix.scrye.com> (if you want to do a retire/removal pr for 01)
2024-12-12 17:53:21 <@seddik:fedora.im> Yes I'll look at this later.. 
2024-12-12 17:53:35 <@nirik:matrix.scrye.com> thanks
2024-12-12 17:54:17 <@nirik:matrix.scrye.com> oh, David Kirwan pushed a initial openshift best practices guide:
2024-12-12 17:54:19 <@nirik:matrix.scrye.com> https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/
2024-12-12 17:54:34 <@humaton:fedora.im> saibug: if you have a list of tickets of interest I dont mind going over and pinging people for updates
2024-12-12 17:54:35 <@Zlopez:matrix.org> @dkirwan:fedora.im++
2024-12-12 17:54:35 <@nirik:matrix.scrye.com> we can suggest changes in pr's or discussion/list/whatever
2024-12-12 17:54:37 <@zodbot:fedora.im> zlopez gave a cookie to dkirwan. They now have 9 cookies, 1 of which were obtained in the Fedora 41 release cycle
2024-12-12 17:55:04 <@nirik:matrix.scrye.com> https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift_bestpractices/ I guess is the doc
2024-12-12 17:57:44 <@nirik:matrix.scrye.com> I don't think I have much else...
2024-12-12 17:58:04 <@nirik:matrix.scrye.com> I will be on pto after tomorrow, but will still be around somewhat
2024-12-12 17:59:05 <@Zlopez:matrix.org> I will be on PTO from tomorrow, I'm not sure how much I will be reacting here
2024-12-12 17:59:56 <@Zlopez:matrix.org> I see that we are already at the end of our time
2024-12-12 18:00:00 <@Zlopez:matrix.org> Thanks for coming everyone
2024-12-12 18:00:08 <@Zlopez:matrix.org> !endmeeting