16:00:39 <eddiejenningsjr> #startmeeting Infrastructure (2022-09-29) 16:00:39 <zodbot> Meeting started Thu Sep 29 16:00:39 2022 UTC. 16:00:39 <zodbot> This meeting is logged and archived in a public location. 16:00:39 <zodbot> The chair is eddiejenningsjr. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:00:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:39 <zodbot> The meeting name has been set to 'infrastructure_(2022-09-29)' 16:00:58 <eddiejenningsjr> #meetingname infrastructure 16:00:58 <eddiejenningsjr> #chair nirik mobrien zlopez bodanel dtometzki jnsamyak computerkid 16:00:58 <zodbot> The meeting name has been set to 'infrastructure' 16:00:58 <zodbot> Current chairs: bodanel computerkid dtometzki eddiejenningsjr jnsamyak mobrien nirik zlopez 16:00:58 <eddiejenningsjr> #info Agenda is at: https://board.net/p/fedora-infra 16:00:58 <eddiejenningsjr> #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 16:00:58 <eddiejenningsjr> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra 16:01:15 <eddiejenningsjr> #topic greetings! 16:01:30 <eddiejenningsjr> Let's see if I can get through an Infra meeting without work interrupting me :D 16:01:34 <eddiejenningsjr> .hello eddiejennings 16:01:35 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net> 16:01:40 <nirik> morn 16:01:49 <erolkskn> .hello erolkeskin 16:01:50 <zodbot> erolkskn: erolkeskin 'Erol Keskin' <erolkeskin.dev@gmail.com> 16:02:04 <Penguinpee> .hello gui1ty 16:02:05 <zodbot> Penguinpee: gui1ty 'Sandro .' <gui1ty@penguinpee.nl> 16:02:37 <eddiejenningsjr> #topic New folks introductions 16:02:37 <eddiejenningsjr> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 16:02:37 <eddiejenningsjr> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 16:02:54 <eddiejenningsjr> Do we have any new folks with us today? If so, chime in and introduce yourself :) 16:02:59 <eddiejenningsjr> We won't bite. I promise! 16:03:20 <Penguinpee> Hi. I'm here primarily for fedora-badges. 16:03:41 <nirik> welcome 16:03:44 <erolkskn> welcome sandro 16:03:50 <nirik> and thanks! 16:03:57 <AustinPowered[m]> .hi 16:03:58 <zodbot> AustinPowered[m]: Sorry, but user 'AustinPowered [m]' does not exist 16:04:01 <eddiejenningsjr> Welcome :) 16:04:15 <Penguinpee> Got some questions. Need access. And so on. 16:04:41 <eddiejenningsjr> By being involved with badges, you've made Nirik's day :D 16:04:56 <AustinPowered[m]> .hello2 austinpowered 16:04:57 <zodbot> AustinPowered[m]: Sorry, but user 'AustinPowered [m]' does not exist 16:05:02 <eddiejenningsjr> Any other new folks with us today? 16:05:07 <Penguinpee> He made mine so often. It's time to give back. ;) 16:06:29 <eddiejenningsjr> Looks like there are no more new folk. Let's move along. 16:06:34 <eddiejenningsjr> #topic Next chair 16:06:48 <eddiejenningsjr> #info magic eight ball says: 16:06:49 <eddiejenningsjr> #info chair 2022-09-29 - eddiejenningsjr 16:06:49 <eddiejenningsjr> #info chair 2022-10-06 - petebuffon 16:06:49 <eddiejenningsjr> #info chair 2022-10-13 - lenkaseg 16:07:08 <eddiejenningsjr> Do we have any takers for 10/20? 16:07:25 <lenkaseg> .hi 16:07:26 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz> 16:07:55 * nirik will not be here on the 20th, so not me 16:08:20 <eddiejenningsjr> For anyone who's new, being chair is a fun, easy way to be involved. You get to drive the bus that is this weekly Infra meeting :) 16:08:42 <darknao> .hi 16:08:43 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja> 16:09:44 <eddiejenningsjr> We are set for the next two weeks, so we can revisit this in a week. 16:09:54 <petebuffon> .hi 16:09:54 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com> 16:10:19 <eddiejenningsjr> Welcome folks who've just popped in :D 16:10:21 <eddiejenningsjr> And now . . . 16:10:35 <eddiejenningsjr> #topic announcements and information 16:10:53 <eddiejenningsjr> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting 16:10:53 <eddiejenningsjr> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3 16:10:53 <eddiejenningsjr> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request 16:10:53 <eddiejenningsjr> #info please help us with improving contribution to fedora infra https://discussion.fedoraproject.org/t/improving-contribution-to-fedora-infrastructure/38294/8 16:10:53 <eddiejenningsjr> #info oncall should also handling #fedora-releng pings if possible 16:11:04 <nirik> #info nirk on PTO 2022-10-06 to 2022-10-09 and 2022-10-15 to 2022-10-23 16:11:36 <nirik> #info f37 final freeze starts next tuesday 16:12:20 <eddiejenningsjr> We'll hold down the fort while you're away -- or at least try :) 16:13:23 <eddiejenningsjr> Any other announcements from anyone? 16:13:26 <nirik> 👍🏻 16:13:32 <Penguinpee> Enjoy your holiday! 16:14:33 <eddiejenningsjr> Hearing none, we'll move along. 16:14:44 <eddiejenningsjr> #topic Oncall 16:14:47 <eddiejenningsjr> Everyone's favorite topic :) 16:14:56 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:14:56 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ 16:14:56 <eddiejenningsjr> ## .oncalltakeeu .oncalltakeus 16:15:30 <eddiejenningsjr> Ha! Element interpreted double hashes as a heading 16:15:41 <eddiejenningsjr> #info eddiejenningsjr is on call from 2022-09-23 to 2022-09-29 16:15:41 <eddiejenningsjr> #info ??? is on call from 2022-09-30 to 2022-10-06 16:15:51 <eddiejenningsjr> Do we have a taker for the next on-call week? 16:16:01 <nirik> yep. matrix does md 16:16:21 <nirik> I can take it if noone else wants it 16:16:25 <eddiejenningsjr> On-call isn't too stressful of a job for anyone who's interested but worried. 16:17:29 <eddiejenningsjr> Any other's interested? Else, it will go to nirik 16:17:50 <darknao> I can take it 16:18:02 <eddiejenningsjr> sold to darknao! 16:18:18 <eddiejenningsjr> #info darknao is on call from 2022-09-30 to 2022-10-06 16:18:21 <darknao> .oncalltakeus 16:18:21 <zodbot> darknao: Kneel before zod! 16:19:09 <Penguinpee> 😏 16:19:14 <erolkskn> well zod gives me some creepy "ai is going to take over world" vibes 16:19:24 <eddiejenningsjr> Any takers for 2022-10-07 to 2022-10-13? 16:19:43 <eddiejenningsjr> I'll be out of town for a chuck of that week, so I cannot take it :( 16:19:51 * nirik can't do that one, out of town part of it 16:20:08 <erolkskn> Is it open to community members ? 16:20:22 <erolkskn> i mean can a community member become on-call ? 16:21:32 <eddiejenningsjr> I'm a community member, so I'd say yes. It helps to be in the fi-apprentice group, so you can SSH into some systems and try to do some limited troubleshooting. 16:21:46 <eddiejenningsjr> nirik can speak more authoritatively though :) 16:21:57 <nirik> sure! 16:22:00 <erolkskn> Well, then, sign me up for it :p 16:22:10 <eddiejenningsjr> sold to erolskn! 16:22:15 <nirik> we have a doc on it... 16:22:39 <eddiejenningsjr> #info erolskn is on call from 2022-10-06 to 2022-10-13 16:23:16 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:23:16 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ 16:23:23 <eddiejenningsjr> doh, 10/7 not 10/6 16:23:28 <eddiejenningsjr> #info erolskn is on call from 2022-10-07 to 2022-10-13 16:23:41 <eddiejenningsjr> #info Summary of last week: (from current oncall ) 16:23:51 <eddiejenningsjr> So I was summoned twice this week. 16:24:13 * Penguinpee ducks 16:24:16 <eddiejenningsjr> First for a mailman issue, which I couldn't help much, for I couldn't SSH Into the server :( 16:24:43 <eddiejenningsjr> Second was for a Koji issue which was early in the morning and I was sleeping, but seemed to solve itself. 16:25:24 <Penguinpee> Part of it solved itself. The other part was done by smooge. 16:25:41 <eddiejenningsjr> Ah, excellent :D 16:25:54 <eddiejenningsjr> So other than those pings, it was a quiet week. 16:26:13 <eddiejenningsjr> And now, time for the nirik show! 16:26:15 <eddiejenningsjr> #topic Monitoring discussion [nirik] 16:26:15 <eddiejenningsjr> #info https://nagios.fedoraproject.org/nagios 16:26:15 <eddiejenningsjr> #info Go over existing out items and fix 16:26:44 <nirik> so, not much here again. We did clear up some ghost mgmt interfaces eariler this week in the switch upgrades. 16:26:51 <eddiejenningsjr> One day, I'll get around to setting up Nagios at home :D 16:26:55 <nirik> so, down to just a few odd things that don't matter much 16:27:48 <eddiejenningsjr> Sounds like generally good news. :) 16:28:51 <nirik> yeah, not too bad... someday we will have time to clean up all the little junk thats listed 16:28:53 <eddiejenningsjr> Last week we had a good learning topic on IPA, so this week we'll look at tickets. Next week, Nirik will be kind enough to discuss Koji as a learning topic. 16:28:58 <nirik> but before that we might get moved to zabbix. ;) 16:29:26 <nirik> oh shoot, I should reschedule that... I am out next week... 16:29:27 <eddiejenningsjr> #topic Revisit blocked tickets 16:29:27 <eddiejenningsjr> #info Check if any blocked ticket is unblocked 16:29:27 <eddiejenningsjr> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status= 16:29:50 <eddiejenningsjr> First let's look at currently blocked tickets. 16:30:14 <eddiejenningsjr> We wont' go through them one at a time, but everyone take a moment to look at the list, and if there's any kind of update (maybe removing the block), let's discuss it. 16:30:42 <nirik> all still blocked, but getting closer to being able to retire ocp3. ;) 16:31:29 <eddiejenningsjr> I have a couple of RHEL 9 E-mails to get caught up on, is there any chance that mailman would be a part of that? 16:31:58 <eddiejenningsjr> .ticket 8455 16:31:59 <zodbot> eddiejenningsjr: Issue #8455: Move mailman to newer release of Fedora or CentOS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8455 16:32:55 <nirik> nope, no mailman in rhel9 at all. ;) 16:34:39 <eddiejenningsjr> Ah. Just found its gitlab page, seem like something worthy of my time to read :) 16:35:00 <eddiejenningsjr> Moving along, let's look at the backlog 16:35:13 <eddiejenningsjr> #topic Fedora Infra backlog refinement 16:35:13 <eddiejenningsjr> #info Refine oldest tickets on Fedora Infra tracker 16:35:13 <eddiejenningsjr> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc 16:35:44 <eddiejenningsjr> Just as with blocked tickets, we won't go through each one, but if attendees would take a look at the list, and chime in if there are any updates on a ticket. 16:36:39 <eddiejenningsjr> .ticket 9892 16:36:40 <zodbot> eddiejenningsjr: Issue #9892: Centralise and update Fedora infra and releng SOPs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9892 16:36:50 <eddiejenningsjr> I do know Infra docs have been moved. Any news on the releng docs? 16:38:11 <nirik> they were going to be moved this week last I heard... 16:38:16 <nirik> but I haven't seen it yet. 16:38:45 <eddiejenningsjr> I'm actually looking for the on-call doc within the new docs site to replace the one I linked earlier :) 16:40:57 <nirik> https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team 16:41:26 <eddiejenningsjr> There we go. Hiding in plain sight. 16:42:08 <eddiejenningsjr> It looks like there's no other discussion for old tickets. Let's move to open floor. 16:42:15 <eddiejenningsjr> #topic Open Floor 16:42:26 <eddiejenningsjr> Anything else to discuss that hasn't already been mentioned? 16:43:18 <erolkskn> Well the things on badges side not seem so great. The latest develop branch is not in a working state and it should be tested heavily before making it to staging 16:43:25 <eddiejenningsjr> Penguinpee: I remember you might have had some questions about badges 16:43:35 <nirik> erolkskn: yeah, I feared as much 16:44:07 <Penguinpee> Well, erolkskn already mentioned me regarding badges. I'm trying to get into fedora-badges Pagure group, but can't get in contact with anyone on it. 16:44:35 <erolkskn> and as far as i can see from the prod logs, there are more couple of issues like fedmsg-hub restarting itself over and over 16:44:57 <Penguinpee> I talked to misc yesterday. I asked him. But he was a bit evasive. 16:45:35 <erolkskn> btw misc was totally right about the performance problems. Some of the badges causes the service to be down 16:45:44 <erolkskn> since it tries to get every record on datanommer with given criteria 16:45:47 <nirik> Penguinpee: the pagure group? 16:45:50 <erolkskn> it causes the memory to ran out of space 16:45:57 <Penguinpee> Further down the road I would like to get commit access on fedora-badges. 16:46:19 <Penguinpee> nirik: Yes. https://pagure.io/group/fedora-badges 16:46:50 <nirik> ok, thats the docs and requests... none of the actual code. ;) but sure I can add you... 16:46:57 <nirik> whats your fas login? 16:47:00 <Penguinpee> I heard something about staging environments, too. 16:47:09 <Penguinpee> .fas gui1ty 16:47:09 <zodbot> Penguinpee: b"gui1ty '' <gui1ty@penguinpee.nl>" 16:47:40 <nirik> erolkskn: would you like to be added there too? 16:47:58 <erolkskn> nirik: it's not needed for me now 16:48:02 <nirik> Penguinpee: the code is all on github (as far as I know) and yes, we have a staging version deployed as well as the production one. 16:48:03 <erolkskn> thank you 16:48:06 <Penguinpee> Commit access on GitHub would also be appreciated. But I can also live with submitting PRs. 16:48:22 <nirik> Penguinpee: can add you there too. whats your github user? 16:48:43 <Penguinpee> nirik: On GitHub it's penguinpee. 16:49:21 <erolkskn> here is my thought regarding the badger. i think rewriting it would be much easier than fixing it 16:50:30 <Penguinpee> Do I need to apply for apprentice for getting read-only access to the production and/or staging machines? 16:50:31 <erolkskn> at least we can just rewrite the "fedbadges" project at first, and proceed to replace other projects like "tahrir" and "tahrir-api" in next. 16:51:17 <eddiejenningsjr> Sometimes rewrites are the way to go. 16:51:28 <nirik> Penguinpee: I can add you to sysadmin-badges if you like... do you have a staging user? 16:51:38 <nirik> erolkskn: yeah, I think we came to that conclusion. 16:51:43 <Penguinpee> I have a much larger plan for tackling things. But I think that's out of scope for this meeting. 16:52:04 <Penguinpee> nirik: Not yet. Where do I need to set this up? 16:52:28 <nirik> accounts.stg.fedoraproject.org 16:52:41 <erolkskn> eddiejenningsjr: in this case i think it's definitely the way to go since either way we should refactor every piece of that software 16:52:58 <erolkskn> in the current state, fedbadges project heavily relies on fedmsg library 16:53:00 <erolkskn> which is deprecated 16:53:17 <erolkskn> and then there is the scripts in ansible. They are relying on fedmsg too 16:53:22 <Penguinpee> Thanks. I'll do that after the meeting, if that's all right. 16:54:01 <erolkskn> and then there is the fact that fedbadges uses datanommer instead of using datagrepper. 16:54:18 <nirik> Penguinpee: sure, just let me know 16:54:19 <Penguinpee> nirik: Can I re-use my existing key for staging or is it preferred to have a separate SSH key? 16:54:36 <nirik> erolkskn / Penguinpee: this may be of interest also: https://pagure.io/fedora-infra/arc/blob/main/f/docs/badges/index.md 16:54:45 <nirik> you can reuse if you like. up to you 16:54:53 <erolkskn> nirik: yeah i've read it 16:55:01 <erolkskn> it pretty much addresses the same problems 16:55:16 * nirik nods. 16:55:30 <erolkskn> nirik: i do have a question regarding the rewriting of this app 16:55:41 <erolkskn> nirik: should we open up this topic in cpe issues ? 16:55:52 <Penguinpee> I might have stumbled on that doc. Will take a closer look. 16:56:37 * Penguinpee stumbled upon so many things the past weeks, he feels like not to be bruised. 16:56:46 <nirik> erolkskn: well, if you want cpe to form a team to try and assist on it... if not, no, probibly not needed. 16:57:25 <Penguinpee> *lucky 16:57:32 <erolkskn> that's great then :) 16:58:19 <erolkskn> Alright then, sandro i'm waiting your thoughts on this too 16:58:34 <Penguinpee> I think that's it from me for now. Feel free to ping me if there are anymore questions. 16:59:10 <Penguinpee> erolkskn: The rewrite? Let's put out the fire first, before rebuilding the house. 16:59:11 <eddiejenningsjr> Excellent. We have about 2 minutes left. Anything else to discuss? 17:00:52 <eddiejenningsjr> We've reached time. The discussion can continue in the Infra channel and elsewhere :) 17:00:58 <eddiejenningsjr> Thanks for coming everyone! 17:01:08 <eddiejenningsjr> #endmeeting