16:00:39 <eddiejenningsjr> #startmeeting Infrastructure (2022-09-29)
16:00:39 <zodbot> Meeting started Thu Sep 29 16:00:39 2022 UTC.
16:00:39 <zodbot> This meeting is logged and archived in a public location.
16:00:39 <zodbot> The chair is eddiejenningsjr. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:39 <zodbot> The meeting name has been set to 'infrastructure_(2022-09-29)'
16:00:58 <eddiejenningsjr> #meetingname infrastructure
16:00:58 <eddiejenningsjr> #chair nirik mobrien zlopez bodanel dtometzki jnsamyak computerkid
16:00:58 <zodbot> The meeting name has been set to 'infrastructure'
16:00:58 <zodbot> Current chairs: bodanel computerkid dtometzki eddiejenningsjr jnsamyak mobrien nirik zlopez
16:00:58 <eddiejenningsjr> #info Agenda is at: https://board.net/p/fedora-infra
16:00:58 <eddiejenningsjr> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:58 <eddiejenningsjr> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:01:15 <eddiejenningsjr> #topic greetings!
16:01:30 <eddiejenningsjr> Let's see if I can get through an Infra meeting without work interrupting me :D
16:01:34 <eddiejenningsjr> .hello eddiejennings
16:01:35 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:01:40 <nirik> morn
16:01:49 <erolkskn> .hello erolkeskin
16:01:50 <zodbot> erolkskn: erolkeskin 'Erol Keskin' <erolkeskin.dev@gmail.com>
16:02:04 <Penguinpee> .hello gui1ty
16:02:05 <zodbot> Penguinpee: gui1ty 'Sandro .' <gui1ty@penguinpee.nl>
16:02:37 <eddiejenningsjr> #topic New folks introductions
16:02:37 <eddiejenningsjr> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:02:37 <eddiejenningsjr> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:02:54 <eddiejenningsjr> Do we have any new folks with us today?  If so, chime in and introduce yourself :)
16:02:59 <eddiejenningsjr> We won't bite.  I promise!
16:03:20 <Penguinpee> Hi. I'm here primarily for fedora-badges.
16:03:41 <nirik> welcome
16:03:44 <erolkskn> welcome sandro
16:03:50 <nirik> and thanks!
16:03:57 <AustinPowered[m]> .hi
16:03:58 <zodbot> AustinPowered[m]: Sorry, but user 'AustinPowered [m]' does not exist
16:04:01 <eddiejenningsjr> Welcome :)
16:04:15 <Penguinpee> Got some questions. Need access. And so on.
16:04:41 <eddiejenningsjr> By being involved with badges, you've made Nirik's day :D
16:04:56 <AustinPowered[m]> .hello2 austinpowered
16:04:57 <zodbot> AustinPowered[m]: Sorry, but user 'AustinPowered [m]' does not exist
16:05:02 <eddiejenningsjr> Any other new folks with us today?
16:05:07 <Penguinpee> He made mine so often. It's time to give back. ;)
16:06:29 <eddiejenningsjr> Looks like there are no more new folk.  Let's move along.
16:06:34 <eddiejenningsjr> #topic Next chair
16:06:48 <eddiejenningsjr> #info magic eight ball says:
16:06:49 <eddiejenningsjr> #info chair 2022-09-29 - eddiejenningsjr
16:06:49 <eddiejenningsjr> #info chair 2022-10-06 - petebuffon
16:06:49 <eddiejenningsjr> #info chair 2022-10-13 - lenkaseg
16:07:08 <eddiejenningsjr> Do we have any takers for 10/20?
16:07:25 <lenkaseg> .hi
16:07:26 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:07:55 * nirik will not be here on the 20th, so not me
16:08:20 <eddiejenningsjr> For anyone who's new, being chair is a fun, easy way to be involved.  You get to drive the bus that is this weekly Infra meeting :)
16:08:42 <darknao> .hi
16:08:43 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:09:44 <eddiejenningsjr> We are set for the next two weeks, so we can revisit this in a week.
16:09:54 <petebuffon> .hi
16:09:54 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:10:19 <eddiejenningsjr> Welcome folks who've just popped in :D
16:10:21 <eddiejenningsjr> And now . . .
16:10:35 <eddiejenningsjr> #topic announcements and information
16:10:53 <eddiejenningsjr> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:10:53 <eddiejenningsjr> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:10:53 <eddiejenningsjr> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:10:53 <eddiejenningsjr> #info please help us with improving contribution to fedora infra https://discussion.fedoraproject.org/t/improving-contribution-to-fedora-infrastructure/38294/8
16:10:53 <eddiejenningsjr> #info oncall should also handling #fedora-releng pings if possible
16:11:04 <nirik> #info nirk on PTO 2022-10-06 to 2022-10-09 and 2022-10-15 to 2022-10-23
16:11:36 <nirik> #info f37 final freeze starts next tuesday
16:12:20 <eddiejenningsjr> We'll hold down the fort while you're away -- or at least try :)
16:13:23 <eddiejenningsjr> Any other announcements from anyone?
16:13:26 <nirik> 👍🏻
16:13:32 <Penguinpee> Enjoy your holiday!
16:14:33 <eddiejenningsjr> Hearing none, we'll move along.
16:14:44 <eddiejenningsjr> #topic Oncall
16:14:47 <eddiejenningsjr> Everyone's favorite topic :)
16:14:56 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:14:56 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:14:56 <eddiejenningsjr> ## .oncalltakeeu .oncalltakeus
16:15:30 <eddiejenningsjr> Ha!  Element interpreted double hashes as a heading
16:15:41 <eddiejenningsjr> #info eddiejenningsjr is on call from 2022-09-23 to 2022-09-29
16:15:41 <eddiejenningsjr> #info ??? is on call from 2022-09-30 to 2022-10-06
16:15:51 <eddiejenningsjr> Do we have a taker for the next on-call week?
16:16:01 <nirik> yep. matrix does md
16:16:21 <nirik> I can take it if noone else wants it
16:16:25 <eddiejenningsjr> On-call isn't too stressful of a job for anyone who's interested but worried.
16:17:29 <eddiejenningsjr> Any other's interested?  Else, it will go to nirik
16:17:50 <darknao> I can take it
16:18:02 <eddiejenningsjr> sold to darknao!
16:18:18 <eddiejenningsjr> #info darknao is on call from 2022-09-30 to 2022-10-06
16:18:21 <darknao> .oncalltakeus
16:18:21 <zodbot> darknao: Kneel before zod!
16:19:09 <Penguinpee> 😏
16:19:14 <erolkskn> well zod gives me some creepy "ai is going to take over world" vibes
16:19:24 <eddiejenningsjr> Any takers for 2022-10-07 to 2022-10-13?
16:19:43 <eddiejenningsjr> I'll be out of town for a chuck of that week, so I cannot take it :(
16:19:51 * nirik can't do that one, out of town part of it
16:20:08 <erolkskn> Is it open to community members ?
16:20:22 <erolkskn> i mean can a community member become on-call ?
16:21:32 <eddiejenningsjr> I'm a community member, so I'd say yes.  It helps to be in the fi-apprentice group, so you can SSH into some systems and try to do some limited troubleshooting.
16:21:46 <eddiejenningsjr> nirik can speak more authoritatively though :)
16:21:57 <nirik> sure!
16:22:00 <erolkskn> Well, then, sign me up for it :p
16:22:10 <eddiejenningsjr> sold to erolskn!
16:22:15 <nirik> we have a doc on it...
16:22:39 <eddiejenningsjr> #info erolskn is on call from 2022-10-06 to 2022-10-13
16:23:16 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:23:16 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:23:23 <eddiejenningsjr> doh, 10/7 not 10/6
16:23:28 <eddiejenningsjr> #info erolskn is on call from 2022-10-07 to 2022-10-13
16:23:41 <eddiejenningsjr> #info Summary of last week: (from current oncall )
16:23:51 <eddiejenningsjr> So I was summoned twice this week.
16:24:13 * Penguinpee ducks
16:24:16 <eddiejenningsjr> First for a mailman issue, which I couldn't help much, for I couldn't SSH Into the server :(
16:24:43 <eddiejenningsjr> Second was for a Koji issue which was early in the morning and I was sleeping, but seemed to solve itself.
16:25:24 <Penguinpee> Part of it solved itself. The other part was done by smooge.
16:25:41 <eddiejenningsjr> Ah, excellent :D
16:25:54 <eddiejenningsjr> So other than those pings, it was a quiet week.
16:26:13 <eddiejenningsjr> And now, time for the nirik show!
16:26:15 <eddiejenningsjr> #topic Monitoring discussion [nirik]
16:26:15 <eddiejenningsjr> #info https://nagios.fedoraproject.org/nagios
16:26:15 <eddiejenningsjr> #info Go over existing out items and fix
16:26:44 <nirik> so, not much here again. We did clear up some ghost mgmt interfaces eariler this week in the switch upgrades.
16:26:51 <eddiejenningsjr> One day, I'll get around to setting up Nagios at home :D
16:26:55 <nirik> so, down to just a few odd things that don't matter much
16:27:48 <eddiejenningsjr> Sounds like generally good news. :)
16:28:51 <nirik> yeah, not too bad... someday we will have time to clean up all the little junk thats listed
16:28:53 <eddiejenningsjr> Last week we had a good learning topic on IPA, so this week we'll look at tickets.  Next week, Nirik will be kind enough to discuss Koji as a learning topic.
16:28:58 <nirik> but before that we might get moved to zabbix. ;)
16:29:26 <nirik> oh shoot, I should reschedule that... I am out next week...
16:29:27 <eddiejenningsjr> #topic Revisit blocked tickets
16:29:27 <eddiejenningsjr> #info Check if any blocked ticket is unblocked
16:29:27 <eddiejenningsjr> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:29:50 <eddiejenningsjr> First let's look at currently blocked tickets.
16:30:14 <eddiejenningsjr> We wont' go through them one at a time, but everyone take a moment to look at the list, and if there's any kind of update (maybe removing the block), let's discuss it.
16:30:42 <nirik> all still blocked, but getting closer to being able to retire ocp3. ;)
16:31:29 <eddiejenningsjr> I have a couple of RHEL 9 E-mails to get caught up on, is there any chance that mailman would be a part of that?
16:31:58 <eddiejenningsjr> .ticket 8455
16:31:59 <zodbot> eddiejenningsjr: Issue #8455: Move mailman to newer release of Fedora or CentOS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8455
16:32:55 <nirik> nope, no mailman in rhel9 at all. ;)
16:34:39 <eddiejenningsjr> Ah.  Just found its gitlab page, seem like something worthy of my time to read :)
16:35:00 <eddiejenningsjr> Moving along, let's look at the backlog
16:35:13 <eddiejenningsjr> #topic Fedora Infra backlog refinement
16:35:13 <eddiejenningsjr> #info Refine oldest tickets on Fedora Infra tracker
16:35:13 <eddiejenningsjr> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:35:44 <eddiejenningsjr> Just as with blocked tickets, we won't go through each one, but if attendees would take a look at the list, and chime in if there are any updates on a ticket.
16:36:39 <eddiejenningsjr> .ticket 9892
16:36:40 <zodbot> eddiejenningsjr: Issue #9892: Centralise and update Fedora infra and releng SOPs - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9892
16:36:50 <eddiejenningsjr> I do know Infra docs have been moved.  Any news on the releng docs?
16:38:11 <nirik> they were going to be moved this week last I heard...
16:38:16 <nirik> but I haven't seen it yet.
16:38:45 <eddiejenningsjr> I'm actually looking for the on-call doc within the new docs site to replace the one I linked earlier :)
16:40:57 <nirik> https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
16:41:26 <eddiejenningsjr> There we go. Hiding in plain sight.
16:42:08 <eddiejenningsjr> It looks like there's no other discussion for old tickets.  Let's move to open floor.
16:42:15 <eddiejenningsjr> #topic Open Floor
16:42:26 <eddiejenningsjr> Anything else to discuss that hasn't already been mentioned?
16:43:18 <erolkskn> Well the things on badges side not seem so great. The latest develop branch is not in a working state and it should be tested heavily before making it to staging
16:43:25 <eddiejenningsjr> Penguinpee: I remember you might have had some questions about badges
16:43:35 <nirik> erolkskn: yeah, I feared as much
16:44:07 <Penguinpee> Well, erolkskn already mentioned me regarding badges. I'm trying to get into fedora-badges Pagure group, but can't get in contact with anyone on it.
16:44:35 <erolkskn> and as far as i can see from the prod logs, there are more couple of issues like fedmsg-hub restarting itself over and over
16:44:57 <Penguinpee> I talked to misc yesterday. I asked him. But he was a bit evasive.
16:45:35 <erolkskn> btw misc was totally right about the performance problems. Some of the badges causes the service to be down
16:45:44 <erolkskn> since it tries to get every record on datanommer with given criteria
16:45:47 <nirik> Penguinpee: the pagure group?
16:45:50 <erolkskn> it causes the memory to ran out of space
16:45:57 <Penguinpee> Further down the road I would like to get commit access on fedora-badges.
16:46:19 <Penguinpee> nirik: Yes. https://pagure.io/group/fedora-badges
16:46:50 <nirik> ok, thats the docs and requests... none of the actual code. ;) but sure I can add you...
16:46:57 <nirik> whats your fas login?
16:47:00 <Penguinpee> I heard something about staging environments, too.
16:47:09 <Penguinpee> .fas gui1ty
16:47:09 <zodbot> Penguinpee: b"gui1ty '' <gui1ty@penguinpee.nl>"
16:47:40 <nirik> erolkskn: would you like to be added there too?
16:47:58 <erolkskn> nirik: it's not needed for me now
16:48:02 <nirik> Penguinpee: the code is all on github (as far as I know) and yes, we have a staging version deployed as well as the production one.
16:48:03 <erolkskn> thank you
16:48:06 <Penguinpee> Commit access on GitHub would also be appreciated. But I can also live with submitting PRs.
16:48:22 <nirik> Penguinpee: can add you there too. whats your github user?
16:48:43 <Penguinpee> nirik: On GitHub it's penguinpee.
16:49:21 <erolkskn> here is my thought regarding the badger. i think rewriting it would be much easier than fixing it
16:50:30 <Penguinpee> Do I need to apply for apprentice for getting read-only access to the production and/or staging machines?
16:50:31 <erolkskn> at least we can just rewrite the "fedbadges" project at first, and proceed to replace other projects like "tahrir" and "tahrir-api" in next.
16:51:17 <eddiejenningsjr> Sometimes rewrites are the way to go.
16:51:28 <nirik> Penguinpee: I can add you to sysadmin-badges if you like... do you have a staging user?
16:51:38 <nirik> erolkskn: yeah, I think we came to that conclusion.
16:51:43 <Penguinpee> I have a much larger plan for tackling things. But I think that's out of scope for this meeting.
16:52:04 <Penguinpee> nirik: Not yet. Where do I need to set this up?
16:52:28 <nirik> accounts.stg.fedoraproject.org
16:52:41 <erolkskn> eddiejenningsjr: in this case i think it's definitely the way to go since either way we should refactor every piece of that software
16:52:58 <erolkskn> in the current state, fedbadges project heavily relies on fedmsg library
16:53:00 <erolkskn> which is deprecated
16:53:17 <erolkskn> and then there is the scripts in ansible. They are relying on fedmsg too
16:53:22 <Penguinpee> Thanks. I'll do that after the meeting, if that's all right.
16:54:01 <erolkskn> and then there is the fact that fedbadges uses datanommer instead of using datagrepper.
16:54:18 <nirik> Penguinpee: sure, just let me know
16:54:19 <Penguinpee> nirik: Can I re-use my existing key for staging or is it preferred to have a separate SSH key?
16:54:36 <nirik> erolkskn / Penguinpee: this may be of interest also: https://pagure.io/fedora-infra/arc/blob/main/f/docs/badges/index.md
16:54:45 <nirik> you can reuse if you like. up to you
16:54:53 <erolkskn> nirik: yeah i've read it
16:55:01 <erolkskn> it pretty much addresses the same problems
16:55:16 * nirik nods.
16:55:30 <erolkskn> nirik: i do have a question regarding the rewriting of this app
16:55:41 <erolkskn> nirik: should we open up this topic in cpe issues ?
16:55:52 <Penguinpee> I might have stumbled on that doc. Will take a closer look.
16:56:37 * Penguinpee stumbled upon so many things the past weeks, he feels like not to be bruised.
16:56:46 <nirik> erolkskn: well, if you want cpe to form a team to try and assist on it... if not, no, probibly not needed.
16:57:25 <Penguinpee> *lucky
16:57:32 <erolkskn> that's great then :)
16:58:19 <erolkskn> Alright then, sandro i'm waiting your thoughts on this too
16:58:34 <Penguinpee> I think that's it from me for now. Feel free to ping me if there are anymore questions.
16:59:10 <Penguinpee> erolkskn: The rewrite? Let's put out the fire first, before rebuilding the house.
16:59:11 <eddiejenningsjr> Excellent.  We have about 2 minutes left.  Anything else to discuss?
17:00:52 <eddiejenningsjr> We've reached time.  The discussion can continue in the Infra channel and elsewhere :)
17:00:58 <eddiejenningsjr> Thanks for coming everyone!
17:01:08 <eddiejenningsjr> #endmeeting