16:02:44 <pcreech> #startmeeting Infrastructure (2023-08-24) 16:02:44 <zodbot> Meeting started Thu Aug 24 16:02:44 2023 UTC. 16:02:44 <zodbot> This meeting is logged and archived in a public location. 16:02:44 <zodbot> The chair is pcreech. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:02:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:02:44 <zodbot> The meeting name has been set to 'infrastructure_(2023-08-24)' 16:02:44 <pcreech> #meetingname infrastructure 16:02:44 <pcreech> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp 16:02:44 <zodbot> The meeting name has been set to 'infrastructure' 16:02:44 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg nb nirik patrikp pcreech zlopez 16:02:44 <pcreech> #info Agenda is at: https://board.net/p/fedora-infra 16:02:44 <pcreech> #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 16:02:44 <pcreech> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra 16:02:46 <pcreech> #topic greetings! 16:02:52 <darknao> .hi 16:02:53 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja> 16:02:57 <pcreech> .hello pcreech17 16:02:58 <zodbot> pcreech: pcreech17 'Patrick Creech' <pcreech@redhat.com> 16:03:00 <patrikp> .hi 16:03:02 <zodbot> patrikp: patrikp 'Patrik Polakovič' <ppolakov@redhat.com> 16:03:04 <eddiejennings> .hi 16:03:05 <nirik> morning. 16:03:05 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net> 16:03:28 <nirik> to quote a horrible star trek episode: "brain and brain. what is brain?" 16:03:40 <pcreech> Sorry again everyone, my silly brain failed me this week. :D 16:03:46 <pcreech> hehe 16:03:53 <pcreech> #topic New folks introductions 16:03:53 <pcreech> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 16:03:53 <pcreech> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 16:03:54 <nirik> no problems at all. 16:03:55 <smooge> EHLO 16:03:56 <eddiejennings> Worry not. 16:04:13 <pcreech> Any new people today? 16:04:46 <pcreech> Don't see any e-mails, moving on :D 16:04:57 <pcreech> #topic Next chair 16:04:57 <pcreech> #info magic eight ball says: 16:04:57 <pcreech> #info chair 2023-08-24 pcreech 16:04:57 <pcreech> #info chair 2023-08-31 eddiejennings 16:04:57 <pcreech> #info chair 2023-09-07 ??? 16:04:58 <pcreech> #info chair 2023-09-14 ??? 16:05:23 <pcreech> Any volunteers for the 7th or the 14th? 16:05:26 <eddiejennings> Unfortunately, I cannot chair on 2023-08-31. I have a family thing going on that day that will require 100% of my attention. 16:05:29 <eddiejennings> I can do the 7th though 16:05:42 <nirik> I can take next week if no one else wants it. 16:06:19 <pcreech> I can also do next week. 16:07:01 <nirik> either way. I could take the 14th too. 16:07:22 <pcreech> k. let's do that. pcreech next week, nirik 14th 16:07:42 <nirik> +1 16:07:57 <pcreech> #topic announcements and information 16:07:57 <pcreech> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting 16:07:57 <pcreech> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3 16:08:15 <nirik> #info we are in f39 beta infrastructure freeze 16:08:46 <eddiejennings> Exciting! 16:08:56 <patrikp> Let's go baby! 16:08:56 <zlopez> #info Flask-OIDC is using authlib now, please upgrade 16:09:40 <pcreech> Nice! Anything else? 16:11:06 <nirik> nothing off hand here. 16:11:32 <pcreech> #topic Oncall 16:11:32 <pcreech> #info https://fedoraproject.org/wiki/Infrastructure/Oncall 16:11:32 <pcreech> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ 16:11:32 <pcreech> ## .oncalltakeeu .oncalltakeus 16:11:32 <pcreech> #info eddiejennings is on call from 2023-08-25 to 2023-08-31 16:11:33 <pcreech> #info ??? is on call from 2023-09-01 to 2023-09-07 16:11:35 <pcreech> #info ??? is on call from 2023-09-08 to 2023-09-14 16:12:22 <pcreech> Looks like we have eddiejennings set up to take over, but nothing going into september. Any volunteers for september? 16:12:55 <eddiejennings> .oncalltakeus 16:12:55 <zodbot> eddiejennings: Kneel before zod! 16:13:02 <nirik> I can do 8th-14th (so I end with running the meeting that week) 16:13:35 <darknao> and can take 1st-7th 16:13:52 <pcreech> awesome! sold! 16:14:03 <pcreech> #info Summary of last week: (from current oncall ) 16:14:10 <pcreech> is aheath1992 around? 16:14:21 <aheath1992> yep 16:14:50 <aheath1992> Nothing from oncall, all was quiet 16:15:01 <pcreech> cool 16:15:09 <pcreech> #topic Monitoring discussion [nirik] 16:15:10 <pcreech> #info https://nagios.fedoraproject.org/nagios 16:15:10 <pcreech> #info Go over existing out items and fix 16:15:29 <nirik> lets see. 16:16:31 <nirik> there was a bvmhost-a64 down this morning, I power cycled it... there's some certs I need to renew and then a few outstanding alerts. 16:16:51 <nirik> note also, that there's a zabbix proof of concept up in staging now. 16:17:17 <nirik> https://discussion.fedoraproject.org/t/deploying-zabbix-monitoring-stack-to-fedora-infra-eventually-replace-nagios/87560/4 16:17:47 <nirik> There's also two copr alerts. I should ask them about fixing those up. 16:17:57 <nirik> Otherwise nothing else to report here. 16:18:04 <pcreech> Nice! 16:18:51 <pcreech> #topic Revisit blocked tickets 16:18:51 <pcreech> #info Check if any blocked ticket is unblocked 16:18:51 <pcreech> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status= 16:19:29 <pcreech> Any blocker updates? 16:20:10 <aheath1992> We were able to star sending logs from log01 to Splunk that ticket was closed 16:20:20 <aheath1992> Start* 16:20:51 <nirik> no news on any of the blocked ones 16:21:16 <pcreech> cool. Backlog time! 16:21:19 <pcreech> #topic Fedora Infra backlog refinement 16:21:19 <pcreech> #info Refine oldest tickets on Fedora Infra tracker 16:21:19 <pcreech> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc 16:21:45 <pcreech> .ticket 10358 16:21:48 <nirik> everone loves log. 16:21:59 <zodbot> pcreech: Issue #10358: IPA backup fails on ipa03 and ipa02.stg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10358 16:22:07 <nirik> pagure is being kinda slow 16:22:47 <pcreech> Yup, made me think I did something wrong for a minute there :D 16:23:14 <pcreech> nirik, i'm guessing this one is still "on the list"? 16:23:28 <nirik> yeah. 16:23:37 <nirik> I'd like to try and do stg soon 16:24:12 <pcreech> .ticket 11323 16:24:13 <zodbot> pcreech: Issue #11323: [Badges] [DB] Create updated database schema and export it for developers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11323 16:24:55 <nirik> not sure here. 16:25:18 <nirik> we got the db, but should ask status... 16:25:31 <pcreech> I'll ping on the ticket for an update 16:25:31 <zlopez> nirik: Could we let somebody else to do the staging deployment for 10358? So somebody could learnt it? 16:26:40 <nirik> well, sure, but I also need to figure it out. was hoping to learn in stg for doing prod later 16:27:37 <zlopez> nirik: understood 16:27:40 <nirik> I think its just doing a new instance, adding it to replication, then replacing the rest... but I haven't looked in a while 16:28:14 <pcreech> one does not simply "ipa" :D 16:28:25 <pcreech> .ticket 11122 16:28:26 <zodbot> pcreech: Issue #11122: Set up maintainer test machine for s390x - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11122 16:28:47 <nirik> not sure on status here either. should ask in ticket. 16:29:06 <zlopez> Hm, I thought this is already done 16:29:21 <zlopez> Or it was another s390? 16:29:28 <pcreech> the last on the ticket looks like there was some waiting on a testing machine from someone 16:29:54 <nirik> not done that I know of... 16:30:45 <pcreech> ping in ticket made 16:31:01 <pcreech> .ticket 11013 16:31:03 <zodbot> pcreech: Issue #11013: Redeploy openQA workers with consistent storage configuration - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11013 16:31:10 <zlopez> nirik: You have better info about hardware than me. So I will trust you 😃 16:31:50 <nirik> never got to this one and we are in freeze... 16:31:57 <nirik> so, after beta ? 16:32:16 <nirik> well, perhaps we can do some at a time 16:32:20 <nirik> I'll ask 16:32:20 <zlopez> Are they in Openshift? 16:32:24 <nirik> no 16:32:37 <pcreech> ack, leaving it to you nirik 16:32:39 <nirik> they are bare metal hw 16:32:46 <zlopez> OK 16:32:51 <nirik> need to be, they run qemu / etc 16:33:05 <zlopez> Makes sense 16:33:16 <pcreech> .ticket 11330 16:33:17 <zodbot> pcreech: Issue #11330: freeipa pagure repo does not sync anymore to github - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11330 16:33:47 <nirik> so I wanted to try and test a fix, but never heard back from them 16:34:15 <nirik> I guess we could duplicate on stg? 16:34:35 <nirik> I'd love for help with this one. ;) 16:35:10 <zlopez> I'm glad to help 16:35:28 <pcreech> if it's reasonably "simple" enough, i could volunteer to help too 16:35:32 <nirik> so, I guess in stg we need to make a project and set it to mirror and see it blow up 16:35:51 <nirik> the problem is the git change that makes it error when repos are not owned by the same user 16:36:22 <nirik> but we don't care about that and want to override it, but I couldn't get it to override. ;) 16:36:29 <zlopez> same user as where? 16:37:47 <nirik> thats a good question. ;) in our case the repo is owned by one user (git/pagure) and the mirroring is another user pagure_mirror 16:37:50 <nirik> https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/ 16:37:59 <nirik> stderr: fatal: detected dubious ownership in repository at 16:38:13 <zlopez> Oh 16:38:23 <nirik> anyhow, if someone can duplicate it on staging pagure we can try and figure out how to fix it better 16:39:09 <zlopez> Never played with pagure_mirror, but can look into that 16:39:36 <nirik> it's on the server, it's an option you can set on a project 16:40:25 <nirik> then there's a celery job that runs the actual mirroring (thats what fails) 16:42:08 <pcreech> Cool, i'll leave the ticket in your hands zlopez! next up: 16:42:14 <pcreech> .ticket 11338 16:42:15 <zodbot> pcreech: Issue #11338: Please upgrade mailman for lists.fedoraproject.org to fix ARC signing - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11338 16:42:42 <nirik> This should just be blocked on the upgrade mailman one. 16:42:46 <pcreech> ah, that one was on the blocked list 16:42:49 <nirik> yeah 16:43:01 <pcreech> .ticket 11324 16:43:05 <zodbot> pcreech: Issue #11324: [Badges] [DB] Dump snapshot of existing Badges database in line with updated schema - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11324 16:43:28 <nirik> not sure on this one either. ;) 16:43:55 <zlopez> t0xic0der is the one running badges show 16:44:15 <pcreech> pinged him on the ticket :D 16:44:41 <pcreech> .ticket 10241 16:44:42 <zodbot> pcreech: Issue #10241: Port apps to OIDC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10241 16:45:32 <nirik> I think we are blocked here too, right zlopez ? 16:45:39 <nirik> we need rhel9 pagure to move forward there. 16:45:44 <zlopez> I don't think there is any progress on this 16:45:57 <zlopez> nirik: correct 16:46:30 <pcreech> So, a "blocked" tag maybe? Don't see where I can add it... 16:47:00 <nirik> yeah, +1 to add blocked tag... 16:47:10 <zlopez> +1 16:48:02 <pcreech> k. I'll have to elave that to one of you, i'm guessing I need some perms in pagure to do such 16:48:12 <nirik> huh, should just be in editing... 16:48:31 <nirik> added 16:48:54 <pcreech> Yeah, I don't see an 'edit' of any kind. I could just be failing at pagure too... 16:49:06 <nirik> might be perms. Can look. 16:49:10 <pcreech> .ticket 10845 16:49:11 <zodbot> pcreech: Issue #10845: reply by email to Pagure issues bounce - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10845 16:50:31 <nirik> need someone to duplicate/dig into this one. It might be specific to that provider tho 16:51:30 <zlopez> I didn't saw it anywhere, I'm getting mails without problem 16:52:05 <pcreech> Yeah, it read like a specific domain issue to me 16:52:53 <nirik> well, getting them is not the problem... it's replying to them and having it update the ticket I think 16:53:37 <nirik> anyhow, help welcome here too if someone can look and try and duplicate it and see if they can see whats wrong 16:54:15 <pcreech> yeah, e-mail issues are pretty far out of my wheelhouse :D 16:55:00 <pcreech> .ticket 10458 16:55:01 <zodbot> pcreech: Issue #10458: Help me move my discourse bots to production? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10458 16:55:14 <pcreech> looks like we're getting into the "updated in the past month" territory 16:55:34 <pcreech> I'm assuming this one is still waiting on mattdm 16:55:44 <zlopez> We are almost at end of our time anyway 16:55:59 <pcreech> Wow, where di the past 10 minutes go... i swear it was just 45 16:56:05 <nirik> yeah... and he's out right now I think. 16:56:19 <pcreech> #topic Open Floor 16:56:26 <pcreech> Ok, open floor time! 16:57:07 <zlopez> Everybody dance now! 16:57:11 <pcreech> !dance 16:57:33 <pcreech> ... yeah, I'm used to having a dance bot 16:57:34 <nirik> shake it! 16:57:34 <pcreech> ᕕ(⌐■_■)ᕗ ♪♬ 16:57:40 <zlopez> .dance 16:58:09 <zlopez> zodbot can learn some new tricks 16:58:18 <nirik> 🕺 16:58:28 <pcreech> Sorry again everyone for being a space case this week! Post-vacation ramp-up is always a fun week. 16:58:35 <pcreech> don't tempt me zlopez :D 16:58:38 <patrikp> I had one thing. Here is the PR for removing mentions of modularity. 16:58:46 <patrikp> https://pagure.io/infra-docs-fpo/pull-request/218#request_diff 16:58:58 <nirik> nice, thanks patrikp! 16:59:10 <patrikp> I only went for the releng section of SOPs. If it looks reasonable to you feel free to merge. 16:59:27 <zlopez> +1 16:59:36 <nirik> can look later today (or someone else can feel free to) 16:59:57 <zlopez> patrikp++ 16:59:57 <zodbot> zlopez: Karma for patrikp changed to 1 (for the release cycle f38): https://badges.fedoraproject.org/tags/cookie/any 17:00:34 <pcreech> Ok everyone! That's the hour! You don't have to go home, and you can stay here if you'd like, but the logs are over! 17:00:37 <patrikp> Should mentions of modularity also be removed from the sysadmin section? 17:00:41 <pcreech> #endmeeting