16:02:44 <pcreech> #startmeeting Infrastructure (2023-08-24)
16:02:44 <zodbot> Meeting started Thu Aug 24 16:02:44 2023 UTC.
16:02:44 <zodbot> This meeting is logged and archived in a public location.
16:02:44 <zodbot> The chair is pcreech. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:02:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:44 <zodbot> The meeting name has been set to 'infrastructure_(2023-08-24)'
16:02:44 <pcreech> #meetingname infrastructure
16:02:44 <pcreech> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
16:02:44 <zodbot> The meeting name has been set to 'infrastructure'
16:02:44 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg nb nirik patrikp pcreech zlopez
16:02:44 <pcreech> #info Agenda is at: https://board.net/p/fedora-infra
16:02:44 <pcreech> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:02:44 <pcreech> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:02:46 <pcreech> #topic greetings!
16:02:52 <darknao> .hi
16:02:53 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:02:57 <pcreech> .hello pcreech17
16:02:58 <zodbot> pcreech: pcreech17 'Patrick Creech' <pcreech@redhat.com>
16:03:00 <patrikp> .hi
16:03:02 <zodbot> patrikp: patrikp 'Patrik Polakovič' <ppolakov@redhat.com>
16:03:04 <eddiejennings> .hi
16:03:05 <nirik> morning.
16:03:05 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:03:28 <nirik> to quote a horrible star trek episode: "brain and brain. what is brain?"
16:03:40 <pcreech> Sorry again everyone, my silly brain failed me this week. :D
16:03:46 <pcreech> hehe
16:03:53 <pcreech> #topic New folks introductions
16:03:53 <pcreech> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:53 <pcreech> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:03:54 <nirik> no problems at all.
16:03:55 <smooge> EHLO
16:03:56 <eddiejennings> Worry not.
16:04:13 <pcreech> Any new people today?
16:04:46 <pcreech> Don't see any e-mails, moving on :D
16:04:57 <pcreech> #topic Next chair
16:04:57 <pcreech> #info magic eight ball says:
16:04:57 <pcreech> #info chair 2023-08-24 pcreech
16:04:57 <pcreech> #info chair 2023-08-31 eddiejennings
16:04:57 <pcreech> #info chair 2023-09-07 ???
16:04:58 <pcreech> #info chair 2023-09-14 ???
16:05:23 <pcreech> Any volunteers for the 7th or the 14th?
16:05:26 <eddiejennings> Unfortunately, I cannot chair on 2023-08-31.  I have a family thing going on that day that will require 100% of my attention.
16:05:29 <eddiejennings> I can do the 7th though
16:05:42 <nirik> I can take next week if no one else wants it.
16:06:19 <pcreech> I can also do next week.
16:07:01 <nirik> either way. I could take the 14th too.
16:07:22 <pcreech> k.  let's do that.  pcreech next week, nirik 14th
16:07:42 <nirik> +1
16:07:57 <pcreech> #topic announcements and information
16:07:57 <pcreech> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting
16:07:57 <pcreech> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:08:15 <nirik> #info we are in f39 beta infrastructure freeze
16:08:46 <eddiejennings> Exciting!
16:08:56 <patrikp> Let's go baby!
16:08:56 <zlopez> #info Flask-OIDC is using authlib now, please upgrade
16:09:40 <pcreech> Nice!  Anything else?
16:11:06 <nirik> nothing off hand here.
16:11:32 <pcreech> #topic Oncall
16:11:32 <pcreech> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:11:32 <pcreech> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:11:32 <pcreech> ## .oncalltakeeu .oncalltakeus
16:11:32 <pcreech> #info eddiejennings  is on call from 2023-08-25 to 2023-08-31
16:11:33 <pcreech> #info ???  is on call from 2023-09-01 to 2023-09-07
16:11:35 <pcreech> #info ???  is on call from 2023-09-08 to 2023-09-14
16:12:22 <pcreech> Looks like we have eddiejennings set up to take over, but nothing going into september.  Any volunteers for september?
16:12:55 <eddiejennings> .oncalltakeus
16:12:55 <zodbot> eddiejennings: Kneel before zod!
16:13:02 <nirik> I can do 8th-14th (so I end with running the meeting that week)
16:13:35 <darknao> and can take 1st-7th
16:13:52 <pcreech> awesome!  sold!
16:14:03 <pcreech> #info Summary of last week: (from current oncall )
16:14:10 <pcreech> is aheath1992 around?
16:14:21 <aheath1992> yep
16:14:50 <aheath1992> Nothing from oncall, all was quiet
16:15:01 <pcreech> cool
16:15:09 <pcreech> #topic Monitoring discussion [nirik]
16:15:10 <pcreech> #info https://nagios.fedoraproject.org/nagios
16:15:10 <pcreech> #info Go over existing out items and fix
16:15:29 <nirik> lets see.
16:16:31 <nirik> there was a bvmhost-a64 down this morning, I power cycled it... there's some certs I need to renew and then a few outstanding alerts.
16:16:51 <nirik> note also, that there's a zabbix proof of concept up in staging now.
16:17:17 <nirik> https://discussion.fedoraproject.org/t/deploying-zabbix-monitoring-stack-to-fedora-infra-eventually-replace-nagios/87560/4
16:17:47 <nirik> There's also two copr alerts. I should ask them about fixing those up.
16:17:57 <nirik> Otherwise nothing else to report here.
16:18:04 <pcreech> Nice!
16:18:51 <pcreech> #topic Revisit blocked tickets
16:18:51 <pcreech> #info Check if any blocked ticket is unblocked
16:18:51 <pcreech> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:19:29 <pcreech> Any blocker updates?
16:20:10 <aheath1992> We were able to star sending logs from log01 to Splunk that ticket was closed
16:20:20 <aheath1992> Start*
16:20:51 <nirik> no news on any of the blocked ones
16:21:16 <pcreech> cool.  Backlog time!
16:21:19 <pcreech> #topic Fedora Infra backlog refinement
16:21:19 <pcreech> #info Refine oldest tickets on Fedora Infra tracker
16:21:19 <pcreech> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:21:45 <pcreech> .ticket 10358
16:21:48 <nirik> everone loves log.
16:21:59 <zodbot> pcreech: Issue #10358: IPA backup fails on ipa03 and ipa02.stg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10358
16:22:07 <nirik> pagure is being kinda slow
16:22:47 <pcreech> Yup, made me think I did something wrong for a minute there :D
16:23:14 <pcreech> nirik, i'm guessing this one is still "on the list"?
16:23:28 <nirik> yeah.
16:23:37 <nirik> I'd like to try and do stg soon
16:24:12 <pcreech> .ticket 11323
16:24:13 <zodbot> pcreech: Issue #11323: [Badges] [DB] Create updated database schema and export it for developers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11323
16:24:55 <nirik> not sure here.
16:25:18 <nirik> we got the db, but should ask status...
16:25:31 <pcreech> I'll ping on the ticket for an update
16:25:31 <zlopez> nirik: Could we let somebody else to do the staging deployment for 10358? So somebody could learnt it?
16:26:40 <nirik> well, sure, but I also need to figure it out. was hoping to learn in stg for doing prod later
16:27:37 <zlopez> nirik: understood
16:27:40 <nirik> I think its just doing a new instance, adding it to replication, then replacing the rest... but I haven't looked in a while
16:28:14 <pcreech> one does not simply "ipa" :D
16:28:25 <pcreech> .ticket 11122
16:28:26 <zodbot> pcreech: Issue #11122: Set up maintainer test machine for s390x - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11122
16:28:47 <nirik> not sure on status here either. should ask in ticket.
16:29:06 <zlopez> Hm, I thought this is already done
16:29:21 <zlopez> Or it was another s390?
16:29:28 <pcreech> the last on the ticket looks like there was some waiting on a testing machine from someone
16:29:54 <nirik> not done that I know of...
16:30:45 <pcreech> ping in ticket made
16:31:01 <pcreech> .ticket 11013
16:31:03 <zodbot> pcreech: Issue #11013: Redeploy openQA workers with consistent storage configuration - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11013
16:31:10 <zlopez> nirik: You have better info about hardware than me. So I will trust you 😃
16:31:50 <nirik> never got to this one and we are in freeze...
16:31:57 <nirik> so, after beta ?
16:32:16 <nirik> well, perhaps we can do some at a time
16:32:20 <nirik> I'll ask
16:32:20 <zlopez> Are they in Openshift?
16:32:24 <nirik> no
16:32:37 <pcreech> ack, leaving it to you nirik
16:32:39 <nirik> they are bare metal hw
16:32:46 <zlopez> OK
16:32:51 <nirik> need to be, they run qemu / etc
16:33:05 <zlopez> Makes sense
16:33:16 <pcreech> .ticket 11330
16:33:17 <zodbot> pcreech: Issue #11330: freeipa pagure repo does not sync anymore to github - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11330
16:33:47 <nirik> so I wanted to try and test a fix, but never heard back from them
16:34:15 <nirik> I guess we could duplicate on stg?
16:34:35 <nirik> I'd love for help with this one. ;)
16:35:10 <zlopez> I'm glad to help
16:35:28 <pcreech> if it's reasonably "simple" enough, i could volunteer to help too
16:35:32 <nirik> so, I guess in stg we need to make a project and set it to mirror and see it blow up
16:35:51 <nirik> the problem is the git change that makes it error when repos are not owned by the same user
16:36:22 <nirik> but we don't care about that and want to override it, but I couldn't get it to override. ;)
16:36:29 <zlopez> same user as where?
16:37:47 <nirik> thats a good question. ;) in our case the repo is owned by one user (git/pagure) and the mirroring is another user pagure_mirror
16:37:50 <nirik> https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/
16:37:59 <nirik> stderr: fatal: detected dubious ownership in repository at
16:38:13 <zlopez> Oh
16:38:23 <nirik> anyhow, if someone can duplicate it on staging pagure we can try and figure out how to fix it better
16:39:09 <zlopez> Never played with pagure_mirror, but can look into that
16:39:36 <nirik> it's on the server, it's an option you can set on a project
16:40:25 <nirik> then there's a celery job that runs the actual mirroring (thats what fails)
16:42:08 <pcreech> Cool, i'll leave the ticket in your hands zlopez!  next up:
16:42:14 <pcreech> .ticket 11338
16:42:15 <zodbot> pcreech: Issue #11338: Please upgrade mailman for lists.fedoraproject.org to fix ARC signing - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11338
16:42:42 <nirik> This should just be blocked on the upgrade mailman one.
16:42:46 <pcreech> ah, that one was on the blocked list
16:42:49 <nirik> yeah
16:43:01 <pcreech> .ticket 11324
16:43:05 <zodbot> pcreech: Issue #11324: [Badges] [DB] Dump snapshot of existing Badges database in line with updated schema - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11324
16:43:28 <nirik> not sure on this one either. ;)
16:43:55 <zlopez> t0xic0der is the one running badges show
16:44:15 <pcreech> pinged him on the ticket :D
16:44:41 <pcreech> .ticket 10241
16:44:42 <zodbot> pcreech: Issue #10241: Port apps to OIDC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10241
16:45:32 <nirik> I think we are blocked here too, right zlopez ?
16:45:39 <nirik> we need rhel9 pagure to move forward there.
16:45:44 <zlopez> I don't think there is any progress on this
16:45:57 <zlopez> nirik: correct
16:46:30 <pcreech> So, a "blocked" tag maybe?  Don't see where I can add it...
16:47:00 <nirik> yeah, +1 to add blocked tag...
16:47:10 <zlopez> +1
16:48:02 <pcreech> k.  I'll have to elave that to one of you, i'm guessing I need some perms in pagure to do such
16:48:12 <nirik> huh, should just be in editing...
16:48:31 <nirik> added
16:48:54 <pcreech> Yeah, I don't see an 'edit' of any kind.  I could just be failing at pagure too...
16:49:06 <nirik> might be perms. Can look.
16:49:10 <pcreech> .ticket 10845
16:49:11 <zodbot> pcreech: Issue #10845: reply by email to Pagure issues bounce - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10845
16:50:31 <nirik> need someone to duplicate/dig into this one. It might be specific to that provider tho
16:51:30 <zlopez> I didn't saw it anywhere, I'm getting mails without problem
16:52:05 <pcreech> Yeah, it read like a specific domain issue to me
16:52:53 <nirik> well, getting them is not the problem... it's replying to them and having it update the ticket I think
16:53:37 <nirik> anyhow, help welcome here too if someone can look and try and duplicate it and see if they can see whats wrong
16:54:15 <pcreech> yeah, e-mail issues are pretty far out of my wheelhouse :D
16:55:00 <pcreech> .ticket 10458
16:55:01 <zodbot> pcreech: Issue #10458: Help me move my discourse bots to production? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10458
16:55:14 <pcreech> looks like we're getting into the "updated in the past month" territory
16:55:34 <pcreech> I'm assuming this one is still waiting on mattdm
16:55:44 <zlopez> We are almost at end of our time anyway
16:55:59 <pcreech> Wow, where di the past 10 minutes go... i swear it was just 45
16:56:05 <nirik> yeah... and he's out right now I think.
16:56:19 <pcreech> #topic Open Floor
16:56:26 <pcreech> Ok, open floor time!
16:57:07 <zlopez> Everybody dance now!
16:57:11 <pcreech> !dance
16:57:33 <pcreech> ... yeah, I'm used to having a dance bot
16:57:34 <nirik> shake it!
16:57:34 <pcreech> ᕕ(⌐■_■)ᕗ ♪♬
16:57:40 <zlopez> .dance
16:58:09 <zlopez> zodbot can learn some new tricks
16:58:18 <nirik> 🕺
16:58:28 <pcreech> Sorry again everyone for being a space case this week!  Post-vacation ramp-up is always a fun week.
16:58:35 <pcreech> don't tempt me zlopez :D
16:58:38 <patrikp> I had one thing. Here is the PR for removing mentions of modularity.
16:58:46 <patrikp> https://pagure.io/infra-docs-fpo/pull-request/218#request_diff
16:58:58 <nirik> nice, thanks patrikp!
16:59:10 <patrikp> I only went for the releng section of SOPs. If it looks reasonable to you feel free to merge.
16:59:27 <zlopez> +1
16:59:36 <nirik> can look later today (or someone else can feel free to)
16:59:57 <zlopez> patrikp++
16:59:57 <zodbot> zlopez: Karma for patrikp changed to 1 (for the release cycle f38):  https://badges.fedoraproject.org/tags/cookie/any
17:00:34 <pcreech> Ok everyone!  That's the hour!  You don't have to go home, and you can stay here if you'd like, but the logs are over!
17:00:37 <patrikp> Should mentions of modularity also be removed from the sysadmin section?
17:00:41 <pcreech> #endmeeting