18:02:41 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:02:41 <zodbot> Meeting started Mon Mar 27 18:02:41 2023 UTC.
18:02:41 <zodbot> This meeting is logged and archived in a public location.
18:02:41 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:02:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:02:41 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:02:42 <nirik> #chair  nirik nb smooge phsmoura
18:02:42 <zodbot> Current chairs: nb nirik phsmoura smooge
18:02:48 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:02:48 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:02:54 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:03:01 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:03:06 <nirik> #info reminder: speak up if you want to work on a ticket!
18:03:14 <nirik> #topic Tickets needing review
18:03:14 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:03:42 <nirik> There's a pile since we were all gone last week. :)
18:04:10 <nirik> .ticket 11195
18:04:11 <zodbot> nirik: Issue #11195: local delete_old_oci_images.py broken by new docker-distribution upgrade - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11195
18:04:37 <nirik> I filed this. It needs some python/ansible work... but it's not super urgent I guess.
18:04:44 <nirik> so med med ops I suppose.
18:05:21 <nirik> .ticket 11197
18:05:22 <zodbot> nirik: Issue #11197: genesisadaptive.com mirrors fail more often than not - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11197
18:05:50 <nirik> I mailed the mirror admin, but have not heard anything.
18:05:56 <nirik> So, I think I will disable it and close the ticket. They can re-enable it later once they sort things out.
18:07:53 <nirik> .ticket 11201
18:07:54 <zodbot> nirik: Issue #11201: Planned outage: Upgrade of Copr servers - 2023-03-30 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11201
18:08:23 <nirik> moving this to waiting on external
18:08:27 <nirik> .ticket 11203
18:08:28 <zodbot> nirik: Issue #11203: Support creating addtional IAM users in AWS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11203
18:08:45 <nirik> it's unclear what they want here... needs more investigation.
18:09:42 <nirik> marking it med/med/waiting on reporter for now
18:09:51 <nirik> .ticket 11204
18:09:52 <zodbot> nirik: Issue #11204: DNF countme is broken for the week 2023-03-13->2023-03-19 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11204
18:10:22 <nirik> so, I am unclear on what we need to do here.
18:10:23 <nirik> I'll ask in the ticket I guess.
18:11:18 <nirik> .ticket 11207
18:11:19 <zodbot> nirik: Issue #11207: Login Issue with transtats.fp.o - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11207
18:11:46 <nirik> this might be related to another ticket we have about waiverdb having the same problem. ;(
18:11:58 <nirik> I asked about that in ticket. In the meantime, med med ops I guess
18:12:25 <nirik> .ticket 11209
18:12:28 <zodbot> nirik: Issue #11209: Planned Outage - Server updates/reboots - 2023-03-29 21:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11209
18:12:28 <smooge> hello
18:12:56 <nirik> I just filed this this morning. Need someone to review it. I am also reworking the mass update SOP...
18:13:13 <nirik> it's quite a blast from the past... it talks about xen hosts, and running func, and machines that no longer exist. ;)
18:14:53 <nirik> anyhow, I plan to update it and also if anyone wants to learn more about it, let me know.
18:14:53 <nirik> Thats all the tickets I think.
18:14:55 <nirik> in infra
18:15:37 <nirik> on to releng.
18:15:44 <nirik> I'm just going to mark the unretires as low/low
18:17:00 <nirik> .releng 11345
18:17:01 <zodbot> nirik: Issue #11345: Missing f38 beta in vagrant box list - releng - Pagure.io - https://pagure.io/releng/issue/11345
18:17:21 <nirik> I have no idea who handles this. I guess I can cc some folks in the ticket...
18:17:29 <nirik> med med ops I suppose?
18:18:15 <nirik> .releng 11346
18:18:16 <zodbot> nirik: Issue #11346: please remove linux-system-roles branch - releng - Pagure.io - https://pagure.io/releng/issue/11346
18:18:41 <nirik> low low ops?
18:18:56 <nirik> .releng 11349
18:18:57 <zodbot> nirik: Issue #11349: Fedora Media Writer 5.0.6 released - releng - Pagure.io - https://pagure.io/releng/issue/11349
18:19:40 <nirik> med med ops... just needs doing by jednorozec and me.
18:20:09 <nirik> I think ideally we would get time to spin up the instances to do this in AWS... then it could be done by anyone with permissions to.
18:20:20 <nirik> .releng 11352
18:20:21 <zodbot> nirik: Issue #11352: SOP documents do not cover all aspects of beta install and cleanup - releng - Pagure.io - https://pagure.io/releng/issue/11352
18:20:48 <nirik> so, docs need updating. I must have missed the perms on that dir. :(
18:21:07 <nirik> but the beta stuff we were keeping for some reason... but I can't find it now, so I guess we could delete them
18:21:59 <nirik> low low ops and cced jednorozec
18:22:33 <nirik> .releng 11353
18:22:34 <zodbot> nirik: Issue #11353: htdig package has licensing issues - releng - Pagure.io - https://pagure.io/releng/issue/11353
18:22:47 <phsmoura> hi nirik, sorry. Forgot we are not in winter time anymore
18:23:28 <nirik> yeah, although it sure seems like winter here. ;)
18:23:49 <nirik> so, I guess this one needs more investigation... high/high ops...
18:26:24 <phsmoura> , now just 11357?
18:27:26 <nirik> sorry, I got sidetracked looking at the last one
18:27:36 <phsmoura> .releng 11357
18:27:37 <zodbot> phsmoura: Issue #11357: Change: Changes of defaults in createrepo_c-1.0.0 - releng - Pagure.io - https://pagure.io/releng/issue/11357
18:28:07 <nirik> This is likely going to need bodhi changes.
18:28:15 <nirik> we were discussing it on the devel list.
18:28:37 <nirik> Might also need pungi changes.
18:28:45 <nirik> SO, I would say med med ops and leave open to investigate changes we need to make
18:29:19 * jednorozec lurking around
18:29:30 <phsmoura> ok, done
18:29:41 <jednorozec> add changes and f39
18:29:53 <nirik> 👋
18:30:06 <nirik> one releng one left...
18:30:59 <smooge> will wait for open flood
18:31:07 <nirik> .releng 11358
18:31:08 <zodbot> nirik: Issue #11358: Sync RCs to alt/stg dl.fp.o - releng - Pagure.io - https://pagure.io/releng/issue/11358
18:31:13 <phsmoura> f39?
18:31:36 <nirik> this just needs doing... med med ops?
18:31:39 <jednorozec> phsmoura, it change for f39
18:31:41 <smooge> nirik did my comment on 11358 make sense about dl not being faster
18:32:27 <nirik> yes, but... I am not sure how it relates to this issue. ;)
18:32:28 <jednorozec> nirik, I will add it to the sync script in the morning
18:32:30 <phsmoura> seems that we dont have a f39 tag  :/
18:32:38 <jednorozec> just was not sure if that is what we want
18:32:59 <jednorozec> phsmoura, create one, I usuall create it when new release change lands
18:33:06 <jednorozec> this is first for f39 it seems
18:33:21 <nirik> smooge: it's just that they want it synced right after it completes, instead of us waiting to manually do it.
18:33:46 <smooge> ah ok. the reason in the original was that they said getting it from dl would be faster
18:34:08 <smooge> i focused on that part too much
18:34:22 <smooge> or i misread completel
18:35:32 <nirik> or I did? alt is actually mirrorred... so thats an improvement over kojipkgs
18:36:08 <nirik> anyhow, if syncing them automatically makes them happy we can leave it at that. ;)
18:36:31 <nirik> Anyhow, we are over time, but I think we are finally thru tickets?
18:36:37 <nirik> #topic work plans / upcoming tasks
18:36:38 <smooge> well it is mirrored to download.fedoraproject.org. Downloading straight from dl.fedoraproject.org is going to not be mirrored. Mirroring also has to wait until the image has gotten to a lot of mirrors.
18:36:47 <smooge> I have +1 your ticket on reboots
18:37:26 <nirik> yes, but if it's in alt it does get mirrored... ie, download-ib01, download-cc-rdu01, mirrors.kernel.org, etc...
18:38:12 <phsmoura> sure, just before we close how can a new tag be created? Trying to find.. looking at the docs
18:38:37 <nirik> My plan for this week: I am going to be updating the mass update SOP, going to update/reboot staging hosts today, other non outage causing ones tomorrow, and then outage on wed. If anyone would like to play along, please let me know. ;)
18:39:00 <nirik> also as part of the outage will be updating bodhi in prod and fixing db-datanommer02 backups.
18:39:25 <nirik> phsmoura: it's in settings... there should be a 'tags' section?
18:39:48 <nirik> I was also thinking of reinstalling bastion02 today, and then doing 01 wed if nothing looks too bad.
18:39:55 <smooge> cool
18:40:00 <nirik> I also wanted to do batcave01, but thats going to be more... difficult, so perhaps after release.
18:41:24 <nirik> phsmoura: you still poking at the planet stuff? anything we can do to help?
18:41:29 <nirik> jednorozec: did you see the fedora media writer ticket? can you do the windows build?
18:42:16 <phsmoura> yes I'm. Probably will will work on the issues I opened in noggin too
18:42:24 <smooge> nirik: would you try a lockbox01 as a 'stage' for batcave01 or just do a reinstall ?
18:42:40 <nirik> Great. Do let us know if you get stuck or have questions...
18:43:09 <jednorozec> nirik, sure it just needs uploading
18:43:25 <nirik> smooge: I'd probibly just save the current lv and reinstall and then copy /home and /srv back... but I am not 100% sure we have everything we need in epel9 yet.
18:43:54 <phsmoura> I think I dont have perms to create tags tryibg to find 'settings' or anything that could make me create a new tag under releng repo but cant find it
18:44:11 <nirik> I guess we could play it by ear... if there's time/energy on wed after the outage stuff is done I could do it. Otherwise punt.
18:44:35 <nirik> huh, ok, let me look
18:46:03 <nirik> pagure seems slow today
18:46:09 <smooge> nirik: yes.
18:46:23 <smooge> i had to restart a lot of stuff on it friday I think
18:46:46 <smooge> it was giving sporadic 500's and then a bunch more
18:47:08 <nirik> phsmoura: as soon as I can get in, will adjust your permissions. ;)
18:48:00 <smooge> i am not sure if I did the 'right' thing on those restarts so it may be buggy because of that
18:48:44 <nirik> I'm restarting it again
18:50:12 <nirik> phsmoura: look now?
18:51:18 <smooge> nirik: I did an update on log01 recently which caught the last selinux policy. That should allow a faster outage
18:51:23 <nirik> Anyhow, anyone have anything else before we close?
18:51:28 <nirik> #info f38 final freeze next tuesday (week from tomorrow)
18:51:39 <phsmoura> there it is
18:52:11 <phsmoura> done
18:52:50 <nirik> cool. ;)
18:53:46 <nirik> ok, shall we all agree to meet again tomorrow? :)
18:54:39 <phsmoura> sure
18:54:40 <nirik> #endmeeting