18:02:41 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:02:41 Meeting started Mon Mar 27 18:02:41 2023 UTC. 18:02:41 This meeting is logged and archived in a public location. 18:02:41 The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 18:02:41 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:02:41 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:02:42 #chair nirik nb smooge phsmoura 18:02:42 Current chairs: nb nirik phsmoura smooge 18:02:48 #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:02:48 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:02:54 #info meeting is 30 minutes MAX. At the end of 30, its stops 18:03:01 #info agenda is at https://board.net/p/fedora-infra-daily 18:03:06 #info reminder: speak up if you want to work on a ticket! 18:03:14 #topic Tickets needing review 18:03:14 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:03:42 There's a pile since we were all gone last week. :) 18:04:10 .ticket 11195 18:04:11 nirik: Issue #11195: local delete_old_oci_images.py broken by new docker-distribution upgrade - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11195 18:04:37 I filed this. It needs some python/ansible work... but it's not super urgent I guess. 18:04:44 so med med ops I suppose. 18:05:21 .ticket 11197 18:05:22 nirik: Issue #11197: genesisadaptive.com mirrors fail more often than not - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11197 18:05:50 I mailed the mirror admin, but have not heard anything. 18:05:56 So, I think I will disable it and close the ticket. They can re-enable it later once they sort things out. 18:07:53 .ticket 11201 18:07:54 nirik: Issue #11201: Planned outage: Upgrade of Copr servers - 2023-03-30 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11201 18:08:23 moving this to waiting on external 18:08:27 .ticket 11203 18:08:28 nirik: Issue #11203: Support creating addtional IAM users in AWS - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11203 18:08:45 it's unclear what they want here... needs more investigation. 18:09:42 marking it med/med/waiting on reporter for now 18:09:51 .ticket 11204 18:09:52 nirik: Issue #11204: DNF countme is broken for the week 2023-03-13->2023-03-19 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11204 18:10:22 so, I am unclear on what we need to do here. 18:10:23 I'll ask in the ticket I guess. 18:11:18 .ticket 11207 18:11:19 nirik: Issue #11207: Login Issue with transtats.fp.o - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11207 18:11:46 this might be related to another ticket we have about waiverdb having the same problem. ;( 18:11:58 I asked about that in ticket. In the meantime, med med ops I guess 18:12:25 .ticket 11209 18:12:28 nirik: Issue #11209: Planned Outage - Server updates/reboots - 2023-03-29 21:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11209 18:12:28 hello 18:12:56 I just filed this this morning. Need someone to review it. I am also reworking the mass update SOP... 18:13:13 it's quite a blast from the past... it talks about xen hosts, and running func, and machines that no longer exist. ;) 18:14:53 anyhow, I plan to update it and also if anyone wants to learn more about it, let me know. 18:14:53 Thats all the tickets I think. 18:14:55 in infra 18:15:37 on to releng. 18:15:44 I'm just going to mark the unretires as low/low 18:17:00 .releng 11345 18:17:01 nirik: Issue #11345: Missing f38 beta in vagrant box list - releng - Pagure.io - https://pagure.io/releng/issue/11345 18:17:21 I have no idea who handles this. I guess I can cc some folks in the ticket... 18:17:29 med med ops I suppose? 18:18:15 .releng 11346 18:18:16 nirik: Issue #11346: please remove linux-system-roles branch - releng - Pagure.io - https://pagure.io/releng/issue/11346 18:18:41 low low ops? 18:18:56 .releng 11349 18:18:57 nirik: Issue #11349: Fedora Media Writer 5.0.6 released - releng - Pagure.io - https://pagure.io/releng/issue/11349 18:19:40 med med ops... just needs doing by jednorozec and me. 18:20:09 I think ideally we would get time to spin up the instances to do this in AWS... then it could be done by anyone with permissions to. 18:20:20 .releng 11352 18:20:21 nirik: Issue #11352: SOP documents do not cover all aspects of beta install and cleanup - releng - Pagure.io - https://pagure.io/releng/issue/11352 18:20:48 so, docs need updating. I must have missed the perms on that dir. :( 18:21:07 but the beta stuff we were keeping for some reason... but I can't find it now, so I guess we could delete them 18:21:59 low low ops and cced jednorozec 18:22:33 .releng 11353 18:22:34 nirik: Issue #11353: htdig package has licensing issues - releng - Pagure.io - https://pagure.io/releng/issue/11353 18:22:47 hi nirik, sorry. Forgot we are not in winter time anymore 18:23:28 yeah, although it sure seems like winter here. ;) 18:23:49 so, I guess this one needs more investigation... high/high ops... 18:26:24 , now just 11357? 18:27:26 sorry, I got sidetracked looking at the last one 18:27:36 .releng 11357 18:27:37 phsmoura: Issue #11357: Change: Changes of defaults in createrepo_c-1.0.0 - releng - Pagure.io - https://pagure.io/releng/issue/11357 18:28:07 This is likely going to need bodhi changes. 18:28:15 we were discussing it on the devel list. 18:28:37 Might also need pungi changes. 18:28:45 SO, I would say med med ops and leave open to investigate changes we need to make 18:29:19 * jednorozec lurking around 18:29:30 ok, done 18:29:41 add changes and f39 18:29:53 👋 18:30:06 one releng one left... 18:30:59 will wait for open flood 18:31:07 .releng 11358 18:31:08 nirik: Issue #11358: Sync RCs to alt/stg dl.fp.o - releng - Pagure.io - https://pagure.io/releng/issue/11358 18:31:13 f39? 18:31:36 this just needs doing... med med ops? 18:31:39 phsmoura, it change for f39 18:31:41 nirik did my comment on 11358 make sense about dl not being faster 18:32:27 yes, but... I am not sure how it relates to this issue. ;) 18:32:28 nirik, I will add it to the sync script in the morning 18:32:30 seems that we dont have a f39 tag :/ 18:32:38 just was not sure if that is what we want 18:32:59 phsmoura, create one, I usuall create it when new release change lands 18:33:06 this is first for f39 it seems 18:33:21 smooge: it's just that they want it synced right after it completes, instead of us waiting to manually do it. 18:33:46 ah ok. the reason in the original was that they said getting it from dl would be faster 18:34:08 i focused on that part too much 18:34:22 or i misread completel 18:35:32 or I did? alt is actually mirrorred... so thats an improvement over kojipkgs 18:36:08 anyhow, if syncing them automatically makes them happy we can leave it at that. ;) 18:36:31 Anyhow, we are over time, but I think we are finally thru tickets? 18:36:37 #topic work plans / upcoming tasks 18:36:38 well it is mirrored to download.fedoraproject.org. Downloading straight from dl.fedoraproject.org is going to not be mirrored. Mirroring also has to wait until the image has gotten to a lot of mirrors. 18:36:47 I have +1 your ticket on reboots 18:37:26 yes, but if it's in alt it does get mirrored... ie, download-ib01, download-cc-rdu01, mirrors.kernel.org, etc... 18:38:12 sure, just before we close how can a new tag be created? Trying to find.. looking at the docs 18:38:37 My plan for this week: I am going to be updating the mass update SOP, going to update/reboot staging hosts today, other non outage causing ones tomorrow, and then outage on wed. If anyone would like to play along, please let me know. ;) 18:39:00 also as part of the outage will be updating bodhi in prod and fixing db-datanommer02 backups. 18:39:25 phsmoura: it's in settings... there should be a 'tags' section? 18:39:48 I was also thinking of reinstalling bastion02 today, and then doing 01 wed if nothing looks too bad. 18:39:55 cool 18:40:00 I also wanted to do batcave01, but thats going to be more... difficult, so perhaps after release. 18:41:24 phsmoura: you still poking at the planet stuff? anything we can do to help? 18:41:29 jednorozec: did you see the fedora media writer ticket? can you do the windows build? 18:42:16 yes I'm. Probably will will work on the issues I opened in noggin too 18:42:24 nirik: would you try a lockbox01 as a 'stage' for batcave01 or just do a reinstall ? 18:42:40 Great. Do let us know if you get stuck or have questions... 18:43:09 nirik, sure it just needs uploading 18:43:25 smooge: I'd probibly just save the current lv and reinstall and then copy /home and /srv back... but I am not 100% sure we have everything we need in epel9 yet. 18:43:54 I think I dont have perms to create tags tryibg to find 'settings' or anything that could make me create a new tag under releng repo but cant find it 18:44:11 I guess we could play it by ear... if there's time/energy on wed after the outage stuff is done I could do it. Otherwise punt. 18:44:35 huh, ok, let me look 18:46:03 pagure seems slow today 18:46:09 nirik: yes. 18:46:23 i had to restart a lot of stuff on it friday I think 18:46:46 it was giving sporadic 500's and then a bunch more 18:47:08 phsmoura: as soon as I can get in, will adjust your permissions. ;) 18:48:00 i am not sure if I did the 'right' thing on those restarts so it may be buggy because of that 18:48:44 I'm restarting it again 18:50:12 phsmoura: look now? 18:51:18 nirik: I did an update on log01 recently which caught the last selinux policy. That should allow a faster outage 18:51:23 Anyhow, anyone have anything else before we close? 18:51:28 #info f38 final freeze next tuesday (week from tomorrow) 18:51:39 there it is 18:52:11 done 18:52:50 cool. ;) 18:53:46 ok, shall we all agree to meet again tomorrow? :) 18:54:39 sure 18:54:40 #endmeeting