18:07:28 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:07:29 <zodbot> Meeting started Thu Oct 20 18:07:28 2022 UTC. 18:07:29 <zodbot> This meeting is logged and archived in a public location. 18:07:29 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 18:07:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:07:29 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:07:29 <phsmoura> #chair nirik mobrien aheath1992 smooge 18:07:29 <zodbot> Current chairs: aheath1992 mobrien nirik phsmoura smooge 18:07:29 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:07:29 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:07:29 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:07:29 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily 18:07:40 <phsmoura> sorry I was late 18:07:52 <phsmoura> #info reminder: speak up if you want to work on a ticket! 18:07:52 <phsmoura> #topic Tickets needing review 18:08:11 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:08:54 <Ebeneezer_Smooge> hrllo phsmoura 18:09:07 <Ebeneezer_Smooge> hope you are doing ok 18:09:39 <phsmoura> hi Ebeneezer_Smooge, yes Im fine thanks and you? 18:10:01 <Ebeneezer_Smooge> I am doing ok. I can work on 10950 18:10:40 <phsmoura> .ticket 10950 18:10:41 <zodbot> phsmoura: Issue #10950: read-only /var/tmp/ on buildvm-ppc64le-23 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10950 18:10:47 <phsmoura> cool 18:10:56 <phsmoura> low,low,ops? 18:11:17 <Ebeneezer_Smooge> yep 18:11:42 <phsmoura> ok 18:11:56 <phsmoura> .ticket 10948 18:11:57 <zodbot> phsmoura: Issue #10948: CentOS Stream 9 VM Request for Minetest game server - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10948 18:12:15 <phsmoura> Im not sure about this one. low gain med trouble ops? 18:12:47 <Ebeneezer_Smooge> low/high/ops/meeting 18:13:46 <phsmoura> meeting? 18:14:41 <Ebeneezer_Smooge> sorry probably the wrong flag. It needs someone with amazon AND it needs approval from bcotton or mattdm that this is important 18:15:13 <Ebeneezer_Smooge> s/with amazon/with amazon rights/ 18:15:30 <Ebeneezer_Smooge> low gain/high trouble/ops 18:15:47 <phsmoura> we have aws flag, the reporter added mattdm in cc 18:16:44 <phsmoura> updated with aws flag 18:17:26 <Ebeneezer_Smooge> ok so buildvm-ppc64le-23 has a bad disk 18:18:26 <phsmoura> you solving it already 18:18:35 <Ebeneezer_Smooge> I am shutting it down as it is not possible to repair without a reboot. that is 10950 18:18:44 <Ebeneezer_Smooge> s/reboot/rebuild/ 18:19:10 <phsmoura> yep. Im going move to releng we have 1 ticket there 18:19:13 <phsmoura> #info https://pagure.io/releng/issues?status=Open 18:19:24 <phsmoura> .releng 11102 18:19:25 <zodbot> phsmoura: Issue #11102: Remove of private-c99-port branches - releng - Pagure.io - https://pagure.io/releng/issue/11102 18:20:42 <phsmoura> low trouble high gain ops I guess? 18:20:49 <Ebeneezer_Smooge> yep. 18:20:55 <Ebeneezer_Smooge> med trouble maybe 18:21:04 <Ebeneezer_Smooge> I don't know how easy it is to remove branches from releng 18:21:24 <jednorozec> med 18:21:58 <phsmoura> done med high ops 18:22:39 <phsmoura> #topic Planning, Upcoming work and Open floor 18:23:01 <jednorozec> f37 was nogo 18:23:04 <phsmoura> anything for open floor? 18:23:09 <jednorozec> so there will be new RC request 18:23:49 <jednorozec> there was a glitch in koji and some packages went unsigned, so I did some clenup today and updates* composes are out for all releases 18:24:10 <Ebeneezer_Smooge> yesterday there was an extended outage on a Power9 system bvmhost-p09-03 18:24:47 <Ebeneezer_Smooge> nirik got it back up but the guests seem to be reporting btrfs errors 18:25:16 <Ebeneezer_Smooge> I would suggest we get them all out of of koji as usable 18:25:17 <phsmoura> what time that happened Ebeneezer_Smooge, do you know? 18:25:57 <Ebeneezer_Smooge> i think kevin got it rebooted at 00:00 UTC or so 18:26:22 <Ebeneezer_Smooge> I had it power cycled earlier in the day but it was stuck on needing a password until he did the final work 18:26:48 <Ebeneezer_Smooge> jednorozec, what is needed to disable builders in koji as I am not sure these boxes are 'working' in any good sense 18:27:05 <phsmoura> I think I had a script running about that time I was gathering info from our infra so we can update the firmware it lasted about 2 hrs to run 18:27:17 <phsmoura> but I only hit the mgmt machines 18:27:17 <Ebeneezer_Smooge> I think -23 was disabled earlier by sharkcz but I just checked -21 and it is also complaining about btrfs errors 18:27:41 <jednorozec> koji disable-host show do the job 18:29:49 <Ebeneezer_Smooge> ok I am going to reboot a couple of these vm's and see if they are recoverable. -23 is definitely a rebuild 18:30:44 <Ebeneezer_Smooge> phsmoura, do you have ability to ack things in nagios? 18:31:24 <phsmoura> sorry, what do you mean ack things? 18:31:53 <Ebeneezer_Smooge> go into nagios and acknowledge an alert so that it doesn't ping #fedora-noc every 10 minutes 18:32:58 <phsmoura> no I dont, but can do it if theres a sop or something 18:35:27 <Ebeneezer_Smooge> ok it might need an FBR to allow you to do that .. I will see if I can still do myself and then talk with you in #fedora-noc about what I am doing 18:35:41 <Ebeneezer_Smooge> that wya you can do it once whatever change is needed 18:36:43 <phsmoura> ok. Im also looking for a sop about that 18:37:07 <phsmoura> we passed a bit the time today thanks all for attending 18:37:07 <Ebeneezer_Smooge> yeah I may never have read one 18:37:15 <Ebeneezer_Smooge> s/read/written/ 18:38:28 <phsmoura> I see, ok. if you ping me we can work together 18:39:13 <phsmoura> #endmeeting