18:00:01 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:01 <zodbot> Meeting started Thu Oct 15 18:00:01 2020 UTC. 18:00:01 <zodbot> This meeting is logged and archived in a public location. 18:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:01 <nirik> #chair mboddu nirik smooge pingou mobrien 18:00:01 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:00:01 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:01 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge 18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:02 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily 18:00:03 <nirik> #topic Tickets needing review 18:00:05 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:07 <nirik> anyone around for standup today? 18:00:18 <smooge> here 18:00:33 <nirik> hey smooge. how did things go at the DC? 18:00:34 <smooge> can handle tickets 18:00:57 <smooge> I got some boxes racked, moved some wiring and got the ips for various boxes 18:01:11 <nirik> cool. 18:01:16 <nirik> .ticket 9391 18:01:17 <zodbot> nirik: Issue #9391: Cannot get root on retrace-stg.aws.fedoraproject.org - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9391 18:01:27 <smooge> so its time to put in tickets and see what work can be done to get dhcp and thngs going on an internal lan 18:01:37 <nirik> med/med/ops ? 18:01:52 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9391: medium-gain, medium-trouble, and ops https://pagure.io/fedora-infrastructure/issue/9391 18:01:53 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9391 https://pagure.io/fedora-infrastructure/issue/9391 18:01:56 <smooge> done 18:02:04 <nirik> I think we can't fix it as long as it's staging. 18:02:29 <smooge> so I think it needs to be not staging for the time being 18:02:30 <nirik> I guess I'd say just add his key to root users there as the easiest way forward 18:02:41 <smooge> ok sounds good 18:02:45 <mboddu> I am here, sorry for the delay 18:02:50 <nirik> hey mboddu 18:03:12 <nirik> there's one private ticket (which I am not sure why it's private). I will poke that one. 18:03:20 <nirik> thats it for infra... anything on releng side? 18:03:27 <mboddu> Yup 18:03:47 <mboddu> .releng 9811 18:03:48 <zodbot> mboddu: Issue #9811: Create detached signatures for the Ignition 2.7.0 release - releng - Pagure.io - https://pagure.io/releng/issue/9811 18:04:04 <mboddu> I will work on it, its med,med,ops? 18:04:23 <nirik> +1 18:04:24 <smooge> done 18:04:55 <mboddu> .releng 9810 18:04:56 <zodbot> mboddu: Issue #9810: Change impact: Python 3.10 - releng - Pagure.io - https://pagure.io/releng/issue/9810 18:05:10 <mboddu> Its just an impact change, they might request for side tag, but anyway, its low,low,ops 18:05:31 * mboddu updating the ticket 18:05:32 <nirik> yeah. 18:05:41 <smooge> oh sorry I was doing updates 18:05:44 <smooge> sorry mboddu 18:05:58 <mboddu> smooge: You are good, I meant update the ticket with a comment 18:06:03 <smooge> ok 18:06:18 <mboddu> Thats all 18:07:30 <mboddu> But I a topic to discuss 18:07:37 <nirik> go for it. 18:07:43 <mboddu> https://pagure.io/releng/failed-composes/issue/1920#comment-696382 18:07:56 <mboddu> nirik: ^ Can we somehow fix it? Now its spread to arm builders :( 18:08:48 <nirik> buy me 35 or so really big ssds? 18:08:59 <mboddu> smooge: ^ :P 18:09:44 <nirik> wait... is that the builder? odd if so 18:10:25 <nirik> Storage: 18:10:25 <nirik> Filesystem Size Used Avail Use% Mounted on 18:10:25 <nirik> /dev/vda2 135G 12G 116G 10% / 18:10:38 <mboddu> nirik: https://tenor.com/view/eleventhdoctor-motivation-youcandoit-finales-matt-smith-gif-5216973 18:10:41 <nirik> I think thats the size of the image, not the builder? 18:10:50 <smooge> well you know what they say.. 116GB is just not as much as it used to be 18:11:12 <mboddu> nirik: Nothing has changed since yesterday, but why it failed today not yesterday? 18:11:36 <mboddu> Something is fishy then 18:12:09 <nirik> fedora-repos is 14k larger. ;) 18:12:39 <mboddu> Oh right, I did push fedora-repos 18:13:00 <mboddu> But I dont think its the reason 18:13:03 <nirik> but yeah, it seems odd to fail one day and not another. 18:13:50 <nirik> ah, it's still there 18:13:57 <nirik> and indeed the image is 100% full 18:14:11 <nirik> /dev/mapper/loop1p3 5.9G 5.3G 0 100% /var/lib/mock/f33-build-23372582-2256825/root/var/tmp/imgcreate-4idff4li/install_rootbtrfs.007 18:14:29 <mboddu> Okay, pwhalen opened https://pagure.io/fedora-kickstarts/pull-request/717#request_diff which should fix it 18:14:42 <smooge> ok I have to head out. back later 18:15:00 <nirik> yeah, worth doing 18:15:03 <nirik> smooge: have fun 18:15:07 <mboddu> Bye smooge, be safe 18:15:09 <fm-admin> pagure.issue.edit -- bellet edited the private fields of ticket fedora-infrastructure#9392 https://pagure.io/fedora-infrastructure/issue/9392 18:15:14 <mboddu> I just merged it 18:15:21 <nirik> ok. 18:15:29 <mboddu> Lets see what happens 18:15:48 <mboddu> Sorry, I thought its the storage on the builder before checking it 18:16:00 <nirik> it's really hard to tell sometimes. ;) 18:16:10 <nirik> ok, anything else to discuss ? 18:16:23 <nirik> #info recharge day tomorrow - no standup until monday 18:16:27 <mboddu> None from me 18:16:55 <nirik> #info staging koji is up and all but one builder (s390x). scratch builds working now. 18:17:28 <mboddu> nirik: Whats the status on bodhi stg? 18:17:30 <nirik> I need to get some firewall rules adjusted for the stg s390x builder. 18:17:46 <nirik> also I need to get pkgs.stg pointed to the right place 18:18:04 <nirik> no idea. I installed it a while back, but haven't looked at it yet 18:18:21 <mboddu> Okay 18:19:10 <nirik> looks like the web pod is down. let me fire a deployment 18:20:08 <nirik> huh... no idea: 18:20:11 <nirik> gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3> 18:20:45 <mboddu> Its not an urgency, just wondering whats going on, thats all 18:21:35 <nirik> yeah, we will need to sort it out. ;) 18:21:39 <nirik> ok, thanks everyone. 18:21:42 <nirik> #endmeeting