18:00:01 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:01 <zodbot> Meeting started Thu Oct 15 18:00:01 2020 UTC.
18:00:01 <zodbot> This meeting is logged and archived in a public location.
18:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:01 <nirik> #chair mboddu nirik smooge pingou mobrien
18:00:01 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:01 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:01 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge
18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:02 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:00:03 <nirik> #topic Tickets needing review
18:00:05 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:00:07 <nirik> anyone around for standup today?
18:00:18 <smooge> here
18:00:33 <nirik> hey smooge. how did things go at the DC?
18:00:34 <smooge> can handle tickets
18:00:57 <smooge> I got some boxes racked, moved some wiring and got the ips for various boxes
18:01:11 <nirik> cool.
18:01:16 <nirik> .ticket 9391
18:01:17 <zodbot> nirik: Issue #9391: Cannot get root on retrace-stg.aws.fedoraproject.org - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9391
18:01:27 <smooge> so its time to put in tickets and see what work can be done to get dhcp and thngs going on an internal lan
18:01:37 <nirik> med/med/ops ?
18:01:52 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9391: medium-gain, medium-trouble, and ops https://pagure.io/fedora-infrastructure/issue/9391
18:01:53 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9391 https://pagure.io/fedora-infrastructure/issue/9391
18:01:56 <smooge> done
18:02:04 <nirik> I think we can't fix it as long as it's staging.
18:02:29 <smooge> so I think it needs to be not staging for the time being
18:02:30 <nirik> I guess I'd say just add his key to root users there as the easiest way forward
18:02:41 <smooge> ok sounds good
18:02:45 <mboddu> I am here, sorry for the delay
18:02:50 <nirik> hey mboddu
18:03:12 <nirik> there's one private ticket (which I am not sure why it's private). I will poke that one.
18:03:20 <nirik> thats it for infra... anything on releng side?
18:03:27 <mboddu> Yup
18:03:47 <mboddu> .releng 9811
18:03:48 <zodbot> mboddu: Issue #9811: Create detached signatures for the Ignition 2.7.0 release - releng - Pagure.io - https://pagure.io/releng/issue/9811
18:04:04 <mboddu> I will work on it, its med,med,ops?
18:04:23 <nirik> +1
18:04:24 <smooge> done
18:04:55 <mboddu> .releng 9810
18:04:56 <zodbot> mboddu: Issue #9810: Change impact: Python 3.10 - releng - Pagure.io - https://pagure.io/releng/issue/9810
18:05:10 <mboddu> Its just an impact change, they might request for side tag, but anyway, its low,low,ops
18:05:31 * mboddu updating the ticket
18:05:32 <nirik> yeah.
18:05:41 <smooge> oh sorry I was doing updates
18:05:44 <smooge> sorry mboddu
18:05:58 <mboddu> smooge: You are good, I meant update the ticket with a comment
18:06:03 <smooge> ok
18:06:18 <mboddu> Thats all
18:07:30 <mboddu> But I a topic to discuss
18:07:37 <nirik> go for it.
18:07:43 <mboddu> https://pagure.io/releng/failed-composes/issue/1920#comment-696382
18:07:56 <mboddu> nirik: ^ Can we somehow fix it? Now its spread to arm builders :(
18:08:48 <nirik> buy me 35 or so really big ssds?
18:08:59 <mboddu> smooge: ^ :P
18:09:44 <nirik> wait... is that the builder? odd if so
18:10:25 <nirik> Storage:
18:10:25 <nirik> Filesystem      Size  Used Avail Use% Mounted on
18:10:25 <nirik> /dev/vda2       135G   12G  116G  10% /
18:10:38 <mboddu> nirik: https://tenor.com/view/eleventhdoctor-motivation-youcandoit-finales-matt-smith-gif-5216973
18:10:41 <nirik> I think thats the size of the image, not the builder?
18:10:50 <smooge> well you know what they say.. 116GB is just not as much as it used to be
18:11:12 <mboddu> nirik: Nothing has changed since yesterday, but why it failed today not yesterday?
18:11:36 <mboddu> Something is fishy then
18:12:09 <nirik> fedora-repos is 14k larger. ;)
18:12:39 <mboddu> Oh right, I did push fedora-repos
18:13:00 <mboddu> But I dont think its the reason
18:13:03 <nirik> but yeah, it seems odd to fail one day and not another.
18:13:50 <nirik> ah, it's still there
18:13:57 <nirik> and indeed the image is 100% full
18:14:11 <nirik> /dev/mapper/loop1p3                         5.9G  5.3G     0 100% /var/lib/mock/f33-build-23372582-2256825/root/var/tmp/imgcreate-4idff4li/install_rootbtrfs.007
18:14:29 <mboddu> Okay, pwhalen opened https://pagure.io/fedora-kickstarts/pull-request/717#request_diff which should fix it
18:14:42 <smooge> ok I have to head out. back later
18:15:00 <nirik> yeah, worth doing
18:15:03 <nirik> smooge: have fun
18:15:07 <mboddu> Bye smooge, be safe
18:15:09 <fm-admin> pagure.issue.edit -- bellet edited the private fields of ticket fedora-infrastructure#9392 https://pagure.io/fedora-infrastructure/issue/9392
18:15:14 <mboddu> I just merged it
18:15:21 <nirik> ok.
18:15:29 <mboddu> Lets see what happens
18:15:48 <mboddu> Sorry, I thought its the storage on the builder before checking it
18:16:00 <nirik> it's really hard to tell sometimes. ;)
18:16:10 <nirik> ok, anything else to discuss ?
18:16:23 <nirik> #info recharge day tomorrow - no standup until monday
18:16:27 <mboddu> None from me
18:16:55 <nirik> #info staging koji is up and all but one builder (s390x). scratch builds working now.
18:17:28 <mboddu> nirik: Whats the status on bodhi stg?
18:17:30 <nirik> I need to get some firewall rules adjusted for the stg s390x builder.
18:17:46 <nirik> also I need to get pkgs.stg pointed to the right place
18:18:04 <nirik> no idea. I installed it a while back, but haven't looked at it yet
18:18:21 <mboddu> Okay
18:19:10 <nirik> looks like the web pod is down. let me fire a deployment
18:20:08 <nirik> huh... no idea:
18:20:11 <nirik> gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
18:20:45 <mboddu> Its not an urgency, just wondering whats going on, thats all
18:21:35 <nirik> yeah, we will need to sort it out. ;)
18:21:39 <nirik> ok, thanks everyone.
18:21:42 <nirik> #endmeeting