15:02:54 <mboddu> #startmeeting RELENG (2020-06-30)
15:02:54 <zodbot> Meeting started Tue Jun 30 15:02:54 2020 UTC.
15:02:54 <zodbot> This meeting is logged and archived in a public location.
15:02:54 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:54 <zodbot> The meeting name has been set to 'releng_(2020-06-30)'
15:02:55 <mboddu> #meetingname releng
15:02:55 <zodbot> The meeting name has been set to 'releng'
15:02:55 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
15:02:55 <mboddu> #topic init process
15:02:55 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:03:19 <sharkcz> hello, I'm here today
15:04:16 <mboddu> Hey sharkcz
15:04:20 <mboddu> How are you doing?
15:04:33 <sharkcz> I'm fine
15:06:48 <mboddu> I think nirik is not here today
15:07:07 <mboddu> Good for him that he is taking some rest
15:07:19 <mboddu> #topic #9562 cloud/container image creation fails on s390x
15:07:25 <mboddu> #link https://pagure.io/releng/issue/9562
15:07:36 <mboddu> So, I haven't dug anything into it yet
15:07:45 <sharkcz> any idea about this one?
15:07:45 <mboddu> sharkcz: Anything you observed?
15:08:11 <sharkcz> I belive it first appeared after updating koji to 1.21 on the builders
15:11:19 <mboddu> sharkcz: What do you mean by https://pagure.io/releng/issue/9562#comment-662551 ?
15:12:09 <sharkcz> I have meant like updating only koji, but not oz (in case koji 1.21 needs newer oz than koji 1.20)
15:12:41 <mboddu> sharkcz: Okay, let me check that quickly
15:13:51 <sharkcz> because missing "image.os_plugin.abort()" should fail on all arches, there is (hopefully) nothing s390 specific there
15:16:18 <mboddu> I compared both s390x and x86_64 builders oz and imagefactory and both of them are running the same versions and the latest one's as well
15:16:44 <sharkcz> ok
15:17:09 <mboddu> Maybe a koji issue?
15:18:21 <sharkcz> it looks as code issue, I will probably need to look closer
15:19:22 <sharkcz> maybe something changed there between 1.20.1 and 1.21 ...
15:21:09 <mboddu> https://koji.fedoraproject.org/koji/taskinfo?taskID=45808274 is when they started failing it seems
15:21:56 <mboddu> And it seems when we moved to iad2 dc and also when we upgraded to 1.21
15:22:11 <sharkcz> yup
15:22:45 <sharkcz> so in theory there could be some firewall rule missing too
15:24:48 <mboddu> Yup
15:25:54 * nirik waves, reads up
15:27:24 <mboddu> nirik: Go back to bed
15:30:53 <nirik> naw, I'm fine
15:32:30 <mboddu> So, we think it might be some firewall issue or koji 1.21 or imagefactory or oz updates or iad2 ....
15:34:41 * nirik has no idea without looking.
15:37:22 <nirik> ok. I know what it is.
15:37:35 <nirik> imagefactory packaging issue.
15:38:53 <nirik> f31 used python 3.7 and f32 uses python 3.8... and imagefactory tries to make these links in /etc/imagefactory/plugins.d/ that point to the files, but it makes them in post and sometimes does the wrong thing on upgrades.
15:40:09 <mboddu> But why s390x only?
15:40:22 <mboddu> Doesn't it should happen on all arches?
15:41:18 <nirik> good question. :) All the other ones were reinstalled fresh with f32.
15:41:27 <nirik> only s390x were upgraded
15:42:39 <mboddu> Ahhh okay, so, uninstalling imagefactory and reinstalling it again on s390 should fix this?
15:42:49 <mboddu> *might* fix this?
15:43:02 <sharkcz> or editing the symlinks by hand
15:43:22 <nirik> yeah, I am testing a reinstall... not sure if it does the right thing or not.
15:43:24 <mboddu> sharkcz: Its easy to write an adhoc command to do this
15:43:34 <sharkcz> ok
15:43:39 <mboddu> this = uninstalling and installing of imagefactory
15:44:50 <nirik> it's just the plugins... but yeah, removing it and installing again fixes it.
15:45:40 <mboddu> Okay, I will remove them and install them on s390x builders
15:46:15 <nirik> mboddu: great, thanks.
15:46:24 <mboddu> #info Kevin found that it might be due to a packaging bug, so mboddu will uninstall and install imagefactory on all s390 builders
15:46:47 <mboddu> #topic Open Floor
15:47:12 <mboddu> nirik: Whats the status on installing the hardware that got shipped from phx2 last week?
15:47:39 <nirik> we are working thru it... I hope to add more builders today/tomorrow.
15:48:58 <mboddu> nirik++
15:49:00 <mboddu> Thanks nirik
15:49:43 <siddharthvipul> oh I have one
15:50:00 <siddharthvipul> mboddu: container base images are failing
15:50:23 <mboddu> siddharthvipul: On s390x or all arches?
15:50:30 <mboddu> If s390x we were just talking about it
15:50:31 <siddharthvipul> s390x
15:50:39 <siddharthvipul> I see
15:50:55 <siddharthvipul> I will skip that in script.. thanks
15:51:25 <mboddu> siddharthvipul: nirik debugged the issue and I am fixing it as we speak
15:51:36 <siddharthvipul> oh you all are awesome
15:51:43 <siddharthvipul> I will bug you later then :)
15:53:36 <mboddu> Anything else?
15:53:45 <siddharthvipul> nothing here :)
15:54:02 <siddharthvipul> mboddu++ nirik++ for s390x issue.. and well, so manyother work
15:54:14 <siddharthvipul> of course I am out of karma :p
15:54:39 <mboddu> Well, its all nirik, I am just running a simple adhoc command to fix it :)
15:55:24 <mboddu> Anyway, if nothing else, I will give back 5 of your min back
15:55:34 <mboddu> Thanks guys for joining
15:55:36 <mboddu> #endmeeting