15:00:51 <mboddu> #startmeeting RELENG (2020-05-05)
15:00:51 <zodbot> Meeting started Tue May  5 15:00:51 2020 UTC.
15:00:51 <zodbot> This meeting is logged and archived in a public location.
15:00:51 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:51 <zodbot> The meeting name has been set to 'releng_(2020-05-05)'
15:00:51 <mboddu> #meetingname releng
15:00:51 <zodbot> The meeting name has been set to 'releng'
15:00:51 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
15:00:51 <mboddu> #topic init process
15:00:51 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:01:20 * sharkcz is here, hi mboddu
15:01:24 <nirik> morning
15:01:48 <mboddu> Morning sharkcz and nirik
15:02:35 <mboddu> I wanna start with couple of things
15:02:44 <mboddu> #topic ODCS compose
15:02:59 <mboddu> Thanks to nirik we think we have everything we need
15:03:26 <mboddu> The recent compose failed due to "RuntimeError: Failed to mount /srv/odcs/odcs-1237/work/global/tmp/tweak_buildinstall_dm2srp12/images/efiboot.img"
15:03:52 <mboddu> This *should* be fixed by adding buildinstall_use_guestmount = True to pungi config
15:04:00 <mboddu> Which I just did and started another compose
15:04:01 <nirik> cool
15:04:18 <mboddu> Hopefully, this is all it needs :)
15:04:55 <mboddu> #info nirik was able to mount /srv/odcs to runroot hosts and we are currently testing it
15:05:12 <mboddu> #topic Bodhi rawhide
15:05:34 <mboddu> A bunch of rawhide builds got stuck in testing https://bodhi.fedoraproject.org/updates/?releases=F33&status=testing&page=7
15:06:11 <mboddu> I pinged pingou about it in #bodhi and he asked me to restart the bodhi-celery
15:06:21 <mboddu> Which I just did before this meeting has started
15:06:48 <mboddu> Maybe I need to re-tag things, but I am giving bodhi couple of min before I start tagging builds again
15:07:24 <nirik> yes, better to be patient. ;)
15:07:53 <mboddu> Sorry nirik about the free-task thing, I was looking at this issue and didn't pay attention to what it was, so I just free'd the task :(
15:08:07 <nirik> no biggie
15:08:32 <nirik> might need cverna to look at bodhi...
15:08:44 <mboddu> nirik: He is on training, thats why I pinged pingou
15:08:53 <nirik> ah, missed that, ok
15:09:47 <mboddu> Well, I technically pinged cverna as well when I pinged pingou stating that cverna is on training :)
15:09:55 <nirik> ha
15:10:00 <mboddu> And now he got more pings :D
15:10:51 <nirik> yeah, it's hard to avoid
15:11:38 <mboddu> #topic #9435 Containers builds fail without a known reason quite often
15:11:44 <mboddu> #link https://pagure.io/releng/issue/9435
15:11:56 <mboddu> sharkcz: Just wondering, do you the reporter of this issue?
15:12:12 <mboddu> ;)
15:12:41 <sharkcz> yes, I do, it's a guy in Brno office, same last name :-)
15:13:02 <mboddu> Or is that your clone :P
15:13:33 <nirik> again (since he's not here) OSBS is for clemont. ;)
15:13:34 <sharkcz> no, there is a better clone, with the same first and last name (and same office) ;-)
15:14:30 <mboddu> Hehe, okay, I wanted to bring it up and see if its related to infra in anyway
15:14:39 <mboddu> I will assign the ticket to him
15:15:48 <nirik> I'm not sure... how to even debug
15:16:27 <mboddu> Okay
15:16:48 <mboddu> #topic #9408 Tag ghc into EPEL8 buildroot for ghc bootstrap
15:16:54 <mboddu> #link https://pagure.io/releng/issue/9408
15:17:19 <nirik> this just needs doing. I had it on my list but I have too many things on my list. ;)
15:17:33 <mboddu> Sure, I can understand
15:18:01 <mboddu> I never pulled in content from rhel before, but I am happy to help
15:18:04 <nirik> I've also never imported things, so not sure.
15:18:19 <mboddu> I guess, all it needs is pulling from centos and adding it as a external repo to epel8 buildroot, right?
15:18:20 <nirik> cool. perhaps we could setup a time to work on it together?
15:18:38 <mboddu> Sure, that would help
15:18:52 <nirik> oh, I suppose we could do that, I was thinking we would import it directly into epel8...
15:19:34 <nirik> but an external repo should work, and might be easier since we can then just delete it after the bootstrap
15:19:46 <mboddu> My reason of doing it as external repo because, in future, we can grab other things if people want and dont need to do any special things
15:19:48 <nirik> hum, but wait...
15:19:49 <mboddu> On koji side
15:19:51 <sharkcz> +1 for external repo
15:20:25 * mboddu is waiting...
15:20:32 <mboddu> :P
15:20:33 <nirik> well, this should be very much the exception.
15:20:35 <sharkcz> if possible, seeing nirik's "wait" :-)
15:20:42 <nirik> we shouldn't just add stuff because people want it.
15:21:02 <mboddu> Definitely, but what if there are other exceptions like this
15:21:12 <nirik> I was pondering if ghc was a module if the external repo would work.
15:21:38 * mboddu hates to say this, but...
15:21:43 <mboddu> grobisplitter? :P
15:22:05 <sharkcz> if you omit the module metadata and treat it as just rpms?
15:22:22 <mboddu> "Buildroot-only package" from the comment
15:22:53 <mboddu> He says its a package, which I *assume* its a normal rpm
15:24:09 <nirik> we have to download it as packages
15:24:15 <nirik> we have no modular repodata
15:24:25 <nirik> so it should work I think.
15:25:28 <mboddu> #info mboddu and nirik will work on adding the package to an external repo for epel8 buildroot
15:25:46 <mboddu> nirik: What day do you think is better for you so that I can schedule a meeting?
15:26:04 <mboddu> Your calendar definitely wont reflect the dc move work, hence the question :)
15:27:00 <nirik> yeah.
15:27:03 * nirik looks
15:27:33 <nirik> how about tomorrow, 17utc?
15:27:39 <mboddu> ack
15:28:48 <mboddu> Meeting invite sent
15:30:06 <nirik> thanks
15:33:49 <mboddu> #topic #9068 Please delete `epel8` and `epel8-playground` branches for `rpms/buildbot`
15:33:55 <mboddu> #link https://pagure.io/releng/issue/9068
15:34:33 * nirik re-reads.
15:34:42 <nirik> were there any builds ever done from those?
15:35:22 <mboddu> nirik: Is there any way to check that? Other than trusting the requester?
15:35:35 <mboddu> And is my understanding in https://pagure.io/releng/issue/9068#comment-644769 correct?
15:36:09 <nirik> well, I mean we can look at koji and see no epel8/epel8-playground builds.
15:36:49 <nirik> yeah, not sure. I don't think this case was considered
15:37:47 <nirik> There was some script someone came up with that could tell if something was deleteable.
15:38:03 <mboddu> I think it was kalev
15:38:18 <mboddu> But I dont think it will check if any builds were submitted or not
15:38:23 <mboddu> It does other thing
15:38:41 <mboddu> More like matching the hashes from master
15:38:54 <mboddu> branch
15:39:09 <nirik> BRANCH=epel8
15:39:09 <nirik> if git rev-list origin/master | grep `git rev-parse remotes/origin/$BRANCH` ; then
15:39:09 <nirik> echo "OK to remove" ; else
15:39:09 <nirik> echo "NOT ok to remove" ;
15:39:09 <nirik> fi
15:39:10 <nirik> NOT ok to remove
15:39:12 <nirik> yeah
15:39:20 <nirik> so, punt back to fesco seems reasonable.
15:39:36 <nirik> you want me to file a fesco ticket? or you want to? or should we ask reporter to?
15:39:51 <mboddu> I can file a ticket :)
15:40:20 <mboddu> nirik: How did you find it so easily?
15:40:28 <mboddu> ^ the remove branch code
15:41:36 <nirik> I remembered it was int he fesco ticket, searched for that and found it.
15:42:05 <mboddu> Ah okay :)
15:42:27 <mboddu> #topic Alternate Arches Update
15:42:31 <mboddu> sharkcz: any updates?
15:43:13 <sharkcz> nope, but https://pagure.io/koji/issue/1974 should/might have debug data collected
15:43:35 <sharkcz> nirik: can you check the logs or is there a way for me to look?
15:43:53 <nirik> I can try... but I am not sure if it was still in debug mode (hopefully so).
15:44:32 <sharkcz> thx, I hope we can get some data ...
15:44:48 <sharkcz> because the eln rebuild was a good chance for it
15:45:18 <nirik> it seems like it happens more often the longer the vm's have been up too...
15:45:49 <sharkcz> hmm
15:46:20 <nirik> but anyhow, I have added to my list to gather logs on it.
15:46:30 <mboddu> Thanks nirik
15:46:36 <mboddu> Let me know and I can help you as well
15:46:52 <nirik> sure, if you want to look, please feel free. ;)
15:47:04 * nirik gets more coffee, back in a min
15:47:16 <mboddu> Also, I have another update in this
15:47:33 * mboddu dont remember when he added something to alt arch updates
15:47:54 <mboddu> sharkcz: https://pagure.io/pungi/issue/1385#comment-649977
15:48:00 <mboddu> So, it needs koji changes
15:48:23 <mboddu> ^ RE: not producing vmdk images for ppc64le and s390x
15:49:02 <sharkcz> mboddu: can't we learn pungi to produce the vmdk format only for select arches instead?
15:49:22 <mboddu> sharkcz: Nope, as the images are generated by koji, not pungi
15:49:36 <mboddu> pungi just asks koji to create the images and gathers them when they are done
15:49:42 <sharkcz> aha, ok
15:50:16 <sharkcz> then we need fixing koji :-)
15:50:31 <mboddu> So, how much do we need this?
15:50:43 <mboddu> Is it really worth asking koji folks to spend their time on this?
15:50:58 <nirik> well, it saves us some compute / storage right?
15:51:02 <sharkcz> I think it is worth
15:51:08 <sharkcz> nirik: right
15:51:13 <mboddu> nirik: Yes
15:51:36 <sharkcz> it would gives us felxibility for future formats too
15:51:49 <mboddu> Okay, I will create a koji ticket
15:51:53 <sharkcz> thx
15:52:03 <mboddu> #info mboddu to create a koji ticket for this
15:52:11 <mboddu> #topic Open Floor
15:52:18 <mboddu> Anybody has anything to share?
15:52:37 <sharkcz> I don't
15:52:54 <nirik> I do. ;)
15:53:10 <mboddu> Its getting warmer here and I can get my motorcycle and start riding ;)
15:53:14 <mboddu> nirik: Sure, go ahead
15:53:32 <nirik> so, theres a number of somewhat related things we need to do... or can do...
15:54:02 <nirik> the new rpm with sqlite is coming out (or already did)? and we wanted to enable bootstrap in mock with that.
15:54:07 <nirik> also there's a koji upstream update
15:54:14 <nirik> and we could move builders to f32
15:54:44 <nirik> or... we could do as little as we can now... and since we are moving to the new dc in 5 weeks just do those then.
15:55:00 <sharkcz> ha, one thing - mock 2.2 doesn't work right on s390x - https://github.com/rpm-software-management/mock/issues/571
15:55:02 <nirik> but that means it migh be harder testing things to make sure they are working or why they are not
15:55:36 <nirik> ah, fun.
15:55:56 <mboddu> nirik: I think we should try to do koji update and builders upgrade to F32 before the move, if possible
15:56:54 <sharkcz> builder upgrades are dangerous in general, see the mock issue, not sure it's only s390x
15:57:02 <mboddu> nirik: If we decide to do this before the move, can we have a video call and you do the work and we will record it (since we dont have time for explanations)?
15:57:04 <nirik> ok. so should we wait on mock update before scheduling something?
15:57:18 <mboddu> Yes, thats what I think
15:57:55 <nirik> well, I am happy to work with folks on it. I don't know that a video is a good idea... I'd prefer irc where everyone can see whats typed and save that... but we have a sop I think.
15:58:06 <nirik> There's a upgrade koji playbook.
15:58:36 <nirik> The builders it's mostly: take them out of koji, destroy them, run the playbook to re-make them with the new version.
15:58:37 <sharkcz> does koji use classic chroot or nspawn in mock?
15:58:56 <sharkcz> I don't recall :-)
15:58:59 <mboddu> sharkcz: chroot
15:59:07 <nirik> currently it's using classic. we could try and move to nspawn, but I know of one case at least that would need to be fixed for that (kernel builds)
15:59:15 <mboddu> Although we thought of moving to nspawn
15:59:26 <sharkcz> hasn't been chroot removed in 2.x?
15:59:30 <mboddu> and ^^ that
15:59:37 <nirik> the problem is we haven't had time to test, and so keep kicking it down the road.
15:59:38 <nirik> nope.
15:59:45 <nirik> the name changed
16:00:14 <nirik> --old-chroot          Obsoleted. Use --isolation=simple
16:00:19 <sharkcz> ah, right
16:01:26 <nirik> so, I guess lets wait for mock, then schedule something (although most of it can be done outside an outage)
16:01:40 <mboddu> ack
16:01:44 <sharkcz> ack
16:01:55 <nirik> enabling bootstrap is kinda needed with the new rpm... or if we don't all our images will have the db the wrong format
16:02:05 <nirik> which I guess isn't the end of the world, but is not great
16:02:38 <nirik> might be ok for a week or two until we update mock tho I suppose. we could ask rpm maintainer...
16:02:46 <nirik> I know they want to land that change soon
16:02:54 <mboddu> #info nirik will work on enabling bootstrap in mock for the new rpm sqlite update and updating koji to new release and moving the builders to F32
16:03:03 <mboddu> mboddu will help as much as he can :)
16:03:04 <nirik> .releng 9308
16:03:07 <zodbot> nirik: Issue #9308: F33 system wide change: Sqlite Rpmdb - releng - Pagure.io - https://pagure.io/releng/issue/9308
16:04:43 <mboddu> Anyway, thats all guys
16:04:50 <mboddu> Thanks for joining
16:05:01 <mboddu> If anything else, lets take it to #fedora-releng
16:05:06 <nirik> thanks mboddu
16:05:07 <mboddu> #endmeeting