16:01:47 <mboddu> #startmeeting RELENG (2019-05-30)
16:01:47 <zodbot> Meeting started Wed May 29 16:01:47 2019 UTC.
16:01:47 <zodbot> This meeting is logged and archived in a public location.
16:01:47 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:47 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:47 <zodbot> The meeting name has been set to 'releng_(2019-05-30)'
16:01:48 <mboddu> #meetingname releng
16:01:48 <zodbot> The meeting name has been set to 'releng'
16:01:48 <mboddu> #chair nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu dustymabe ksinny jednorozec
16:01:48 <mboddu> #topic init process
16:01:48 <zodbot> Current chairs: dustymabe jednorozec ksinny masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
16:02:25 <bcotton> .hello2
16:02:29 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
16:04:15 <nirik> morning
16:04:16 * mboddu waves at bcotton
16:04:25 <mboddu> nirik: Morning
16:04:50 <mboddu> I haven't prepared for this meeting at all, so, lets see
16:07:17 <mboddu> So, lets get to some old tickets
16:07:21 <mboddu> #topic #7400 Rethink how we handle networking config with createImage (oz / ImageFactory)
16:07:28 <mboddu> #link https://pagure.io/releng/issue/7400
16:07:32 * mboddu reading back
16:08:21 * nirik reads, not sure I ever saw this one
16:08:39 <nirik> oh wait, yeah
16:09:32 <mboddu> I remember seeing this, but didn't get to it
16:10:38 <nirik> so right now, the oz images install with a ensa263152761 whatever interface
16:10:56 <nirik> which gets written to a ifcfg file by NM
16:11:15 <nirik> so we have to nuke it in the kickstart to allow for someone using that image to get their own interface
16:11:42 <nirik> if we just set everything to use eth0 it's easier to nuke
16:13:20 <nirik> so, I'm in favor of making these changes...
16:14:17 <mboddu> nirik: So, set to use eth0 and rm -f /etc/sysconfig/network-scripts/ifcfg-eth0?
16:14:27 <nirik> yeah
16:14:34 <mboddu> Thats a change to koji
16:14:54 <mboddu> I can file a ticket against koji
16:15:37 <mboddu> Or say we agree to it and ping mikem on the ticket
16:16:22 <nirik> sure, or adamw might do a PR if he's in the mood
16:19:11 <mboddu> #info mboddu commented on the ticket to check with mikem and adamw and if anyone can make the necessary changes to koji
16:20:59 <mboddu> #topic #7802 Several EPEL7 package conflicts with RHEL/CentOS 7.5
16:21:05 <mboddu> #link https://pagure.io/releng/issue/7802
16:21:11 <mboddu> I dont think its an issue anymore
16:21:16 <mboddu> nirik: Can I close the ticket?
16:22:41 <nirik> I've not looked... well, we should probibly look at those packages and see if they are still/ever causing problems.
16:23:27 <nirik> perhaps we could give reporter more time to chime in
16:23:47 <nirik> and ask them... what packages in EPEL7 are causing problems specifically.
16:24:03 <nirik> since those are the only ones we can do anything about... by removing them.
16:24:13 <mboddu> nirik: Okay, I will ping him on the ticket again
16:24:26 <nirik> the lists are a bit confusing to me.
16:24:34 <mboddu> Yes, exactly
16:24:52 <nirik> I just want: here's epel7 packages that overlap with any rhel repos we care about... so we can say 'yes, thats ok' or 'no, we should remove X'
16:25:47 <mboddu> nirik: I know there is no communication involved when packages are moved to RHEL and we have to remove them in EPEL
16:26:28 <mboddu> Cant we ask RHEL maintainers to send us the list whenever they do so, or once a week/month?
16:26:57 <mboddu> Or at the least when there is a release?
16:28:10 <nirik> we have a list of packages in rhel
16:28:37 <nirik> https://infrastructure.fedoraproject.org/repo/json/pkg_el7.json
16:28:42 <nirik> https://infrastructure.fedoraproject.org/repo/json/pkg_el6.json
16:29:44 <mboddu> nirik: Yes, I know them, but they contain all of them, not anything newly added
16:30:07 <mboddu> Or we can write our own thing to compare list of packages in epel vs those json files
16:30:26 <nirik> sure.
16:30:41 <nirik> but we need to whitelist the limited arch ones...
16:30:50 <mboddu> Yes
16:30:55 <bcotton> do we know how often that changes? because if it's infreqent, I can talk to my counterparts who work on RHEL to notify me/someone
16:32:03 <nirik> it changes at point releases...
16:32:21 <nirik> I'm not sure notifying us is great.
16:32:22 <mboddu> bcotton: ^ mostly at point releases
16:32:34 <mboddu> nirik: Why?
16:32:52 <nirik> there is supposed to be a step when adding a new package to rhel to file a bug on the epel package if it exists and getting the maintainer to retire it.
16:33:05 <nirik> because releng doesn't scale, and we should get this as automated as possible.
16:33:10 <nirik> manual workflows suck.
16:33:26 <bcotton> agreed. if there's something that's supposed to be happening that isn't, i can bring that up, too
16:34:16 <nirik> I suppose the best we might do is a script that runs every day and checks rhel against epel and has a whitelist. When something new comes up we retire it or whitelist it. but thats still manual... so dunno.
16:34:24 <mboddu> That is true, but we can add something like "Talk to epel maintainer....blah blah....if the epel maintainer doesn't respond then talk to RelEng"
16:35:29 <mboddu> I am okay with either one rather then being in the dark unless someone files a ticket
16:38:00 <nirik> well, sure, but it needs script writing...
16:38:55 <mboddu> nirik: I am okay with it
16:39:03 <mboddu> I will create a ticket for it
16:39:15 <nirik> so, yeah... lets put the generic solution in a ticket...
16:39:25 <nirik> and in this one ask reporter for specfic details
16:40:02 <mboddu> #info mboddu will ping the maintainer about the specific epel packages that that are causing issues
16:40:53 <mboddu> #info mboddu will create a ticket to create a script that will compose epel packages with rhel package list json file and report back with results
16:41:17 <nirik> ack
16:42:19 <cverna> I think there the is-it-rhel project that pingou maintains
16:42:28 <cverna> not sure if that solves the issue
16:42:33 <nirik> cverna: yeah, thats the thing that makes those json files. ;)
16:42:39 <nirik> (I thought)
16:42:46 <cverna> ah ok :)
16:42:55 <cverna> https://pagure.io/is-it-in-rhel
16:43:18 <cverna> so we need a is-it-epel and compare both :P
16:43:19 <nirik> ah no, thats a interface to it. ;)
16:43:25 <nirik> yeah
16:43:38 <nirik> or is-it-in-both-epel-and-rhel
16:43:47 <cverna> :)
16:43:52 <mboddu> haha :)
16:44:17 <nirik> at least for epel8 we won't have to deal with the limited arch stuff.
16:45:07 <mboddu> Yes
16:47:03 <mboddu> #topic #7763 define and implement retention policy for nightly composes
16:47:09 <mboddu> #link https://pagure.io/releng/issue/7763
16:47:28 <mboddu> I am not sure if we need to change any policy here
16:47:33 <mboddu> 14 days seems good to me
16:47:53 <mboddu> and when we really need one, we can use koji event to clone the tag and create a compose
16:48:07 <mboddu> But that means, we have to keep track of those koji events
16:48:12 <mboddu> cverna: ^ fpdc :D
16:49:13 <nirik> yeah. that all sounds good to me.
16:50:22 <mboddu> cverna: So, can you add that to the requirements doc?
16:50:28 <mboddu> for fpdc
16:51:43 <mboddu> #info We will not change the policy but we will keep track of the koji event id (probably in fpdc) from which we can clone a koji tag and run a compose from it if and when its really needed.
16:53:03 <mboddu> #topic Open Floor
16:53:09 <mboddu> I got 2 things
16:53:18 <nirik> I wanted to share my progress on builders this week...
16:53:26 <mboddu> #info F28 went EOL yesterday
16:54:26 <mboddu> #info FLOCK CFP selection starts on Jun 1st, so please submit your talks
16:54:29 <mboddu> nirik: Go ahead
16:54:59 <nirik> so, our builders currently are mostly fedora29, some f28 (the ppc64 ones) and some f27 (armv7)
16:55:31 <nirik> I'm hoping to move armv7 to f30 and all the f29's to f30 and drop the ppc64 ones
16:55:41 <mboddu> +1
16:55:48 <nirik> I'll need testing on armv7 to try and find a stable os/kernel combo.
16:56:27 <nirik> also, oz and ImageFactory in rawhide have moved to python3, and I am going to try and move them for us too in f30-infra for now.
16:56:29 <pwhalen> nirik, or f29 w/ 4.20 GA , that seems the most stable
16:56:45 <nirik> which means that koji also has to be moved to python3
16:57:02 <nirik> pwhalen: ok. I was hoping f30 with 4.20 would work, but I haven't gotten to testing yet. ;(
16:57:28 <pwhalen> I had a question about the workstation repo, do we still plan on moving workstation to use Everything?
16:57:30 <nirik> I've got oz/imagefactory python3 builds in infratags... also koji, but it needs a few more patches.
16:57:54 <nirik> pwhalen: I thought so.. yes.
16:58:33 <nirik> I'm not sure the timeframe for builders... I am going to work on it more this week, but I am out next week and then traveling the week after, so it might not be until the week after that.
16:59:46 * nirik looks for when the mass rebuild is.
17:00:04 <nirik> 2019-07-24... ok, cool.
17:00:27 <mboddu> Yeah, we have time
17:00:29 <nirik> also, I am continuing to push about moving our s390x builders to kvm... there's some progress on it...
17:00:39 <mboddu> Oh finally
17:00:51 <nirik> thats all I had
17:02:08 <pwhalen> nirik, ok. I've seen 4.20 GA with f30 userspace pause and the 'prefetch abort' error. F29 looked much better for some reason.
17:02:35 <nirik> bummer, ok. I was hoping to move them all to f30... but ok, can deal with 29 I guess
17:02:41 <pwhalen> me too
17:03:08 <nirik> but hopefully we can get to the bottom of it and solve it upstream once and for all.
17:03:10 <pwhalen> I really hope it can be solved and not worked around
17:03:13 <pwhalen> agreed!
17:03:33 * nirik wonders why no one else hit this. I guess not too many armv7 vm's around?
17:03:39 <mboddu> #info nirk is looking into upgrading builders to F30 and killing ppc64 and also working on moving s390x builders to kvm
17:03:59 <mboddu> #info nirik is working on - Oz and ImageFactory in rawhide moved to python3 and trying to move them to f30-infra and if everything works out, then moving to python3 koji.
17:04:22 <nirik> I did disable ppc64 in bodhi pungi for epel6/7. we need to remove it from koji tags before we retire those machines.
17:06:32 <mboddu> nirik: I guess epel7-build and dist-6E-epel-build build tags, right?
17:06:37 <nirik> yeah
17:07:02 <mboddu> nirik: Do you want me to do it now or does it have to wait?
17:07:16 <nirik> go ahead if you want... or I can do it sometime
17:07:22 <mboddu> Oh we went overboard
17:07:31 <mboddu> nirik: I will do it now and then go to lunch
17:07:34 <nirik> meeting overboard!
17:07:38 <mboddu> haha :)
17:07:45 <mboddu> Thanks everyone for joining
17:07:48 <mboddu> #endmeeting