21:00:39 <tdawson> #startmeeting EPEL (2020-09-11)
21:00:39 <zodbot> Meeting started Fri Sep 11 21:00:39 2020 UTC.
21:00:39 <zodbot> This meeting is logged and archived in a public location.
21:00:39 <zodbot> The chair is tdawson. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
21:00:39 <zodbot> The meeting name has been set to 'epel_(2020-09-11)'
21:00:39 <tdawson> #meetingname epel
21:00:39 <tdawson> #chair nirik tdawson bstinson Evolution pgreco carlwgeorge
21:00:39 <tdawson> #topic aloha
21:00:39 <zodbot> The meeting name has been set to 'epel'
21:00:39 <zodbot> Current chairs: Evolution bstinson carlwgeorge nirik pgreco tdawson
21:01:18 <carlwgeorge> howdy folks
21:01:34 <bstinson> hey all
21:01:44 <orionp> hello
21:01:45 <pgreco> hey all
21:01:57 <pgreco> bstinson: jinx ;)
21:02:00 <nirik> morning
21:02:04 <tdawson> Hi carlwgeorge
21:02:08 <tdawson> Hi bstinson
21:02:11 <tdawson> Hi pgreco
21:02:17 <tdawson> Hi nirik
21:02:39 <tdawson> Wow ... we've got most everyone here already.
21:02:57 <michel_slm> .hello salimma
21:02:58 <zodbot> michel_slm: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
21:03:05 <tdawson> Hi michel_slm
21:03:07 <michel_slm> hi all!
21:03:28 <michel_slm> apologies for sending my proposal we talked about last week at the last minute :/
21:04:40 <tdawson> That's ok.
21:05:17 <tdawson> #topic Old Business
21:05:17 <tdawson> epel8-playground
21:05:25 <tdawson> Hopefully this will be quick
21:05:55 <tdawson> I've got one more draft of the documentation, and hopefully that will be good.
21:06:07 <tdawson> nirik: Were you going to do the fedpkg work?
21:06:15 <tdawson> Or did you need someone else to give that a shot?
21:06:47 <michel_slm> vvfvddfrhevlftiubfdfgkilhblitutgvkidbbkbnlid
21:07:03 * michel_slm curses yubikey
21:07:24 <pgreco> michel_slm: I thought that format looked familiar
21:07:25 <nirik> not it.
21:07:46 <nirik> There is an upstream (some small number of developers) so we could just file a issue and hopefully they can do it.
21:08:34 <tdawson> That sounds good, spread the work around even more. :)
21:09:26 * nirik nods
21:09:56 <tdawson> I was going to file a releng ticket, so that the script that creates the dist-git branches quick putting the package.cfg file in the repo.
21:10:29 <tdawson> I have enough privilidges to take the file out from all the epel repos, but if they keep making them, that will be a pain.
21:10:38 <tdawson> nirik: Do you mind filing the bug then?
21:10:50 <tdawson> Bug for fedpkg I mean
21:11:01 <nirik> sure, I can do that.
21:11:43 <tdawson> OK, after we get those done, we can worry about the next steps, which is cleaning up the auto-built packages.
21:12:21 <pgreco> do we want to keep the different %dist in playground?
21:12:43 <tdawson> pgreco: as far as I know, yes.  So it's obvious that it's a different repo.
21:13:02 <nirik> https://pagure.io/fedpkg/issue/414
21:13:12 <pgreco> ack, I'm going to ask this again when we get to -next ;)
21:13:21 <tdawson> :)
21:13:29 <tdawson> Thanks nirik
21:13:41 <tdawson> Anything else about playground before we move to -next ?
21:15:16 <tdawson> OK, epel8-next
21:15:32 <tdawson> carlwgeorge: I'll let you lead the discussion for this.
21:16:13 <carlwgeorge> i won't re-hash what i sent to the mailing list too much
21:16:44 <carlwgeorge> seems like most of the feedback was around the name, but i still have a strong preference for the name -next for the reasons given in the thread
21:17:11 <pgreco> ok, time to ask again, what %dist do we want for it?
21:17:24 <nirik> also there was some dicussion about if the repo package would be installed by default or not or enabled or not.
21:17:50 <carlwgeorge> i'm fine with keeping it .el8, but i'm not opposed to mangling it to always be higher than the epel8 build
21:18:06 <carlwgeorge> i don't think it's strictly required
21:18:24 <pgreco> I like using el8 too, since anything we add would be newer than el8
21:18:33 <pgreco> and would force a rebuild in point releases
21:18:49 <carlwgeorge> ideally yes, and like fedora we can have guidelines that packages should be mindful of the upgrade path
21:18:53 <pgreco> but if we keep the el8, I guess it would be handled as a tag
21:19:20 <nirik> well, keeping el8 makes it harder to see where it came from.
21:20:01 <nirik> but thats been an age old issue.
21:20:07 <carlwgeorge> repoquery is a better tool for that imo, but like i said i'm ok either way
21:20:50 <pgreco> I prefer el8 to avoid forcing the rebuild, but not exactly a strong opinion
21:20:55 <pgreco> wrt installed/enabled, could we make it be a weak dep of centos-release-stream?
21:21:33 <michel_slm> the only time we have to use a higher version is when -next builds against the upcoming major release, right?
21:21:34 <pgreco> so if you have epel-release and centos-release-stream, it would be suggested, otherwise ignored
21:21:34 <carlwgeorge> maybe weak reverse (supplements)
21:21:44 <pgreco> yeap, that's what I had in mind
21:21:54 <nirik> well, thats against guidelines I think...
21:22:50 <carlwgeorge> michel_slm: epel8-next would always build against c8s, so the upcoming rhel minor release
21:22:54 <tdawson> Do we expect it to have it's own dist-git?  If we do, then I'm fine with .el8 ... otherwise I think you're going to need to have a different %dist
21:23:21 <nirik> it should have it's own branch. (That will be some work to setup also)
21:23:26 <carlwgeorge> tdawson: do you mean dist-git branch?
21:23:35 <tdawson> carlwgeorge: yes
21:24:24 <tdawson> The reason is this.  If you have package foo, that depends on something that get's bumped in CentOS Stream.  How are you going to have a build against both regular EPEL and epel-next with the same dist-git.
21:24:24 <carlwgeorge> nirik: i'm happy to help with whatever i can on the releng side, but i'm sure permissions will make that challenging
21:25:27 <nirik> carlwgeorge: cool. I was going to suggest perhaps we could add a steps thing to your proposal... the things that need doing (like tdawson did for playground rework) and see what and who can do them each...
21:25:35 <tdawson> Because there might be a time there foo has to be rebuilt, let's say a security issue, after they've already done the work building on stream.
21:26:29 <carlwgeorge> tdawson: maybe have that in the guidelines about upgrade path
21:26:30 <tdawson> If there is a different %dist, then thay can simply build on both epel and -next, but otherwise, it's a bit of a pain having to do that update.
21:26:44 <carlwgeorge> i.e. do the security update in epel8 first, then epel8-next with release+1
21:27:26 <nirik> I don't think one shared dist-git branch is a good idea or possible.
21:27:29 <carlwgeorge> i think nirik mentioned that even with the same nvr dnf is probably clever enough to pick the right one based on libraries available
21:27:52 <carlwgeorge> i'm also in favor of using a separate epel8-next branch
21:27:57 <tdawson> They can't have the same nvr, koji won't allow it.
21:28:03 <carlwgeorge> oh yeah true
21:28:13 <michel_slm> so thinking about the future where we have epel and epel-next ... would it make sense to change %dist to have both major and minor version then?
21:28:18 <nirik> unless the dist tag makes them different.
21:28:22 <michel_slm> unless we don't always want to do a full rebuild
21:28:25 <pgreco> that's why I said we'd only need to tag it
21:28:33 <pgreco> and it would go into epel8
21:28:53 <tdawson> I'm ok with doing what you said above (do epel8 first, then bump the release 1 for -next)
21:29:00 <nirik> michel_slm: I'm not sure a 'full rebuild' is possible / desireable
21:29:02 <carlwgeorge> we could do it similar to rhel and use el8_3 style disttag
21:29:23 <pgreco> tdawson: let's use your kde as an example
21:29:43 <nirik> tdawson: I think that will get confusing very fast
21:29:44 <pgreco> when stream bumps you rebuild in epel8-next
21:30:00 <tdawson> Ya ... bumping 300 packages would be a pain.
21:30:10 <pgreco> when 8.x is released, you tag your epel8-next builds in epel8
21:30:27 <pgreco> and they are literally the same, which won't bother koji
21:30:56 * nirik thinks bodhi will not handle this workflow at all. ;)
21:31:03 <carlwgeorge> yeah i was just about to bring up bodhi
21:31:19 * pgreco goes away to cry alone....
21:31:39 <carlwgeorge> i had been wondering the same as pgreco, could there be a way to "promote" a build from epel8-next to epel8 after a rhel point release?
21:31:40 * tdawson pats pgreco on the head.  "There there ... it will be ok"
21:31:45 <bstinson> probably not now, but is that a workflow we want for the future?
21:32:28 <michel_slm> so... only use -next if there's a package in epel that doesn't work with c8s, and in that case, mangle the %dist otherwise leave it alone?
21:32:34 <carlwgeorge> i have no idea what bodhi supports there, but it could be a nice workflow
21:32:37 <nirik> when you submit a bodhi update, bodhi knows those builds are in that update. You can never submit them again in any update because they are already in that one
21:33:18 <carlwgeorge> eh, a release bump rebuild isn't that bad to handle the "promotion"
21:33:30 <nirik> I think we will need to rebuild again in epel8 once the 'next' change that is incompatible lands...
21:33:49 <carlwgeorge> yes
21:34:01 <nirik> but then we are starting over from testing somewhat. it's not the same exact artifact.
21:34:18 <carlwgeorge> michel_slm: that's what i'm thinking
21:34:19 <nirik> of course the minor release could be different from stream? (or should that never be the case?)
21:34:55 <pgreco> and that also complicates things for people who have the same spec everywhere
21:35:03 <pgreco> fedora/epel
21:35:23 <pgreco> forcing a bump I mean
21:35:53 <bstinson> stream will probably never contain all individual minor-release NVRs lined up at the exact same time
21:35:55 <carlwgeorge> nirik: do you mean the minor release we inject into the disttag on epel8-next builds?
21:36:53 <carlwgeorge> pgreco: yes things could get confusing, but it works well enough between fedora branches so i'm not too worried there
21:37:02 <nirik> I'm not sure I am following...
21:37:27 <bstinson> nirik: are you asking if there's ever a CentOS Stream that has the exact same content as CentOS Linux 8.3 GA?
21:38:09 <nirik> yeah.
21:39:02 <tdawson> Well, there is the problem that Stream, is much like a stream.  It keeps on flowing.  Lets say this month bash is updated, and next month glibc is updated.  What about the package that was built against the newer bash, but older glibc.
21:39:19 <tdawson> Sorry, those are terrible example packages, but you get what I mean.
21:39:26 <bstinson> ok, the answer there is 'no'. we've likely built/released all of the content at some point, but the exact GA content won't exist together at a point in time
21:40:02 <nirik> so say you maintain foo in epel8. it's fine, then some stream change happens and it won't work anymore there. You request a epel8-next branch, modify your spec if needed, build it in epel8-next, submit to bodhi, get out, fine. rhel 8.x lands. Is the 8.x minor release just like the stream it was built against.
21:40:20 <carlwgeorge> matching the exact nvrs isn't my concern, the problem being solved for is library bumps
21:40:39 <nirik> sure.
21:40:40 <pgreco> but even if they are not the same, they should always be close enough not to have mixed sonames
21:40:44 <carlwgeorge> nirik: from a library perspective, probably
21:41:31 <carlwgeorge> i think there could be a small window where for example c8s has 8.3 and 8.4 content immediately prior to 8.3 ga
21:41:44 <bstinson> fwiw, i'd be against elx_y disttags for reasons like ^this
21:42:12 <tdawson> It's looking like we're not going to get this totally resolved in this meeting.  I'm giving us 3 more minutes so we can move on to the other agenda item (epel packaging sig)
21:42:24 <nirik> yeah, which might be another reason it's good for epel8 and epel8-next to be different things... so when 8.3 lands you can rebuild epel8 against it, and keep epel8-next building against the newer thing
21:42:55 <carlwgeorge> bstinson: fair, but it would be correct the majority of the time
21:43:23 <nirik> anyhow, perhaps carlwgeorge could fold in feedback and add a 'things we need to do' and a example workflow?
21:43:35 <carlwgeorge> can do
21:44:16 <tdawson> So ... are we ready to move to the next topic?
21:45:09 <carlwgeorge> sure
21:45:17 * nirik nods
21:45:20 <tdawson> carlwgeorge: Thanks for leading that.  Very good points were brought up, and as nirik said, if you can fold that into an email that would be great.
21:45:33 <carlwgeorge> 👍
21:45:44 <tdawson> michel_slm: Did you want to lead the discussion on EPEL packaging sigs?
21:45:45 <michel_slm> one last thing - would it make sense to always have a -next branch? otherwise there might be a coordination issue in getting a bunch of packages rebiult
21:46:09 <michel_slm> tdawson: sure. oh the epel SIG topic might touch on how to do -next rebuilds anyway
21:46:16 <tdawson> True
21:46:17 <nirik> well, I am still not sure how often this happens...
21:46:20 <carlwgeorge> if we're making -playground optional, i think -next should be optional as well
21:46:48 <carlwgeorge> some epel maintainers may just not want to participate at all, and for noarch packages it's likely a complete waste
21:46:57 <michel_slm> so TL;DR - some orgs (e.g. Fedora Infra and Facebook) rely on a lot of EPEL packages. and currently there's a bit too much friction in getting packages in EPEL if the maintainers themselves are not interested
21:47:07 <pgreco> epel8 is basically epel, the rest is optional
21:48:04 <michel_slm> the proposal is to have something similar to the language-based SIGs (e.g. Python) but with a slight twist -- make it so that if there's no response from the maintainers to a branch request, the EPEL SIG get added directly (but with a whitelist so they only have access to the EPEL branches)
21:49:18 <carlwgeorge> currently dist-git doesn't support per-branch permissions, if you have commit rights you can commit to any branch
21:49:19 <pgreco> basically an epel* only provenpackager
21:49:37 <nirik> carlwgeorge: it does now actually...
21:49:41 <pgreco> according to the ML, we do have that
21:49:49 <carlwgeorge> oh neat, glad that got fixed
21:49:52 <michel_slm> it should make it smoother to upgrade from one major EL release to the next (right now, at least at FB we tend to find out after the fact that service X depends on package Y and oops, it's not branched for epel8) and also pulls in the companies / orgs depending on packages to help maintain them
21:50:01 <michel_slm> pgreco: yep, epel-provenpackager is a good name actually
21:50:16 <bstinson> didn't we intend to do this 5...6? years ago?
21:50:28 <bstinson> we started epel-wranglers
21:50:34 <nirik> but... collaborators I don't think have perms to request branches, so that would have to be done by someone else
21:50:40 <bstinson> as a FAS group, but never connected up the perms
21:50:40 <michel_slm> carlwgeorge: yep, I keep forgetting what it's called so I never mentioned it myself. bstinson huh, I guess it's time for it now. happy to help resurrect that
21:50:52 <tdawson> Oh ... is that what epel-wranglers was ... I never knew.
21:51:01 <michel_slm> nirik: committers do have access to request branches. oh, only if they are direct committers?
21:51:10 <bstinson> 2014-09-19 16:17:57 CDT
21:51:23 <nirik> I'm not sure, but yes, I think thats the case... only commiters/admins
21:51:27 <carlwgeorge> nirik: i think that got fixed too, now if you have any permissions on a dist-git repo you can request branches
21:51:35 <bstinson> heh, almost 6 years ago
21:51:45 <nirik> could be. I haven't had time to look into it.
21:51:48 <carlwgeorge> or maybe that was just admin via group i'm thinking of...
21:52:00 <michel_slm> if it's like provenpackager, maybe we can grant access as committer/admin but with the understanding that we don't touch the Fedora branches unless it's an emergency
21:52:22 <tdawson> michel_slm: I'd feel much better with it just being EPEL permissions.
21:52:34 <michel_slm> or can we fix dist-git so collaborators have branch request access matching their whitelist?
21:52:36 <nirik> part of the problem is all the workflows different people do in fedora.
21:52:52 <michel_slm> tdawson: me too. might be easier to get buy in from the packagers who don't want to support epel too
21:53:28 <nirik> group a: everything in master, add conditionals to spec to make it work on all branches, merge back to all branches. group b: every branch is it's own spec. Only make changes for that branch there. and everything between
21:53:37 <tdawson> Yep.  If packager X doesn't want to deal with EPEL, and then we say the epel-provenpackagers can mess with his rawhide branch ... packager X is going to fight this all the way.
21:54:17 <carlwgeorge> packager X would need to be reminded that it's not their package, it's fedora/epel's package :D
21:54:32 <tdawson> carlwgeorge: It's Fedora's package ... not EPEL's.
21:55:04 <carlwgeorge> whynotboth.jpg
21:55:08 <tdawson> :)
21:55:09 <nirik> well ideally we adjust to whatever workflow the maintainer(s) want, but that makes it harder to automate and always do the same workflow
21:55:23 <tdawson> Yep
21:55:53 <michel_slm> carlwgeorge: I hope we're less tribal on this than other distros anyway (in Debian you have to do an NMU to force an update IIRC?)
21:55:54 <carlwgeorge> i think tackling the general ability to do epel stuff via the group is step 1, automation can come later
21:56:00 <tdawson> I've found that the majority of packages, if they don't have any %if's, just build on epel8, as long as the dependencies are there.
21:56:20 <michel_slm> yeah, the delta between rawhide and what's needed on epel8 is mostly minimal
21:56:44 <nirik> well, right now anyhow. :) In ~5 years...
21:56:53 <michel_slm> in most cases it's ok if epel-wrangler/provenpackager only have access to epel* branches and just have the spec diverge over time - if there's something we need in rawhide we can always just PR it
21:56:53 <tdawson> True
21:56:53 <carlwgeorge> i've butted heads with proven packagers removing epel conditionals from the master branch of packages i maintain before
21:57:41 <michel_slm> or ... if the Fedora maintainers want to they can make epel-wrangler a full admin/committer. just that it's having it be added as a collaborator by default if there's no response from maintainers
21:57:45 <tdawson> Yep, there is that.  EPEL is a "enterprise" release, and some things we want very stable, for Fedora and master are going to diverge.
21:58:08 <nirik> anyhow. I like the idea of epel-sig or reviving epel-wranglers... I'd prefer time to review the proposal more before +1ing tho... I just only glanced at it before this meeting.
21:58:20 <carlwgeorge> same here
21:58:45 <nirik> more people invested in fixing a collection of packages can only help us out. :)
21:59:00 <pgreco> sounds like the basic idea is, looks like we want this, we need to give it the right shape
21:59:11 <tdawson> Well, we're pretty much out of time ... how about michel_slm ... if you do what carlwgeorge is going to do, and wrap the feedback into an email and we'll discuss is during the week.
21:59:42 <tdawson> I agree with both nirik and pgreco
21:59:52 <pgreco> wow, I didn't see the time, it just flew by....
21:59:53 <bstinson> if we do decide to reclaim epel-wranglers, we should make sure we contact everyone who's already sponsored
22:00:05 <carlwgeorge> this was a good meeting, thanks everybody for coming and helping me and michel_slm hone our proposals
22:00:06 <michel_slm> tdawson: yup, will follow up there, thanks for the discussion!
22:00:15 <michel_slm> I have an electrician who just arrived so have to show him around
22:00:24 <nirik> shocking!
22:00:30 * nirik shows himself out.
22:00:35 <carlwgeorge> lol
22:00:37 <tdawson> Thank you everyone.  Great discussions and meeting.
22:00:40 <pgreco> nirik: that hurt.....
22:00:48 <tdawson> #endmeeting