17:59:50 <smooge> #startmeeting EPEL (2020-01-08)
17:59:50 <zodbot> Meeting started Wed Jan  8 17:59:50 2020 UTC.
17:59:50 <zodbot> This meeting is logged and archived in a public location.
17:59:50 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:59:50 <zodbot> The meeting name has been set to 'epel_(2020-01-08)'
17:59:50 <smooge> #meetingname epel
17:59:50 <zodbot> The meeting name has been set to 'epel'
17:59:51 <smooge> #topic aloha
17:59:51 <smooge> #chair nirik smooge tdawson bstinson Evolution pgreco sgallagh merlinm
17:59:51 <zodbot> Current chairs: Evolution bstinson merlinm nirik pgreco sgallagh smooge tdawson
17:59:51 <smooge> #info Meeting is run from https://board.net/p/epel
17:59:58 <nirik> morning
18:00:04 <sgallagh> .hello2
18:00:05 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
18:00:09 <pgreco> and we're back!! :)
18:00:15 <tdawson> Howdy.
18:01:10 <smooge> #topic Call for Topics
18:01:31 <smooge> #info please add items to the https://board.net/p/epel or email me what topics need to be discussed
18:01:45 <smooge> #topic EPEL-6
18:01:45 <smooge> #info EPEL-6 is End of Life in 2020-11. It will be moved to archives in 2020-12
18:01:45 <smooge> #info THIS IS NOT A DRILL.
18:01:47 * sgallagh defers to merlinm who has been more active than I of late
18:02:03 <smooge> any other epel-6 items?
18:03:36 <smooge> #topic EPEL-7
18:03:58 <smooge> anyone have anything EPEL-7 related?
18:04:13 <nirik> I had one thing
18:04:19 <smooge> cool
18:04:25 <nirik> we dropped aarch64, but we have the repo still there...
18:04:40 <nirik> on the one hand, it's nice for people still using it to have the rpms around easily.
18:04:49 <nirik> on the other it's confusing since it's not updating anymore.
18:04:59 <nirik> should we move/remove it? or just leave it there?
18:05:03 <smooge> I actually thought the compose was going to delete it
18:05:10 <smooge> but I would think we delete it
18:05:18 <pgreco> nirik, I already have the hardware to start building aarch64 the same way I build armhfp
18:05:38 <nirik> pgreco: awesome!
18:05:38 <smooge> and put a pointer to the centos repos
18:05:49 <nirik> https://dl.fedoraproject.org/pub/epel/7/aarch64/
18:05:50 <pgreco> I was hoping to make some progress after I return from fosdem
18:06:11 <nirik> https://dl.fedoraproject.org/pub/epel/testing/7/aarch64/
18:06:34 <pgreco> if we move it to archive, will the users be redirected? or fail?
18:07:16 <nirik> it would just disappear...
18:07:28 <nirik> if we removed it from mm they would just get an error
18:07:55 <smooge> we would need to hack mm to work around this
18:08:20 <pgreco> well, an empty repo would be preferable I think
18:08:57 <nirik> any option will likely cause some problem. ;(
18:09:44 <smooge> if we have an empty repo then people will never know it was moved
18:10:01 <pgreco> yeap, different kind of failure..
18:10:03 <nirik> I guess we could discuss on list...
18:10:34 <nirik> I can write up the options there?
18:10:59 <smooge> I think we should discuss on list. I think that getting CentOS-alt-EPEL going, removing the items from our EPEL with a big fat README is our 'best' of 'worst' worlds
18:11:01 <sgallagh> Point it at that x86_64 mirror! :-P
18:11:18 <nirik> ha.
18:11:27 <smooge> ... did that before
18:11:42 <sgallagh> ... intentionally?
18:11:52 <smooge> --force --nodeps makes the pain even worse
18:12:03 <smooge> no not intentionally
18:12:31 <smooge> #action nirik will open thread on epel-devel on what to do with the aarch64 epel-7 repos
18:12:38 <pgreco> well, includepkgs=*noarch* while pointing to x86 from armhfp has saved me many times...
18:13:02 <smooge> #topic EPEL-8
18:13:21 <smooge> ok sgallagh merlinm this is your space since the magic dropped since our last meeting
18:13:50 <sgallagh> Has anything broken visibly since then?
18:13:56 <nirik> I have a question: Is there anything still being tested in stg? or would it be ok next week to sync prod->stg koji?
18:14:43 <sgallagh> I don't think we have anything going on that would suffer from a sync. merlinm ?
18:15:01 <smooge> I don't think they are on irc at the moment
18:15:08 <sgallagh> hmm, this may be lunchtime for him
18:15:56 <sgallagh> nirik: I'd recommend double-checking with merlinm, but I suspect it's fine to move ahead
18:16:35 <nirik> ok, thanks.
18:16:40 <tdawson> I know when we were testing the epel8 modules, we would have really liked a sync ... but whether merlinm is in the middle of something or not, I don't know.
18:17:15 <sgallagh> The only thing not complete for EPEL 8 modularity as far as I know is how to deal with epel8-playground
18:17:55 <sgallagh> When last we left our heroes, they were debating how to deal with module stream expansion to support playground
18:18:11 <nirik> well, and we need an announcement too right?
18:18:13 <sgallagh> Since they both have to run atop the same platform:el8 base
18:18:55 <sgallagh> nirik: The plan was to soft-launch in December and do an announcement probably around DevConf once we were sure the sky wasn't going to fall from it
18:19:20 <sgallagh> I haven't heard any yelling yet about things being broken
18:19:27 <nirik> me either too
18:19:29 <sgallagh> But I also haven't gotten through my entire email backlog yet
18:20:16 <smooge> i have not seen any emails of yells
18:20:45 <sgallagh> I will note that stream-expansion from Fedora modules is working. I (inadvertently) built the latest nodejs:12 module for F30-F32 and EPEL 8 this morning
18:20:59 <smooge> cool
18:21:15 <sgallagh> But I won't push it to epel8 repos because it would override RHEL 8
18:21:33 <smooge> that is going to be a problem
18:21:45 <sgallagh> But that's good, as it means people should be able to see that it's easy to build for EPEL 8
18:21:57 <sgallagh> smooge: Why? I just won't submit it to Bodhi
18:22:06 <smooge> well you are good about it
18:22:16 <pgreco> sgallagh: somewhat related, do we have any "logic" to add packages o a stream in RHEL?
18:22:20 <sgallagh> We had planned from the beginning that building modules between Fedora and EPEL8 was desirable and should be opt-out.
18:22:32 <pgreco> kind of what we currently do with php-extras for 7
18:22:44 <sgallagh> pgreco: Sorry, to inject EPEL content into a RHEL stream?
18:22:58 <pgreco> yeap
18:23:10 <sgallagh> No, but there's nothing preventing us from building an epel version that is a superset of the RHEL version and suggesting that people use that
18:23:15 <smooge> i don't think that is possible
18:23:23 <pgreco> using php as an examplo
18:23:26 <pgreco> example
18:23:34 <smooge> injecting content into a stream that is
18:23:50 <smooge> but i am probably wrong
18:23:53 <pgreco> rhel doesn't build, let's say php-imap
18:23:54 <sgallagh> Or creating an entirely new stream to enable and having the packages in it "Supplements:" the RHEL one
18:24:22 <pgreco> my first idea was to add, but if there is another option, I'm ok
18:24:24 <sgallagh> smooge: It's *technically* possible, but it would be painful.
18:25:08 <sgallagh> And it would arguably be "replacing RHEL content"
18:25:08 <sgallagh> (since the metadata would change)
18:25:08 <smooge> yeah
18:25:14 <pgreco> my idea is this, currently for 7 we build php-extras to generate php-imap and others for php 5.4, right?
18:25:24 <pgreco> how could we achieve something like that for 8?
18:25:42 <sgallagh> pgreco: php-extras is an SRPM that produces multiple binary subpackages?
18:25:52 <pgreco> yes
18:26:21 <sgallagh> Do you want to produce versions that work with the different streams of PHP in RHEL, or just the default stream?
18:26:47 <pgreco> let's say the default for now (assuming it is easier)
18:27:07 <sgallagh> In that case, you can do it exactly like you did in EPEL 7
18:27:47 <pgreco> built outside a module
18:27:49 <sgallagh> If you want to have it work with the alternative streams as well, then you need to make it a module and allow it to stream-expand
18:27:59 <sgallagh> Right
18:28:16 <sgallagh> Now, it gets trickier if the thing you want to build is a subpackage of an SRPM that RHEL is shipping.
18:28:32 <sgallagh> For example, I have a BZ asking me to ship libuv-devel in EPEL
18:29:06 <sgallagh> The main package `libuv` is shipped by RHEL BaseOS but they aren't delivering libuv-devel in any channel
18:29:24 <pgreco> yeah, that's a different beast
18:29:38 <sgallagh> I have some ideas about how this might work, but they need to be explored carefully
18:30:05 <pgreco> yeap, let's stick to the really added stuff
18:30:09 <sgallagh> But your case is fairly straightforward. And yes, if you only care about the default PHP version for now, non-modular is simplest
18:30:33 <smooge> the problem I have found is to make the libuv-devel you have to make the libuv also which replaces that
18:30:36 <sgallagh> (Though the modular version wouldn't be terribly hard either)
18:30:49 <pgreco> ok, and there is an option to expand the original module if we needed to
18:30:57 <sgallagh> smooge: Want to #topic this and we can discuss the issues and options?
18:31:15 <sgallagh> pgreco: Sorry, can you rephrase "expand the original module"?
18:31:24 <sgallagh> I'm not sure what you mean by that
18:31:25 <smooge> #topic Modularity and other things
18:31:30 <pgreco> I don't need to know how, just that it "could" be done, because I guess it will come up
18:31:43 <pgreco> sgallagh: what you said "then you need to make it a module and allow it to stream-expand"
18:32:01 <sgallagh> pgreco: The word "original" there threw me.
18:32:17 <sgallagh> What I said above was creating a new module in EPEL, not extending the RHEL one
18:32:19 <pgreco> original==RHEL for my interpretation ;)
18:32:45 <sgallagh> OK, so you misunderstood me, I think
18:32:59 <pgreco> that sounds very likely :)
18:33:16 <pgreco> moving on...
18:33:26 <sgallagh> While creating a new stream of the php module *could* be done, it would be more trouble than it's worth
18:33:33 <sgallagh> Both from the user and packager sides.
18:33:41 <sgallagh> So let's just rule that out from the start :)
18:34:27 <sgallagh> OK, let's discuss the libuv-devel case in detail. (But give me two minutes; this might take a bit and I need a quick bio-break)
18:34:47 <tdawson> May I interject going back to the beginning of what we were talking about.  Someone wanted libuv-devel in EPEL8, and I do as well, along with about 10 other "missing" devel packages.  Weren't we making a big module that had all those missing packages?  Wasn't that something to do with centos?
18:35:44 <sgallagh> That was one approach I was looking at. There are complications, which is why I want to discuss things here. (Though I may be rubber-ducking a bit)
18:36:13 <smooge> 2 minute bio break
18:36:24 <smooge> smoke'm if you got them
18:36:30 <tdawson> OK, so I'm not crazy, it is one approach.  OK, then proceed, when everyone's back. :)
18:36:46 <pgreco> tdawson, if it was centos related, I guess bstinson would be the guy to talk to :)
18:38:05 <sgallagh> I'll try to describe the problem space concretely first. << EOF
18:38:46 <sgallagh> RHEL has shipped the libuv event library in BaseOS (read: the RHEL 8 non-modular repositories)
18:39:34 <sgallagh> There are packages that would like to ship in EPEL that would like to be able to build/link against libuv, but the required headers and unversioned .so file are unavailable because RHEL is not shipping them.
18:40:18 <sgallagh> We want EPEL 8 to ship libuv-devel that can be used by packagers and users to build their own libuv-consuming software.
18:40:54 <sgallagh> EPEL 8 is not permitted to override RHEL 8 in the non-modular repos or any default module stream
18:40:55 <sgallagh> EOF
18:41:03 <sgallagh> (Did I miss anything in the problem statement?)
18:41:14 <tdawson> Sounds right to me.
18:41:27 <jsmith> Makes sense to me... and I've seen similar libraries where I'd like to do the same thing
18:41:55 <sgallagh> Right, I'm just using libuv as a stand-in because it's a nice, easy example (that I also happen to be directly responsible for)
18:43:06 <jsmith> :-)
18:43:39 <sgallagh> One idea I had (as tdawson mentioned) was to create a special module whose purpose is to rebuild packages from RHEL 8 BaseOS and AppStream that have filtered out some of their subpackages.
18:44:13 <sgallagh> In the long-term, I'd like to have such a stream automated, but for the first pass let's worry about one package at a time, manually.
18:44:39 <sgallagh> So we'll look at what it would take to build libuv again for EPEL 8.
18:45:24 <sgallagh> Problem 1: We want to rebuild the exact same sources as RHEL 8, so we need to be able to identify the git commit in git.centos.org that matches the release.
18:45:40 <sgallagh> (Problems meaning "things that need to be solved" and not "things we can't solve")
18:46:12 <sgallagh> We then would create a module stream that rebuilds this exact commit in the EPEL infrastructure.
18:47:12 <sgallagh> Problem 2: Nearly all `*-devel ` subpackages in Fedora follow the guideline that they must match the full ENVR of the parent package. If we rebuild within a module, it would get a %{dist} tag that includes the modular suffix. This would not match the base package.
18:48:06 <sgallagh> I have some wild ideas about how to solve that one, but those are entirely untested as of yet and may require MBS enhancements.
18:49:25 <sgallagh> We would also need to filter out any packages from this module stream that were included in RHEL, so we don't override them.
18:50:26 <sgallagh> Lastly, we would probably want to make this -devel stream a default stream (or at least a default in the EPEL buildroot) to enable dependent package building
18:51:25 <tdawson> Those last two don't really sound like problems, rather steps that will need to get done.
18:51:41 <tdawson> But, go on.  I'm following.
18:51:46 <sgallagh> Problem 1 has a corollary that we don't usually get git.centos.org content until release day.
18:52:07 <sgallagh> Unless RHT does us a favor and gives CentOS advance access
18:52:15 <sgallagh> But I wouldn't expect that.
18:53:25 <sgallagh> Problem 1 is harder than it sounds, because we don't have a public record of what commits match which package build in RHEL
18:53:29 <smooge> the problem is that if we remove the libuv from the stream.. we won't be able to install the libuv-devel
18:53:44 <sgallagh> smooge: Why?
18:54:04 <merlinm> I'm just catching up post-lunch... Please go ahead with a sync anytime.
18:54:10 <x3mboy> .hello2
18:54:11 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
18:54:11 <sgallagh> Or do you just mean because of Problem 2?
18:55:30 <smooge> sgallagh, when I have tried it.. ENVR of the two do not match and dnf/yum/rpm balk
18:55:50 <sgallagh> smooge: Right, you just restated Problem 2 above.
18:56:00 <smooge> well 2 says Fedora guidelines
18:56:43 <sgallagh> Maybe I was unclear. I was noting that they have a strict dependency because they've always been told to do that. (Which is right and proper)
18:57:27 <sgallagh> I should list another constraint: If at all possible, we must not need to modify individual specfiles from RHEL in order to accomplish our goal.
18:57:59 <smooge> there is another one which is build system related
18:58:31 <sgallagh> Rephrased: we should be able to use the specfiles from RHEL unmodified. If this is not achievable, it must be possible to automate the modification so it is not done manually for every package.
18:58:36 <sgallagh> smooge: Please go ahead
18:59:37 <smooge> even if you have ENVR being the same, koji kicks out checksum mismatches (part of why we can't put CentOS-7.7 aarch64 as something for noarch packages do not match the RHEL-7.7 x86_64 ones)
19:00:27 <smooge> i am unclear on what exactly has to be the same but even if they had the same ENVR, koji had a fit
19:00:44 <sgallagh> ah, hmm
19:01:10 <tdawson> I don't think koji would care.  The RHEL8 packages wouldn't be in Koji's database, only in the external repo's.
19:01:13 <sgallagh> smooge: I think they still build fine but could not be tagged because the tag matching that ENVR was already in use
19:01:16 <sgallagh> But...
19:01:22 <sgallagh> yeah, I was just about to say what tdawson said
19:02:04 <pgreco> if it has the same ENVR, just assume it won't work unless it is exactly the same package
19:02:22 <tdawson> If we were to do this in Red Hat's internal BREW builders (which are koji based), then we would have problems.  But not with Fedora's.
19:04:31 <sgallagh> If it came to it, we could probably manage special tagging rules.
19:04:42 * sgallagh handwaves
19:05:01 <smooge> this was in getting the buildroots so you could even try to build the package versus tagging
19:05:19 <tdawson> Anyway, go on.  It's something we can/should try ... see what works.
19:06:51 <sgallagh> That's pretty much where I am right now.
19:07:35 <sgallagh> The only alternative I can think of is manually curating individual packages to bilf just the missing subpackages and mangle their dependency to ensure they work with the RHEL base package.
19:07:42 <sgallagh> But that's ugly and unsustainable
19:09:52 <sgallagh> If there's nothing else, we can move on
19:09:57 <smooge> my only take was to make a separate repository which just builds these packages and does not have the 'we do not replace' rule on it
19:10:28 <smooge> but also ugly and unsustainable
19:10:34 <smooge> I have nothing else
19:10:38 <smooge> #topic Open Floor
19:10:53 <smooge> ok anyone have anything for the floor?
19:13:10 <smooge> #endmeeting