21:00:16 <tdawson> #startmeeting EPEL (2021-11-17)
21:00:16 <zodbot> Meeting started Wed Nov 17 21:00:16 2021 UTC.
21:00:16 <zodbot> This meeting is logged and archived in a public location.
21:00:16 <zodbot> The chair is tdawson. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
21:00:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
21:00:16 <zodbot> The meeting name has been set to 'epel_(2021-11-17)'
21:00:16 <tdawson> #meetingname epel
21:00:16 <zodbot> The meeting name has been set to 'epel'
21:00:16 <tdawson> #chair nirik tdawson bstinson pgreco carlwgeorge michel dcavalca
21:00:16 <tdawson> #topic aloha
21:00:16 <zodbot> Current chairs: bstinson carlwgeorge dcavalca michel nirik pgreco tdawson
21:00:31 <c4t3l> hello
21:00:39 <dcavalca> .hi
21:00:40 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
21:01:28 <tdawson> Hi c4t3l
21:01:28 <nirik> morning. I'm around, but starting an outage, so might be distracted
21:01:33 <tdawson> Hi dcavalca
21:01:42 <tdawson> Hi distracted nirik
21:02:42 <MichelAlexandreS> .hi
21:02:43 <zodbot> MichelAlexandreS: Sorry, but user 'MichelAlexandreS' does not exist
21:02:48 <carlwgeorge> .hi
21:02:49 <zodbot> carlwgeorge: carlwgeorge 'Carl George' <carl@redhat.com>
21:02:53 <MichelAlexandreS> .hello salimma
21:02:53 <zodbot> MichelAlexandreS: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
21:03:16 <MichelAlexandreS> sigh, so unless my display name in Matrix is exactly my FAS zodbot is confused
21:03:19 <tdawson> Hi MichelAlexandreS
21:03:24 <tdawson> Hi carlwgeorge
21:04:04 <tdawson> MichelAlexandreS: That's ok, we know it's you
21:04:06 <dcavalca> MichelAlexandreS: I think you can fix that in FAS if you add your nicks to the account profile
21:04:18 <c4t3l> .hi
21:04:19 <zodbot> c4t3l: Sorry, but user 'c4t3l' does not exist
21:04:30 <c4t3l> ugh sorry
21:04:49 <carlwgeorge> i believe in you, you exist to me
21:04:56 <c4t3l> :)
21:05:36 <MichelAlexandreS> dcavalca: I think I did already
21:05:53 <tdawson> #topic Old Business
21:06:29 <MichelAlexandreS> oh, I accidentally marked salimma as IRC, woops
21:06:35 <tdawson> I don't see pgreco around ... so I don't know about the macros he's been working on.
21:07:30 <tdawson> Sorry, I'm a bit distracted ... but I'll get over it.
21:07:56 <tdawson> Willit, no progress this week.
21:08:44 <tdawson> As for other old business, I believe it all falls into the regular topics, so I will address them there.
21:08:49 <tdawson> #topic EPEL-7
21:08:50 <pgreco> tdawson: I'm at a conference all week, though I had mentioned it, sorry
21:09:12 <tdawson> pgreco: You might have, it's been a crazy week for me.
21:09:55 <tdawson> pgreco: Any progress on the macros?  Or has the conference soaked up your time?
21:10:36 <pgreco> Nothing, last week was a mess preparing for this week of doing nothing....
21:10:37 <MichelAlexandreS> pgreco: SC21?
21:10:48 <pgreco> adoptingbitcoin
21:11:18 <tdawson> Going back to EPEL 7 ... anything for it this week?
21:12:16 <tdawson> I'll take that as a no ...
21:12:22 <tdawson> #topic EPEL-8
21:12:44 <tdawson> I do have one thing I'd like to bring up ... the centos-devel repo in our buildroot.
21:12:57 <nirik> it seems to now be empty?
21:12:59 <MichelAlexandreS> #info I have openssl3 put up for review, Conan Kudo has some feedback so I'm reworking it right now
21:13:38 <tdawson> Correct, not only is it empty, but even if they put stuff in it, it wouldn't be updated.
21:14:16 <tdawson> We need to take it out at somepoint, I'm wondering if now (sooner) is better than later.
21:15:19 <carlwgeorge> i agree, we should yank it now that we have solid guidance for requesting devel packages in crb, and proof that it works (albeit slowly)
21:16:08 <tdawson> It also makes the epel8 build repo less complicated, less prone to failures.
21:17:04 <tdawson> Does anyone want to keep it longer?   Or are we ok if I open a releng ticket for it's removal?
21:17:23 <carlwgeorge> i can work with releng to make that happen
21:17:49 <MichelAlexandreS> if we want to be super careful we can disable it first and remove later, but we can just remove it now too
21:18:25 <tdawson> True, at the very least, disable it.  But, it's currently empty.
21:19:17 <carlwgeorge> it doesn't affect the currently shipped packages, just new builds going forward
21:19:45 <nirik> I can remove it right now. :)
21:19:48 <carlwgeorge> and those maintainers should be pushing for crb inclusion of their buildreqs if they haven't already
21:20:47 <tdawson> I haven't heard any voices against it ... are we ok moving forward on this?
21:21:09 <tdawson> If nirik can disable/remove it, I'll send out an email.
21:21:10 <nirik> FWIW it broke epel8 buildroot when it was empty...
21:21:31 <MichelAlexandreS> oh, if being empty broke buildroot, yeah, +1 remove
21:21:34 <nirik> removed. :)
21:21:45 <carlwgeorge> wheee
21:21:51 <tdawson> OK, I'll send out an email after this meeting.
21:22:04 <carlwgeorge> now we need to update the docs about what epel8 builds against, i can do that
21:22:14 <tdawson> carlwgeorge Thanks
21:22:56 <tdawson> MichelAlexandreS: You said you were getting a review on your openssl3 package, which is great.  Did you need any more help or do you think it's moving forward?
21:24:48 <tdawson> Is there any other EPEL8 business?
21:26:03 <tdawson> OK, we might have lost MichelAlexandreS ... I'm going to move on for now.
21:26:08 <tdawson> #topic EPEL-9
21:26:21 <tdawson> How are things going on the epel9-next front ?
21:26:21 <carlwgeorge> i'd like to kick all the epel9-next stuff (well the entire epel9 topic really) to the end if we can, mboddu is gonna be late and he wants to be here for it
21:26:33 <tdawson> OK, I can do that.
21:26:38 <MichelAlexandreS> tdawson no, just FYI
21:26:39 <mboddu> Hehe, right on time :)
21:26:47 <carlwgeorge> sweet
21:26:55 <tdawson> Or ... since mboddu is here ... how about now :)
21:27:30 <carlwgeorge> so, i've been having a hell of a time explaining our epel9-next, mass branch/rebuild to epel9 plan to people inside red hat.
21:27:53 <tdawson> Do they not like it?  Or just not understand it?
21:28:03 <carlwgeorge> the latter
21:28:25 <carlwgeorge> also there is a lot of pressure to have epel9 "ready" at the rhel9 ga,  despite me assuring people it will be available faster than we've ever had before.
21:29:04 <nirik> for that we would need to have it before GA. :) which I suppose is possible, but...
21:29:29 <mboddu> rhel9 beta....
21:29:37 <carlwgeorge> my "what if" thought was, how about we go ahead and set up epel9, built against rhel9 beta content, and launch with consistent instructions as we have for epel8/epel8-next, that is target the main epel9, unless you find a problem that necessitates building against stream.
21:29:43 <MichelAlexandreS> carlwgeorge: time for flowcharts
21:29:59 <tdawson> But RHEL9 beta is not RHEL9
21:30:15 <carlwgeorge> it's closer than we've ever been before
21:30:36 <tdawson> CentOS Stream 9, at GA is closer to RHEL9 GA than RHEL9 Beta.
21:30:38 <carlwgeorge> my current understanding is it's basically 9.-1, i.e. a point release before 0
21:30:55 <dcavalca> what's the downside here?
21:31:18 <dcavalca> meaning, are there things in RHEL 9 that, were they sufficiently different from Beta, make our life harder?
21:31:23 <dcavalca> or our users life harder
21:31:28 <carlwgeorge> like every point release, there will likely be a few packages that need to be rebuild at the ga time.  but the repo will largely be populated with working packages.
21:31:31 <tdawson> Basically if they change any library between RHEL9 Beta and RHEL9 GA, you have to rebuild all your packages.
21:31:58 <dcavalca> do you have a sense of how wide is usually the Beta -> GA delta?
21:31:58 <mboddu> Yes, and I suggested if we need to I can run the mass rebuild at ga
21:32:06 <tdawson> No, nope nope ... There is a HUGE difference between beta and GA
21:32:13 <dcavalca> mass rebuild seems a reasonable way to approach this
21:32:15 <nirik> and beta also never gets any updates... so if there's insecure thing in it we are built against it.
21:32:21 <carlwgeorge> so that's not true anymore
21:32:34 <carlwgeorge> rhel9 beta gets errata
21:32:39 <nirik> huh, ok.
21:32:42 <tdawson> That IS TRUE ... There have been lots of builds against very important libraries every day
21:32:47 <mboddu> nirik: Apparantly beta is getting updates, thats a news to me, just got to know an hour ago
21:33:16 <carlwgeorge> yeah i only learned this via my epel talks with product and marketing people, who would really like to say epel9 is "ready" at ga launch
21:33:19 <tdawson> What about all the packages added and removed between Beta and GA
21:33:32 <MichelAlexandreS> yeah, that's surprising, I still here the old line of "beta might not be upgradable to GA" on Reddit - by RH engineers
21:33:39 <MichelAlexandreS> s/here/hear
21:33:45 <carlwgeorge> the branch requests are still checked against what's in the c9s composes
21:34:33 <carlwgeorge> as a cya, sure, but that's for the whole distro.  on a package by package level, most things for sure upgrade fine.
21:34:44 <tdawson> Why don't we simply grab a snapshot of CentosStream 9 when they cut for GA, and use that?
21:35:17 <carlwgeorge> well that would get back to the static problem of the old betas, but i'm not necessarily opposed to it
21:35:21 <tdawson> I'm totally against using BETA, despite what the marking department says
21:35:55 <MichelAlexandreS> yeah, seems using C9S for now makes it easier to debug too, since we can see what's in it
21:36:01 <tdawson> carlwgeorge Are you not seeing the constant builds, the packages added and removed every day?
21:36:04 <mboddu> I still feel like the original plan is better, but...
21:36:44 * nirik liked the orig plan too.
21:36:45 <carlwgeorge> my main goal is having consistent instructions for users between epel8/epel8-next and epel9/epel9-next, not saying one set of instructions now and a different one later
21:37:13 <tdawson> carlwgeorge Wow ... you totally have your mind set and are not listening to anything anybody here is saying.
21:37:16 <carlwgeorge> i.e. i'd like to go ahead and set up epel9 now and launch epel9 and epel9-next together
21:37:25 <carlwgeorge> no i assure you i'm listening
21:37:26 <tdawson> NNNNNNNOOOOOOOO!!!!!!!!!!!!11
21:37:38 <carlwgeorge> there is no need to yell
21:37:50 <tdawson> Well ... you didn't address a single one of my concerns.
21:37:59 <carlwgeorge> things are scrolling kinda fast
21:38:02 <nirik> I won't say that, but I would need a lot of convincing to be in favor.
21:39:03 <carlwgeorge> i understand that things are changing between the beta and ga.  but in the context of epel9 package build requirements, how significant are those changes (now that openssl3 has landed)?
21:39:04 <nirik> is there any documentation on differences allowed between beta and final available?
21:39:17 <dcavalca> tdawson: is your concern primarily about churn?
21:39:43 <tdawson> My concern is that we'll do all that work, and a majority of the packages won't install anyway.
21:39:52 * Eighth_Doctor waves
21:39:57 <mboddu> I think tdawson's concern is having a bunch of epel builds that might not work on rhel9 ga due to the diff in beta and ga content
21:40:01 <nirik> hey Eighth_Doctor
21:40:04 <carlwgeorge> how many soname bumps have we had since the beta?
21:40:09 <tdawson> Hi Eighth_Doctor
21:40:33 <dcavalca> nirik: if there is documentation on allowable changes I'd love to see it
21:40:34 <Eighth_Doctor> carlwgeorge: I only know of a couple, there hasn't been too much of it that _I've_ seen
21:40:35 <nirik> perhaps we could quantify or have a better idea if we gathered some stats? but I guess we can't really know until GA happens
21:40:37 <mboddu> Hello Eighth_Doctor
21:40:52 <dcavalca> agreed, this kind of decision is a lot easier with data in hand
21:41:17 <tdawson> carlwgeorge What is wrong with grabbing a snapshot of CentOS Stream 9 when they branch for GA?  That will be much closer than Beta.
21:41:21 <carlwgeorge> and this isn't something i'm locked into, i ran it by mboddu right before the meeting to see if it was technically possible from the infra side, but i'm bringing it here to discuss for a reason
21:41:25 <mboddu> Even we take rhel8 historical data, I think we changed a lot between rhel8 beta and ga due to some special cases
21:41:27 <carlwgeorge> tdawson: i'm not opposed to that
21:41:49 <carlwgeorge> like i said, the goal isn't to build against the beta, that's just a means to an end
21:42:39 <nirik> I'd prefer the orig plan to mass rebuild at GA time... if speed is needed we can just push everything stable... I would expect it wouldn't take very long to build things
21:42:46 <dcavalca> carlwgeorge: thanks for the clarification; if the goal isn't building against beta, I think I agree with tdawson that using a c9s snapshot might be a better option
21:43:16 <tdawson> On a related note about epel9-next ... We still can't get branches through due to scm-admins - https://pagure.io/fedscm-admin/issue/73
21:43:49 <dcavalca> oh that's unfortunate
21:43:58 <dcavalca> we were hoping to be able to start working on iptables-extra soonist
21:44:01 <dcavalca> * soonish
21:44:19 <carlwgeorge> all they have to do is update their fedscm-admin checkouts, i think
21:44:29 <mboddu> tdawson: Does RH can announce/say to the outside world when they cut the rhel9 ga from c9s?
21:44:59 <tdawson> mboddu: As far as I know ... no ... but this is still sorta new territory
21:45:16 <Eighth_Doctor> well, we can kind of figure it out based on what happened for beta
21:45:28 <Eighth_Doctor> beta was cut off from c9s at the end of August
21:45:32 <carlwgeorge> people can observe that in 8 the centos-stream-release package shifted to a new minor release 1-2 months before the minor release it was targeting was released
21:45:34 <nirik> we can just ask limb to update...
21:45:36 <Eighth_Doctor> and it was released in November
21:45:44 <mboddu> So, working out of the c9s snapshot might not be really possible. Maybe all we can do is guesstimate
21:46:14 <nirik> so if we do this... you would want a mass rebuild asap, and then _another_ mass rebuild for GA? or just let maintainers sort it out?
21:46:57 <Eighth_Doctor> well, I suppose I could look at the data for rebuilds in the EPEL9Dev OBS project I have
21:47:07 <Eighth_Doctor> https://build.opensuse.org/project/show/home:Pharaoh_Atem:CS9EPELDev
21:47:10 <carlwgeorge> we could treat it like every other point release and let them sort it, or we could standarize doing a mass rebuild at each point release
21:47:26 <Eighth_Doctor> there's rebuild history with information on what triggered the rebuilds
21:47:32 <tdawson> Ya ... regardless of where we get it from ... when would we start this?
21:47:38 <dcavalca> I kinda like the idea of doing mass rebuilds on point releases
21:48:09 <Eighth_Doctor> so treat them like branching mass builds?
21:48:10 <carlwgeorge> if i dig into the soname delta between the beta and c9s, what would be the threshold to be comfortable using the beta for the buildroot?  10%?  5%?
21:48:11 <dcavalca> if nothing else because it would trigger FTBFS and make it explicit if packages need help
21:48:25 * nirik shudders
21:48:29 <Eighth_Doctor> carlwgeorge: probably ~10%
21:48:45 <tdawson> Not really ... a package might still be installable and usable, but not buildable.
21:48:50 <Eighth_Doctor> the main ones to watch out for are the language stacks
21:49:08 <tdawson> Some of the epel7 packages were built at the very beginning, and are still used today.
21:49:11 <Eighth_Doctor> e.g. rhel8 beta and rhel8 final had default php switch from php 7.1 to 7.2
21:49:13 <Eighth_Doctor> and 7.1 disappeared entirely
21:49:32 <nirik> mass rebuilds are a lot of churn
21:49:33 <mboddu> dcavalca: If we are doing mass rebuilds for every point release, we need to keep in mind about the storage, also the rhel lifetime is 10 years, so, clean up would be hard
21:49:34 <tdawson> carlwgeorge Would you mind sending out an email, and we can discuss this on it?
21:49:43 <mboddu> s/clean up/archive/
21:49:58 <dcavalca> mboddu: that's a great point and something I had not thought of at all
21:50:08 <nirik> I'm not sure how you would handle them with bodhi either
21:50:20 <carlwgeorge> another point, using the beta would not give maintainers a false sense of security that their package will build with a buildroot devel package that isn't in the shipped content
21:50:20 <nirik> one "big update" ? :)
21:50:50 <carlwgeorge> the "rebuild on point releases" is a related idea but not tied at the hip with the initial thing
21:50:57 <carlwgeorge> tdawson: yes i can start an email about this on the list
21:51:16 <tdawson> carlwgeorge Thanks ... cuz I think it's going to need more discussion than we have time here for.
21:51:34 <MichelAlexandreS> if we do mass rebuild every point release, we might need a policy of saying e.g. only the latest two point release (plus -next for stream) is supported, and those on older ones will need to mirror on their own before we wipe the tree
21:51:37 * nirik really thinks we should just wait. If they want to seed us with GA content before relelase, great, we can mass rebuild then.
21:51:41 <carlwgeorge> yes to be clear i didn't come to the meeting today expecting an answer, this idea is in the early stages
21:51:49 <MichelAlexandreS> maybe epel9-prev, epel9 and epel9-next for symmetry
21:52:31 <mboddu> MichelAlexandreS: NOOOOOOOOOOOOOOOOOOOOOOOO (not shouting, but its my pain)
21:52:40 <mboddu> :)
21:52:55 <tdawson> We're getting close on time, I'm going to move on incase there are other things to bring up.
21:52:59 * nirik invents a new term: repo fatigue
21:53:04 <tdawson> #topic EPEL-Packaging-SIG
21:53:07 <tdawson> *laughs*
21:53:10 <mboddu> Lol
21:53:52 <tdawson> So, I did go through my bugs and tried to document the prefered way to request packages and go through the stalled package workflow.
21:54:20 <MichelAlexandreS> and I plan to get back to working on the tool for bulk branch request - but I probably need to sort through urgent tasks for the next few days first
21:54:31 <tdawson> But, evidently I got my asciidoc formatting mixed up - https://tdawson.fedorapeople.org/epel-docs/public/epel/epel-package-request/
21:55:23 <nirik> nice.
21:55:23 <MichelAlexandreS> Davide found the neat mdapi.fp.o site for querying package metadata, which would simplify my tool a lot. no need to require a Pagure token just to see if a package is branched
21:55:23 <tdawson> I'll have a pull request ready once I get the formatting fixed.
21:55:35 <MichelAlexandreS> tdawson++
21:56:22 <tdawson> I also have a pull request to clarify our Stalled EPEL Release Policy - https://pagure.io/epel/pull-request/129
21:57:01 <MichelAlexandreS> LGTM on the PR, I'll +1 there
21:57:40 <tdawson> Ya, we aren't really changing the policy, just clarifying what we originally discussed.
21:58:10 <tdawson> Because, as mboddu correctly pointed out, "no response" is not the same as "no action"
21:59:19 <tdawson> Looks like we have alot of +1's ... so we'll get that merged.
21:59:28 <tdawson> #topic General Issues / Open Floor
21:59:34 <MichelAlexandreS> yeah. someone waffing with "I don't know, I'll think about it" is a response that's not actionable
21:59:37 <MichelAlexandreS> .hi
21:59:38 <zodbot> MichelAlexandreS: Sorry, but user 'MichelAlexandreS' does not exist
21:59:52 <MichelAlexandreS> sigh, zodbot not synced yet
22:00:00 <tdawson> I know we dont' have much time for open floor ... does anyone have anything they want to bring up?
22:00:44 <nirik> so what is the next steps for epel-next ?
22:00:52 <nirik> we have builds now, announce?
22:01:02 * nirik may have missed it.
22:01:05 <mboddu> Did we announce c9s?
22:01:11 <mboddu> I know we sorta did a soft announce
22:01:14 <carlwgeorge> i'm working on coordinating the epel9-next launch with c9s
22:01:26 <carlwgeorge> rich said he wants to do that by the end of the month
22:01:33 <mboddu> Okay
22:01:52 <nirik> so just the announcement left? :) awesome.
22:01:55 <tdawson> So, sorta soft announce epel9-next ?
22:02:07 <dcavalca> carlwgeorge: for c9s I think we have most of what we need for the announcement lined up
22:02:12 <tdawson> Well ... and getting limb to update so we can request packages.
22:02:13 <mboddu> tdawson: Yeah, that works :)
22:02:14 <carlwgeorge> no, real announcement at the same time as c9s
22:02:16 <dcavalca> iirc there was just some artwork stuff remaining
22:02:32 <mboddu> tdawson: I will work with limb
22:02:36 <carlwgeorge> yeah in c9s there is no wallpaper
22:02:37 <tdawson> mboddu: Thanks
22:03:39 <tdawson> Well, we are over time.  Thank you all for coming and the good discussion.
22:03:45 <tdawson> And sorry for my yelling. :(
22:03:56 <c4t3l> bye yall
22:04:03 <tdawson> We'll talk next week, if not sooner.
22:04:13 * carlwgeorge hugs tdawson
22:04:14 <tdawson> #endmeeting