16:02:28 <sgallagh> #startmeeting ELN (2021-03-26)
16:02:28 <sgallagh> #meetingname eln
16:02:28 <sgallagh> #topic Init Process
16:02:28 <zodbot> Meeting started Fri Mar 26 16:02:28 2021 UTC.
16:02:28 <zodbot> This meeting is logged and archived in a public location.
16:02:28 <zodbot> The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:28 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:28 <zodbot> The meeting name has been set to 'eln_(2021-03-26)'
16:02:28 <zodbot> The meeting name has been set to 'eln'
16:03:04 <cyberpear> o/
16:03:23 <dcavalca> .hi
16:03:24 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
16:03:33 <cyberpear> I'd had it an hour later in my calendar, but I'm here now
16:03:45 * tdawson is here.
16:03:52 <cyberpear> .hi
16:03:53 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:03:58 <claudiozz> o/
16:04:15 <sgallagh> #chair sgallagh michel_slm cyberpear dcavalca tdawson claudiozz
16:04:15 <zodbot> Current chairs: claudiozz cyberpear dcavalca michel_slm sgallagh tdawson
16:04:19 <michel_slm> .hi salimma
16:04:20 <zodbot> michel_slm: Sorry, but you don't exist
16:04:21 * cyberpear just learned the new command!
16:04:29 <michel_slm> .hello salimma
16:04:30 <zodbot> michel_slm: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
16:04:32 <sgallagh> .hi sgallagh
16:04:33 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:05:02 <sgallagh> #topic Agenda
16:05:08 <tdawson> .hi tdawson
16:05:09 <zodbot> tdawson: tdawson 'None' <tdawson@redhat.com>
16:05:26 <sgallagh> We don't have any tagged items for today.
16:05:35 <sgallagh> Anyone have a topic they'd like to discuss?
16:06:01 <dcavalca> I wanted to talk about https://github.com/fedora-eln/eln/issues/33
16:06:25 <dcavalca> there was some discussion on devel@ around this as well, but I didn't see anything conclusive
16:06:57 <michel_slm> can we get our own calendar on fedocal?
16:07:13 <dcavalca> I did file https://pagure.io/fedora-infrastructure/issue/9730 as a followup from the devel discussion, but that also didn't seem to get much traction
16:07:21 <sgallagh> #info Agenda Item: ELN Mirroring
16:07:45 <sgallagh> michel_slm: Calendar isn't working quite right at the moment (due to FAS upgrade)
16:07:57 <michel_slm> ahh
16:08:00 <sgallagh> I will get us a calendar entry once it's working again
16:08:17 <sgallagh> #topic ELN Mirroring
16:08:22 <dcavalca> basically I'm looking for a way to host a mirror of ELN on mirror.facebook.net
16:08:23 <michel_slm> the FAS upgrade probably explains the lack of traction on ELN mirroring too
16:08:36 <sgallagh> Most likely, yes.
16:08:46 <sgallagh> They've been "all hands on deck" for a while now
16:08:54 <cyberpear> I'd also find it valuable to have rsync access for ELN.
16:08:57 <dcavalca> both so we can consume it internally, but also that can serve as a mirror for anybody else that wants to consume ELN without taking up bandwidth from the main mirror
16:09:31 <dcavalca> there was some concern on devel that the main mirror server might not have enough resources to support rsync for something changing as fast as ELN
16:09:33 <jforbes> .hello2
16:09:34 <zodbot> jforbes: jforbes 'Justin M. Forbes' <jforbes@redhat.com>
16:10:07 <sgallagh> dcavalca: Right, it (usually) updates many times per day.
16:10:16 <sgallagh> However, right now it's also completely broken :-(
16:10:26 <sgallagh> It hasn't had a successful compose since March 2nd
16:10:31 <dcavalca> so, one option could be to do push mirroring instead of pull
16:10:37 <sgallagh> For different reasons each time
16:10:53 <dcavalca> either via rsync, or just by dumping stuff into say a S3 bucket (which I'd happily provide if needed)
16:11:02 <dcavalca> sgallagh: yeah, I meant to ask about that too
16:11:14 <michel_slm> sgallagh: would it make sense to compose it only once a day, like Rawhide?
16:11:16 <jforbes> Is it really different resource wise than rawhide?
16:11:26 <dcavalca> also the last compose is marked FINISHED_INCOMPLETE which doesn't inspire confidence
16:11:57 <sgallagh> dcavalca: That one had a failure in a container image build, I think. The RPM repos were fine.
16:12:00 <dcavalca> jforbes: my understanding is that because ELN regenerates every few hours, there's a concern that one might not be able to rsync it down before the new one kicks off
16:12:01 <sgallagh> (ostensibly)
16:12:19 <sgallagh> Though some key packages also appear to be missing.
16:12:42 <sgallagh> I've been flat-out this last two weeks on other things, so I haven't had a chance to look into it.
16:13:07 <sgallagh> The latest compose failure appears to be related to the maven module not being available for ELN.
16:13:31 <sgallagh> I rebuilt it today.
16:13:45 <tdawson> Is there a need for the compose building every few hours?  Can we turn that down to once a day, like rawhide?
16:14:04 <tdawson> As much as I like composes all the time, if we are over-using resources, I'm a little concerned.
16:14:17 <sgallagh> You can view the logs for the composes at https://odcs.fedoraproject.org/composes/ FTR
16:14:45 <michel_slm> tdawson: yeah, I was asking that too
16:14:47 <jforbes> I suppose the real question is, what doe so many composes buy us?
16:14:52 <sgallagh> I remember having a really good reason originally for the compose rate, but I forget what it was right now.
16:15:02 <tdawson> Hmm ... maybe I need to get my compose status page public
16:15:12 <sgallagh> bookwar[m]: Do you remember?
16:15:13 <michel_slm> bonus: wait for Rawhide to compose, then build that, so we have an apples to apples comparison
16:15:35 <sgallagh> I think she's on PTO, actually.
16:15:51 <bookwar[m]> here
16:16:00 <bookwar[m]> but i don't know the answer
16:16:20 <sgallagh> Ha
16:16:38 <bookwar[m]> I think we wanted the content resolver and generally fast turnaround when we were actively building things
16:16:59 <sgallagh> Ah right, it was so the Content Resolver would never be more than a couple hours out of date.
16:17:14 <sgallagh> So yeah, probably okay for us to slow it down
16:17:33 <sgallagh> At least until the RHEL 10 ramp-up hits.
16:18:27 <bookwar[m]> also we have the possibility to trigger compose manually when needed. So we can slow down, but then if something important lands and we want to try it, we can trigger a new build not waiting for anything.
16:19:05 <cyberpear> who can ours
16:19:14 <jforbes> That seems to take the risk out of slowing down
16:19:14 <cyberpear> press that button?
16:19:15 <sgallagh> Aside: the maven thing was fixed, now we have a different issue (xorg-x11-server-utils no longer exists)
16:19:24 <sgallagh> cyberpear: I can
16:20:02 <sgallagh> cyberpear: also jkaluza and contyk, IIRC
16:20:10 <sgallagh> I suspect bookwar[m] probably also
16:20:27 <cyberpear> 👍
16:20:44 <michel_slm> this ties in with the documentation topic in the agenda, we should probably list this so people who need a rebuild know who to ask
16:20:57 <sgallagh> michel_slm++
16:20:57 <zodbot> sgallagh: Karma for salimma changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:21:23 <bookwar[m]> the doc is here https://docs.fedoraproject.org/en-US/eln/compose/#_generating_the_fedora_eln_compose but it doesn't say about permissions
16:21:26 <dcavalca> sorry, my connection dropped
16:21:51 <sgallagh> "Members of the eln-sig FAS group (and eln-sig FAS staging group) can generate Fedora ELN compose using the odcs command line client."
16:21:58 <sgallagh> So yeah, apparently it does say :)
16:22:04 <bookwar[m]> oh, right :)
16:22:26 <sgallagh> Can someone remind me what the zodbot command is to list that group?
16:22:41 <bookwar[m]> That's the deal with Jan, he actually writes docs.
16:22:54 <nirik> NOTE: zodbot is still pointing to fas... so info may be out of date.
16:23:01 <nirik> zodbot: members eln-sig
16:23:02 <zodbot> nirik: Members of eln-sig: asamalik @bookwar jwboyer merlinm +psabata @sgallagh tdawson
16:23:50 <tdawson> Ya, I was thinking I had permissions.
16:23:50 <sgallagh> nirik: Thank you
16:24:12 <dcavalca> should we update that to be in sync with github org membership?
16:24:55 <sgallagh> Hmm, I'd prefer not to have those be exactly the same membership.
16:25:12 <sgallagh> Our SIG membership is *very* open, but I don't think we want literally anyone being able to fire off composes.
16:25:22 <jforbes> agreed
16:25:32 <dcavalca> sgallagh: works for me, as long as there's a process for folks to get added to the FAS group if they need to
16:26:14 <sgallagh> dcavalca: "Ask me and if I trust you I'll say yes" :)
16:26:31 <dcavalca> surewhynot
16:26:45 <dcavalca> also, I'm sorry, my connection dropped off earlier and I missed a bunch; did we reach a conclusion on the mirroring stuff?
16:27:09 <sgallagh> We ended up discussing the rate of composes and that it was too high for mirroring to be safe.
16:27:20 <sgallagh> We then established that we should be able to reduce that rate
16:27:48 <dcavalca> cool, thanks
16:28:01 <dcavalca> should we have a ticket to track that or just use https://github.com/fedora-eln/eln/issues/33 ?
16:28:34 <sgallagh> Probably good to get into the habit of tracking individual tasks separately
16:28:50 * sgallagh is bad about that
16:29:39 <dcavalca> alright, lemme file another ticket for it then
16:29:44 <sgallagh> dcavalca: Would you mind creating it and assigning it to me?
16:29:50 <sgallagh> I'll look into it on Monday.
16:30:28 <sgallagh> Once that adjustment is made, we can reopen discussions with Infra about setting up a mirroring arrangement.
16:30:43 <sgallagh> Note: I still don't think we necessarily want to add it to MirrorManager for general use.
16:30:48 <sgallagh> ELN is a development tool, not an OS
16:30:52 <dcavalca> sgallagh: https://github.com/fedora-eln/eln/issues/39
16:30:59 <dcavalca> doesn't look like I can assign it though
16:31:10 <dcavalca> sgallagh: yeah, agreed
16:31:34 <sgallagh> But there's definitely value in letting developers/COPR maintain a local mirror for performance.
16:31:40 <dcavalca> I think we can have this as an opt-in mirror for a subset of sites, and I'd expect most to ignore it
16:32:03 <dcavalca> or, if you prefer, we can just add the rsync endpoint and that's it
16:32:34 <sgallagh> I don't have a sufficiently informed opinion. Best to discuss with Infra
16:32:48 <cyberpear> I'd hope mirrormanager load would be trivial?
16:33:43 <cyberpear> it's convenient to redirect to local mirrors without changing the config on each deploy...
16:34:28 <dcavalca> I'm fine either way, but I tend to concur with cyberpear that if adding this to mirrormanager is cheap, there's not really any downside to it
16:34:31 <sgallagh> cyberpear: I *think* that requires the data to be on the primary Fedora mirror first, though I'm not certain.
16:35:12 <sgallagh> I'd summon some Infra folks to this discussion, but today is a bad day to distract them, I think.
16:35:27 <dcavalca> sgallagh: oh yeah let's wait for the AAA madness to be over
16:35:36 <michel_slm> so if/when we flip mirrormanager on, ideally there's at least a few mirrors up, right, so the main mirror doesn't take the brunt of traffic
16:36:38 <dcavalca> I'm sure I can make it work on our end if we end up the only ones mirroring this and need to shelter the load
16:37:24 <sgallagh> "shelter the load"?
16:37:39 <sgallagh> (Not familiar with that idiom, sorry)
16:38:03 <dcavalca> sgallagh: what I meant is that if we only have the master mirror and our mirror, I'm fine with our mirror taking up most/all of the traffic from mirrormanager is we need to
16:38:22 <sgallagh> Ohh, okay.
16:38:31 <sgallagh> FYI, I think you meant "shoulder the load" :)
16:38:47 <dcavalca> lol yes, you're definitely right
16:38:51 <dcavalca> ESL perks :)
16:39:15 <sgallagh> Completely understood. English is my first language and I'm *still* terrible at it.
16:41:03 <dcavalca> something else that could be useful, once we get the composes pipeline working again, is having a stable symlink to the latest successfully completed compose
16:41:25 <sgallagh> dcavalca: We (mostly) have that.
16:41:55 <dcavalca> sgallagh: oh ok, that makes it easy then :)
16:42:16 <sgallagh> I think there was an issue at one point where the last-successful compose got cleaned due to age, but ISTR Jan fixed that.
16:42:42 <sgallagh> https://odcs.fedoraproject.org/composes/latest/
16:42:43 <bookwar[m]> https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/ this supposed to be it, i think, but the definition of "success" may vary
16:43:04 <sgallagh> Oh right, the one bookwar[m] put up is correct
16:43:25 <sgallagh> The one I have is the "latest that FINISHED[_INCOMPLETE]" but not the latest that passed any sort of validation.
16:43:28 <dcavalca> yeah that's the FINISHED_INCOMPLETE one
16:43:56 <sgallagh> FINISHED_INCOMPLETE just means that all of the artifacts marked as "critical" completed.
16:43:59 <dcavalca> I'm less worried about validation at this stage, and more about "are the packages even there at all in this compose"
16:44:06 <dcavalca> yeah that's good enough
16:44:07 <sgallagh> And that optional output had one or more failures
16:44:11 <dcavalca> also something else we should document somewhere
16:44:30 <sgallagh> dcavalca: That's the same as for any other compose in Fedora
16:45:50 <dcavalca> sgallagh: yeah, I meant more in general, I was researching this a while ago and the only reference I found was https://laptrinhx.com/kevin-fenzi-rawhide-notes-from-the-trail-a-milestone-no-yes-1617947199/
16:46:10 <sgallagh> Quick straw poll: Does anyone oppose the plan to reduce the compose rate and look into limited mirroring?
16:46:47 <michel_slm> I almost +1 as a knee jerk reaction then I noticed the question is phrased as a negative :)
16:46:49 <jforbes> not I
16:47:10 <tdawson> I'm for the reduction.  Once a day is good enough for me.
16:47:37 <sgallagh> michel_slm: I didn't think it needed a formal vote, just a check that no one had opposing feelings
16:48:34 <bookwar[m]> i agree to reduce the rate
16:48:48 <sgallagh> #info ELN SIG will look into reducing the compose rate to enable mirroring of the content
16:49:47 <sgallagh> Any other topics for the last ten minutes?
16:50:21 <dcavalca> we should probably discuss the eln-extra proposal that tdawson posted, but I don't know if we can do that in 10 min
16:50:49 <michel_slm> we have to start somewhere, I guess. maybe how to get packages onboarded?
16:51:30 <dcavalca> on my end, I do have a list of potential candidate packages, but I need to stack rank and trim it a bit before it's useful
16:51:35 <michel_slm> e.g. can we do something like "someone proposes X" -> system computes the transitive dependencies, saying "ok, you also need Y and Z that are not in ELN yet" -> user confirms
16:51:59 <sgallagh> michel_slm: https://tiny.distro.builders/
16:52:13 <dcavalca> ^^ yes, this one
16:52:27 <cyberpear> I'd like ELN-extra to be built very similarly to EPEL, without access to hidden -devel packages, etc. to catch any issues early
16:52:28 <michel_slm> and have them tracked somewhere separately and add the ability to generate EPEL requests in Bugzilla for them
16:52:42 <dcavalca> I don't even think we need the EPEL integration at this stage
16:53:02 <tdawson> cyberpear: that would be painful to setup ... very very painful
16:53:30 <sgallagh> cyberpear: I don't think that's realistic, frankly.
16:53:47 <sgallagh> We'd have to set up a completely separate Koji, I think
16:53:52 <cyberpear> fair enough... maybe I'll find cycles to attempt it
16:53:53 * michel_slm slightly surprised httpd is not in ELN
16:54:10 <bookwar[m]> on the automation side we need something like is_eln_extra() function here https://github.com/fedora-ci/eln-build-pipeline/blob/master/eln-rebuild.py#L60 and it will be quite easy to add as soon as we have package list provided by content resolver
16:54:15 <sgallagh> michel_slm: It is... https://koji.fedoraproject.org/koji/buildinfo?buildID=1714362
16:54:36 <michel_slm> odd, https://tiny.distro.builders/workloads.html says rawhide for httpd
16:54:41 <tdawson> We could have json/yaml file in a git repo that people can do pull requests on.
16:54:42 <sgallagh> We also need a Koji target set up that will tag it appropriately
16:54:57 <michel_slm> oh the name is different, for some reason
16:55:07 <dcavalca> tdawson: yeah, an authoritative list in a repo would be best
16:55:18 <dcavalca> and with a name tied to each package, so we know who's responsible for what
16:55:22 <michel_slm> being able to PR is nice. that means the user submitting would need to keep track of dependencies themselves, I guess?
16:55:44 <michel_slm> but taking care of dependencies, we can do that with tooling later on
16:56:30 <dcavalca> yeah, we don't have to solve everything in the first implementation
16:56:52 <tdawson> It will depend on if the requestor cares about build dependencies or not.  Some might be fine using Rawhide, some will want ELN all the way.
16:57:20 <tdawson> But ya, getting a list started, and then populated would be a good first step.
16:57:24 <dcavalca> well, I think if we build X in eln-entra, we need all its dependencies to be included in eln + eln-extra
16:57:37 <michel_slm> right. it's the ones who care about EPEL that will care, but... it's easy for them to test too
16:57:43 <michel_slm> install ELN package, see what Rawhide packages get pulled in
16:57:43 <dcavalca> so, if X requires stuff that's not currently in eln, it'll have to be added to eln-extra
16:57:45 <tdawson> dcavalca: But which dependencies, runtime only, or buildroot also?
16:58:05 <michel_slm> tdawson: oof, good point. the build dependencies too
16:58:36 <tdawson> But, that's for further down the road.  Just getting the first package in is the hardest part.
16:58:41 <dcavalca> tdawson: I think both?
16:59:26 <sgallagh> We're out of time this week, so dcavalca: would you mind taking this to a fedora-devel mailing list thread?
16:59:31 <dcavalca> tdawson: yeah, and we can pick an easy package to start with and validate things, like say screen
16:59:42 <tdawson> :)
16:59:45 <sgallagh> Picked that one at random, did you? :-)
16:59:48 <dcavalca> sgallagh: sure, we can follow up on the thread that tdawson had started
17:00:22 <michel_slm> blasphemy. it should be tmux! :p
17:00:45 <dcavalca> michel_slm: tmux is already in base :)
17:00:56 <michel_slm> ah, doh
17:01:20 * michel_slm actually worked at the same place as the screen authors for a while
17:01:54 <sgallagh> Alright, thank you for coming, folks. We'll meet again in two weeks!
17:02:02 <sgallagh> #endmeeting