16:00:18 <sgallagh> #startmeeting Fedora ELN SIG (2021-05-11)
16:00:18 <zodbot> Meeting started Fri May  7 16:00:18 2021 UTC.
16:00:18 <zodbot> This meeting is logged and archived in a public location.
16:00:18 <zodbot> The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:18 <zodbot> The meeting name has been set to 'fedora_eln_sig_(2021-05-11)'
16:00:18 <sgallagh> #meetingname eln
16:00:18 <zodbot> The meeting name has been set to 'eln'
16:00:18 <sgallagh> #chair sgallagh bookwar
16:00:18 <zodbot> Current chairs: bookwar sgallagh
16:00:18 <sgallagh> #topic init process
16:00:26 <sgallagh> .hello2
16:00:27 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:00:36 <tdawson> .hi
16:00:37 <zodbot> tdawson: tdawson 'None' <tdawson@redhat.com>
16:00:37 <dcavalca> .hi
16:00:40 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
16:00:45 <tstellar_> .hi
16:00:46 <zodbot> tstellar_: Sorry, but you don't exist
16:01:04 <tstellar_> .hi tstellar
16:01:05 <zodbot> tstellar_: Sorry, but you don't exist
16:01:23 <tdawson> tstellar_: You exist to me ... I believe in you :)
16:02:11 <sgallagh> .hello tstellar
16:02:12 <zodbot> sgallagh: tstellar 'Tom Stellard' <tstellar@redhat.com>
16:03:04 <sgallagh> OK, smaller crowd than I expected today, but I suppose we can get started.
16:03:14 <sgallagh> #topic #topic #40 Create infrastucture for building alternative composes based on ELN - https://github.com/fedora-eln/eln/issues/40
16:03:26 <sgallagh> @tstellar_ I believe you wanted to talk about this a bit?
16:03:57 <tstellar_> sgallagh: Yes, we discussed this last meeting and decided to try to gather data about resource usage which I did here: https://github.com/fedora-eln/eln/issues/47
16:04:21 <sgallagh> Thank you for doing that
16:04:56 <michel_slm> .hello salimma
16:04:57 <zodbot> michel_slm: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
16:04:59 <sgallagh> Would you mind associating some context to those numbers for the meeting?
16:05:09 <sgallagh> #link https://github.com/fedora-eln/eln/issues/47
16:06:13 <tstellar_> sgallagh: Sure, so I was looking at the costs for two different scenarios: 1) How much resources does it take to do a mass rebuild (First table) and how much resources does it take to mantain the sync between rawhide (second table).
16:06:25 <sgallagh> tstellar_: So, IIRC the idea was to determine how much it would "cost" if we were to spin up another system similar to ELN to test new features?
16:07:26 <tstellar_> sgallagh: Yes, or seeing if Fedora koji had the capacitiy to accomodate this amount of usage.
16:07:51 <sgallagh> tstellar_: So, I have two bits of information to add there
16:08:27 <sgallagh> 1) We've tried doing things like this on a "secondary" Koji in the past. We had something called "shadow-koji" for doing builds on non-primary architectures.
16:08:49 <sgallagh> This proved to be an absolute nightmare for Infra to maintain and upkeep
16:09:36 <sgallagh> So I'd *much* prefer that we stick to primary Koji (and work on getting additional builder/storage resources there if needed)
16:10:10 <tstellar_> sgallagh: Ok, and that's what bookwar[m] suggested last time too.
16:10:23 <sgallagh> 2) I'd like to hear some examples of alternative composes that you're thinking of, because I wonder if they couldn't be fit into the ELN mold anyway.
16:10:25 <dcavalca> yeah, I'd love to not add yet another koji for this if we can avoid it
16:10:42 <jforbes> ,hello2
16:10:52 <jforbes> .hello2
16:10:53 <sgallagh> #info General consensus is that no one wants to stand up another Koji infrastructure
16:10:53 <zodbot> jforbes: jforbes 'Justin M. Forbes' <jforbes@redhat.com>
16:11:00 <sgallagh> Welcome, Justin
16:11:04 <tstellar_> sgallagh: My first idea, which I've mentioned before is a compose with clang as the default compiler.
16:11:48 <tstellar_> sgallagh: I think it could also be used to test new gcc versions, new compiler flags, etc. too.
16:12:24 <sgallagh> I'd argue that the new flags/versions things could (probably *should*) be done in ELN proper.
16:12:36 <jforbes> I thought new toolchain was doing sidetags first anyway?
16:12:48 <sgallagh> Let's keep in mind that our goal is not for ELN to be a distribution, it's a development tool.
16:13:08 <sgallagh> Essentially prepping for what will eventually become the next CentOS Stream and RHEL major releases.
16:13:48 <tdawson> One thing to note, each of those (clang, or gcc flags) would require a mass rebuild, so looking at your tables, you incure both resources, a mass rebuild and montly builds as you go along.
16:13:51 <dcavalca> tstellar_: is the goal to test a clang compose to gather extra signal, or to test a clang compose because we eventually want to switch to clang?
16:13:55 <sgallagh> From my perspective, I'd like to *encourage* bold changes early on (and get more restrictive the closer we get to the next RHEL branch)
16:14:10 <dcavalca> if the goal is the latter, I agree with sgallagh this should go in ELN proper
16:14:32 <sgallagh> tdawson: They don't necessarily require a mass-rebuild, though that's the best way to test *everything*
16:15:09 <tdawson> True, but they would need some type of initial build ... just not the "mass" of the full mass.
16:15:33 * sgallagh nods
16:15:37 <tstellar_> dcavalca: My idea is that experimenting with ELN would help make decissions about what to do for future RHEL.  So we might try something in ELN, but then based on the results decide not to use that feature for future RHEL.
16:16:40 <tstellar_> This is why I've been envisioning something separate from the official ELN compose.
16:16:51 <sgallagh> I have some thoughts on that, but it's actually leading me to a slightly different topic.
16:16:56 <dcavalca> so, one benefit of doing this in a separate compose is that we'd avoid potentially mixing up different signals
16:17:25 <tstellar_> dcavalca: Right or we could experiment with 2 different features in parallel.
16:17:28 <dcavalca> meaning, if there's multiple things changing / being tested at the same time, I could see trouble telling apart if an issue is because of clang or of some other change
16:18:12 <dcavalca> which then brings the question: do we do a different compose for *every* major thing? and how do we define "major"?
16:18:21 <tstellar_> tdawson: Right, the assumption is there is 1 mass rebulid at the begining then ongoiong sync costs.
16:18:22 <sgallagh> Doing such composes wouldn't be terribly hard from a strictly technical perspective
16:19:34 <sgallagh> In fact, I suspect strongly that we could do such composes directly from on-demand side-tags.
16:20:32 <sgallagh> I think we'd need jkaluza to help us get ODCS permissions set up, but once they're in place, we'd really only need to tweak the pungi/lorax config
16:20:49 <sgallagh> Just to tell it which tag to pull from
16:21:58 <sgallagh> Sorry, I'm getting off in the weeds a bit here.
16:22:22 <dcavalca> I quite like the idea of tying this to side-tags fwiw
16:22:42 * sgallagh nods
16:23:02 <sgallagh> It has the definite benefit that if and when we decide the experiment is a success, we can merge it back in easily
16:24:18 <michel_slm> one nice feature is if anyone can create labeled side tags (or at least create side tags with known prefixes)
16:24:36 <michel_slm> otherwise all side tags look virtually identical
16:24:41 <michel_slm> (tangential issue, though)
16:24:46 <sgallagh> tstellar_: Since this is something you're clearly interested in, would you be willing to take point on trying to scrape together the documentation we'd need to add new ODCS composes against a side-tag?
16:24:58 <tstellar_> sgallagh: Sure
16:25:02 <sgallagh> You'd want to talk to jkaluza, lsedlar and bookwar I think
16:26:15 <sgallagh> #action tstellar_ to reach out to ODCS and OSCI folks to gather information on generating composes from side-tags.
16:26:34 <sgallagh> tstellar_: Anything else you want to raise about this topic today or should we move to the next agenda item?
16:26:44 <tstellar_> sgallagh: That's it for now.
16:27:01 <sgallagh> Okay, thank you!
16:27:10 <sgallagh> #topic #19 Generate AWS images in Fedora Infrastructure - https://github.com/fedora-eln/eln/issues/19
16:27:48 <sgallagh> So, I'm of two minds on this; on the one hand, being able to quickly spin up ELN images could make development and testing easier.
16:28:25 <sgallagh> On the other hand, I don't really want ELN ending up as the platform for anything; that's what Fedora, CentOS Stream and RHEL are for.
16:28:59 <jforbes> Sure, but the test value is interesting
16:29:15 <sgallagh> So there's a definite part of me that feels it would be better to provide a simple tool/script to convert a Rawhide AMI into an ELN instance than it would be to maintain pre-created images.
16:29:25 <jforbes> I mean, I would hope no one is using rawhide as the platform for anything either
16:29:45 <sgallagh> jforbes: I will say nothing that would disabuse you.
16:29:56 <dcavalca> do we already ship a rawhide AMI ?
16:30:17 <sgallagh> https://aws.amazon.com/marketplace/pp/B07YWL3ZJM
16:30:27 <sgallagh> Oh wait. Didn't read far enough...
16:30:49 <dcavalca> "This product charges for seller support" lol
16:31:14 <dcavalca> but yeah, if we do ship rawhide, ELN doesn't seem that much worse
16:31:25 <dcavalca> as long as it's clearly documented you shouldn't use it for prod
16:31:41 <sgallagh> We don't have Rawhide images on AWS marketplace that I can find
16:31:52 <sgallagh> Though we do produce QCOW2 images that can be imported
16:32:35 <jforbes> Right, I don't think there is need or want to have them in marketplace
16:34:19 <nirik> FYI, we do upload all the rawhide images to aws. ;)
16:34:28 <tdawson> Speaking of this, we are still generating images every 3 hours ... before we do this (if we do this) we need to turn that down to once a day.
16:34:33 <sgallagh> oh, interesting
16:34:34 <nirik> fedimg.image.publish -- Fedora-Cloud-Base-Rawhide-20210506.n.0.aarch64 published in region, eu-west-2 (ami-04ccc55a460ca8985, hvm, gp2)
16:34:57 <nirik> fedimg.image.publish -- Fedora-Cloud-Base-Rawhide-20210506.n.0.x86_64 publ
16:34:57 <nirik> ished in region, eu-west-2 (ami-07e3ddaf31b47e3fe, hvm, gp2)
16:34:59 <sgallagh> tdawson: I'm actually not sure how to adjust that. I'll ask jkaluza
16:35:46 <tdawson> https://github.com/fedora-eln/eln/issues/39
16:36:20 <sgallagh> Yeah, I just sent him a direct email
16:36:45 <tdawson> Didn't want to interupt the discussion, just wanted to point out if we decide to do this, we should wait until the frequency goes down.
16:36:56 <sgallagh> It's a good point
16:38:06 <sgallagh> nirik: If we wanted to add ELN to whatever service is pushing Rawhide to AWS, how would we do that?
16:39:06 <nirik> it's fedimg... I guess an infra ticket, although as far as I know no one really is handling that service, so someone would have to figure it out. :)
16:40:21 <sgallagh> ack
16:41:21 <sgallagh> Assuming that "someone" means "someone from the ELN SIG", does anyone want to volunteer to bring this to an Infra ticket and help drive it through?
16:41:55 <jforbes> I can do that
16:42:23 <sgallagh> #action jforbes to work with Infra on delivery of AWS image for ELN
16:42:26 <sgallagh> Thanks!
16:42:39 <sgallagh> #topic Open Floor
16:43:05 <jforbes> Though I need to wait until the composes are on a normal schedule
16:43:19 <nirik> I had one thing to mention... but perhaps it's a wider thing, but wanted to bring it up here.
16:43:30 <sgallagh> I have one quick item for Open Floor: I'm working with the lorax maintainer to get `lorax-templates-rhel` packaged in Fedora so we can escape an ugly hack our composes have to do right now.
16:43:47 <sgallagh> I suspect this is also a prerequisite for the additional alternate composes we discussed earlier.
16:43:52 <sgallagh> EOM
16:43:56 <sgallagh> nirik: Go ahead
16:44:54 <nirik> the FTBFS scripts seem to file FTBFS bugs against fedora/rawhide for packages that fail to build in eln. ;( This resulted in one being filed for grub2 (which was building fine in rawhide), it was later moved with all the rawhide bugs pre-branch to f34, then it resulted in it being orphaned because grub2 maintainers left it alone since it was a eln bug...
16:45:32 <jforbes> Is that what happened with nouveau too?  It orphaned all the kernel bugs for nouveau :)
16:45:36 <nirik> so, we need to figure out a way to seperate out these or not fille them or something
16:45:40 <nirik> jforbes: yep.
16:45:44 <sgallagh> Yikes
16:46:04 <sgallagh> I don't even know where those scripts are
16:46:37 <nirik> releng repo
16:47:00 <nirik> anyhow, just a heads up that we should really fix the flow here somehow...
16:47:05 <jforbes> Did someone specifically add eln to those scripts, or why is it picking up eln at all?
16:47:14 <sgallagh> nirik: If you could file an ELN bug (or tag me on an appropriate releng one), I'll try to have a look next week
16:47:28 <nirik> jforbes: I am not sure.
16:47:45 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1924713 is the grub2 one
16:48:07 <nirik> sgallagh: sure, where at?
16:48:33 <sgallagh> nirik: https://github.com/fedora-eln/eln/issues/
16:48:39 <nirik> ok
16:48:43 <sgallagh> Thank you
16:49:09 <nirik> except I have to be a sig member to file a ticket. ok.
16:49:22 <jforbes> Yeah, scripts are very broken, nothing in the bug mentions eln until after it was orphaned
16:49:30 <dcavalca> oh, we should open up the tracker to allow anybody to file issues
16:49:38 <sgallagh> nirik: Wait, really?
16:49:41 <sgallagh> That can't be right
16:49:59 <nirik> sgallagh: yep, it's asking me to join.
16:50:02 <tstellar_> nirik: I filed a ticket before I was a a member.
16:50:05 <sgallagh> Please hold, that's a bug
16:50:20 <tstellar_> nirik: I think what you saw was the bug template for requesting membership.
16:50:31 <nirik> oh wait, there is a 'open a blank issue' link
16:50:37 <tstellar_> There is a link under that to create a bug (I know it's confusing, I was confused too).
16:50:47 <nirik> yeah, there is. confusing
16:52:16 <sgallagh> Yeah, I need to add another template
16:52:20 <sgallagh> For general issues
16:52:22 <sgallagh> Sorry about that
16:52:33 <dcavalca> nirik: while we sort this out, any update on rsync for ELN? https://pagure.io/fedora-infrastructure/issue/9730
16:53:54 <nirik> dcavalca: oh yeah, that... so, I need to talk to copr and internal rhel people.
16:54:16 <nirik> basically I can make it work but they will have to change their link to have another /path/ in it.
16:54:29 <nirik> I'll try and move it forward.
16:54:37 <dcavalca> sweet, thanks!
16:55:15 <nirik> https://github.com/fedora-eln/eln/issues/54 filed
16:56:39 <tdawson> Just wanted to note that Adam made good progress on the ELN Extras in Content Resolver - https://tiny.distro.builders/view--view-eln-extras.html
16:57:00 <tdawson> He's still working on the buildroot, but runtime dependencies are there.
16:57:21 <tdawson> Those are runtime dependencies, that are not in ELN
16:57:41 <sgallagh> OK, I added a generic "New Task" template for issues
16:58:42 <tdawson> Also, I don't really want nedit in ELN Extras, it's just a good test package.
16:59:13 <sgallagh> OK, we are coming to the top of the hour; does anyone have any last-minute urgent business?
16:59:24 <sgallagh> (this being literally the last minute)
17:01:08 <tdawson> Nothing from me
17:01:17 <sgallagh> Thank you for coming, everyone.
17:01:28 <sgallagh> Next meeting should be in two weeks.
17:01:32 <sgallagh> #endmeeting