2024-10-02 16:30:19 <@gurssing:matrix.org> !startmeeting fedora_coreos_meeting
2024-10-02 16:30:26 <@gurssing:matrix.org>  !topic roll call
2024-10-02 16:30:34 <@jbtrystram:matrix.org> !hi
2024-10-02 16:30:43 <@gurssing:matrix.org> !hi gursewak
2024-10-02 16:31:21 <@dustymabe:matrix.org> !hi
2024-10-02 16:31:21 <@dustymabe:matrix.org> 
2024-10-02 16:31:48 <@ravanelli:matrix.org> !hi ravanelli
2024-10-02 16:32:26 <@gurssing:matrix.org> !topic Action items from last meeting
2024-10-02 16:32:29 <@siosm:matrix.org> !hi
2024-10-02 16:32:48 <@gurssing:matrix.org> I don't see the last meeting's minutes in https://discussion.fedoraproject.org/tag/coreos-wg
2024-10-02 16:33:15 <@ravanelli:matrix.org> umm is the bot working?
2024-10-02 16:33:21 <@aaradhak:matrix.org> !hi aaradhak 
2024-10-02 16:33:36 <@gurssing:matrix.org> That's a manual step.
2024-10-02 16:34:16 <@ravanelli:matrix.org> I don't mean the last minutes, but all other actions
2024-10-02 16:34:20 <@jbtrystram:matrix.org> Yeah the bot seems out
2024-10-02 16:34:55 <@ravanelli:matrix.org> Let me send a message in the infra channel, or seem if something is there already
2024-10-02 16:35:21 <@dustymabe:matrix.org> gursewak: I think maybe Jonathan Lebon didn't send them out? 
2024-10-02 16:35:52 <@ravanelli:matrix.org> a jbtrystram already sent a message in the infra channel, thanks ;)
2024-10-02 16:36:16 <@gurssing:matrix.org> Going through the minutes, I don't see any new action items there
2024-10-02 16:37:39 <@gurssing:matrix.org> Moving on.
2024-10-02 16:37:40 <@gurssing:matrix.org> !topic Latest next VMWare OVA Fails To Boot
2024-10-02 16:37:49 <@gurssing:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1802
2024-10-02 16:40:22 <@dustymabe:matrix.org> I tagged this with the meeting label. We've characterized this a bit with Hristo Marinov and fifofonix. It looks like it's somehow related to the OVA we create. We need someone to dig in deeper. It needs to be someone who has access to VMWare and can run `cosa` to develop/iterate on a fix. 
2024-10-02 16:41:15 <@dustymabe:matrix.org> for testing purposes it seems like VMWare fusion on MAC or Windows is good enough, but I think once we've developed a fix we need to make sure someone runs it on ESXi too
2024-10-02 16:41:27 <@siosm:matrix.org> So this is a fresh FCOS 41 failing to boot when a fresh F41 (package mode) image works?
2024-10-02 16:41:29 <@dustymabe:matrix.org> them both being able to first reproduce the issue and then verify the images with the "fix" solve the problem
2024-10-02 16:41:51 <@meetbot:fedora.im> Meeting started at 2024-10-02 16:30:19 UTC
2024-10-02 16:42:02 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2024-10-02 16:42:03 <@dustymabe:matrix.org> travier: kind of
2024-10-02 16:42:11 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his
2024-10-02 16:42:38 <@dustymabe:matrix.org> travier: let me explain the info we have so far
2024-10-02 16:42:47 <@siosm:matrix.org> !topic Latest next VMWare OVA Fails To Boot
2024-10-02 16:42:57 <@hricky:fedora.im> !hi
2024-10-02 16:43:06 <@dustymabe:matrix.org> travier: I've got someone to test F41 server (i.e. Anaconda from ISO) and they said it worked
2024-10-02 16:43:23 <@dustymabe:matrix.org> also got someone to test installing with our ISO (bare metal workflow) - it works
2024-10-02 16:43:29 <@ravanelli:matrix.org> If it is just a matter of testing VMWare fusion on MAC, maybe I can give it a try:
2024-10-02 16:43:40 <@jlebon:fedora.im> !hi
2024-10-02 16:43:48 <@dustymabe:matrix.org> got someone to test by first using `testing` F40 OVA and then rebasing to `next` F41 OVA - it works
2024-10-02 16:43:54 <@zodbot:fedora.im> Gursewak Singh (gursewak)
2024-10-02 16:43:54 <@zodbot:fedora.im> None (jlebon)
2024-10-02 16:43:55 <@zodbot:fedora.im> Hristo Marinov (hricky) - he / him / his
2024-10-02 16:43:56 <@zodbot:fedora.im> Renata Ravanelli (ravanelli)
2024-10-02 16:44:19 <@dustymabe:matrix.org> F41 OVA doesn't work
2024-10-02 16:44:20 <@jlebon:fedora.im> is anyone else having matrix issues?
2024-10-02 16:44:25 <@dustymabe:matrix.org> F41 `next` OVA doesn't work
2024-10-02 16:44:51 <@dustymabe:matrix.org> though hmm.. I think fifofonix did mention that after the rebase from `testing` to `next` a `bootupctl update` caused the system to stop working
2024-10-02 16:44:53 <@hricky:fedora.im> Me.
2024-10-02 16:45:04 <@siosm:matrix.org> Jonathan Lebon: the bot is in slow mode
2024-10-02 16:45:14 <@ravanelli:matrix.org> They just restarted the channel bot Jonathan Lebon , other than that, it is normal here
2024-10-02 16:46:07 <@zodbot:fedora.im> Timothée Ravier (siosm) - he / him / his
2024-10-02 16:46:13 <@dustymabe:matrix.org> so at times during this I've thought it was the bootloader (GRUB was upgraded to 2.12, which was a large update) and at times I've thought it was something to do with the OVA
2024-10-02 16:46:14 <@zodbot:fedora.im> Sorry, I can only look up one username at a time
2024-10-02 16:46:52 <@siosm:matrix.org> Hristo Marinov said in the last comment that booting next directly was failing. This is really weird.
2024-10-02 16:48:00 <@siosm:matrix.org> we used to install the bootloader from cosa to the VM but we no longer do AFAIK?
2024-10-02 16:48:33 <@dustymabe:matrix.org> travier: correct. It should be the version exactly from the payload
2024-10-02 16:49:55 <@siosm:matrix.org> Ah, so it likely is something with the bootloader or shim
2024-10-02 16:50:37 <@dustymabe:matrix.org> Yes. but I think we need more of a commitment than that. i.e. we can get people already to try specific things, but what we really need is for someone with access to try a bunch of different things and also test potential fixes
2024-10-02 16:50:43 <@siosm:matrix.org> really weird that this would not affect classic Fedora. Maybe it's a difference in the GRUB configs? But that would still boot and fail, not error out directly
2024-10-02 16:51:00 <@dustymabe:matrix.org> travier: I thought so, but installing via bare metal workflow works if reports are correct 
2024-10-02 16:51:11 <@dustymabe:matrix.org> should be the same as the OVA??
2024-10-02 16:51:30 <@jlebon:fedora.im> hmm, wonder if it's related to the versioning info in the OVA + e.g. some changes in the kernel
2024-10-02 16:51:51 <@jlebon:fedora.im> e.g. if we actually needed to bump the min version
2024-10-02 16:51:55 <@siosm:matrix.org> yeah, installing bare metal from ISO should be 100% like the OVA minus the command line kargs change?
2024-10-02 16:52:02 <@ravanelli:matrix.org> I can help with the rest as well.
2024-10-02 16:52:15 <@dustymabe:matrix.org> Thanks Renata Ravanelli ! 
2024-10-02 16:52:36 <@dustymabe:matrix.org> ideally we'd have some VMWare infra just accessible to us in Fedora too 
2024-10-02 16:52:49 <@hricky:fedora.im> I can also help with testing.
2024-10-02 16:53:37 <@dustymabe:matrix.org> !action Renata Ravanelli to help us dig down and diagnose the issue further to find the root cause
2024-10-02 16:54:14 <@jlebon:fedora.im> https://github.com/coreos/fedora-coreos-config/blob/2d3404c98ae3daeb8914894ed99cfaa7f8e9938f/image-base.yaml#L24
2024-10-02 16:54:14 <@jlebon:fedora.im> e.g. right now we have: `vmware-hw-version: 17`
2024-10-02 16:54:31 <@ravanelli:matrix.org> Yeah, it would be really good to even  have a CI test running with VMWare
2024-10-02 16:56:00 <@gurssing:matrix.org> Moving on
2024-10-02 16:56:14 <@gurssing:matrix.org> !topic Roadmap to Fedora Bootable Containers
2024-10-02 16:56:21 <@gurssing:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1726
2024-10-02 16:56:41 <@jlebon:fedora.im> gursewak: i untagged that one earlier today, but i think the meeting ticket was already generated
2024-10-02 16:56:51 <@jlebon:fedora.im> i forgot to untag it immediately after the meeting last week
2024-10-02 16:57:06 <@gurssing:matrix.org> Got it.
2024-10-02 16:57:17 <@gurssing:matrix.org> That's all the issues tagged.
2024-10-02 16:57:34 <@siosm:matrix.org> For this issue, we talked about it in the bootc meeting
2024-10-02 16:58:17 <@siosm:matrix.org> I suggested doing a tier-y with just enough ignition support to be able to re-use our kola + cosa workflow with the bootc images to leverage the infra and testing integration that we have right now but for bootc
2024-10-02 16:58:33 <@siosm:matrix.org> For this issue, we talked about it in the bootc meeting earlier this week
2024-10-02 16:58:57 <@siosm:matrix.org> (we would not add ignition to bootc images, this would be just for testing)
2024-10-02 16:58:59 <@dustymabe:matrix.org> so now tier-x and tier-y ?
2024-10-02 16:59:39 <@siosm:matrix.org> Yes. It's a mean to get gating for packages in bodhi on "some" bootc testing as soon as possible
2024-10-02 17:00:30 <@siosm:matrix.org> I think it's where we'll get the most value. The manifest duplication is not great but it's not the most painful part. If we can avoid breaking package update landing in Fedora then we get much more
2024-10-02 17:00:52 <@jlebon:fedora.im> there's a wide gap right now with how CI is done in the rest of Fedora.
2024-10-02 17:00:52 <@jlebon:fedora.im> travier: my concern with doing that is that we would very likely be primary maintainers for it. ideally this is owned by a wider group.
2024-10-02 17:01:15 <@dustymabe:matrix.org> but we already have FCOS tests running on packages in Fedora.. where we could just make the results of those test gate already? 
2024-10-02 17:01:50 <@siosm:matrix.org> Agree that we would likely be the one maintaining it. Hopefully, this would significantly lighten the load on the FCOS side and compensate that.
2024-10-02 17:01:53 <@dustymabe:matrix.org> For those interested - please follow https://matrix.to/#/#jenkins-coreos:fedoraproject.org 
2024-10-02 17:03:34 <@jlebon:fedora.im> 1. add fedora bootc CI setup for running in bodhi updates (using e.g. testing farm or tmt, which i think is what is used downstream as well in rhel-bootc and centos-bootc)
2024-10-02 17:03:34 <@jlebon:fedora.im> 3. change proposal to turn on gating. maintain CI together
2024-10-02 17:03:34 <@jlebon:fedora.im> 2. upstream tests that make sense from FCOS to the fedora bootc level. this might require reworking things to work in the new framework
2024-10-02 17:03:34 <@jlebon:fedora.im> roughly the steps I see are:
2024-10-02 17:03:52 <@siosm:matrix.org> Gating on FCOS tests would also help but does not bring us closer to sharing tests.
2024-10-02 17:04:59 <@siosm:matrix.org> From my experience with Zuul (I'm not sure this fully applies to tmt/testing farm), 1 would be a lot of work
2024-10-02 17:05:26 <@jbtrystram:matrix.org> step 2 is a lot of work as well from what I understand (moving from ignition configs to SSH)
2024-10-02 17:05:38 <@jlebon:fedora.im> travier: yeah, that's exploration to do there for sure. the QE folks on the bootc team have a lot of experience with it
2024-10-02 17:05:41 <@siosm:matrix.org> Zuul is centered around booting an pre-built VM image or a container an running Ansible playbooks or scripts in it
2024-10-02 17:07:29 <@dustymabe:matrix.org> we are literally like 1 step away from success with this 👆️ though
2024-10-02 17:07:46 <@dustymabe:matrix.org> I realize that has nothing to do with bootc, but man we got so close
2024-10-02 17:07:58 <@siosm:matrix.org> agree
2024-10-02 17:07:59 <@jlebon:fedora.im> there's a lot of tradeoffs, but basically i think for this to work as intended, we need a CI platform designed and brought up by the wider group
2024-10-02 17:08:04 <@siosm:matrix.org> how flaky is it right now?
2024-10-02 17:08:19 <@dustymabe:matrix.org> travier: look at the history in the channel
2024-10-02 17:08:41 <@dustymabe:matrix.org> it's not bad right now 
2024-10-02 17:08:56 <@jlebon:fedora.im> dustymabe: appearances can be deceiving :) there's a wide gap to cross before we can scale it up. probably work in bodhi, and work in the community
2024-10-02 17:08:58 <@dustymabe:matrix.org> but of course would increase pressure if some tests were flaking for whatever reason
2024-10-02 17:09:39 <@jlebon:fedora.im> unless we're ok with keeping the scope where it is (i.e. a small subset of packages)
2024-10-02 17:09:40 <@dustymabe:matrix.org> I guess I'm missing the remaining pieces
2024-10-02 17:09:46 <@siosm:matrix.org> we can probably pursue both in parallel.
2024-10-02 17:10:01 <@dustymabe:matrix.org> it's a good start anyway
2024-10-02 17:10:01 <@jlebon:fedora.im> the way it's implemented right now is really awful
2024-10-02 17:10:53 <@siosm:matrix.org> what do you mean?
2024-10-02 17:11:00 <@jlebon:fedora.im> (i say this as the person that implemented it :) )
2024-10-02 17:12:07 <@jlebon:fedora.im> travier: we have this regex that combines all the packages to watch for the messages we want: https://github.com/coreos/coreos-ci/blob/c088dcc406170d2eee426505e33bcd7088bbb2cd/jobs/bodhi-trigger.Jenkinsfile#L62-L65
2024-10-02 17:12:50 <@jlebon:fedora.im> this works because we don't currently watch that many packages yet, but it doesn't scale
2024-10-02 17:13:13 <@jlebon:fedora.im> there's also the fact that gating and critical path packages are concepts that are linked in ways we may not want to for this
2024-10-02 17:13:48 <@dustymabe:matrix.org> re: gating/critical path - we'd have to figure that out with `bootc` too? 
2024-10-02 17:14:22 <@jlebon:fedora.im> we're hitting all this because we're doing it on the side. with bootc, i think the goal would be to integrate into the existing Fedora CI effort
2024-10-02 17:14:46 <@jlebon:fedora.im> or at least not bring up another type of CI
2024-10-02 17:15:42 <@dustymabe:matrix.org> I think that's fine, but I don't think it's going to be here soon (maybe Fedora 43 or 44) ?
2024-10-02 17:16:10 <@jlebon:fedora.im> i guess this is straying from the topic a bit :)
2024-10-02 17:16:10 <@jlebon:fedora.im> what i'd say is: let's talk with the bootc QE folks and where their minds are at
2024-10-02 17:17:33 <@siosm:matrix.org> not really
2024-10-02 17:18:04 <@siosm:matrix.org> if we rebase to the tier-x manifest then we also depend on their CI setup
2024-10-02 17:19:27 <@jlebon:fedora.im> travier: not sure i follow the connection
2024-10-02 17:19:56 <@dustymabe:matrix.org> 
2024-10-02 17:19:56 <@dustymabe:matrix.org> > rebase to the tier-x manifest
2024-10-02 17:19:56 <@dustymabe:matrix.org> are you talking about https://github.com/coreos/fedora-coreos-config/pull/3177 ?
2024-10-02 17:20:17 <@siosm:matrix.org> yes
2024-10-02 17:20:58 <@dustymabe:matrix.org> ehh. I don't think so. We're still relying on our CI. We're just inheriting the package list (mind that it's not locked NVRs or depsolved)
2024-10-02 17:21:41 <@jlebon:fedora.im> once we have fedora bootc tier-x with gating CI, then we can actually work on derivation for real, and the tier-x manifest stuff in that PR would be gone
2024-10-02 17:21:51 <@dustymabe:matrix.org> but I think we do want the upstream definition to be more scrutinized - hence: https://gitlab.com/fedora/bootc/tracker/-/issues/40 
2024-10-02 17:22:44 <@jlebon:fedora.im> that PR is a way to share now, with the objective that the goal should be that we share at the image or "lockfile" level eventually
2024-10-02 17:23:11 <@jlebon:fedora.im> (or some other level that gives us guarantees that CI on tier-x remains meaningful even as we add our things)
2024-10-02 17:23:44 <@dustymabe:matrix.org> I think if we know the goal is to share in the future it's a good first step, but we do have that open request out to them about branches that will make it easier to deal with the submodules 
2024-10-02 17:24:15 <@jlebon:fedora.im> yeah, that's a big one
2024-10-02 17:24:23 <@dustymabe:matrix.org> https://gitlab.com/fedora/bootc/tracker/-/issues/39
2024-10-02 17:25:41 <@siosm:matrix.org> what I mean is that we need some form of CI that validates changes in bootc for downstream consumers. That can be upstreaming some of tests in bootc or directly running FCOS tests on bootc PRs
2024-10-02 17:26:15 <@siosm:matrix.org> if we have neither, then we rely on bootc CI only for changes in bootc, which we'll only really test when we bump the submodule "downstream" in fcos
2024-10-02 17:26:21 <@siosm:matrix.org> which means is not ideal
2024-10-02 17:26:42 <@siosm:matrix.org> what I mean is that we need some form of CI that validates changes in bootc for downstream consumers. That can be upstreaming some of FCOS tests in bootc or directly running FCOS tests on bootc PRs
2024-10-02 17:26:54 <@siosm:matrix.org> which means it is not ideal
2024-10-02 17:26:59 <@dustymabe:matrix.org> agree. We want changes to upstream to be tested
2024-10-02 17:27:24 <@jlebon:fedora.im> basically, we (and probably e.g. Iot) have a lot of tests that aren't actually FCOS-specific and we would want to upstream those
2024-10-02 17:27:24 <@jlebon:fedora.im> right, gotcha. yes, i see this as a prerequisite to derivation.
2024-10-02 17:28:13 <@dustymabe:matrix.org> shall we close off this topic? 
2024-10-02 17:28:47 <@gurssing:matrix.org> Almost time anyways.
2024-10-02 17:28:49 <@gurssing:matrix.org> !topic Open Floor
2024-10-02 17:29:40 <@gurssing:matrix.org> Thanks everyone for coming.
2024-10-02 17:29:54 <@gurssing:matrix.org> !endmeeting