16:31:01 <travier> #startmeeting fedora_coreos_meeting
16:31:01 <zodbot> Meeting started Wed Sep 20 16:31:01 2023 UTC.
16:31:01 <zodbot> This meeting is logged and archived in a public location.
16:31:01 <zodbot> The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:31:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:05 <travier> #topic roll call
16:31:09 <travier> .hello siosm
16:31:12 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:31:47 <dustymabe> .hi
16:31:48 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:54 <dwalsh> .hi
16:31:55 <zodbot> dwalsh: dwalsh 'Daniel J Walsh' <dwalsh@redhat.com>
16:31:59 <marmijo> .hello marmijo
16:32:00 <zodbot> marmijo: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:32:26 <jlebon> .hello2
16:32:27 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:31 <aaradhak> .hello aaradhak
16:32:32 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:32:48 <fifofonix> .hello
16:32:48 <zodbot> fifofonix: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:32:58 <fifofonix> .hello2
16:32:59 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:33:05 <apiaseck> .hi c4rt0
16:33:06 <zodbot> apiaseck: Sorry, but user 'apiaseck' does not exist
16:33:36 <apiaseck> .hello c4rt0
16:33:37 <zodbot> apiaseck: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:35:34 <travier> let's start
16:35:53 <travier> We have some guests so we'll prioritize their topic but we have another urgent one
16:36:16 <travier> #chair dwalsh dustymabe marmijo jlebon aaradhak fifofonix apiaseck
16:36:16 <zodbot> Current chairs: aaradhak apiaseck dustymabe dwalsh fifofonix jlebon marmijo travier
16:36:23 <travier> #topic Action items from last meeting
16:36:44 <travier> * dustymabe to work with baude to open a podman issue for exploring the
16:36:44 <travier> options mentioned in
16:36:44 <travier> https://github.com/coreos/fedora-coreos-tracker/issues/1557#issuecomment-1716137519
16:37:22 <travier> Reading https://github.com/coreos/fedora-coreos-tracker/issues/1557#issuecomment-1720041318, it seems that this has been addressed
16:37:42 <travier> let's move on
16:37:50 <dustymabe> yeah I opened an issue in the podman repo too to follow up on those future threads
16:37:59 <travier> +1
16:38:04 <jlebon> +1
16:38:07 <travier> Cna you link it in the ticket above?
16:38:09 <travier> can*
16:38:22 <dustymabe> https://github.com/containers/podman/issues/19982
16:38:29 <dustymabe> will link there
16:38:37 <travier> thanks
16:38:39 <travier> #topic F39: Docker Run Fails
16:38:48 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1578
16:39:11 <travier> Running any Docker container on F39 fails. It works on our latest F38 images.
16:39:25 <travier> any container via Docker*
16:39:39 <travier> dustymabe suggested we paused the rollout
16:40:15 <travier> This is weird as all releases are using the same version of docker according to https://src.fedoraproject.org/rpms/moby-engine
16:40:36 <dwalsh> SELinux issue?
16:40:46 <dustymabe> travier: we pinned a while back in our f38 streams
16:41:03 <dustymabe> dwalsh: I think the reporter said the problem also happens when selinux is permissive
16:41:24 <jlebon> can confirm, just tested it too
16:41:31 <fifofonix> correct.  i'm the OP. was concerned i was missing SELinux warnings due to auditd
16:41:52 <fifofonix> (btw some SELinux denials re auditd also)
16:42:21 <fifofonix> has this been replicated on other distros?
16:42:34 <travier> I don't know
16:42:54 <jlebon> this is basic functionality. i agree we should pause the rollout.
16:43:28 <travier> Any opposition to pausing the rollout? Anyone up to doing the work to pause the roolout?
16:43:40 <jlebon> at least until we know more. if e.g. there's a manual workaround, maybe we don't respin and unpause
16:43:42 <fifofonix> only alternative would be to recommend stopping/disabling zincati
16:44:00 <fifofonix> speaking for myself this is what i've done
16:44:14 <jlebon> travier: i can pause it if agreed
16:45:02 <dustymabe> +1 to pause - would be nice to know if there was some sort of time frame on getting a fix
16:45:35 <fifofonix> if you're going to pause for this there has to be argument to add a test?
16:46:07 <dustymabe> we should have had tests all along
16:46:23 <dustymabe> there are some that exist but they got disabled some time ago and no one ever went back in to enable them
16:46:25 <fifofonix> meaning there are tests and we don't know how they passed?
16:46:33 <fifofonix> x-wires
16:46:34 <dwalsh> Docker is working for me on F39
16:46:53 <jlebon> dwalsh: cloud/server?
16:46:55 <dustymabe> dwalsh: what version of the RPM?
16:46:58 <dwalsh> Laptop
16:47:10 <dwalsh> f39 not fcos
16:47:18 <dustymabe> fifofonix: no - there are tests from the container linux days and they got disabled at some point
16:47:31 <dustymabe> dwalsh: right, that's why I asked what version of the RPM :) - will be good information
16:47:36 <dwalsh> But I might be using docker-ce
16:47:52 <dwalsh> Yup I am
16:48:03 <dwalsh> So never mind
16:48:06 <dustymabe> +1
16:48:13 <travier> thanks for checking
16:48:27 <travier> would be helpful to reproduce in a Fedora Cloud image
16:48:34 <jlebon> downloading image right now :)
16:49:00 <travier> #proposed We'll pause the rollout of the next stream due to https://github.com/coreos/fedora-coreos-tracker/issues/1578
16:49:49 <dustymabe> thanks fifofonix for testing.. if you want more comprehensive docker tests to get run in the future we'd be open to reviewing PRs too :)
16:50:14 <dustymabe> i think travier just opened a basic one in https://github.com/coreos/fedora-coreos-config/pull/2622/files
16:50:34 <travier> yes, I made the simplest test I could do
16:50:45 <fifofonix> yw & thanks travier
16:51:10 <dustymabe> +1
16:51:14 <fifofonix> +1
16:51:14 <dustymabe> for the #proposed
16:51:17 <jlebon> +1
16:51:35 <travier> #agreed We'll pause the rollout of the next stream due to https://github.com/coreos/fedora-coreos-tracker/issues/1578
16:51:57 <dustymabe> once we get a fix I'd propose we just promote everything and do a new next release
16:52:03 <travier> I'll let jlebon dop the PR to pause the rollout
16:52:08 <travier> dustymabe: +1
16:52:19 <ravanelli> .hi
16:52:20 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:52:26 <dustymabe> also thanks fifofonix for opening a BZ too
16:52:45 <dustymabe> I added myself as cc to watch the conversation there
16:52:49 <fifofonix> np. ami will still exist though right. so, despite pause it will be specifically reachable for other testing.
16:53:23 <fifofonix> i want to dig into the aws v2 thing more.
16:53:54 <travier> let's move to next issue?
16:54:05 <dustymabe> yeah, unless we switch back - at least right now I'm not proposing we do that
16:54:18 <jlebon> pause: https://github.com/coreos/fedora-coreos-streams/pull/772
16:54:19 <dustymabe> `next` is kinda there for some of the more experimental pieces
16:55:12 <travier> #topic New Package Request: crun-wasm
16:55:12 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1375
16:55:38 <travier> podman (desktop) folks want us to include crun-wasm and the wasm-edge runtime that comes with it
16:56:08 <travier> the python deps were a blocker, those have been apparently removed
16:56:27 <travier> we should now decide if we include this or not
16:57:15 <lsm5> wasmedge-rt rpm should be free of python deps
16:57:32 <dwalsh> This was requested by podman-desktop team, via me
16:57:43 <benoitf> I confirm crun-wasm + wasmedge-rt = no python dependency
16:58:00 <dwalsh> They want to be able to run and build WASM based containers in podman-desktop
16:58:16 <dwalsh> Their customers any ways.
16:58:30 <flouthoc> +1 I think `crun-wasm` should easily work with `wasmedge-rt` and it has no python deps.
16:59:02 <dwalsh> Hopefully the work Dusty and Brent and others are doing to allow podman-machine be based off of FCOS in the future can make these package additions disappear in the future.
16:59:28 <dwalsh> Using OCI Images.
16:59:46 <dwalsh> How big is addition of crun-wasm and wasmedge-rt?
16:59:52 <dustymabe> I don't have a strong opinion here.
17:00:28 <dustymabe> theoretically other people other than podman desktop could want this, we just haven't had any requests for it
17:01:13 <jlebon> it sounds like the team is starting to invest in the layering tech
17:01:28 <jlebon> based on the latest convos. but i'm guessing you haven't fully switched over yet?
17:01:41 <dwalsh> Well crun-wasm would work with Docker.  Can people use wasmedge-rt to run workloads outside of container.
17:01:59 <dwalsh> jlebon, Yup have not switched yet.
17:02:15 <benoitf> How big is addition of crun-wasm and wasmedge-rt? <--- it says "Will download: 4 packages (723.8?kB)" when doing the install
17:02:27 <benoitf> so it's still very small
17:03:21 <travier> https://github.com/coreos/fedora-coreos-tracker/issues/1375#issuecomment-1728123185
17:04:03 <jlebon> if we add it, we could still remove it in the future (once layering is the default). but the messaging is the tricky bit if users start depending on it.
17:04:24 <dustymabe> looks like the biggest file is `1.4M Jan  1  1970 /usr/lib64/libwasmedge.so.0.0.3`
17:04:43 <dustymabe> uncompressed ^^
17:04:54 <jlebon> another approach, if this is a minority of users asking for this is to have them opt into the experimental layering path
17:05:36 <travier> Approximately 2-3MB uncompressed in total
17:06:13 <travier> I'm OK adding it. Hopefully we can flesh out the missing bits for the layering path and podman desktop can start relying on that
17:06:31 <dustymabe> I'm +0.5
17:06:37 <travier> If I remember correctly, the main concern was about getting boot images from containers?
17:06:55 <travier> (I would prefer we don't add it but I don't really see a better option)
17:07:29 <dustymabe> yeah I think it's fine,, just would prefer to not have "runtime flavor of the week" get added every few months or so
17:07:39 <travier> agree
17:07:47 <dustymabe> i'm super up to speed on wasm, but I assume it has staying power?
17:07:48 <jlebon> yup
17:07:58 <dustymabe> i'm NOT
17:08:04 <travier> :)
17:08:19 <benoitf> hopefully we would not require any new package in mid-term due to the switch to OCI layers
17:08:31 <dustymabe> it certainly is nice that this set of packages is small
17:08:36 <travier> It's been here for a while but the more it goes the less it feels like it's going to replace uses cases outside of the browser
17:08:38 <benoitf> but customers are asking right now
17:08:44 <jlebon> travier: are boot images required though? can't they switch over on first boot?
17:08:46 <dustymabe> as opposed to the gvisor addition we recently did (which we can hopefully remove in the future)
17:09:14 <dustymabe> jlebon: I think the UX they are going for doesn't involve a lot of things happening on boot (i.e. hitting the network and rebasing)
17:09:34 <travier> jlebon: they can, but if they want to have their VM start quickly on first setup then they need pre-made boot images
17:09:45 <jlebon> dustymabe: right, but if it's opt-in and catering to a minority of users, that seems acceptable
17:10:16 <dustymabe> i don't know if they have a good "opt-in" mechanism right now
17:10:21 <travier> local layering could very well be an option as well
17:10:29 <travier> but podman would have to learn to do that
17:10:54 <jlebon> local layering was brought up in the past as well for other pkgs. don't quite remember why we didn't go that way
17:11:18 <jlebon> i guess those pkgs at the time was for broader functionality/higher impact of slow boot time
17:11:21 <dustymabe> at least for the gvforwarder thing - it was needed to even set up the network so the VM could talk  to the internet
17:11:30 <dustymabe> so that's why it was needed in the image
17:11:40 <travier> from memory it was for something needed for all setups
17:11:41 <dustymabe> these wasm packages could probably be layered fine
17:11:45 <travier> which is not the case here
17:12:25 <travier> I would prefer local layering but considering the size, I don't feel like it's worth the work
17:12:50 <dustymabe> yeah, and.. let's say wasm takes off.. now FCOS is a good place for it :)
17:12:56 <travier> :)
17:13:22 <dustymabe> i'm happy either way - i'll stick at +0.5
17:13:42 <travier> #proposed We will include crun-wasm & wasmedge-rt in Fedora CoreOS. We will revisit this discussion once the coreos layering story has matured.
17:14:24 <jlebon> i'm -0.25
17:14:32 <dustymabe> "revisit" -> we'd probably need to keep it in even if podman started building their own thing - or maybe we could drop it out on a major version rebase
17:14:44 <dustymabe> +0.5
17:15:00 <dustymabe> anybody else want to weigh in?
17:15:56 <travier> jlebon: real question: how should we procede if we don't add it?
17:16:08 <travier> how should they*
17:16:17 <jlebon> (to explain my negative: i would rather more time be spent working on the alternatives, and providing ad-hoc temporary solutions even if they're not great)
17:17:02 <jlebon> more packages is more overhead, and personally feels like we're adding them much more easily than in the earlier days of FCOS
17:17:58 <travier> agree :/
17:18:00 <dustymabe> I agree with the second part of that statement - but then again, for this package, unlike the gvforwarder, I feel like there could be other users
17:18:35 <dustymabe> so i don't look at this as 100% for podman
17:18:51 <dustymabe> now.. if someone else had requested it I do think we would have suggested to them to layer for now
17:19:24 <jlebon> right, exactly. if it became the hot thing and there was a lot of clamour for it, that makes the argument easier
17:19:28 <dustymabe> but I do think the podman machine/desktop use case (there's gravity there) does carry some weight here which leans me to +0.5
17:19:47 <dustymabe> and we are already talking to them about how they "build their own thing" in the future
17:19:58 <dustymabe> so I know they want to do it too
17:20:10 <travier> any other votes? marmijo aaradhak fifofonix apiaseck
17:20:35 <travier> (I'm assuming dwalsh benoitf and lsm5 are +1)
17:20:36 <dustymabe> ravanelli spresti
17:20:51 <jlebon> right, podman machine is a valid and great use of FCOS, but very much unlike other users
17:20:58 <travier> #chair ravanelli spresti dwalsh benoitf lsm5
17:20:58 <zodbot> Current chairs: aaradhak apiaseck benoitf dustymabe dwalsh fifofonix jlebon lsm5 marmijo ravanelli spresti travier
17:20:59 <benoitf> yes I'm +1
17:21:02 <aaradhak> +1
17:21:05 <lsm5> travier I'm +1 but I didn't know I had voting rights here :)
17:21:13 <benoitf> +1
17:21:24 <apiaseck> +1
17:21:34 <dustymabe> ehh - it's a loose system - i think we try to get consensus as much as we can
17:21:42 <lsm5> ack
17:21:45 <dustymabe> we probably should make it more formal
17:22:57 <travier> Maybe it would be nice to get a basic test in our config repo for it so that we at least know it's working
17:23:11 <dustymabe> yes - we did add a test for pasta networking when it was added
17:23:21 <dustymabe> let's try to get someone from the podman team to do the same with this
17:23:40 <jlebon> (now, the reason for the 0.25 is that yes in *this specific instance* the pkg is small and could have other users. but there's definitely a trend going on here.)
17:24:29 <travier> #proposed We will include crun-wasm & wasmedge-rt in Fedora CoreOS with a basic test for functionnality. We will revisit this discussion once the coreos layering story has matured.
17:24:52 <dustymabe> should that be agreed?
17:24:54 <flouthoc> @dustymabe do you mean basic e2e for running wasm workload with podman ?
17:24:55 <dustymabe> +0.5
17:25:10 <travier> I tweaked the proposed so did not want to make it a surprise
17:25:22 <travier> flouthoc: yes
17:25:29 <dustymabe> flouthoc: something along the lines of https://github.com/coreos/fedora-coreos-config/pull/2622 but for this use case
17:25:35 <flouthoc> We have that somewhere in crun ( https://github.com/containers/crun/blob/main/tests/wasmedge-build/run-tests.sh ) can be easily extended to podman repo.
17:25:43 <flouthoc> Sure that can be done.
17:25:49 <dustymabe> flouthoc: nothing crazy, just a sanity check
17:25:54 <flouthoc> cool
17:26:13 <dustymabe> ok let's close this out - would like to discuss https://github.com/coreos/fedora-coreos-tracker/issues/1575 today too
17:26:15 <travier> #agreed We will include crun-wasm & wasmedge-rt in Fedora CoreOS with a basic test for functionnality. We will revisit this discussion once the coreos layering story has matured.
17:26:30 <travier> #topic F39 drops some firmware packages that provide wifi/BT firmware
17:26:36 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1575
17:26:41 <dustymabe> 👋
17:26:41 <travier> we have 4 minutes
17:26:44 <travier> let's make it fast
17:27:06 <dustymabe> TL;DR wifi card firmware drivers got dropped as a requires from the linux-firmware package in f39+
17:27:28 <travier> I think we should remove them in F39 and add it to the release notes list
17:27:32 <dustymabe> by default we don't ship with wifi networking support (i.e. no tools to start up a Wifi NIC card)
17:28:11 <dustymabe> but if a user had layered the packages for wifi (i.e. NetworkManager-wifi) and were using wifi dropping these firmwares would cause their system to not have network any longer
17:28:22 <dustymabe> on upgrade
17:29:02 <jlebon> i think if we agreed not to ship NetworkManager-wifi, it makes sense to drop the firmware too. but indeed, we need to give a heads up
17:29:39 <dustymabe> I guess we could check with Networkmanager to see if they wanted to require those packages if NetworkManager-wifi gets installed
17:29:57 <travier> that could be an option as well
17:30:08 <travier> require would not work however
17:30:28 <travier> as I could have a card that does not need those firmware and thus not have them installed
17:30:34 <dustymabe> another option is more complicated but we could have a migration systemd unit that detects if Networkmanager-wifi is layered and add a request to those 4 firmware packages so they would continue to get layered
17:30:44 <jlebon> on one hand, it's suboptimal to pull all firmwares instead of just yours. on the other, you might not even know which pkg you need and their names are obscure
17:30:50 <dustymabe> the last option is drop it but give the users some warning (CLHM) of sorts
17:30:59 <dustymabe> yeah
17:31:22 <travier> We have a warning for modules, maybe we can extend it?
17:31:30 <dustymabe> travier: correct
17:31:40 <travier> (Note we're at time, moving to proposed)
17:31:45 <dustymabe> but I think I would want that warning to last for some time
17:31:54 <dustymabe> i.e. we make the change in f40 instead of f39
17:32:06 <dustymabe> ^^ example of "some time"
17:32:20 <jlebon> warning SGTM. don't think we should auto-migrate
17:32:30 <dustymabe> jlebon: that's fine
17:33:02 <travier> #proposed We'll remove those firmware in an upcoming FCOS release (timing to be determined) after we've announced the change. We'll add a CLHM warning.
17:33:24 <jlebon> +1
17:33:25 <apiaseck> +1
17:33:29 <travier> Announce, wait 3 months, do it?
17:33:43 <dustymabe> travier: is it safe to say for now (if we do another `next` release in the next week) that we can leave them in?
17:34:06 <travier> I'd say yes?
17:34:17 <travier> is that ambiguous from my proposed?
17:34:22 <dustymabe> travier: I like the major version bump as a good place to make changes like this - since we have a long time in `next`.
17:34:34 <travier> agree, but you want more warning time
17:34:37 <travier> we can't have both
17:34:39 <dustymabe> travier: not exactly ambiguous, ok cool
17:34:40 <dustymabe> +1
17:34:41 <travier> I'm for dropping now
17:35:06 <travier> but I don't care and can wait 3 months
17:35:07 <dustymabe> "more warning time" for me would be if we did it in F40
17:35:26 <dustymabe> so we have the CLHM helper for 6 months and then the users get the change
17:35:28 <travier> we can wait 6 months. At this points, it does not really matter
17:35:32 <dustymabe> exactly
17:35:38 <jlebon> +1
17:35:39 <dustymabe> OK I think all points have been resolved then
17:35:47 <travier> we wanted to keep the CLHM running only until F39 however
17:35:52 <travier> it does have some cost
17:36:07 <dustymabe> I would just put it in now for everywhere
17:36:19 <dustymabe> i.e. if release < 40 add CLHM helper
17:36:38 <jlebon> yeah, that was my understanding too
17:36:56 <dustymabe> the CLHM helper can detect if they've already layered in the firmware and not warn in that case
17:37:19 <travier> can it?
17:37:33 <travier> ok, this is taking too long
17:37:33 <dustymabe> yeah.. `rpm-ostree status --json` has a lot of info in it
17:37:54 <travier> #proposed We'll keep the firmware in the upcoming next release. We'll revisit this discussion next week
17:38:04 <dustymabe> +1
17:38:06 <jlebon> +1
17:38:06 <travier> #proposed We'll keep the firmware in the upcoming next releases. We'll revisit this discussion next week
17:38:33 <travier> I'm +1
17:38:38 <dustymabe> +1
17:38:44 <travier> #agreed We'll keep the firmware in the upcoming next releases. We'll revisit this discussion next week
17:38:50 <travier> #topic Open Floor
17:38:55 <jlebon> i know we're over but mind if i suggest something for the previous convo (wasm)?
17:39:01 <travier> We're past the time, will close in 2 minutes
17:39:03 <fifofonix> comment: thanks team, for pro-actively pausing `next` release.  annoying for you all i'm sure, but good to see this kind of a decision being taken speedily. (sorry gtg)
17:39:08 <travier> jlebon: go ahead
17:39:24 <dustymabe> fifofonix: +1 - thanks (again) for running `next`
17:39:27 <travier> fifofonix:
17:39:28 <travier> 👋
17:39:39 <dustymabe> #info we have a test day/week coming up soon: https://github.com/coreos/fedora-coreos-tracker/issues/1565
17:39:44 <jlebon> we could add the pkgs in next only for now. that makes it clearer that we *may* remove them in the future but still provides bootimages for podman.
17:39:58 <dustymabe> we also have a TODO list of items we'd like to get done before the test week starts: https://github.com/coreos/fedora-coreos-tracker/issues/1571
17:40:17 <dustymabe> Please volunteer for some of the TODO items (talk to ravanelli) if you'd like to get involved
17:40:32 <ravanelli> +1
17:41:06 <travier> jlebon: how would that work once we are out of beta?
17:41:07 <dustymabe> jlebon: that's an option - i'm not sure what stream they currently consume
17:41:22 <travier> they offer all streams from what I remember
17:41:24 <dustymabe> travier: we could keep building next separate from testing
17:41:29 <travier> stable by default
17:41:49 <dustymabe> yeah, i think it just complicates things - not sure it's really worth it
17:41:57 <jlebon> right, so they'd have to get users to opt-in for choosing the next stream
17:42:00 <travier> jlebon: could you propose that in the ticket?
17:42:31 <travier> If that does not take off, we have a weak agreement to add it
17:42:32 <jlebon> and it doesn't only apply to wasm but could be done for other pkgs too that podman wants soon
17:42:39 <jlebon> sure thing, will add
17:43:01 <travier> ok, will close
17:43:04 <travier> #endmeeting