16:30:50 <travier> #startmeeting fedora_coreos_meeting
16:30:50 <zodbot> Meeting started Wed Sep 27 16:30:50 2023 UTC.
16:30:50 <zodbot> This meeting is logged and archived in a public location.
16:30:50 <zodbot> The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:50 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:53 <travier> #topic roll call
16:31:21 <dustymabe> .hi
16:31:26 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:16 <travier> .hello
16:32:16 <zodbot> travier: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:32:20 <travier> .hello siosm
16:32:24 <mnguyen> .hello mnguyen
16:32:26 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:31 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:38 <travier> #chair dustymabe mnguyen
16:32:38 <zodbot> Current chairs: dustymabe mnguyen travier
16:32:42 <ydesouza> .hello ydesouza
16:32:48 <zodbot> ydesouza: ydesouza 'Yasmin Valim de Souza' <ydesouza@redhat.com>
16:32:51 <marmijo> .hello marmijo
16:32:56 <zodbot> marmijo: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:33:00 <travier> #chair marmijo ydesouza
16:33:00 <zodbot> Current chairs: dustymabe marmijo mnguyen travier ydesouza
16:33:30 <jlebon> .hello2
16:33:36 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:34:19 <travier> #chair jlebon
16:34:19 <zodbot> Current chairs: dustymabe jlebon marmijo mnguyen travier ydesouza
16:35:39 <travier> let's start
16:35:47 <travier> #topic Action items from last meeting
16:36:17 <travier> no action items as far as I can see
16:36:49 <travier> #topic F39 drops some firmware packages that provide wifi/BT firmware
16:36:55 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1575
16:37:01 <travier> Let's resume last week discussion
16:37:34 <dustymabe> summary of last week's discussion: https://github.com/coreos/fedora-coreos-tracker/issues/1575#issuecomment-1728277613
16:37:59 <travier> The problem with adding another CLHM module that checks for installed packages is that it calls rpm-ostree status on  every boot
16:38:25 <travier> See https://github.com/coreos/fedora-coreos-config/pull/2510
16:38:47 <dustymabe> what's the problem exactly? multiple things calling `rpm-ostree status` ?
16:38:56 <travier> and 99% of the time, it will just be a waste of CPU & IO
16:39:19 <travier> for modularity it was OK as this was limited in time, here, it's going to be for 6 months
16:39:55 <travier> and cri-o was the main package affected, which is clearly a core use case for FCOS
16:40:02 <travier> but wifi isn't
16:40:15 <aaradhak> .hi aaradhak
16:40:22 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:40:29 <travier> thus why I'm advocating for a smaller window or just a mail and no CLHM warning
16:40:34 <travier> #char aaradhak
16:40:42 <dustymabe> travier: we don't have to call `rpm-ostree status` - can use `rpm -q` too
16:41:01 <jlebon> do you have more context on the IO waste part? i don't think it's that bad
16:41:07 <travier> Then I guess it's marginally better, but not by a lot
16:41:12 <jlebon> also note the daemon autoexits
16:41:30 <travier> yes, it's not much, but is it worth it?
16:41:35 <dustymabe> I think so, yes
16:41:47 <dustymabe> my $.02
16:42:37 <dustymabe> i guess one thing we can do is "record" if we get a negative hit and not run any more
16:42:42 <jlebon> i'm pretty sure `rpm-ostreed` actually always starts on every boot just because of the way the unit is setup. probably shouldn't do that but... wrt this it's not really adding much overhead since it'll already be running
16:43:02 <dustymabe> i.e. if we run once and don't detect NetworkManager-wifi then we won't run again
16:43:13 <dustymabe> it's not as correct, but...
16:43:32 <dustymabe> i'd just prefer to run on every boot (by default happens once every two weeks)
16:43:40 <jlebon> try this:
16:43:46 <jlebon> $ cosa run -x 'systemctl status rpm-ostreed' | grep 'Active'
16:43:55 <jlebon> Active: active (running) since Wed 2023-09-27 16:43:33 UTC; 1s ago
16:44:48 <ravanelli> .hi
16:44:54 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:45:44 <travier> OK then I guess it does not matter then
16:46:18 <apiaseck> .hi c4rt0
16:46:24 <zodbot> apiaseck: Sorry, but user 'apiaseck' does not exist
16:46:32 <apiaseck> .hello c4rt0
16:46:38 <zodbot> apiaseck: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:46:44 <travier> #chair apiaseck
16:46:44 <zodbot> Current chairs: apiaseck dustymabe jlebon marmijo mnguyen travier ydesouza
16:46:48 <travier> #chair ravanelli
16:46:48 <zodbot> Current chairs: apiaseck dustymabe jlebon marmijo mnguyen ravanelli travier ydesouza
16:47:10 <travier> indeed, zincati will start the daemon
16:47:26 <travier> ok, then I get we don't have this concern then
16:47:48 <dustymabe> 👍
16:48:15 <dustymabe> so of the options:
16:48:33 <dustymabe> 1. having NetworkManager-wifi require the wifi firmware packages in the future:
16:48:35 <dustymabe> - involves another team, not great because it requires firmwares that aren't needed
16:48:37 <dustymabe> 2. running a migration that detects if NetworkManager-wifi is layered and add the 4 firmware packages to the layering requests
16:48:39 <dustymabe> 3. adding a CLHM helper that detects when NetworkManager-wifi is layered and warning/instructing the user on necessary action.
16:48:54 <dustymabe> - do we agree those are the options? any other options to add?
16:49:08 <dustymabe> - what option do people think we should pursue
16:49:14 <travier> I'm for 3
16:49:52 <mnguyen> Vote for 3
16:49:58 <dustymabe> 3 WFM
16:50:04 <jlebon> also for 3, though i think there's a more generic issue here that doesn't affect just us
16:50:37 <ravanelli> 3 for me too
16:50:43 <jlebon> i wonder if in the anaconda path, there's logic somewhere that knows to only pull in the relevant pkgs?
16:50:45 <dustymabe> agree. actually this is probably less of a corner case for IoT because I think they include wifi by default
16:51:03 <dustymabe> though they might mention those specific firmwares in there manifest list?
16:51:22 <travier> they explicitely list the firmware in their manifest
16:51:41 <dustymabe> ahh - ok :)
16:52:07 <jlebon> i wonder if it's pulled in on fedora workstation/server too
16:52:20 <jlebon> then it would really just be an issue for us
16:52:21 <dustymabe> well it's still a recommends
16:52:30 <travier> https://pagure.io/fedora-iot/ostree/blob/main/f/fedora-iot-base.json
16:52:32 <dustymabe> and I think those platforms honor recommends
16:52:50 <jlebon> NM-wifi recommends the firmwares?
16:53:01 <dustymabe> no. the main package (linux-firmware) recommends them
16:53:04 <travier> linux-firmware recommends the others
16:53:13 <travier> https://src.fedoraproject.org/rpms/linux-firmware/blob/rawhide/f/linux-firmware.spec
16:53:15 <jlebon> gotcha, right
16:53:17 <dustymabe> https://src.fedoraproject.org/rpms/linux-firmware/c/be92a95e169422f1108bce0aee70ed96fc26fb0b?branch=rawhide
16:53:24 <travier> It's mostly IoT & us not shipping recommends by default
16:54:22 <travier> in fact they don"r
16:54:24 <travier> don't
16:54:26 <travier> it's from the comps
16:54:43 <jlebon> ok. the ux here is really awkward. i guess we can have the CLHM just recommend to layer all four back?
16:55:13 <jlebon> and users who know what's up will know which one in specific they actually need
16:55:14 <dustymabe> yeah, we can probably debate the wording later
16:55:19 <travier> https://pagure.io/fedora-comps/pull-request/860
16:55:32 <dustymabe> but we can have the CLHM just point to an issue for more information (i.e. where a lot more context can be added)
16:55:53 <jlebon> yup, can discuss later
16:56:36 <travier> #proposed We'll keep the packages in FCOS until the rebase to F40. We'll add a CLHM module to warn for systems where wifi packages are layered and we'll send a warning mail.
16:56:51 <travier> (I still thinks we're doing too much here but meh)
16:57:21 <dustymabe> yeah, it seems like a lot, but it has to do with basic networking
16:57:41 <travier> In short: we can not "support" layering cases, there are too many combinations
16:57:44 <dustymabe> requiring someone to move a machine in order to debug an issue is a bad experience :(
16:58:17 <dustymabe> travier: I agree, but there is always a but...
16:58:32 <dustymabe> :)
16:58:34 <travier> We should write our stance regarding wifi support somewhere
16:58:36 <travier> +
16:58:54 <jlebon> travier: +1
17:00:04 <travier> https://github.com/coreos/fedora-coreos-docs/issues/589
17:00:17 <travier> can I get some votes on the proposed?
17:00:28 <travier> (sorry this was too long)
17:00:36 <travier> apiaseck dustymabe jlebon marmijo mnguyen ravanelli travier ydesouza
17:01:05 <jlebon> +1 (but personally I'd also be happy with just the warning email)
17:01:16 <apiaseck> I'd rather stay on sidelines here :)
17:01:19 <dustymabe> oh sorry - +1 on the proposed
17:01:26 <dustymabe> didn't realize I hadn't voted
17:01:37 <marmijo> +1
17:02:11 <dustymabe> apiaseck: that's fair - you can always pipe up and just say +0 when you want to stay on the sidelines
17:02:19 <apiaseck> +0
17:02:19 <dustymabe> it shows you are there and you have no opinion
17:02:23 <travier> I guess the CLHM module will happen if someone does it
17:02:41 <mnguyen> +1
17:02:54 <dustymabe> I can do it if no one else wants to. though it's probably a good learning experience if someone else did want to
17:03:46 <travier> #agreed We'll keep the packages in FCOS until the rebase to F40. We'll add a CLHM module to warn for systems where wifi packages are layered and we'll send a warning mail.
17:04:30 <travier> #topic Split Ignition support in an Ignition RPM sub-package (like ignition-edge)
17:04:36 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1582
17:05:10 <travier> Ignition support in Fedora CoreOS is now mature and less likely to drastically change. Let's split all the units, config, services, etc. in an ignition RPM sub-package.
17:05:10 <travier> This would let us more easily include Ignition support in Project Sagano images: gitlab.com/CentOS/cloud/sagano
17:05:10 <travier> This will also enable us to de-duplicate the effort with fedora-iot/ignition-edge.
17:05:10 <travier> To reduce the impact on development speed, we would enable CI in the new repo and add auto-packaging updates with packit for the Fedora package.
17:05:21 <travier> (Copy/pasted from the issue text)
17:06:09 <jlebon> > Another alternative is to teach rpm-ostree to handle overlays during compose and use that instead of having the logic in COSA.
17:06:31 <jlebon> this came up before too. it would actually be a really cleanup of our overlay stuff
17:06:58 <dustymabe> jlebon: but how does that help us share code with other projects? git submodules?
17:08:23 <jlebon> maybe? but yeah, the main thing is it avoids packaging things
17:08:56 <travier> Now that packit CI is going at pushing Fedora updates, I guess that would make it less painful?
17:08:57 <jlebon> sort of another way to work around the "everything must be in RPMs" requirement of osbuild
17:09:12 <travier> It's not just osbuild, it's UKI, etc.
17:09:28 <travier> we're workaround things here :)
17:09:43 <dustymabe> yeah, honestly if we were going to extend the "overlay" model doing work in rpm-ostree would be useful, but I think we'd be shoehorning that model into different workflows too often
17:10:20 <dustymabe> I think we'll have an easier time if we just comply
17:10:23 <dustymabe> :)
17:10:53 <jlebon> yeah, probably. not strongly against or anything, just having the discussion
17:11:00 <dustymabe> +1
17:11:11 <travier> +1
17:11:19 <dustymabe> in projects where you're not necessarily trying to share code and you're not bound to RPMs I think the overlay model is sooo great
17:11:27 <dustymabe> so I wouldn't mind more support in rpm-ostree being baked in
17:11:55 <dustymabe> I just don't think it's the right answer for this purpose
17:12:35 <dustymabe> +1 for a new subpackage.. though I will say this probably won't be straightforward either
17:12:46 <jlebon> travier: to clarify, you're proposing a new upstream repo, right?
17:12:51 <travier> yes
17:13:10 <dustymabe> ahh, not just like a subfolder if existing Ignition repo?
17:13:10 <travier> or we add this to the Ignition repo
17:13:17 <travier> both could work
17:13:32 <jlebon> hmm, i don't think it should be in the Ignition repo
17:13:52 <jlebon> well, it depends what we mean by "Ignition support". that itself will be a challenge
17:13:58 <dustymabe> I don't have a strong opinion
17:14:01 <travier> It's easier to do in the Ignition repo but less "distro" independant
17:14:08 <jlebon> there's a lot of things in there of varying "tightness" to CoreOS
17:15:06 <travier> What's in ignition-edge right now would be a good start I guess
17:15:14 <travier> I've not looked at it deeper yet
17:15:20 <jlebon> given that Ignition is used by not just us, i think it wouldn't look good
17:15:35 <jlebon> so I'd suggest making it a separate repo instead
17:16:34 <jlebon> making it a subpackage on the Fedora/RHEL side OTOH seems fine
17:17:35 <dustymabe> WFM
17:18:42 <travier> #proposed We'll split Ignition support config and services from fedora-coreos-config into another repo and include them in FCOS via an Ignition subpackage, similarly to how ignition-edge does it today, eventually replacing it
17:19:34 <apiaseck> +1
17:19:46 <dustymabe> +1
17:19:51 <mnguyen> +1
17:20:00 <ydesouza> +1
17:20:17 <jlebon> generally +1, but I think before going much farther we need to define "Ignition support". i.e. what subset of FCOS/RHCOS features
17:21:08 <travier> jlebon: agree
17:21:23 <marmijo> +1
17:21:23 <travier> I'll make a PoC before we commit to the change
17:21:35 <jlebon> sounds good
17:22:01 <travier> #agreed We'll split Ignition support config and services from fedora-coreos-config into another repo and include them in FCOS via an Ignition subpackage, similarly to how ignition-edge does it today, eventually replacing it
17:22:11 <travier> Does anyone have something urgent?
17:22:17 <travier> otherwise we go to open floor
17:22:28 <travier> something else*?
17:22:39 <travier> I have another ticket but it's going to be too long
17:23:45 <jlebon> is it urgent? probably better to keep it for next time
17:24:25 <travier> for next time, it's been waiting for a while already ;)
17:24:38 <travier> #topic Open Floor
17:24:45 <dustymabe> #info we have the test week going on this week. if you want to participate check out what test cases haven't been executed to see if you can try them out: https://testdays.fedoraproject.org/events/159
17:25:14 <dustymabe> also jlebon travier can we come to some sort of conclusion on https://github.com/coreos/fedora-coreos-tracker/issues/1375 ?
17:25:43 <dustymabe> i.e. I'm not sure the podman team even knows if just `next` is OK, but maybe we can try that first? I don't know
17:26:25 * dustymabe notes that after we switch other streams to f39 if we keep only those packages in `next` then we'll have to keep `next-devel` running (slight waste of resources IMO)
17:26:49 <jlebon> IIRC from last week, someone said that podman machine users can choose the stream to install, right?
17:27:27 <dustymabe> I think they can, but will they know? might cause more pain/confusion than it is worth.
17:28:38 <travier> let's start with adding it in next?
17:28:49 <jlebon> well, if the podman team is fine with it, i'd say that's good enough for me :)
17:28:56 <dustymabe> ok
17:29:15 <dustymabe> when we do switch testing-devel to f39, i'm going to bring this up again, though
17:29:31 <travier> agree, it's not great
17:29:44 <dustymabe> I don't think running all our processes for a few packages is worth the extra pipeline and CI churn
17:30:13 <jlebon> i mean, that's kinda what `next` is for. trying out experimental things
17:30:20 <travier> but do we really need next-devel? we could keep next seperate but not have devel
17:30:23 <jlebon> what's the point if we really don't want it to differ from `testing`
17:30:53 <dustymabe> jlebon: I think we mostly just need to weigh the costs when we choose to have it differ, that's all
17:31:13 <dustymabe> we can try to minimize the cost though
17:31:22 <dustymabe> so it's not as much of a problem
17:31:24 <jlebon> right, agreed (was typing that :) )
17:31:34 <dustymabe> +1
17:32:21 <jlebon> i'm trying really hard to keep FCOS minimal where we can. it's ballooned a lot over the years, but i don't want to fully throw in the towel either.
17:33:20 <travier> alright, will close soon
17:33:57 * dustymabe notes anyone can feel free to volunteer to work on https://github.com/coreos/fedora-coreos-tracker/issues/1375 now that we have that figured out
17:34:10 <jlebon> i'll try to get benoit's opinion since it seems like they're the ones closest to the issue
17:34:16 <dustymabe> +1
17:37:14 <travier> #endmeeting