16:29:39 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:39 <zodbot> Meeting started Wed Nov 23 16:29:39 2022 UTC.
16:29:39 <zodbot> This meeting is logged and archived in a public location.
16:29:39 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:43 <travier> .hello siosm
16:29:44 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:29:45 <dustymabe> #topic roll call
16:29:51 <dustymabe> .hi
16:29:52 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:14 <fifofonix> .hi
16:30:15 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:30:18 <c4rt0> .hi
16:30:18 <zodbot> c4rt0: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:30:23 <walters> .hi
16:30:24 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:30:41 <dustymabe> #chair travier fifofonix c4rt0 walters
16:30:41 <zodbot> Current chairs: c4rt0 dustymabe fifofonix travier walters
16:30:46 <mnguyen> .hello mnguyen
16:30:47 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:30:56 <dustymabe> #chair ravanelli mnguyen
16:30:56 <zodbot> Current chairs: c4rt0 dustymabe fifofonix mnguyen ravanelli travier walters
16:31:02 <lorbus> .hi
16:31:03 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:31:20 <dustymabe> #chair lorbus
16:31:20 <zodbot> Current chairs: c4rt0 dustymabe fifofonix lorbus mnguyen ravanelli travier walters
16:31:51 <jlebon> .hello2
16:31:52 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:03 <anthr76[m]> .hello
16:32:03 <zodbot> anthr76[m]: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:32:14 <anthr76[m]> .hello anthr76
16:32:15 <zodbot> anthr76[m]: anthr76 'Anthony Rabbito' <hello@anthonyrabbito.com>
16:32:18 <dustymabe> #chair jlebon anthr76[m]
16:32:18 <zodbot> Current chairs: anthr76[m] c4rt0 dustymabe fifofonix jlebon lorbus mnguyen ravanelli travier walters
16:32:21 <dustymabe> welcome all
16:32:34 <dustymabe> #topic Action items from last meeting
16:32:47 <dustymabe> Looks like there were no action items from last meeting, so we can move on
16:32:55 <dustymabe> #topic FOSDEM 2023: Image Based Linux dev room
16:32:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1348
16:33:05 * dustymabe passes floor to travier
16:33:11 <travier> Heya
16:33:19 <c4rt0> Hi all
16:33:37 <dustymabe> 👋
16:33:41 <travier> The UAPI group / image based summit crowd is organizing a dev room at FOSDEM
16:34:09 <travier> Feel free to submit a talk about FCOS (or anything else related to the topic) there if you intend to be there
16:34:13 <travier> that's it :)
16:34:26 <travier> (I likely won't be at FOSDEM this year)
16:35:11 <dustymabe> yeah, since devconf moved to the summer it makes going to FOSDEM for non-Europe based harder
16:35:47 <ravanelli> .hi
16:35:49 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:35:54 <dustymabe> thanks for sharing travier
16:36:04 <travier> 👍
16:36:07 <dustymabe> #chair ravanelli
16:36:07 <zodbot> Current chairs: anthr76[m] c4rt0 dustymabe fifofonix jlebon lorbus mnguyen ravanelli travier walters
16:36:24 <dustymabe> #topic amd-gpu-firmware no longer being installed between stable to testing
16:36:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1345
16:36:58 <dustymabe> For this one the linux-firmware package broke out the GPU firmware it was previously shipping in the main package into subpackages
16:37:17 <dustymabe> in f36 the subpackages were requires, in f37+ they are recommends
16:37:25 <dustymabe> so when we moved to f37 we dropped them out
16:37:47 <dustymabe> the question is: should we add them back?
16:37:57 <dustymabe> I think the answer is probably yes
16:38:02 <jlebon> how many new subpackages fall in this category?
16:38:14 <dustymabe> but I think it's worth us discussing how much we think they are used
16:38:48 <dustymabe> jlebon: amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware
16:39:01 <travier> $ rpm -qi amd-gpu-firmware-20221109-144.fc37.noarch | grep Size
16:39:01 <travier> Size        : 22564513
16:39:47 <dustymabe> is that ~ 22MiB ?
16:40:05 <travier> $ rpm -qi intel-gpu-firmware-20221109-144.fc37.noarch| grep Size
16:40:05 <travier> Size        : 8329555
16:40:05 <travier> $ rpm -qi nvidia-gpu-firmware-20221109-144.fc37.noarch| grep Size
16:40:05 <travier> Size        : 1262337
16:40:26 <jlebon> dustymabe: looks like it
16:40:35 <travier> 22 + 8 +1
16:41:07 <dustymabe> definitely not insignificant
16:41:52 <travier> I think we can probably skip intel & nvidia
16:42:05 <travier> it's unlikely someone will use them right now
16:42:25 <jlebon> travier: can you expand on that?
16:42:34 <anthr76[m]> travier: What is your reasoning?
16:42:52 <dustymabe> ehh, I think we probably shouldn't split hairs - probably should do all or nothing
16:43:07 <jlebon> yeah, agreed
16:44:17 <jlebon> this is technically a regression, so i'm inclined to add them back before it hits stable even if we decide later on to remove them with some window
16:44:28 <travier> this is for GPU/OpenCL workloads
16:44:39 <travier> NVIDIA will require the binary drivers
16:44:57 <dustymabe> jlebon: yeah, that's a good point
16:45:00 <travier> I don't think there is Intel cards for that although we have the ARC ones now
16:45:24 <travier> so I really think we don't need the NVIDIA ones
16:45:35 <travier> intel maybe, amd yes
16:46:06 <jlebon> travier: couldn't some users be pulling in the drivers themselves though?
16:46:20 <travier> what do you mean?
16:46:24 <anthr76[m]> FWIW I know quite a few folks using iGPUs with Intel way more then AMD in fact. https://github.com/intel/intel-device-plugins-for-kubernetes
16:46:55 <anthr76[m]> The real question (to me) is do all GPUs require direct firmware loading? If not which ones don't
16:47:21 <jlebon> travier: are the nvidia drivers easily available in a way that we could expect some users to layer them?
16:47:57 <travier> jlebon: you get them from RPM Fusion and usually you need to freeze the kernel as it moves too fast in Fedora for NVIDIA
16:48:19 <anthr76[m]> Also, IIRC these really aren't drivers more or less the layer before the driver. mesa (a driver for example) can't start if the firmware isn't initialized.
16:48:25 <travier> I really don't see users running NVIDIA workloads on nouveau
16:48:34 <travier> that would be a complete waste of maney
16:48:36 <travier> money*
16:49:20 <travier> (but if you want to add all firmware I won't mind, just trying to reduce the size of what we add)
16:49:37 <jlebon> travier: ack ok, so it's possible they do pull it in, but unlikely
16:49:52 <dustymabe> yeah even so i think nvidia is working on making the open source side of things better and I'd prefer not to track progress there. I think I'd rather just include it especially since it's the smallest of the 3
16:49:52 <jlebon> yeah, the 1M cost doesn't seem worth being inconsistent
16:50:02 <travier> the binary NVIDIA drivers do not use those firmwares files
16:50:11 <travier> they use their own
16:50:18 * dustymabe wishes all of the drivers were just open source
16:50:27 <travier> soon!
16:50:29 <travier> :)
16:50:38 <anthr76[m]> ™️
16:50:49 <fifofonix> fifofonix: actually experimenting with nvidia's open kernel for the kernel interface piece presently. :o)
16:50:53 <dustymabe> so OK - maybe we should open a new issue to really discuss what we think we should do in the future here
16:51:23 <dustymabe> but for now maybe we should do what jlebon suggests and get them back in before we ship the next `stable`
16:51:31 <travier> +1
16:52:13 <jlebon> +1
16:52:28 <travier> do we have numbers for image size between f36 & f37?
16:52:47 <anthr76[m]> +1
16:53:05 <dustymabe> #proposed we will add the amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware packages back to FCOS to restore the previous functionality we had and open a new ticket to discuss the merits of these packages and if we ever want to remove them in the future.
16:53:26 <dustymabe> travier: we could compare the meta.json size values
16:54:01 <jlebon> 1.55G vs 1.57G for the uncompressed qemu qcow2 it looks like
16:54:25 <jlebon> ack to proposal
16:54:33 <travier> ack
16:55:49 <dustymabe> any other inputs?
16:55:55 <dustymabe> vote
16:56:52 * dustymabe will +1
16:57:03 <jlebon> anthr76[m]: nice work noticing this before it gets to stable!
16:57:09 <dustymabe> #agreed we will add the amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware packages back to FCOS to restore the previous functionality we had and open a new ticket to discuss the merits of these packages and if we ever want to remove them in the future.
16:57:15 <dustymabe> yes thank you anthr76[m]
16:57:24 <anthr76[m]> Happy to help and be here :)
16:58:01 <dustymabe> we generally do a thorough analysis of added/removed packages (see https://github.com/coreos/fedora-coreos-tracker/issues/1221) but I just didn't have time to do it this cycle
16:58:27 <dustymabe> #topic open floor
16:58:37 <travier> this one is tricky as it's not removed, it's "not added" :)
16:59:04 <dustymabe> travier: right, but rpm-ostree db diff would report on it anyway
16:59:04 <travier> as it's a new sub package
16:59:11 <dustymabe> so I would have caught it
16:59:17 <travier> would it?
16:59:21 <travier> I don't think so
16:59:26 <dustymabe> absolutely it would
16:59:26 <jlebon> travier: the subpackages are in f36
16:59:39 <dustymabe> this ^^
16:59:40 <travier> oh, my bad then
16:59:47 <travier> 👍
16:59:50 <dustymabe> travier: sorry, you would have been right otherwise
16:59:57 <travier> 👍
17:00:36 <travier> https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization
17:00:37 <travier> ?
17:00:46 <dustymabe> note that soon we'll try to start evaluating the change proposals for f38
17:01:11 <dustymabe> travier: interesting.. maybe we overlooked that
17:01:32 <travier> https://koji.fedoraproject.org/koji/buildinfo?buildID=2089044 > I can see them here indeed
17:01:39 <dustymabe> weird. I don't see it in https://github.com/coreos/fedora-coreos-tracker/issues/1222
17:01:56 <anthr76[m]> I saw the breakout on my desktop, but wasn't aware of the change proposal. So I didn't think much of it on my FCOS nodes until GPUs stop initializing :)
17:02:28 <dustymabe> hmm. it's not in the list of 37 changes: https://fedoraproject.org/wiki/Releases/37/ChangeSet#Fedora_Linux_37_Accepted_System-Wide_Changes
17:02:29 <travier> https://src.fedoraproject.org/rpms/linux-firmware/blob/rawhide/f/linux-firmware.spec#_23
17:02:42 <travier> It's a F37 change :)
17:03:12 <dustymabe> travier: isn't there a difference in a proposed change and an accepted change ?
17:03:29 <travier> it should be indeed
17:03:32 <jlebon> looks like it was withdrawn based on bcotton's edit here: https://fedoraproject.org/w/index.php?title=Changes/Linux_Firmware_Minimization&diff=649159&oldid=649137
17:03:32 <travier> it's really weird
17:04:05 <travier> in the spec the change is there and it impacts us as we don't ship recommends
17:04:17 <travier> so they did not do the full change, just a part of it
17:05:01 <travier> so it's indeed both not a F37 change and a F37 change :D
17:05:02 <dustymabe> yeah the change proposal talks about using a `Supplements metadata` to automatically install appropriate firmware based on the hardware of the machine
17:05:09 <travier> well, I'll see myself out :)
17:05:34 <dustymabe> i imagine that the all of the change wasn't done, but some pieces they thought were harmless were
17:05:44 <travier> 👍
17:05:59 <dustymabe> i.e. changing to recommends is pretty harmless for most users, but we're obviously doing something that most users aren't
17:06:28 <dustymabe> anyway :) - we're on top of it now
17:06:44 <dustymabe> and our "stream" model is working great at giving us clues about what's to come
17:07:05 <travier> +1
17:07:25 <dustymabe> for example: I would bet that not a lot of people use this firmware given it took til this late in the cycle
17:07:26 <jlebon> dustymabe: weird we didn't notice it when we moved testing-devel to f37
17:07:33 <dustymabe> though I guess that could just mean not a lot of people try `next`
17:07:56 <jlebon> it is in the git diff indeed of https://github.com/coreos/fedora-coreos-config/commit/08ace84d0104661970047642dc49c91bc764943e
17:08:10 <dustymabe> we didn't move testing devel over to f37 until after `testing` had shipped
17:08:55 <dustymabe> https://github.com/coreos/fedora-coreos-config/pull/2083#issuecomment-1312052842
17:09:20 <dustymabe> summary: we'll do better next time :)
17:09:42 <dustymabe> hopefully we aren't migrating our internal build systems to a whole new model next release too
17:10:29 <dustymabe> anything else for open floor?
17:10:37 <jlebon> dustymabe: right, not arguing that we should've caught this for testing or next. it probably came in through bump-lockfiles which we don't inspect really. for testing-devel, we could've had CI that detected this in that PR.
17:11:00 <jlebon> though really even for bump-lockfile, one thing I've wanted to do was defer to PR the result if the pkgset changed
17:11:20 <jlebon> i think there's an issue somewhere for that
17:11:29 <dustymabe> yeah that would be nice
17:11:39 <anthr76[m]> I have something about native containers..
17:11:41 <dustymabe> but I think where we really want to catch it is in rawhide
17:12:00 <jlebon> dustymabe: +1 indeed
17:12:13 <dustymabe> but I think that would imply using lockfiles for rawhide too, which we could do
17:12:16 <dustymabe> anthr76[m]: go for it!
17:12:28 <jlebon> we could compare against the previous rawhide build
17:13:48 <anthr76[m]> I just wanted a sanity check here before I open an issue (I suppose against ignition?)... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/1881daf39cfb1145c7a770cd90ae27e12810fffc>)
17:14:14 * dustymabe will copy the rest of that here so the meeting log will have it
17:14:21 <dustymabe> I'm building a derivation like so: https://github.com/anthr76/infra/blob/de2d532938eb168be6672ef035d58a9ccec611ab/armature/prod/fcos-layers/k8s-node/Dockerfile#L16-L19
17:14:26 <dustymabe> Basically applying this big butane file. https://github.com/anthr76/infra/blob/main/armature/prod/fcos-layers/k8s-node/config.bu.yaml
17:14:31 <dustymabe> Though, when rebasing units that are enabled aren't actually getting enabled. For example: https://github.com/anthr76/infra/blob/de2d532938eb168be6672ef035d58a9ccec611ab/armature/prod/fcos-layers/k8s-node/config.bu.yaml#L159
17:14:35 <anthr76[m]> Sorry :) how does that show on IRC?
17:14:35 <dustymabe> I assume the symlink is getting lost somewhere..
17:14:50 <dustymabe> anthr76[m]: matrix posts a link to the full message
17:14:57 <anthr76[m]> Is it more preferred to attend meetings on IRC?
17:15:08 <dustymabe> not really, i'm happy either way
17:15:23 <jlebon> anthr76[m]: yeah, i would file an issue
17:15:23 <dustymabe> i just wanted the full message to be in the meeting log
17:15:46 <jlebon> might be some interaction between layering and systemd's firstboot preset logic
17:16:20 <jlebon> or an issue with ignition-liveapply
17:17:43 <anthr76[m]> ack'd will do. It's simple to reproduce in a vm, and you can probably even reproduce it in a container with enough fiddling
17:18:34 <dustymabe> any other topics for open floor?
17:18:46 <jlebon> short-term hack, you can `systemctl enable` the units directly in your Dontainerfile
17:18:51 <jlebon> whoops
17:18:54 <jlebon> Dockerfile*
17:19:12 <jlebon> had started to type Containerfile and then saw it was a Dockerfile
17:19:51 <anthr76[m]> I do believe I tried that at some point and it was still no good. I like to try to actually make the symlink to multi-user.target
17:19:55 <dustymabe> and thus Dontainerfile was born 🌟
17:20:18 <anthr76[m]> I'm all for de-dockering everything but it's really hard :P
17:20:40 <dustymabe> any other topics here?
17:20:48 <anthr76[m]> I gave up on Containerfile when I had to hack renovatebot around it all the time :/ and instructing LSPs and what not
17:20:54 <anthr76[m]> Good on my side
17:21:06 * dustymabe closes out the meeting in 60s
17:21:22 <fifofonix> thanks
17:21:23 <jlebon> anthr76[m]: interesting. add that bit to the issue too. `systemctl enable` is used in some of the the coreos-layering-examples
17:22:04 <anthr76[m]> Sure. I will double check before hand and post it up in about a hour
17:22:11 <dustymabe> #endmeeting