16:29:42 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:42 <zodbot> Meeting started Wed Nov 16 16:29:42 2022 UTC.
16:29:42 <zodbot> This meeting is logged and archived in a public location.
16:29:42 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:42 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:48 <dustymabe> #topic roll call
16:29:51 <dustymabe> .hi
16:29:52 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:04 <aaradhak[m]> .hi
16:30:05 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:30:31 <lucab> .hi
16:30:32 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:30:43 <fifofonix> .hi
16:30:44 <aaradhak> .hi
16:30:44 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:30:46 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:30:56 <jlebon> .hello2
16:30:57 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:25 <dustymabe> #chair aaradhak lucab fifofonix jlebon
16:32:25 <zodbot> Current chairs: aaradhak dustymabe fifofonix jlebon lucab
16:33:52 <dustymabe> #topic Action items from last meeting
16:33:59 <dustymabe> * travier will send an email to coreos devel list and cc fedora devel to try to find volunteers to help with moby-engine maintainership in Fedora
16:34:07 <dustymabe> #info travier sent communications about docker maintenance: https://discussion.fedoraproject.org/t/moby-engine-also-known-as-docker-maintenance-in-fedora/44037
16:34:50 <dustymabe> #topic open floor
16:34:55 <dustymabe> we don't have any `meeting` tickets today
16:34:58 <jlebon> travier++
16:35:00 <dustymabe> so here we are at open floor
16:35:04 <ravanelli> .hi
16:35:05 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:35:09 <dustymabe> anyone with any topics they would like to bring up?
16:35:14 <dustymabe> #info The `testing` stream is now on Fedora 37 content: https://discussion.fedoraproject.org/t/fedora-coreos-streams-rebasing-to-fedora-linux-37/44085
16:35:26 <lucab> that was quick!
16:35:30 <dustymabe> lucab: yep
16:36:55 <dustymabe> in the development space of FCOS - `testing-devel` stream and the `fcos-buildroot` container have been moved over to Fedora 37
16:37:00 <Nemric> hi :)
16:37:11 <dustymabe> so you might see some fallout there as a result
16:37:18 <dustymabe> Nemric: 👋
16:38:02 <jlebon> dustymabe: thanks for writing that!
16:38:18 <dustymabe> np
16:38:29 <dustymabe> Nemric: any topics you'd like to discuss today?
16:38:42 <Nemric> yes ...
16:38:55 <walters> .hi
16:38:58 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:39:25 <Nemric> I 'd like to promote FCOS at work, for production env
16:40:04 <Nemric> of course we would start with tests VM but my biggest concern is about support
16:40:43 <fifofonix> we run fcos in production.  i have found the team to be very supportive. :o)
16:40:47 <Nemric> How could I have support with SLA ans so on ?
16:41:33 <dustymabe> Nemric: Fedora CoreOS is community supported. There are no means to pay for support and receive an SLA
16:41:37 <Nemric> I'm on confidence, but my boss want to pay and having someone to blame :D
16:41:54 <lucab> (This weekend I found out I had an old RPi3 sitting around in a bin. I tried to UEFI-boot FCOS on it and I was very surprised to see it properly booted with sshd et al. without any failed units on less than 1 GiB of RAM)
16:42:32 <dustymabe> lucab: that is impressive :) - did you enable the swapOnZram support?
16:42:42 <dustymabe> Nemric: sorry - it's a tough situation to be in
16:42:56 <Nemric> dustymabe: thanks
16:43:23 <fifofonix> so, i have a scenario question.  right now i've had to pin my AI GPU fcos nodes because 6.x is not NVIDIA supported yet.
16:43:27 <dustymabe> Unfortunately there also isn't an equivalent of FCOS in RHEL land either. RHCOS is packaged up as part of OpenShift and isn't sold separately
16:43:41 <dustymabe> but you could request it and people may listen.
16:44:42 <fifofonix> luckily the last 5.x kernels were SSL3 vulnerability patched.  but if there was a new security issue i think i'd be scratching my head as to what to do.
16:45:01 <Nemric> ok, I'll fight for using fcos for less critical prod env ... and perhaps a day ....
16:45:09 <lucab> Nemric: I guess that you could ask around and find some freelancer willing to do support work, but then defining SLAs is funky because there isn't a single brain governing FCOS development
16:45:20 <dustymabe> Nemric: yeah PM me and I'll give you the email of someone to ask
16:45:30 <jlebon> fifofonix: interesting. do you know what the timeline is for getting it supported?
16:45:58 <fifofonix> i don't.  of course NVIDIA only suport fedora 35 officially.
16:46:04 <dustymabe> fifofonix: the joy of out of tree kernel modules :)
16:46:39 <jlebon> i would've thought NVIDIA would be on top of that given the popularity of linux in AI
16:46:43 <travier> .hello siosm
16:46:44 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:46:51 <lucab> I guess it's the same topic as doing pure Fedora or Debian support work. There are plenty of freelancers doing that, with relaxed SLAs though
16:47:23 <fifofonix> there are a lot of 6.x kernel NVIDIA issues registered against desktop drivers.  i am sure they are working hard.
16:47:44 <Nemric> thanks dustymabe but for know I have a lot of work to make somethig like fcos accepted, we do not have any containerized app so the road/way is long
16:48:11 <dustymabe> Nemric: either way I'm glad you do like FCOS (and the model)
16:48:27 <fifofonix> nemric: all i will say is that i have found the team's unoffical support to be way superior to any paying software.  just saying.
16:48:51 <dustymabe> fifofonix: ha - don't tell anyone
16:49:14 <jlebon> fifofonix: no good solution here unfortunately. :(  i guess if it's really bad, you could try rebuilding 5.x with patches and doing an `rpm-ostree override replace`, but that's a non-trivial task to take on
16:49:15 <travier> :)
16:49:30 <fifofonix> so, fyi crowdstrike are inching towards FINALLY providing a FCOS image that can be run as a systemd/privileged container.
16:49:55 <lucab> dustymabe: vanilla FCOS with just an SSH pubkey in Ignition. I didn't try running any workloads, those may need zram-swap indeed. I'm even curious to see if a plain install can successfully pull and apply an upgrade.
16:49:57 <fifofonix> s/FCOS image/container image/ (actually ubi)
16:50:27 <lucab> fifofonix: yay!
16:50:43 <dustymabe> lucab: I know if you try to layer packages you might get an OOM (since processing the package metadata can consume a lot of RAM)
16:50:54 <fifofonix> jlebon: yeah, i figured, that would be tricky and i may end up wishing I was paying for support.
16:50:56 <travier> this ^^
16:51:09 <travier> classic DNF is basically impossible now on those smaller boards
16:51:23 <travier> you need a large swap spave
16:51:25 <travier> space*
16:51:31 <fifofonix> it seems that crowdstrike are also prioritizing flatcar over fcos because of its relative popularity.
16:51:37 <dustymabe> swapOnZram helps a lot there, but yeah
16:52:30 <dustymabe> fifofonix: understood
16:53:00 <dustymabe> anyone with any other topics for open floor? reminder to pleace open/add the meeting label to issues so we can discuss here.
16:53:10 <dustymabe> we should probably revisit our priorities here soon
16:53:19 <lucab> on a different topic, there has been a bit of talk about Hetzner cloud here https://github.com/coreos/fedora-coreos-tracker/issues/1324 but I think we haven't yet reached a common middleground.
16:53:32 <dustymabe> recently we've been buried with release engineering/tooling work to re-architect how we do downstream builds and that has taken a lot of time/effort
16:54:40 <dustymabe> lucab: any thoughts on path forward there?
16:56:31 <lucab> dustymabe: convincing them to let us build an image for them to import, in whatever format they prefer, I guess
16:57:18 <dustymabe> It seems as if they are pretty stuck on the password thing.
16:58:40 <lucab> I think we could find some compromise there
16:58:46 <jlebon> meh, doesn't seem like it's worth blocking on this IMO
16:59:07 <dustymabe> but how do we support it?
16:59:12 <jlebon> if they won't budge, having the password as a fallback seems reasonable if it's not emphasized as they say
16:59:22 <lucab> and if not, at least people would be able to directly import FCOS images on their own
16:59:27 <dustymabe> currently we pull SSH key from the metadata server
16:59:33 <jlebon> we'd need afterburn integration for it i think
16:59:45 <dustymabe> that seems icky
17:00:12 <dustymabe> for ssh key at least what you are pulling from the metadata server is the public data (not private key)
17:00:26 <dustymabe> for password that would be the actual password?
17:00:37 <jlebon> you'd be pulling the hash
17:00:55 <lucab> The salted hash, likely
17:00:57 <dustymabe> I see
17:01:22 <lucab> but yes, I'm also not a fan of that. Also, it's plaintext HTTP.
17:01:40 <dustymabe> do they already provide that as part of their instance metadata or would it require work on their end?
17:02:01 <jlebon> we could have FCOS emit a warning about it
17:02:19 <lucab> but maybe they also have it via a qemu backchannel
17:02:41 <dustymabe> I see `root_password` in https://docs.hetzner.cloud/#server-metadata
17:02:46 <dustymabe> the example doesn't look hashed
17:04:12 <jlebon> dustymabe: i think that's in the response body for the API
17:04:13 <lucab> dustymabe: I think you may be looking at the (authenticated) cloud API
17:04:44 <dustymabe> oh ok - ignore me :)
17:05:54 <lucab> but anyway, I think that's secondary. If they can't bless it without passwords, I'd be already happy to self-import images (like we have in some other clouds).
17:06:23 <dustymabe> yeah, I mean if we can add afterburn support (assuming we can get a hash from the API) then I'd be OK with that
17:06:45 <dustymabe> but it requires us to do work, and prioritize it
17:06:58 <jlebon> it seems like the server metadata doesn't have a separate entry for the password, so possibly they modify the user-data directly to inject it? so then yes they would have to add Ignition support for it
17:07:04 <dustymabe> either way having a clear set of steps to achieve the goal would be useful
17:07:29 <dustymabe> jlebon: in which case it would be easier for them to just require an SSH key to start an FCOS instance
17:07:58 <jlebon> dustymabe: easier yes, but i guess they want to provide a consistent UX
17:08:15 <jlebon> which isn't unreasonable
17:08:46 <dustymabe> regarding the ignition password injection, would they parse the users provided ignition and add their own?
17:09:46 <jlebon> it's defined as the root password, so they'd have to (1) add a root user if one isn't defined already, (2) add an sshd dropin to enable root login (3) add an sshd dropin to enable passwd login
17:09:46 <lucab> or serve a stub with a merging entry to the original user-provided one
17:10:19 <dustymabe> yep
17:10:36 <jlebon> it's definitely not great
17:11:03 <dustymabe> it also means now starting an instance requires them to not "mess things up" in their backend when mucking with the Ignition config
17:11:37 <dustymabe> or I guess they would ONLY do this if you didn't provide an SSH key
17:12:26 <jlebon> short-term, it doesn't seem like it'd be much work for them to notice the config isn't cloud-init and error out if no ssh key is provided since it can't inject passwds
17:12:58 <jlebon> i can add a comment about this
17:13:20 <dustymabe> +1
17:13:24 <dustymabe> any other topics for open floor?
17:16:30 <dustymabe> #endmeeting