16:29:31 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:31 <zodbot> Meeting started Wed Apr 12 16:29:31 2023 UTC.
16:29:31 <zodbot> This meeting is logged and archived in a public location.
16:29:31 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:31 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:35 <dustymabe> #topic roll call
16:29:37 <dustymabe> .hi
16:29:38 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:35 <gursewak> .hi
16:30:36 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:30:54 <marmijo[m]> .hello marmijo
16:30:56 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:32:40 <dustymabe> #chair gursewak marmijo[m]
16:32:40 <zodbot> Current chairs: dustymabe gursewak marmijo[m]
16:33:01 <travier> .hello siosm
16:33:02 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:33:31 <dustymabe> #chair travier
16:33:31 <zodbot> Current chairs: dustymabe gursewak marmijo[m] travier
16:33:33 <jlebon> .hello2
16:33:34 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:37 <dustymabe> #chair jlebon
16:33:37 <zodbot> Current chairs: dustymabe gursewak jlebon marmijo[m] travier
16:33:46 <dustymabe> #chair bgilbert
16:33:46 <zodbot> Current chairs: bgilbert dustymabe gursewak jlebon marmijo[m] travier
16:34:06 <bgilbert> .hi
16:34:07 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:34:23 <dustymabe> will get started soon
16:34:23 <ravanelli> .hi
16:34:24 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:35:15 <dustymabe> #chair ravanelli
16:35:15 <zodbot> Current chairs: bgilbert dustymabe gursewak jlebon marmijo[m] ravanelli travier
16:35:17 <dustymabe> #topic Action items from last meeting
16:35:25 <dustymabe> * jlebon to open a new issue related to the "regular bootloader updates"
16:35:26 <dustymabe> feature
16:35:28 <dustymabe> * dustymabe to open a new issue related to the "regular dbx updates"
16:35:30 <dustymabe> feature
16:35:34 <dustymabe> #action dustymabe to open a new issue related to the "regular dbx updates" feature
16:35:39 <dustymabe> ^^ I still need to do that
16:36:27 <jlebon> #action jlebon to open a new issue related to the "regular bootloader updates"
16:36:30 <jlebon> :)
16:36:51 <dustymabe> :)
16:37:07 <dustymabe> #topic New Package Request: ipcalc
16:37:13 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1460
16:37:58 <dustymabe> I *think* I'm interested to know more about their use case here
16:38:19 <dustymabe> "Often in network setup scripts" implies they are hacking around on their own outside of NetworkManager
16:38:30 <bgilbert> +1
16:39:19 <dustymabe> the reporter appears to be a Red Hat person belonging to OpenShift org
16:39:42 <dustymabe> anyone have any more context on this?
16:40:56 <jlebon> not me
16:41:02 <jlebon> +1 to asking for more context
16:41:45 <jlebon> might just be what they're trying to do requires outside glue code and not natively supported by NM
16:41:50 <spresti[m]> .hello2 spresti
16:41:50 <marmijo[m]> +1 from me for more info, but i also have no context
16:41:50 <zodbot> spresti[m]: Sorry, but user 'spresti [m]' does not exist
16:41:56 <dustymabe> should we have a discussion here anyway - i.e. assume "network setup scripts" outside of NM is somethign we'd want to enable
16:42:02 <dustymabe> #chair spresti[m]
16:42:02 <zodbot> Current chairs: bgilbert dustymabe gursewak jlebon marmijo[m] ravanelli spresti[m] travier
16:42:16 <dustymabe> or should we just ask for more information?
16:43:09 <dustymabe> i'm good with the more information route and kick it to next time
16:43:35 <dustymabe> #topic Boot partition can easily run out of space on upgrade
16:43:40 <jlebon> yeah, let's do that +1
16:43:42 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1247
16:43:56 <dustymabe> ok there's a reason I bring this one up (again)
16:44:02 <dustymabe> new link:
16:44:16 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1464
16:44:36 <dustymabe> #info we are blocked on future updates on aarch64 because of `/boot/` running out of space
16:45:35 <bgilbert> this is getting pretty bad :-(
16:45:39 <dustymabe> there are two options we're talking through in the ticket for a short term solution to our problem (on aarch64)
16:45:52 <dustymabe> bgilbert: yeah
16:46:11 <dustymabe> i think we'll need to work towards changing our /boot size in the future
16:46:18 <jlebon> "384M should be enough for everyone" <- us 4 years ago :)
16:46:27 <dustymabe> jlebon: right
16:46:36 <dustymabe> I think there are a few things we didn't consider then (at least I didn't)
16:46:48 <dustymabe> 1. multi-arch (things are slightly different in various ways)
16:47:05 <dustymabe> 2. the need to store 3 copies of all boot files temporarily while the upgrade is happening
16:47:39 <dustymabe> even now we're still completely fine with 2 copies of all files
16:47:44 <jlebon> yeah definitely. we did what we did with the information we had
16:48:04 <jlebon> and things were moving quite fast back then
16:48:27 <jlebon> i'd add 3. artifacts will likely keep getting larger
16:49:00 <dustymabe> so.. does anyone disagree that the long term solution is likely that we need to increase our boot partition size?
16:49:28 <bgilbert> I'm not sure how we _can_
16:49:33 <dustymabe> (obviously we'll want to coninue supporting existing deployed nodes)
16:49:51 <dustymabe> bgilbert: expand?
16:50:01 <bgilbert> are we going to force everyone to redeploy?  there's no point increasing it for new nodes if old ones are still going to choke on upgrade
16:50:09 <dustymabe> bgilbert: yes
16:50:12 <dustymabe> good point
16:50:17 <dustymabe> so here's my idea there
16:50:25 <bgilbert> unless the plan is to have really aggressive workarounds and then push larger /boot to new installs so the workarounds aren't necessary
16:50:31 <dustymabe> 1. new nodes get new increased /boot/ partition
16:50:42 <dustymabe> 2. existing nodes use the autoprune ostree logic
16:50:54 <dustymabe> https://github.com/ostreedev/ostree/issues/2670#issuecomment-1431793075
16:50:55 <bgilbert> yeah, okay
16:51:42 <dustymabe> at that point the complexity then is isolated to people's existing butane/ignition scripts possibly not working right for partiioning after the transition
16:51:58 <bgilbert> and also reprovisioning clobbering their data partitions
16:52:08 <dustymabe> +1
16:52:17 <dustymabe> we'd need to collect the "problems"
16:52:23 <dustymabe> and think through them
16:52:32 <bgilbert> which is bad enough that we'd probably want coreos-installer code to detect that case
16:52:37 <dustymabe> +1
16:53:06 <bgilbert> scan the target partition table, look for CoreOS, scan the new partition table, check partition sizes, fail if we'd clobber
16:53:43 <jlebon> bgilbert: and add an override switch for forcing it
16:53:48 <bgilbert> yeah
16:53:57 <dustymabe> bgilbert: even with the unfortunate complexity we'd have to push there, do you agree changing the partition table of our generated images is probably the right long term move?
16:54:07 <bgilbert> yeah, I think it's unavoidable
16:54:56 <bgilbert> coreos-installer would need to *start installing* before doing the check, since it needs to get enough of the image to see the GPT
16:55:10 <dustymabe> bgilbert: maybe let's open a dedicated ticket to increasing the paritition size where we can collect the things we'd need to do in order to pull it off safely
16:55:16 <bgilbert> +1
16:55:31 <travier> +1
16:55:57 <bgilbert> dustymabe: want to take an #action?
16:56:22 <jlebon> bgilbert: good point, yuck. i guess that could queue into the 1M block deferring we do
16:56:29 <jlebon> cue*
16:56:40 <dustymabe> #proposed It's becoming obvious that our limited /boot partition sizing is causing us increased issues over time. In the long term we will work towards increasing the /boot/ partition size. We will open a dedicated ticket for that work.
16:56:50 <bgilbert> jlebon: there's already some infrastructure for that because of partition saving, yeah.  just noting it
16:56:56 <jlebon> +1
16:57:23 <bgilbert> dustymabe: to open the ticket, I mean
16:57:32 <bgilbert> +1 to proposed
16:57:34 <jlebon> ack to proposed
16:57:46 <dustymabe> bgilbert: I can open the ticket, unless you'd like to
16:57:53 <dustymabe> #agreed It's becoming obvious that our limited /boot partition sizing is causing us increased issues over time. In the long term we will work towards increasing the /boot/ partition size. We will open a dedicated ticket for that work.
16:57:59 <bgilbert> go for it
16:58:19 <dustymabe> #action dustymabe to open ticket for increasing /boot/ partition size in the future.
16:58:35 <dustymabe> ok now we switch our attention to short term
16:59:00 <dustymabe> there are two options proposed in https://github.com/coreos/fedora-coreos-tracker/issues/1464#issuecomment-1505566902
16:59:02 <jlebon> i'm hoping the ostree auto-pruning stuff should be enough to support updating nodes in the long-term, unless artifacts get really much larger
16:59:47 <dustymabe> jlebon: +1 - i think that's a great long term solution to the "old /boot/ partition size" problem
17:00:11 <dustymabe> as for the current two options:
17:00:19 <dustymabe> A) ship an update with no new kernel (i.e. no new usage in /boot) along with new ostree with autopruning (being developed by @jlebon) turned on (this unblocks ppc64le too)
17:00:27 <dustymabe> B) identify some dtb files that we can remove that we don't care about for FCOS aarch64 usages
17:01:19 <bgilbert> probinson proposed some DTBs to remove
17:01:22 <dustymabe> i know jlebon is close on the autopruning work, but isn't super enthusiastic about rushing that code out
17:01:38 <dustymabe> bgilbert: I see that
17:01:49 <jlebon> just removing qcom alone would be enough
17:02:15 <dustymabe> jlebon: could we theoretically hardlink dtb files if they were the same across two kernel versions (I'm not sure how much those files change
17:02:48 <jlebon> dustymabe: we could. it'd require ostree work (also not sure how much they change)
17:03:33 <dustymabe> so that's at least something else we could investigate
17:03:41 <dustymabe> but maybe not worth it if we have autoprune
17:03:49 <bgilbert> are hardlinks likely to blow up in the future if the files do change?
17:03:57 <bgilbert> i.e. we'd suddenly need the extra space
17:04:07 <jlebon> i had a larger idea somewhere about having an ostree repo live in /boot
17:04:14 <dustymabe> bgilbert: right. i.e. in the worst case all of the dtb files changes
17:04:48 <jlebon> i think if we could remove qcom at least temporarily to remove the ostree auto-prune stuff from the hot path, that'd be great
17:05:05 <jlebon> and then add it back once it's in and we're happy with it
17:05:09 <travier> there are no hardlinks on FAT32 right?
17:05:18 <walters> and probably for us, apple at least
17:05:18 <dustymabe> jlebon: ok let's try that
17:05:20 <jlebon> boot is ext4
17:05:36 <walters> (i'm assuming if you pay a premium for a mac, it's not to run as a server...)
17:05:37 <dustymabe> walters: yeah, but that one is much smaller
17:05:40 <travier> 👍
17:06:53 <dustymabe> ok let's try to remove the qcom dtb files and see if that gets us unblocked for now and then we'll deploy the ostree autoprune in the coming weeks as a more permanent "short-term" solution.
17:07:33 <bgilbert> +1
17:07:47 <jlebon> want to do a proposed?
17:07:48 <bgilbert> will the autoprune trigger automatically?
17:07:56 <bgilbert> i.e. only if needed
17:08:28 <jlebon> that's the plan, yeah
17:08:29 <dustymabe> #proposed we will try to remove the qcom dtb files and see if that gets us unblocked for now and then we'll deploy the ostree autoprune in the coming weeks as a more permanent "short-term" solution.
17:08:40 <bgilbert> +1 to proposed
17:08:53 <jlebon> ack
17:09:07 <dustymabe> #agreed we will try to remove the qcom dtb files and see if that gets us unblocked for now and then we'll deploy the ostree autoprune in the coming weeks as a more permanent "short-term" solution.
17:09:17 <jlebon> (i guess no one is concerned about users who may be deploying on these boards?)
17:09:41 <jlebon> is it worth a coreos-status post?
17:09:45 <bgilbert> do we need a coreos-status post for that purpose? ^
17:09:45 <bgilbert> ...yes
17:09:49 <jlebon> :)
17:10:10 <bgilbert> I guess we're not documenting running on those boards, but people do things
17:10:22 <dustymabe> yeah I'm good with that
17:10:30 <dustymabe> anyone want to write it up?
17:11:12 <bgilbert> I can edit if someone wants to write a draft
17:11:14 <jlebon> i can type stuff up and then let bgilbert chisel out the extraneous :)
17:11:18 <dustymabe> :)
17:11:23 <bgilbert> :-)
17:11:47 <jlebon> dustymabe: maybe before though, let's sanity-check it gets us out of this pickle
17:11:52 <bgilbert> +1
17:11:58 <dustymabe> yeah definitely
17:12:05 <dustymabe> ok next topic:
17:12:20 <dustymabe> #topic systemd-oomd for Fedora CoreOS
17:12:27 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/840
17:12:45 <dustymabe> a user brought this up recently in channel
17:13:18 <dustymabe> wondering if we should re-attempt to get back in line with what other variants are doing here
17:13:57 <jlebon> i think we need a refresher on the state there :)
17:14:05 <dustymabe> same
17:14:28 <dustymabe> i think at the time it was a 'oomd isn't great for kubernetes' discussion
17:15:29 <jlebon> right yeah
17:15:50 <dustymabe> here travier said he wanted to start phasing in systemd-oomd https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-858713659
17:17:23 <dustymabe> sorry i'm just reading through the issue
17:17:38 <dustymabe> looks like we decided to tie systemd-oomd to swapOnZRam too
17:17:44 <travier> From memory, it's fine for non kubernetes
17:18:10 <travier> it's better if you have swap on zram as it won't kill as many things
17:18:20 <travier> less likely to kill things
17:18:30 <jlebon> it sounds like swap is better integrated into kubernetes nowadays
17:18:40 <dustymabe> jlebon: link?
17:18:50 <jlebon> https://github.com/coreos/fedora-coreos-tracker/issues/859#issuecomment-897856382
17:19:47 <jlebon> i'm trying to find whether it's still swap in latest k8s
17:19:48 <dustymabe> jlebon: yeah, i'm just wondering what a more recent update of that is
17:19:52 <jlebon> it's still alpha*
17:19:58 <travier> hum, we still don't have a docs page for kubernetes oerators on FCOS :/
17:20:07 <dustymabe> nope
17:20:29 <dustymabe> :(
17:20:38 <jlebon> still in alpha it seems: https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory
17:21:40 <dustymabe> ideas on moving this forward? Should we consider enabling it in next in a month (after the 38 release) and see what shakes out?
17:22:03 <dustymabe> should we put effort into a kubernetes operators docs page where we can document this stuff?
17:22:10 <bgilbert> I'm still not convinced that this solves a problem
17:22:39 <dustymabe> bgilbert: swap or oomd or BOTH?
17:22:44 <bgilbert> oomd
17:22:45 <jlebon> i think we need to do https://github.com/coreos/fedora-coreos-tracker/issues/880#issuecomment-884565096 first
17:22:55 <bgilbert> but testing it in next could make sense
17:23:10 <jlebon> bgilbert: can you elaborate?
17:23:15 <travier> Made https://github.com/coreos/fedora-coreos-docs/issues/530
17:23:42 <bgilbert> jlebon: mainly just that the default is not to do things
17:24:01 <travier> it kills misbehaving processes/containers before the system slows down to a unusable state
17:24:15 <travier> it's supposed to*
17:25:09 <jlebon> bgilbert: i.e. it should be left opt-in?
17:25:27 <walters> A key difference is the kernel always just kills a process, not a container/cgroup
17:25:33 <walters> which can leave things in a broken state
17:26:03 <bgilbert> I don't have a strong or informed opinion.  I just don't feel 100% comfortable with livelocking a system in process restarts rather than letting things use the available RAM
17:26:34 <bgilbert> will defer to others here
17:26:57 * dustymabe wonders if systemd-oomd and swaponzram were enabled in RHEL9
17:27:09 <jlebon> bgilbert: isn't systemd-oomd just doing what the kernel was going to have to do anyway?
17:27:25 <bgilbert> yes, but necessarily earlier
17:28:06 <dustymabe> bgilbert: i.e. perhaps too much earlier (the process could have recovered?)
17:28:10 <jlebon> ok, IIUC your point is that there's a loss of RAM efficiency there by not allowing processes to use up a bit more?
17:28:17 <bgilbert> I definitely ended up turning off oomd on my desktop when I had a workload that was using a decent % of RAM (not all of it) and it kept getting killed
17:28:42 <dustymabe> yeah maybe the defaults are a bit aggressive
17:28:49 * dustymabe notes we are running out of time
17:28:56 <jlebon> it sounds like the knobs might need to be adjusted
17:29:02 <dustymabe> thoughts on takeaways from this conversation?
17:29:57 <jlebon> bgilbert: but presumably if we get the knobs right, it may lead to a better experience than the status quo (that's the assumption it seems at least, haven't used systemd-oomd much myself either)
17:30:20 <bgilbert> _if_, sure.  but I don't know that we want to be in the oomd tuning business either
17:30:36 <jlebon> gotcha ok
17:30:48 * dustymabe switches to open floor soon
17:31:14 <dustymabe> #topic open floor
17:31:16 <jlebon> dustymabe: my takeaway is: still some investigations to do re. systemd-oomd, and also we probably want that docs page and heads up before changing any defaults
17:31:23 <bgilbert> +1
17:31:24 <jlebon> heads up on the list*
17:32:27 <dustymabe> we did get some feedback in the ticket about how it was working.. a few comments starting here: https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-867813496
17:32:57 <dustymabe> so we need a docs page and announcements
17:33:33 <dustymabe> maybe we can also get bgilbert to play with it some on FCOS to see if there are features that are undesirable that outweigh the positives
17:33:57 <dustymabe> anyone with anything for open floor?
17:34:16 <travier> Workshop accepted for devconf!
17:34:21 <dustymabe> travier: woot woot!
17:34:48 <dustymabe> travier: are you good to run it by yourself (since I probably won't be able to come)?
17:34:54 <travier> sure
17:35:03 <travier> 😿
17:35:17 <dustymabe> for the last one I wrote some automation for bringing up an instance in AWS that all the "students" could log in to to run the lab
17:35:29 <dustymabe> it's dead simple (and uses FCOS/Ignition)
17:35:36 <travier> I'll happily steal your script :)
17:35:38 <dustymabe> I'll have to see if I can find it
17:35:46 <dustymabe> I know it's on my filesystem somewhere
17:36:20 * dustymabe closes out the meeting soon
17:36:55 <dustymabe> #endmeeting