16:30:46 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:46 <zodbot> Meeting started Wed Sep 28 16:30:46 2022 UTC.
16:30:46 <zodbot> This meeting is logged and archived in a public location.
16:30:46 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:46 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:51 <dustymabe> #topic roll call
16:30:54 <dustymabe> .hi
16:30:55 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:51 <travier> .hello siosm
16:31:52 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:08 <jbrooks> .hello jasonbrooks
16:32:10 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:39 <jmarrero> .hi
16:32:41 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:32:45 <dustymabe> #chair travier jbrooks jmarrero
16:32:45 <zodbot> Current chairs: dustymabe jbrooks jmarrero travier
16:33:50 <ravanelli> .hi
16:33:51 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:34:04 <jlebon> .hello2
16:34:05 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:35:19 <dustymabe> #chair ravanelli jlebon
16:35:19 <zodbot> Current chairs: dustymabe jbrooks jlebon jmarrero ravanelli travier
16:35:40 <dustymabe> #topic Action items from last meeting
16:35:48 <dustymabe> #info Looks like there were no action items from last meeting!
16:36:01 <dustymabe> #topic alias quay.io/fedora/fedora-coreos:stable to :latest in quay
16:36:06 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1309
16:37:13 <dustymabe> looks like this is a request for a `:latest` tag in our quay repo for the FCOS OSContainer
16:38:20 <jlebon> it'd be cool if you could mark a tag as default in a repo and runtime tools picked up on that
16:38:34 <dustymabe> yeah
16:38:49 <jlebon> i like how clean it is that we only have a tag per stream, but to be practical having a :latest tag makes sense
16:38:54 <dustymabe> i have some concerns
16:39:31 <dustymabe> I think we have a nice delineation today and forcing people to be thoughtful about which stream they are using makes sense IMO
16:40:34 <dustymabe> This isn't a normal container image - it is an OS that is versioned/released in a unique way and I think forcing them to know what tag to choose also forces them to have some understanding of what they are doing
16:40:42 <jmarrero> I think most people expect to get something when they try to pull an image without a tag, I would expect the project maintainer to give me their default flavor.
16:41:23 <dustymabe> jmarrero: but what context are you operating in?
16:42:33 <dustymabe> `sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable` is nicely explicit and it's clear what "stream" we are following
16:42:55 <dustymabe> what would `sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos` even mean? how do we derive the stream information from that?
16:43:42 <jlebon> it'd be in the image metadata and commit metadata, but yeah point taken that it's nice to have it in the ref
16:44:01 <jlebon> it'd be the only place the stream would show up in a plain `rpm-ostree status`
16:44:23 <aaradhak> .hi
16:44:24 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:44:31 <dustymabe> #chair aaradhak
16:44:31 <zodbot> Current chairs: aaradhak dustymabe jbrooks jlebon jmarrero ravanelli travier
16:44:34 <jmarrero> I think if someone cares about the stream, version they will go find it out. If we are trying to use fcos a container that we can use in a dockerfile or run it to do quick test I think we should behave like all other containers.
16:45:33 <travier> Do we expect the latest -> stable alias to ever change? I don't think it will (unlikely)
16:45:43 <travier> to me it's just a shortcut
16:46:04 <dustymabe> ehh. I kind of feel like we should behave like other remotes here.. i.e. there is no `latest` analogue for the OSTree repo
16:46:26 <Nemric> hi :)
16:46:33 <dustymabe> travier: probably not. but you never know
16:46:34 <travier> 👋
16:46:51 <dustymabe> #chair Nemric
16:46:51 <zodbot> Current chairs: Nemric aaradhak dustymabe jbrooks jlebon jmarrero ravanelli travier
16:48:58 <jlebon> i'm not too concerned about users not typing `:stable`, but do think it should be easy to tell that that's what they got. in the rpm-ostree context, we'd have to enhance `status` to surface it better
16:49:53 <jlebon> if someone asks me for FCOS on a USB stick, i'd give them stable. they don't have to know all about our streams upfront. :)
16:50:14 <jlebon> if other places where we push FCOS had a concept of defaults, it'd make sense to default to stable there too
16:50:29 <jlebon> that might be the case already in clouds that have a "family" concept, not sure
16:50:51 <dustymabe> in GCP the image family is section off by stream
16:51:01 <dustymabe> so there are different image families per stream..
16:51:13 <dustymabe> in all cases today you have to make a stream choice for FCOS up front
16:51:44 <dustymabe> I guess the download page is special
16:51:55 <dustymabe> in that `stable` is shown to you by default
16:51:58 <jlebon> right, but if we had more access points where defaults were a thing, would we make use of them?
16:52:27 <dustymabe> I don't personally think so (bgilbert and lucab usually have strong opinions about stuff like this)
16:52:58 <dustymabe> For FCOS I think it's important people know about the streams and what is available to them. It's part of how our community supports itself
16:53:24 <jlebon> a counterpoint is `coreos-installer download` defaults to stable
16:53:40 <dustymabe> yes, but the resulting artifact knows that it is following the stable stream
16:53:55 <jlebon> that'd be the case here too, right?
16:54:06 <dustymabe> would it?
16:54:14 <jlebon> the stream is in the commit metadata. that's what zincati uses today
16:54:19 <dustymabe> it's following a container in a registry with `:latest`
16:54:53 <Nemric> Agree with that, default should be stable, and aother strems to a bit more advanced users ... so dowload page show stable first and other stream are shown to and it's fine
16:55:01 <jlebon> right, but the artifact does know it's stable
16:55:23 <jlebon> i do agree though the lack of visibility in `status` would be something that'd need to be addressed
16:55:58 <jlebon> we could even have rpm-ostree do a thing where it just auto-dereferences and uses :stable
16:56:23 <dustymabe> "auto-dereferences" how?
16:56:32 <dustymabe> hardcoded and only works for FCOS?
16:57:24 <jlebon> it could look for the canonical tag name in the image metadata and substitute it
16:57:33 <jlebon> not an FCOS-specific thing
16:58:21 <dustymabe> but that assumes the tag exists in the container repo
16:58:32 <jlebon> dustymabe: to be fair, i'm mostly playing devil's advocate here. i do see value in having users being more thoughtful upfront. :)
16:58:37 <dustymabe> it might not always be the case (especially for people's derived container images)
16:58:59 <dustymabe> I guess my thing here is that I see a lot of downsides and not a lot of benefit
16:59:21 <jlebon> i see this being useful to mesh better with existing container tooling
16:59:34 <dustymabe> there are a lot of open questions about rpm-ostree displaying the status, possibly making assumptions, and how zincati will work (which we still haven't figured out for CoreOS Layering) and could complicate things there.
16:59:35 <jlebon> we could also defer and re-evaluate as layering actually gains more traction
17:00:39 <dustymabe> yes - I think that may be a good idea. I'm not totally opposed to doing this eventually, but I'm concerned about some of the implications, especially early on in the lifecycle of us using containers for delivering FCOS
17:01:07 <dustymabe> does anyone want to make a `#proposed` here?
17:01:14 <jlebon> people using it at this stage know what they're getting into so there's less pressure to appeal to the kicking-the-tires crowd
17:01:28 <jlebon> where this might be seen as friction
17:02:13 <dustymabe> note: if there was a mechanism where container tooling would pull `quay.io/fedora/fedora-coreos` and get `quay.io/fedora/fedora-coreos:stable` automagically then I'd be for it
17:02:23 <dustymabe> i.e. almost like a redirect
17:03:25 <walters> .hello2
17:03:26 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
17:03:36 <dustymabe> 👋 walters
17:03:43 <dustymabe> #chair walters
17:03:43 <zodbot> Current chairs: Nemric aaradhak dustymabe jbrooks jlebon jmarrero ravanelli travier walters
17:04:10 <dustymabe> should we try to come to a conclusion on this or let walters catch up and join the discussion?
17:04:27 <walters> only skimming back but one thing I'd say is that if we'd done container-native from the start, I suspect our stable stream would have been *called* `:latest`
17:05:03 <walters> what would be the logical distinction between a redirect and "re-tagging"?
17:05:26 <walters> I imagine at a technical level it could indeed be a HTTP 304 but to the user, that's the same thing
17:05:38 <walters> unless podman/docker printed something when they got a 304 but...
17:06:36 <dustymabe> I'm not sure what exactly "re-tagging" is (might need to clarify), but to me a redirect would tell the tooling "you want to use this tag" and the resulting pulled image would be `quay.io/fedora/fedora-coreos:stable`, not `quay.io/fedora/fedora-coreos` or `quay.io/fedora/fedora-coreos:latest`
17:06:43 <walters> anyways my 2c is I don't care really strongly about doing this right now, it's fine to just have it documented
17:06:45 <dustymabe> I feel like "re-tagging" involves making assumptions
17:07:20 <dustymabe> client side
17:07:29 <walters> oh, I see.  yeah I could imagine something like that being added to the container stack
17:07:50 <dustymabe> 👍
17:09:20 <dustymabe> typing up something now - one moment
17:10:01 <dustymabe> #proposed While this would provide benefit to new users who just want to pull FCOS and see what's inside we could see some potential problems with it and would like to defer implementing something like this until our update story for CoreOS Layering (https://github.com/coreos/fedora-coreos-tracker/issues/1263) is figured out.
17:10:17 <dustymabe> I'll go into more detail in the issue comment
17:10:24 <dustymabe> I'll go into more detail/summary in the issue comment
17:11:16 <dustymabe> aside: it would be nice if container registry redirects worked in general.. would make moving repos around easier (think like moving a git repo and old links still work)
17:11:21 <jlebon> ack
17:11:36 <walters> sure, +1 to that
17:11:59 <jmarrero> +1
17:12:05 <aaradhak> ack
17:12:08 <jlebon> (i don't fully see how this intersects with the update story FWIW, but let's leave that for the issue)
17:12:29 <travier> ack
17:12:32 <dustymabe> #agreed While this would provide benefit to new users who just want to pull FCOS and see what's inside we could see some potential problems with it and would like to defer implementing something like this until our update story for CoreOS Layering (https://github.com/coreos/fedora-coreos-tracker/issues/1263) is figured out.
17:12:51 <travier> I don't see where we would have issues but it's not a priority either now so we can wait
17:13:04 <dustymabe> jlebon: +1 - maybe it doesn't cause any problems, but maybe it does - we just haven't fleshed out that full story yet
17:13:20 <dustymabe> #topic Document /boot requirements and constrains when installing/upgrading kernels
17:13:24 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1247
17:14:11 <dustymabe> ok - new update on this one.. turns out the zstd change doesn't appear to be a silver bullet for the new ppc64le artifacts we are generating
17:14:28 <dustymabe> see https://github.com/coreos/fedora-coreos-tracker/issues/987#issuecomment-1249794907
17:15:02 <dustymabe> I'm wondering if we could/should also add priority to https://github.com/ostreedev/ostree/issues/2670 to facilitate making this less of a problem
17:15:23 <dustymabe> (added a comment to the bottom of that issue earlier today)
17:16:00 <walters> yeah probably not too hard though I would still like to investigate shrinking the initramfs too
17:16:54 <jlebon> aside: do we know why ppc64le has heavier kernel+initrd?
17:17:09 <jlebon> i wonder if we're shipping something in the initrd there that we don't need
17:17:26 <dustymabe> I'm +1 for shrinking the initramfs, but definitely prefer options with minor complexity (i.e. finding generic "decrease size of static binary" wins)
17:17:42 <dustymabe> jlebon: good question - i'd love for someone to investigate
17:18:19 <walters> everyone has the powerpc to do it
17:18:30 <dustymabe> 🤣
17:18:45 <jlebon> heh
17:18:55 <dustymabe> but really.. anyone should be able to do this on their laptop (just files, no specific hardware required)
17:19:13 <dustymabe> I know walters knows that, just making sure we don't confuse anyone
17:19:55 <dustymabe> ok so immediate steps:
17:20:14 <dustymabe> 1. investigate why ppc64le initramfs is larger than say x86_64 or aarch64
17:20:32 <dustymabe> 2. maybe someone can start working on https://github.com/ostreedev/ostree/issues/2670 ?
17:20:47 <dustymabe> 3. we can look at options for reducing the size of the initramfs
17:21:08 <dustymabe> "generically reducing the size"
17:21:34 <jlebon> dustymabe: in the current state, can that system you pasted from even update? given that the zstd size seems greater than available
17:21:50 <dustymabe> jlebon: :)
17:22:24 <travier> We still have the option to make making /boot larger on first boot easier
17:22:30 <dustymabe> that was after the last update so I have tried with that exact configuration.. but what I will tell you is that for this system I've been deleting the rollback deployment before updating it
17:22:56 <dustymabe> which is why I knew this was a problem to begin with
17:23:16 <dustymabe> and blocked sending ppc64le to production streams until it's fixed
17:23:29 <dustymabe> this is our ppc64le builder (dogfooding is good)
17:23:43 <dustymabe> *haven't tried with that exact configuration"
17:23:44 <jlebon> ouch, ok right. so this is blocking ppc64le then
17:24:26 <dustymabe> jlebon: I think after the next update (i.e. the rollback and booted deployment have ZSTD compressed initrd) we might be able to update, but just barely
17:24:37 <dustymabe> it's too close for comfort
17:25:07 <jlebon> yeah, was looking at that, but not even sure it could. we'd get 9M, so 110M free, but the kernel+initrd is 112M
17:25:40 <dustymabe> right.. i was thinking we'd save more like 20M rather than ~10M with the zstd compression, but that doesn't seem to be the case
17:25:54 <dustymabe> travier: indeed
17:26:11 <jlebon> ok, let's signal that in the tracker issue
17:26:30 <dustymabe> jlebon: in https://github.com/coreos/fedora-coreos-tracker/issues/1247 ?
17:26:55 <jlebon> dustymabe: yeah. probably should retitle it at this point
17:27:11 <dustymabe> +1 - suggestions for new title?
17:27:19 <jlebon> i don't think we were aware this was effectively blocking FCOS ppc64le
17:27:52 <dustymabe> yeah, it wasn't really at first, but just happened to be working on ppc64le at the same time and stumbled on it
17:28:51 <jlebon> maybe something like "Boot partition can easily run out of space" ?
17:29:00 <dustymabe> once we figure out a mechanism to make the initramfs smaller we should make a test to verify the generated initramfs for each arch is under a certain threshold size, so we get fair warning of future issues
17:29:31 <dustymabe> jlebon: what about: "Boot partition can easily run out of space on upgrade"?
17:29:51 <jlebon> SGTM
17:30:07 <jlebon> still hoping long-term we can bump the partition size
17:30:31 <dustymabe> i'll try to put a summary in the ticket of this discussion and link to it
17:30:45 <dustymabe> #topic tracker: Fedora 37 changes considerations
17:30:50 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1222
17:30:55 <dustymabe> ok I updated the list this morning
17:30:56 <travier> my thoughts are: right now, only ppc64le is impacted. we need to make improvements for all arches but we can make a short term "increase /boot" workaround for ppc64le until we have better options
17:31:18 <travier> and it's not like we have a large ppc installation base
17:31:48 <dustymabe> there were a few changes that got removed from F37
17:32:00 <ravanelli> I need to check, but I recall a bug in ppc64le where it took a lot of time to boot rhcos, with some loop and a bunch of logs it could be related. + it uses petiboot instead of grub for boot things
17:32:01 <dustymabe> and then one that was added (late I guess?)
17:32:22 <dustymabe> subtopic 228. Minizip Renaming
17:32:45 <dustymabe> #info we don't ship minizip in FCOS, so this should be a noop for us
17:32:51 <dustymabe> :)
17:33:03 <jlebon> nice :)
17:33:09 <dustymabe> anything else Changes related, that we should discuss?
17:33:41 <travier> I don't think so
17:33:44 <dustymabe> #topic open floor
17:33:44 <travier> (we're over time)
17:33:49 <dustymabe> sorry we are over time
17:34:26 <walters> anyone know if the NM team has "daemonize in initramfs" on their radar?  re https://issues.redhat.com/browse/OCPBUGS-1327
17:34:34 <dustymabe> #info there is a FCOS 37 test day happening tomorrow!
17:34:40 <dustymabe> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/message/4J5XDZN6L2UL5U4W5SN7RM5J7U65YAQK/
17:35:10 <dustymabe> walters: NM already runs as a daemon in the initramfs in Fedora and RHEL9 IIUC
17:35:49 <walters> oh hmm
17:35:54 <dustymabe> `nm-initrd.service`
17:36:05 <dustymabe> Nemric: did you have something for open floor?
17:37:09 <dustymabe> who all can come and help us with the test day tomorrow?
17:37:13 <travier> dustymabe: nice, would have to look at that on RHEL 9
17:37:21 <dustymabe> thank you jbrooks for helping to organize
17:37:24 <dustymabe> jbrooks++
17:37:57 <dustymabe> we're going to have a video/screen share session tomorrow at 9:30 AM - 11 AM EDT (1:30 PM - 3:00 PM UTC)
17:38:04 <jbrooks> 👍
17:38:43 <jlebon> should be able to join
17:39:00 <dustymabe> jlebon++
17:39:00 <zodbot> dustymabe: Karma for jlebon changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:39:07 <dustymabe> come one come all :)
17:39:48 <dustymabe> I'll force mnguyen_ to join too (via threats of violence) :)
17:40:00 * dustymabe closes the meeting in 30s if inactivity ensues
17:40:02 <mnguyen_> i'll come
17:40:39 <dustymabe> #endmeeting