16:31:13 <travier> #startmeeting fedora_coreos_meeting
16:31:13 <zodbot> Meeting started Wed Aug 31 16:31:13 2022 UTC.
16:31:13 <zodbot> This meeting is logged and archived in a public location.
16:31:13 <zodbot> The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:31:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:13 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:17 <travier> #topic roll call
16:31:18 <bgilbert> .hi
16:31:19 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:22 <travier> .hello siosm
16:31:23 <spresti[m]> hi
16:31:23 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:31:28 <mnguyen_> .hello mnguyen
16:31:28 <fifofonix> .hi
16:31:28 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:31:31 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:31:34 <spresti[m]> .hi
16:31:34 <zodbot> spresti[m]: Sorry, but user 'spresti [m]' does not exist
16:32:10 <lucab> .hi
16:32:10 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:33:15 <pehunt> hello
16:33:32 <aaradhak> .hi
16:33:33 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:33:38 <pehunt> .hi
16:33:39 <zodbot> pehunt: Sorry, but user 'pehunt' does not exist
16:33:42 <davdunc> .hello2
16:33:43 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:35:04 <travier> 👋
16:35:21 <spresti[m]> 👋
16:35:37 <travier> We have a good bunch of folks, let's start
16:35:56 <travier> #chair bgilbert lucab aaradhak pehunt mnguyen_ davdunc
16:35:56 <zodbot> Current chairs: aaradhak bgilbert davdunc lucab mnguyen_ pehunt travier
16:36:07 <travier> #chair spresti[m] fifofonix
16:36:07 <zodbot> Current chairs: aaradhak bgilbert davdunc fifofonix lucab mnguyen_ pehunt spresti[m] travier
16:36:21 <travier> (hope I did not forgot someone)
16:36:28 <travier> forget*
16:36:33 <travier> #topic Action items from last meeting
16:36:55 <travier> No actions 🎉
16:37:05 <travier> Let's move one
16:37:06 <travier> one*
16:37:08 <travier> on*
16:37:25 <travier> #topic New Package Request: conmon-rs
16:37:31 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1288
16:37:48 <travier> We have guest(s?) today so let's start with this one
16:38:01 <travier> It's about adding a new package to FCOS
16:38:14 <travier> pehunt: can you give us a short overview?
16:38:18 <pehunt> Hey all
16:38:35 <travier> 👋
16:38:44 <pehunt> We're looking to add conmon-rs to fcos. the plan is to have it replace conmon as the oci runtime monitor cri-o and podman use
16:39:25 <pehunt> it's been a long time since we've done this, so we're not really familiar with the whole process. Via the ticket I see we need to request a fedora package first
16:39:48 <travier> yes, all packages that are included in FCOS come from Fedora
16:40:08 <travier> So the first step would probably be to get it into Fedora
16:40:14 <pehunt> makes sense :)  I don't think I have any specific items to discuss, does anyone have any questions for me on it?
16:40:39 <travier> From the issue tracker: how would the migration happen?
16:41:03 <travier> How do users try out the new one?
16:41:17 <travier> Are they 100% compatible?
16:41:57 <pehunt> so the migration would happen from the crio config. Once we have full support in (we're looking at experimental support now), a user would update the crio config to use a runtime of type `pod` instead of `oci` (naming is hard)
16:42:19 <cyberpear> sounds like a win to me! can both be installed in parallel? podman config?
16:42:28 <lucab> I was looking at the current COPR package and I think it currently installs a binary with a different name. I think the interface compatibility is at the CLI level, is that right?
16:42:54 <pehunt> both can be installed in parallel. I think containers.conf will need a new field, or it would be specified via podman CLI
16:43:14 <pehunt> CLI compatiblity is not in the cards. It's a whole new way of interacting with a container monitor
16:43:43 <pehunt> both podman and cri-o require an update to know how to communicate (over a RPC mechanism called cap'n proto)
16:44:00 <pehunt> cri-o's had support since 1.24, podman is approaching support now
16:45:09 <travier> can you run a container with the new conmon and then go back to classic conmon?
16:45:33 <travier> or do you need to re-create it?
16:45:38 <pehunt> need to re-create it
16:45:48 <travier> ouch
16:45:57 <pehunt> conmon{,rs} requires being the direct parent of the container process, so there's no simple migration path
16:46:18 <travier> Let's consider this scenario:
16:46:50 <travier> FCOS with conmon only, I create a container. I update & reboot to FCOS with conmon-rs only. Wht happens to my container?
16:47:04 <pehunt> since you rebooted, the container would restart anyway
16:47:16 <travier> restart yes, but rm & create?
16:47:26 <travier> maybe that's more for podman than cri-o
16:47:48 <pehunt> I don't know exactly how podman handles a restart, but cri-o removes all containers on a restart, as there's state in tmpfs that is needed to track the execution, and that's obviously lost
16:48:49 <pehunt> under the hood, AFAICT, podman containers are recreated in a similar mechanism.
16:49:27 <pehunt> it just looks like they're the same because they're the same name and everything, but the underlying container process is different. I am not sure how the podman community intends on handling it, but I imagine there may need to be a `podman system migrate` call or the like
16:49:59 <travier> OK, so we will have to make sure that we reach out to the podman team
16:50:38 <travier> #action travier Reach out to the podman team or the conmon-rs transition
16:51:07 <travier> This makes it a bit weird to include it in FCOS before we get podman support
16:51:28 <travier> But submitting this for packaging etc. will take some time anyway
16:52:00 <pehunt> I see a situation where cri-o uses conmon-rs experimentally before podman uses it at all. podman would still use conmon (which we don't intend on dropping for a while). is that problematic?
16:52:03 <bgilbert> #undo
16:52:03 <zodbot> Removing item from minutes: ACTION by travier at 16:50:38 : travier Reach out to the podman team or the conmon-rs transition
16:52:07 <bgilbert> #action travier Reach out to the podman team for the conmon-rs transition
16:52:16 <bgilbert> (typo might cause semantic confusion)
16:52:29 <lucab> pehunt: thanks for the additional details, I didn't initially notice that it has its own new RPC mechanism
16:52:48 <travier> bgilbert: 👍
16:54:41 <travier> What do other folks think?
16:54:48 <bgilbert> it sounds as though adding the new package won't create any compat concerns, but removing the old one will?
16:54:51 <bgilbert> assuming we don't change the default
16:55:02 <pehunt> correct, and removing the old is not yet on our radar
16:56:05 <pehunt> there would need to be full feature completeness in cri-o/podman, a while of conmorns being the default (with conmon being a backup to revert back to) and migration paths for both managers before we'd consider removing conmon
16:56:32 <bgilbert> but the plan is definitely to remove conmon in the long run, right?
16:56:50 <pehunt> yeah
16:57:14 <bgilbert> it seems pretty harmless to ship conmon-rs as an aid to testing/migration, provided that we clearly document the situation
16:57:34 <bgilbert> we generally favor long, careful migration paths, which this is
16:58:05 <travier> +1
16:58:43 <bgilbert> without podman support, this would be a leaf package, which is a bit odd
16:58:46 <bgilbert> since we don't ship cri-o
16:58:57 <travier> Maybe we might wait until we have initial podman support to include it and let users give it a try?
16:59:15 <pehunt> that's a good point
16:59:34 <bgilbert> yeah, that's what I'm wondering.  any rough sense of ETA for (at least experimental) podman support?
17:00:26 <pehunt> I don't have an accurate estimate. Some of the work has been done, but podman needs more than cri-o so there are a number of pieces still missing
17:00:39 <pehunt> on the magnitude of months, not weeks
17:01:22 <pehunt> I am fine if we want to wait, and only install conmon-rs optionally alongside the cri-o rpm. We can still go through the process of adding conmon-rs to fedora so we're ready to include it when podman is ready
17:02:10 <lucab> it sounds like OKD may want to start providing this at first, maybe even earlier than FCOS itself?
17:02:32 <pehunt> I can see that being the case
17:02:35 <bgilbert> if there are users who actively want to use their own cri-o with conmon-rs (such as OKD), I'm not 100% opposed to including it sooner.  but without podman support, we'd have no way to know whether the conmon-rs we're shipping even works
17:02:40 <travier> +1 for lucab comment. Maybe this is of interest for early inclusion in OKD
17:03:02 <pehunt> that route works for me too
17:03:30 <travier> And they are working on CentOS Stream CoreOS for OKD too so they will be interested in that too
17:03:53 <travier> alright, should we vote?
17:04:27 <bgilbert> travier: need a proposal :-)
17:04:31 <travier> yep :)
17:05:05 <lucab> IIRC cri-o comes from a module, right?
17:05:28 <pehunt> correct!
17:05:59 <travier> # proposal We will not add conmon-rs to FCOS now but we will as soon as we have basic support in podman to help with the transition
17:06:13 <travier> (open to changes
17:06:14 <travier> )
17:06:19 <lucab> if yes, version-coupling the two in the same module may also help stabilization (if unforeseen RPC/API breakage may still happen)
17:07:03 <bgilbert> travier: that may be stronger than necessary
17:07:23 <bgilbert> we could add conmon-rs as soon as podman supports it, even if there's no migration support
17:07:28 <travier> sure, I'm bad at proposals
17:07:29 <bgilbert> and just document that -rs is a separate world for now
17:07:53 <travier> agree
17:08:48 <lucab> proposal: suggest shipping conmon-rs in the cri-o module for now, and let OKD pick it up directly bypassing FCOS. Re-evaluate FCOS inclusion once stabilized and podman supports that?
17:09:03 <bgilbert> lucab: once what is stabilized?
17:09:07 <walters> OKD and RHCOS?
17:09:37 <travier> We'll probably add it to RHCOS but that's another discussion?
17:09:43 <pehunt> I do intend on having an RHCOS conversation, but i guessed it wasn't in scope here
17:09:44 <lucab> bgilbert: conmon-rs API surface, sorry
17:10:04 <travier> yes, I think the RHCOS discussion can happen after this one
17:11:20 <lucab> I guess the interesting discussion is for Centos9 stream, i.e. either to have it in the main stream or in a dedicated module/sig/copr.
17:11:32 <bgilbert> I'm unclear on the relationship between podman gaining support and the API surface stabilizing
17:11:46 <bgilbert> i.e., whether Luca's proposal is stronger or weaker
17:12:19 <lucab> which is similar to the discussion here about plain Fedora or side-module
17:12:25 <pehunt> podman having full support implies the API surface has stabalized. At that point, any new features would be purely additive, which is handled by the IPC mechanism
17:12:37 <bgilbert> pehunt: +1
17:13:10 <pehunt> I am guessing for C9S we'd include in OKD SIG until it is ready for podman, same as for FCOS
17:13:17 <travier> # proposal We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The transition timeline will be discussed then.
17:13:29 <lucab> bgilbert: that was a minor detail for version-compatibility. I agree it isn't well formulated and we drop that part
17:13:37 <bgilbert> lucab: +1
17:14:20 <bgilbert> travier: for the last sentence, maybe "The timeline for migrating away from legacy conmon will be discussed then." for clarity?
17:14:29 <travier> 👍
17:14:59 <travier> #proposal We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:15:32 <lucab> ack
17:15:34 <bgilbert> +1
17:16:34 <bgilbert> pehunt: does that meet your needs?
17:16:40 <pehunt> +1
17:16:41 <travier> other folks? anybody can vote 🙂
17:17:10 <spresti[m]> +1
17:18:22 <travier> #accepted We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:18:35 <travier> not sure about the meetbot commands anymore
17:18:38 <pehunt> thanks everyone :)
17:18:41 <travier> Thanks pehunt for joining us
17:18:48 <bgilbert> pehunt: thanks for the info and discussion!
17:18:56 <bgilbert> #agreed We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:19:06 <travier> 👍
17:19:34 <walters> (not strictly related but worth bearing in mind that testing out conmon-rs with crio on FCOS should be as easy as writing a Dockerfile that installs both and then booting it, don't even need an RPM of conmon-rs either, could just drop the binary out of a CI build)
17:20:04 <travier> I don't think we have to discuss the other tickets as we already did last week so I think I'll go to open floor
17:20:10 <travier> +1 walters
17:20:14 <bgilbert> one moment travier
17:20:18 <travier> ok
17:20:24 <bgilbert> #undo
17:20:24 <zodbot> Removing item from minutes: AGREED by bgilbert at 17:18:56 : We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:20:25 <bgilbert> #undo
17:20:25 <zodbot> Removing item from minutes: ACCEPTED by travier at 17:18:22 : We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:20:27 <bgilbert> #agreed We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then.
17:20:34 <bgilbert> both cmds exist but are slightly different
17:20:37 <bgilbert> ok, carry on
17:20:40 <travier> ok thanks
17:20:59 <travier> #topic Open Floor
17:21:18 <lucab> #link https://github.com/coreos/fedora-coreos-tracker/issues/1287
17:22:03 <lucab> this was initially a request to change FCOS / systemd defaults, but I think we steered away from that
17:23:17 <lucab> let's see how this progress on the musl side, but I expect this to become a common annoyance in musl-based images in the future
17:24:05 <travier> 👍
17:26:21 <lucab> (that's all from my side)
17:26:36 <travier> probable needs to be reported to musl :/
17:26:41 <travier> probably*
17:29:28 <travier> OK, will close this one if we don't have anything else
17:29:37 <travier> Nice digging indeed lucab for this one
17:30:12 <travier> #endmeeting