16:30:22 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:22 <zodbot> Meeting started Wed Feb 16 16:30:22 2022 UTC.
16:30:22 <zodbot> This meeting is logged and archived in a public location.
16:30:22 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:22 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:22 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:27 <dustymabe> #topic roll call
16:30:30 <dustymabe> .hi
16:30:33 <travier> .hello siosm
16:30:33 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:36 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:31:03 <jlebon> .hello2
16:31:04 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:27 <ravanelli> .ji
16:31:30 <ravanelli> .hi
16:31:30 <jbrooks> .hello jasonbrooks
16:31:31 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:31:34 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:37 <miabbott> .hello miabbott
16:32:38 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:01 <jaimelm> .hello2
16:33:02 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:33:11 <dustymabe> #chair travier jlebon ravanelli jbrooks miabbott jaimelm
16:33:11 <zodbot> Current chairs: dustymabe jaimelm jbrooks jlebon miabbott ravanelli travier
16:34:25 <dustymabe> #topic Action items from last meeting
16:34:39 * cverna is kind of around o/
16:34:40 <dustymabe> we had a few actions from last meeting
16:34:45 <dustymabe> #chair cverna
16:34:45 <zodbot> Current chairs: cverna dustymabe jaimelm jbrooks jlebon miabbott ravanelli travier
16:34:55 <dustymabe> cverna: if you're around you might as well sit in a chiar
16:35:00 <dustymabe> chair rather
16:35:05 <dustymabe> * dustymabe: we think we can pick up DNSoverTLS changes passively but dustymabe will open a ticket to record the discussion here and provide a space for any issues that come up to be discussed.
16:35:07 <dustymabe> * dustymabe to open an issue for investigation into missing packages preventing auto-updates from working
16:35:09 <dustymabe> * jlebon to open issue to investigate "102 Introduce module Obsoletes and EOL"
16:35:11 <dustymabe> * miabbott to open a tracking ticket for early testing of golang 1.18 when it's available
16:35:13 <dustymabe> I had two actions
16:35:17 <dustymabe> #info dustymabe opened ticket for DNSoverTLS tracking: https://github.com/coreos/fedora-coreos-tracker/issues/1098
16:35:24 <dustymabe> #info dustymabe opened ticket for investigation of missing packages affecting auto updates: https://github.com/coreos/fedora-coreos-tracker/issues/1099
16:35:35 <jlebon> ouff, can we re-action mine?
16:35:39 <miabbott> #info miabbott opened ticket for Golang 1.18 testing https://github.com/coreos/fedora-coreos-tracker/issues/1096
16:35:57 <aaradhak[m]> .hi
16:35:58 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:36:16 * cverna sits down and relaxes his legs
16:36:41 <lucab> .hi
16:36:42 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:36:57 <dustymabe> #action jlebon to open issue to investigate "102 Introduce module Obsoletes and EOL"
16:37:01 <mnguyen_> .hello mnguyen
16:37:02 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:37:05 <dustymabe> thanks miabbott
16:37:08 <cverna> #info cverna is trying to get started a "This month in Fedora CoreOS newsletter" --> https://hackmd.io/fITa6rtNTy6Ub1WO4nkdnw
16:37:12 <dustymabe> #chair aaradhak[m]
16:37:12 <zodbot> Current chairs: aaradhak[m] cverna dustymabe jaimelm jbrooks jlebon miabbott ravanelli travier
16:37:16 <dustymabe> #chair lucab
16:37:16 <zodbot> Current chairs: aaradhak[m] cverna dustymabe jaimelm jbrooks jlebon lucab miabbott ravanelli travier
16:37:18 <dustymabe> #chair mnguyen_
16:37:18 <zodbot> Current chairs: aaradhak[m] cverna dustymabe jaimelm jbrooks jlebon lucab miabbott mnguyen_ ravanelli travier
16:37:32 <dustymabe> cverna++ - thank you so much for that
16:37:32 <zodbot> dustymabe: Karma for cverna changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:38:16 <miabbott> cverna++
16:38:16 <zodbot> miabbott: Karma for cverna changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:38:18 * cverna eats his cookies while seating :)
16:38:48 <dustymabe> ok lets get into meeting tickets
16:38:59 <dustymabe> #topic Add factory reset capability
16:39:08 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/399
16:40:14 <dustymabe> jlebon: want to intro this one
16:40:27 <jlebon> sure thing
16:40:35 <jlebon> we discussed this a while ago, but a lot has changed since
16:41:03 <jlebon> notably, we did split out the rootfs in a separate artifact
16:41:24 <jlebon> anyway, i've been working on a POC which uses kexec to re-install the system: https://github.com/coreos/coreos-installer/pull/712
16:42:05 <jlebon> there's some interest, but I think we need to clarify what the constraints of a re-installation feature are
16:42:18 <jlebon> before we can go forward with that POC or something else
16:42:28 <jlebon> a few things for example:
16:42:37 <jlebon> - do we want to support fully offline re-installs?
16:42:55 <jlebon> - do we want to broaden the scope to "installing CoreOS on the system we're running on" ?
16:43:29 <jlebon> - are we comfortable with kexec as a re-installation mechanism?
16:43:56 <dustymabe> A) do we want to support fully offline re-installs?
16:44:02 <dustymabe> B) do we want to broaden the scope to "installing CoreOS on the system we're running on" ?
16:44:07 <dustymabe> C) are we comfortable with kexec as a re-installation mechanism?
16:44:18 <jlebon> thanks
16:44:43 <jlebon> not sure if we should discuss each item here or take it back to the ticket for now
16:44:51 <travier> Could you expand B?
16:44:52 <dustymabe> for A) - is that pick up pieces from the existing FS and install the same version?
16:45:08 <travier> I'm ok with C
16:45:46 <jlebon> ping -- is this going through?
16:45:58 <dustymabe> yep
16:46:01 <dustymabe> pong
16:46:05 <jlebon> k sorry, network blip
16:46:23 <jlebon> re. A, yes exactly. though we'd need to get creative for the rootfs
16:46:23 <travier> Could you expand B?jlebon
16:46:35 <travier> jlebon: Could you expand B?*
16:46:48 <lucab> on C, I know kernel_lockdown(7) does impose some limits on kexec (which means on SecureBoot)
16:46:55 <dustymabe> jlebon could we consider A a "nice to have/extra credit functionality"?
16:47:15 <jlebon> re. B, the kexec approach is generically useful for the case where you're on some other OS, and you want to re-install on the disk on which the current OS is installed
16:47:35 <jlebon> so it applies to CoreOS of course, but any OS where kexec is functional
16:47:39 <travier> ok
16:47:42 <dustymabe> jlebon: ehh
16:47:48 <travier> that's a broad scope
16:47:58 <dustymabe> yeah I mean people can try it
16:48:04 <travier> I would consider that a future enhancement
16:48:08 <dustymabe> but I would say we shouldn't tell people to
16:48:32 <jlebon> this has definitely already been requested
16:49:14 <dustymabe> ok so let's see if we can come to some conclusions
16:49:28 <travier> A) would be nice. We could do an integrity check for the ostree content from the rootfs and then copy to RAM and then rewrite etc. But that reuires mounting the rootfs which may come with its own set of issues
16:49:33 <jlebon> if we limit it to CoreOS, that obviously reduces support complexity
16:49:50 <dustymabe> real quick. for C) - I think we're generally of the opinion that kexec is fine.. if we find that there are problems with it we can discontinue the practice?
16:50:52 <jlebon> i think to be safe, we can make it experimental to test the waters before fully supporting it
16:51:17 <jlebon> it'd be hard to go back on the kexec approach without changing some other things
16:51:40 <dustymabe> +1
16:51:50 <dustymabe> ok for A and B
16:51:51 <jlebon> travier: i really like the idea of A as well. though ideally it'd be reusing the same coreos-installer flow
16:52:34 <jlebon> the big issue there of course if the rootfs squashfs. we'd need to rebuild it, which is awkward
16:52:35 <dustymabe> I consider A to be optional - could we do our implementation initially pulling from network but consider A in our implementation so we don't make it so we can't do it in the future?
16:52:35 <cverna> I read `to test the walters` and I was confused :D :D
16:52:52 <jlebon> cverna: haha
16:52:57 <travier> :)
16:53:22 <dustymabe> The walter is great.. come on in
16:53:34 <cverna> :D :D
16:53:42 <walters> um
16:53:48 <jlebon> i wanted to investigate tackling compression and the ability to regenerate the squashfs using some osmet thing
16:54:03 <travier> or we can download and do A or ask the user to place the squashfs somewhere and do A
16:54:19 <travier> (as workarounds)
16:54:22 <jlebon> right, the current model supports pointing to local or remote files
16:54:38 <jlebon> so they could download/copy from usb as a preliminary step
16:54:44 <jlebon> s/model/POC/
16:54:59 <travier> I think that's a nice future enhancement too, as it does not need this to be useful right now
16:55:09 <dustymabe> +1
16:55:52 <travier> (in the spirit of breaking things into smaller steps to make shipping them easier)
16:55:57 <dustymabe> for B) I'd say we just doc this as reinstalling your existing FCOS systems - we could have a section that mentions doing it on other hosts but say if you have trouble you'll need to re-install from scratch
16:56:33 <travier> +1
16:56:39 <miabbott> +1 to travier's note; keep the initial scope small to local/remote files + FCOS only
16:56:39 <dustymabe> i definitely don't want to get into "Suse's version of kexec isn't compatible" problems
16:56:57 <jlebon> dustymabe: to clarify, you're saying we would support it then?
16:57:30 <dustymabe> What I'm saying is that if it works for someone then it works. But if it doesn't work we won't fix it.
16:57:57 <dustymabe> is that confusing?
16:58:01 <jlebon> hmm
16:58:11 <travier> +1. We don't want to spend time investigating other distro packages and failures
16:58:18 <travier> s/We/I/
16:58:53 <travier> We could support Fedora -> FCOS and RHEL -> RHCOS for example
16:59:29 <jlebon> it does influence the UI whether we support that flow, so it'll need some thought
16:59:36 <travier> (time check, 1/2 hour mark)
16:59:59 <dustymabe> jlebon: in that case (if it's a lot of work) I'd just say not try to support it
17:00:06 <dustymabe> there's just too many variables
17:00:14 <jlebon> travier: right, agreed.  i think if we say we support it to some extent, we should at least test it somehow. those would be good targets
17:00:16 <dustymabe> and we have plenty of options for doing installs
17:01:04 <dustymabe> yeah in that case I say we don't support it :)
17:01:07 <jlebon> dustymabe: yeah, that's what i'm leaning towards as well personally even though i initially thought it'd be really cool
17:01:12 <dustymabe> we've got too much on our plate already
17:01:31 <travier> So I would vote on keeping the scope small initially and then see what we get asked for and go from there. A, B and C look fine.
17:01:40 <jlebon> ok overall it doesn't seem like there's much opposition to the current direction of the POC
17:01:58 <travier> +1
17:02:13 <dustymabe> #proposed For our factory reset capability we'll use kexec and initially require either network access or local copies of the install media to exist. In the future we may generate intall media from the existing system. We also will limit our scope to just re-installing FCOS on FCOS and disallow/discourage running it from other distros.
17:02:26 <travier> +1
17:02:57 <jlebon> +1
17:03:30 * dustymabe +1
17:03:42 <copperi[m]> +1
17:04:26 <dustymabe> #agreed For our factory reset capability we'll use kexec and initially require either network access or local copies of the install media to exist. In the future we may generate intall media from the existing system. We also will limit our scope to just re-installing FCOS on FCOS and disallow/discourage running it from other distros.
17:04:35 <dustymabe> next topic
17:04:44 <dustymabe> #topic tracker: Fedora 36 changes considerations
17:05:15 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/918
17:05:21 <dustymabe> ok some new updates from last week
17:05:43 <dustymabe> there are several items that have been moved to f37 and one that has been dropped completely
17:05:52 <dustymabe> all of those are now strikethrough in the description
17:06:06 <dustymabe> there are a few new items
17:06:34 <dustymabe> subtopic 127 New 128-bit IEEE long double ABI for IBM 64-bit POWER LE
17:06:45 <dustymabe> looks like a change for PPC64le
17:06:58 <dustymabe> I think it should be mostly transparent to us
17:07:17 <dustymabe> ravanelli: do you mind taking a look when you get a chance ^^
17:08:18 * dustymabe notes that we obviously aren't shipping ppc64le right now - but we are in the process
17:09:14 <ravanelli> dustymabe: sure
17:09:39 <dustymabe> #action ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
17:09:44 <dustymabe> ravanelli++
17:09:44 <zodbot> dustymabe: Karma for ravanelli changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:09:53 <dustymabe> subtopic 232 podman 4.0
17:10:13 <dustymabe> so this change came in late which is why it wasn't on my list last time
17:10:29 <dustymabe> podman 4.0 landed in f36 and rawhide over the past week
17:11:02 <dustymabe> though hmm. did it land in f36 - i know it did in rawhide (will check)
17:11:16 <dustymabe> either way I know we had at least one package that needed to be added aardvark-dns
17:11:52 <dustymabe> and travier has done a ton of work investigating options for keeping podman v3 and podman v4 both installable on the same system
17:12:09 * dustymabe notes the tests in rawhide pass at least
17:12:48 <dustymabe> though I did have to disable the toolbox test because there's currently no f37 toolbox
17:12:56 <dustymabe> any comments on this one?
17:13:12 <travier> I should ping toolbox folks for the F37 one
17:13:28 <dustymabe> yeah I dropped a line on IRC last week but didn't hear back
17:14:37 <travier> For podman co-installable v3 and v4, I have a PoC that would enable us to move quickly on the inclusion as it would not break things by default
17:15:06 <travier> We should work on the announcement anyway as it will happen with the rebase to F36 and we should warn folks as soon as possible
17:15:27 <dustymabe> travier: i.e. the breaking podman API changes?
17:16:09 <travier> yes, and the config compatibility changes (would prevent a rollback) and need to clean storage to use some podman v4 features
17:16:53 <travier> storage and config AFAIK
17:17:08 <dustymabe> is there a ticket we should open for this investigation/tracking or should we re-use the existing ticket
17:17:43 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/1070
17:17:53 <dustymabe> that one is specifically for Fedora 35
17:19:02 <travier> It's the same impact in F35
17:19:20 <travier> It's more about warning users in advance than investigation at this point
17:19:29 <travier> (from my perspective)
17:19:42 <travier> We can make another one if you prefer
17:19:58 <dustymabe> i'm cool either way (especially if someone wants to drive the effort) :)
17:20:45 <dustymabe> #action dustymabe to create a f36 changes ticket for podman 4.0 investigation/announcement
17:21:16 <dustymabe> let's move into the next topic then
17:21:27 <dustymabe> #topic podman v4.0, Fedora 35 and CoreOS
17:21:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1070
17:21:53 <dustymabe> so this is more specifically about f35
17:22:10 * dustymabe notes that the f36 beta is coming soon and we'll switch `next` over then
17:22:45 <dustymabe> which is another way of saying.. if the f36 beta is coming soon enough do we even need to do anything special?
17:23:46 <travier> I still think it has value to ship it in all the streams opt-in
17:24:24 <dustymabe> travier: and the current blocker is: https://github.com/coreos/fedora-coreos-tracker/issues/1070#issuecomment-1033025748
17:24:40 <dustymabe> basically you've done the PoC
17:24:51 <dustymabe> but it needs work to get it over the finish line
17:25:01 <dustymabe> and we also don't want to be blocked in doing that work
17:25:22 <dustymabe> i.e. we probably won't be able to make a podman4 package in f35 if the podman team doesn't want it
17:25:30 <jlebon> even with the PoC, we're still planning to move to v4 by default in f36, right?
17:25:37 <dustymabe> correct
17:25:44 <travier> yes, f36 will only have v4
17:26:09 <jlebon> ack ok
17:26:11 <travier> This is for shipping stable with v4 ahead of the f36 rebase to enable podman v4 to use stable for podman machine
17:26:29 <travier> otherwise they are next only
17:26:38 <travier> until we move to f36
17:26:56 <travier> so when they ship podman v4, we will get a big influx of next users
17:27:07 <dustymabe> ehh, we didn't explicitly discuss shipping both v3 and v4 in stable when we originally talked about this\
17:27:14 <jlebon> travier: if the podman team is ok using next once it's on f36, would you still want to have v3 and v4 co-installable?
17:27:20 <dustymabe> it does add extra size to the base
17:27:37 <travier> it add 60MB
17:27:40 <travier> adds*
17:28:37 <travier> I don't really want anything at this point. I think it would have been nice if we had a transition period but this is getting shorter quickly
17:28:40 <dustymabe> ok, I can get over the size thing since it's timeboxed
17:29:07 <jlebon> travier: +1
17:29:12 <dustymabe> my current stance: if the items listed in https://github.com/coreos/fedora-coreos-tracker/issues/1070#issuecomment-1033025748 can happen then I'm game
17:29:19 <jlebon> i.e. we should send an email ASAP regardless
17:29:26 <travier> if they are fine with next only then fine. We will get a lot of next users. This will put pressure on us to fix bugs in F36 too
17:30:21 <dustymabe> should I try to come up with a proposal?
17:31:28 <jlebon> WFM
17:32:30 <dustymabe> #proposed It would be nice if we could have podman 3 and 4 co-installable in F35 so users and podman-machine could use our more stable streams with podman v4. However, there is definitely some work left to do there. If that work doesn't get done and if the podman team is happy with throwing their users on `next` when the F36 beta lands then that may be the approach that wins.
17:33:08 <dustymabe> i'll remove the "throwing" word
17:33:14 <jaimelm> :)
17:33:45 <dustymabe> votes?
17:33:49 <jaimelm> +1
17:33:51 <jlebon> should we mention making an announcement?
17:34:24 <dustymabe> jlebon: we need to make a decision on the co-installable thing before we make an announcement?
17:34:41 <dustymabe> or could we just limit the announcement to talking about `next`+F36beta+podman4
17:34:58 <jlebon> yeah that's what I mean
17:35:31 <copperi[m]> +1
17:35:35 <travier> +1
17:35:48 <dustymabe> #action jlebon to send out an announcement about podmanv4 coming to F36 and the beta coming to next in the coming weeks
17:35:53 <dustymabe> jlebon: :)
17:35:56 <travier> s/throwing/moving
17:36:05 <dustymabe> #proposed It would be nice if we could have podman 3 and 4 co-installable in F35 so users and podman-machine could use our more stable streams with podman v4. However, there is definitely some work left to do there. If that work doesn't get done and if the podman team is happy with their users being on `next` when the F36 beta lands then that may be the approach that wins.
17:36:15 <dustymabe> travier: I updated it to remove the throwing word
17:36:19 <travier> +1
17:36:26 <dustymabe> jlebon: you good with that ^^
17:36:42 <jlebon> +1
17:36:47 <travier> I really hope we won't have aarch64 specific bugs as it would be hard to figure them out
17:36:55 <jlebon> hmm, i think we could combine it with the iptables-nft email
17:37:01 <dustymabe> lean on me and travier for the announcement
17:37:06 <jlebon> they're both f36-based
17:37:07 <dustymabe> ooh - let's not :)
17:37:19 <dustymabe> let's keep each communication consumable
17:37:24 <jaimelm> +1
17:37:28 <travier> +1 for discrete mails
17:37:34 <dustymabe> #agreed It would be nice if we could have podman 3 and 4 co-installable in F35 so users and podman-machine could use our more stable streams with podman v4. However, there is definitely some work left to do there. If that work doesn't get done and if the podman team is happy with their users being on `next` when the F36 beta lands then that may be the approach that wins.
17:37:36 <jlebon> heh, sure wfm
17:37:44 <dustymabe> #topic open floor
17:37:52 <dustymabe> sorry for late open floor again :(
17:37:53 <travier> one topic per mail makes it easier to skip
17:37:57 <dustymabe> jaimelm: has given uip on me
17:38:08 <dustymabe> up rather
17:38:15 <dustymabe> any topics for open floor?
17:38:26 <travier> https://github.com/coreos/fedora-coreos-tracker/issues/1093 > Has anyone submitted a talk?
17:38:43 <dustymabe> I have not :(
17:38:57 <dustymabe> we should make jbrooks do one!
17:39:09 <travier> CfP is closed
17:39:13 <dustymabe> ahh
17:39:15 <jbrooks> :)
17:39:22 <travier> but you can still try!
17:39:26 <lucab> I couldn't figure out how often the Fedora kernel signing key is rotated, but that may make kexec across major versions harder
17:39:41 <jbrooks> I do know the organizers, though ;)
17:39:54 <dustymabe> I was a lot more excited about conferences when I could attend them in person
17:40:23 <dustymabe> or maybe it was because I didn't have two kids and 0 free time :)
17:40:30 * dustymabe ends meeting in 60 seconds
17:41:27 <jlebon> lucab: good thought. can you add a comment to the POC ?
17:41:32 <dustymabe> #endmeeting