16:30:19 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:19 <zodbot> Meeting started Wed Mar  2 16:30:19 2022 UTC.
16:30:19 <zodbot> This meeting is logged and archived in a public location.
16:30:19 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:19 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:23 <dustymabe> #topic roll call
16:30:28 <travier> .hi siosm
16:30:29 <zodbot> travier: Sorry, but user 'travier' does not exist
16:30:34 <travier> .hello siosm
16:30:35 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:30:35 <fifofonix> .hello
16:30:38 <zodbot> fifofonix: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:30:39 <dustymabe> .hi
16:30:39 <jbrooks> .hello jasonbrooks
16:30:41 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:45 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:30:45 <fifofonix> .hi
16:30:50 <aaradhak[m]> .hi
16:30:50 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:30:53 <jlebon> .hello2
16:30:53 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:30:56 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:01 <davdunc> .hello2
16:31:03 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:31:39 <dustymabe> #chair travier fifofonix jbrooks aaradhak[m] jlebon davdunc
16:31:39 <zodbot> Current chairs: aaradhak[m] davdunc dustymabe fifofonix jbrooks jlebon travier
16:31:50 <aaradhak> .hi
16:31:51 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:32:25 <nemric> hi :)
16:32:31 <miabbott> .hello miabbott
16:32:32 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:32:42 <miabbott> congrats on not blowing up zodbot
16:33:38 <dustymabe> #chair nemric miabbott
16:33:38 <zodbot> Current chairs: aaradhak[m] davdunc dustymabe fifofonix jbrooks jlebon miabbott nemric travier
16:33:41 <dustymabe> welcome nemric
16:33:46 <dustymabe> #chair skunkerk
16:33:46 <zodbot> Current chairs: aaradhak[m] davdunc dustymabe fifofonix jbrooks jlebon miabbott nemric skunkerk travier
16:34:21 <dustymabe> #topic Action items from last meeting
16:34:23 <lucab> .hi
16:34:24 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:34:27 <dustymabe> #chair lucab
16:34:27 <zodbot> Current chairs: aaradhak[m] davdunc dustymabe fifofonix jbrooks jlebon lucab miabbott nemric skunkerk travier
16:34:36 <dustymabe> I think ravanelli has a holiday today
16:34:41 <dustymabe> * ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:34:45 <dustymabe> so I'll re-action
16:34:51 <dustymabe> #action ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:35:00 <dustymabe> and move on to meeting tickets
16:35:08 <dustymabe> #topic AWS m4.large instances fail to boot starting with 35.20211226.20.0
16:35:12 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1066
16:35:52 <dustymabe> ok this was the issue where certain instance types would fail to boot FCOS (well, they booted, just had no networking)
16:36:07 <davdunc> yes.
16:36:08 <dustymabe> we identified the issue and pinned our kernel in FCOS
16:36:14 <dustymabe> but we can't pin forever
16:36:48 <dustymabe> I propose that we identify some time frame to move on
16:36:49 <davdunc> and I haven't asked Justin about pinning it for longer, but need 8 to 12 weeks for updates to support it.
16:37:15 <walters> Did we loop in the person who introduced the commit?
16:37:27 <dustymabe> davdunc: ^^
16:37:31 <davdunc> I did not.
16:37:46 <davdunc> but it's a good commit.
16:37:57 <walters> OK let's not be shy about that, most maintainers I think will be helpful for this type of stuff
16:38:01 <davdunc> it just conflicts with an older version of Xen.
16:38:09 <dustymabe> from chatting with davdunc he thought the chance of it getting reverted upstream were small
16:38:21 <davdunc> walters: agreed I am happy to include them.
16:38:42 <walters> Let's please at least make sure they are aware
16:38:44 <travier> Linus is known to revert breaking changes
16:39:06 <travier> walters: +1
16:39:30 <davdunc> but dustymabe is right. it's a beneficial change, even on the other infrastructure on Amazon, it just affects a specific group of hypervisors running a specific version of Xen - 4.2
16:40:15 <davdunc> the team at Amazon has plans to upgrade these systems that are affected and they are working on a very tight timeline.
16:40:39 <dustymabe> so our options seem to be:
16:40:44 <davdunc> it would be ideal if we could carry the revert for just a couple of months.
16:40:50 <dustymabe> davdunc: yes
16:41:12 <dustymabe> davdunc: has agreed to reach out to justin to see if he can pick up the revert again for us
16:41:29 <dustymabe> #action davdunc to see if we can get the revert applied to the latest fedora kernels again
16:41:53 <davdunc> yes. I meant to complete that yesterday, but didn't have time. I'll get that off post haste.
16:41:56 <dustymabe> I propose if we says "no" that we warn our users and move on
16:42:08 <dustymabe> s/we/he
16:42:22 <davdunc> carrying a known issue about it wouldn't hurt.
16:42:40 <walters> another option is to carry a patch that makes this into a kernel module parameter, then we turn that on by default (potentially even just in AWS)
16:43:37 <dustymabe> that is another option, but I doubt justin would be up for that (sounds like more work than a revert)
16:43:45 <travier> Note that by doing that at the Fedora level, anybody else that uses the upstream kernel will hit this issue.
16:43:53 <davdunc> that's an interesting alternative walters
16:44:10 <dustymabe> travier: can you be more specific
16:44:12 <davdunc> yea we could petition for the change upstream.
16:44:41 <travier> This should be raised upstream and as walters says, the author should be warned
16:45:15 <dustymabe> travier: indeed
16:45:31 <dustymabe> ok so should we revisit next week once we know what justin says?
16:45:37 <travier> At the same time, I won't be doing it (just setting up the mail config is discouraging enough) so I won't blame anyone if they don't want to do ti
16:46:09 <dustymabe> yeah, honestly the "not using a git forge" thing is hurting the kernel IMO
16:46:16 <travier> +1
16:46:35 <dustymabe> davdunc: maybe just email sr@denx.de
16:46:38 <davdunc> yes. I'll loop in the original author as well and see if there is a good path forward.
16:46:42 <davdunc> thanks dustymabe
16:46:45 <dustymabe> thank you davdunc
16:46:50 <dustymabe> moving on...
16:47:03 <dustymabe> #topic tracker: Fedora 36 changes considerations
16:47:07 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/918
16:47:21 <dustymabe> here's our large tracker for f36 changes
16:47:32 <dustymabe> let's run through the ones real quick (at least one new one)
16:47:47 <dustymabe> subtopic 102 "Introduce module Obsoletes and EOL"
16:47:49 <dustymabe> #link https://github.com/coreos/rpm-ostree/issues/3465
16:47:57 <dustymabe> jlebon - anything to discuss with regards to https://github.com/coreos/rpm-ostree/issues/3465 ?
16:48:58 <jlebon> dustymabe: i don't think there's anything pressing there
16:49:24 <dustymabe> ok, basically the proposal is to behave the same way we do today, but enhance the error message
16:49:41 <jlebon> the status quo is to error out, which i think matches better the rpm-ostree principles than auto-converting like dnf
16:49:52 <dustymabe> subtopic 103 "DNS Over TLS"
16:49:52 <jlebon> dustymabe: yes
16:49:57 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1098
16:50:14 <dustymabe> for this one we're just tracking if any issues come up related to the change. nothing to do for now
16:50:21 <dustymabe> subtopic 127 "New 128-bit IEEE long double ABI for IBM 64-bit POWER LE"
16:50:35 <dustymabe> This is the one ravanelli has an action item for. So we'll wait to hear from her next week.
16:50:39 <dustymabe> subtopic 128 "GNU Toolchain Update (gcc 12, glibc 2.35)"
16:50:45 <dustymabe> This one is new ^^
16:51:03 <dustymabe> Update the Fedora 36 GNU Toolchain to gcc 12 and glibc 2.35.
16:51:27 <dustymabe> any investigation or changes we need to do as a result of ^^
16:52:04 <jlebon> i think similar to 103. no action, but be on the lookout for fallout
16:52:46 <dustymabe> +1 - should we create a tracker issue for that fallout then?
16:53:24 <lucab> the wiki page confirms in a few place this should be backward-compatible
16:53:27 <lucab> *places
16:53:48 <dustymabe> ok
16:53:59 <jlebon> i prefer doing that Just-In-Time, but ok also preemptively doing it
16:54:02 <dustymabe> in that case I'll say we should be good and we'll open an issue later if we find problems
16:54:29 <dustymabe> #info for 128 "GNU Toolchain Update (gcc 12, glibc 2.35)" the wiki says it should be a backwards compatible change. We'll monitor, but for now there is nothing for us to do.
16:54:38 <dustymabe> subtopic 208 "Retired Packages"
16:54:43 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1099
16:55:36 <dustymabe> looks like that one is pending discussion from us. We're worried about systems getting into a state where it won't be able to update
16:55:50 <dustymabe> I'll add the meeting label to it and we can discuss it more in depth next time.
16:56:18 <dustymabe> subtopic 232 "Podman 4.0"
16:56:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1106
16:56:51 <dustymabe> For this one we sent out a communication to our users (see https://github.com/coreos/fedora-coreos-tracker/issues/1106#issuecomment-1057117220
16:57:21 <dustymabe> also thank you to travier for making a docs page for major changes to FCOS: https://docs.fedoraproject.org/en-US/fedora-coreos/major-changes/#_podman_v4_0
16:57:24 <dustymabe> travier++
16:57:30 <travier> :)
16:58:27 <dustymabe> ok we're out of topics with the explicit meeting label but there was one recently that we dropped the meeting label from and we should revisit
16:58:43 <dustymabe> #topic Consider releasing FCOS with a predictable release cadence
16:58:51 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1062
16:59:08 <dustymabe> travier: are you able to talk about this one today?
16:59:17 <travier> Sure
17:00:10 <travier> The general idea is to try to make the time when new releases are sent to users more predictable in advance
17:00:35 <travier> This won't change any of our messaging, but would help folks to plan for testing new releases in advance
17:01:28 <travier> This is very similar to what other big projects are doing, like Microsoft patch Tuesday, Chrome/Firefox/Rust releases every X weeks
17:01:46 <travier> This would never be a guarentee
17:02:34 <dustymabe> right now we already try to release every two weeks on a tuesday
17:02:40 <travier> And we would still have out of cadence releases for security fixes
17:02:55 <travier> yes, I think we are mostly there
17:03:03 <nemric> +1
17:03:32 <dustymabe> what do you propose we do differently?
17:03:49 <travier> We don't have to do it right now. The more we improve our testing and release process, the more it will naturally happen to
17:03:51 <travier> too*
17:03:53 <nemric> That could be a nice option to have a "tuesday" or else that would let me know when it come
17:04:10 <lucab_> low-effort: start by standardizing the start+duration for "normal" rollouts
17:04:23 <travier> lucab_: +1
17:04:53 <travier> Always start the rollout at a given time, and if passed for the day, then the next day
17:05:05 <travier> (one option)
17:05:33 <dustymabe> I think what benjamin said in https://github.com/coreos/fedora-coreos-tracker/issues/1062#issuecomment-1006030612 is useful
17:05:34 <travier> or try to make it happen for the given time, otherwise delay
17:05:40 <dustymabe> "Since we'll never be completely predictable, I actually think there's value in remaining somewhat inconsistent, so we don't set incorrect expectations."
17:06:11 <travier> Completely agree with that too
17:06:18 <dustymabe> if users need their systems to not update at a certain time then shouldn't that configure zincati accordingly
17:06:25 <dustymabe> they*
17:06:31 <travier> I think it's orthogonal
17:07:06 <travier> it's more about maximizing the time window for users to test the next testing/next release
17:07:31 <dustymabe> i.e. 14 days versus 12 or 13 ?
17:07:38 <lucab> also keep in mind that for most nodes the update server will pick an arbitrary wariness and spread it over the rollout window
17:07:52 <travier> but really I don't want to push it to much. I think it will happen naturally as we improve things
17:08:30 <dustymabe> yeah. I think i'm ok with a statement like "we try to release every two weeks and we try to have releases go out on tuesdays"
17:08:40 <travier> yes, that would be for folks with a wariness of 0 on truly testing servers
17:08:44 <dustymabe> but if we add more contraints I just think it makes us worry about the details too much
17:08:50 <travier> +1
17:08:58 <dustymabe> and focus on that vs other work
17:09:06 <nemric> yes
17:09:40 <dustymabe> should we make any sort of statement here #proposed/#agreed or #info ?
17:09:48 <travier> Agree, let's not spend too much time on that
17:10:11 <nemric> ok to say deployed when ready
17:11:43 <travier> Let's move on? I have a potentially long OD topic
17:12:08 <dustymabe> #proposed We try to release every two weeks and we try to have releases go out on Tuesday. We don't want to write down any more contraints for ourselves than that so we don't waste time with the details and focus on other work.
17:12:13 <dustymabe> sorry was writing that up ^^
17:12:19 <nemric> As a end user I would prefer having a choosen time, but I don't want to be disapointed ^^ I'm happy when it comes, it's a better feeling ;)
17:12:29 <dustymabe> nemric: :)
17:12:51 <dustymabe> ack/nack?
17:12:59 <travier> +1
17:13:12 <nemric> ack
17:13:48 <lucab_> ack
17:13:50 <jlebon> sgtm
17:13:51 <travier> nemric: What's your use case for having a chosen time? Feel free to write into the issue
17:14:02 <dustymabe> #agreed We try to release every two weeks and we try to have releases go out on Tuesday. We don't want to write down any more contraints for ourselves than that so we don't waste time with the details and focus on other work.
17:14:33 <dustymabe> #topic open floor
17:14:40 <dustymabe> travier: you had one topic for open floor?
17:14:51 <travier> OKD is using a Fedora 34 FCOS right now for OKD 4.10 and will switch to Fedora 35 FCOS for 4.11
17:14:57 <nemric> my weechat runs in a container on FCOS, just want to know if I will be able to see the "dustyVM" ip in the morning
17:15:07 <travier> But we will soon switch to F36 too
17:15:20 <dustymabe> nemric: :)
17:15:47 <travier> Essentially, they are running into the "rebase to next Fedora while staying on an OKD release"
17:15:49 <travier> issue
17:16:23 <travier> nemric: you probably want to make sure that everything reconnects automatically on reboot/update then
17:16:39 <dustymabe> hmm - i don't know if I'm familiar with that "rebase to next Fedora while staying on an OKD release" issue
17:16:44 <travier> So some users asked for OKD to stay on a given Fedora release for the entire lifecycle of an OKD release
17:16:44 <nemric> not really, I don't want to miss messages
17:17:09 <dustymabe> nemric: https://github.com/dustymabe/weechat-demo
17:17:12 <travier> nemric: why would you miss messages if it reconnects?
17:17:42 <travier> The rebase to F34 introduced a change that broke some configuration in OKD AFAIK
17:17:58 <nemric> travier: all my services works great after reboot, but I don't have spare apps like on k8s to handle always on service
17:18:25 <travier> I've suggested that they focus on testing the next FCOS for OKD and add testing for their use cases
17:18:32 <nemric> weechat is not set as a servce ^^ that my fault
17:18:35 <dustymabe> nemric: you'd only miss messages for a few minutes
17:18:42 <dustymabe> see https://github.com/dustymabe/weechat-demo
17:18:48 <travier> Otherwise, they'll just end up not truly using FCOS
17:19:39 <dustymabe> travier: so you're encouraging them to test the next version of OKD with the next version of FCOS?
17:20:04 <dustymabe> so that they are not so far behind when they do a release?
17:20:05 <travier> CoreOS layering should enable us to more easily test future updates / rebases
17:20:09 <nemric> dustymabe: I'm waitong for the one that dive into user-units :D
17:20:13 <travier> dustymabe: yes
17:20:25 <dustymabe> travier: that makes me happy
17:20:43 <travier> But that's not the case right now
17:21:22 <jlebon> yeah, that's very similar to what we deal with in OCP with RHCOS
17:21:38 <travier> I'm afraid it will end up in a "pick a Fedora release for a given OKD release" situation and they will essentially not be running FCOS
17:21:46 <jlebon> except there we do have to support those old bootimages and e.g. have to respin them sometimes to fix bugs
17:22:28 <travier> We had the issue where OKD installs where broken, as we removed the F34 key from the installer
17:22:37 <jlebon> which takes us to https://github.com/openshift/enhancements/pull/201
17:23:09 <travier> Essentially, it makes it impossible for users to use the FCOS tracker to report issues if they are not truly using an FCOS release
17:23:35 <dustymabe> IMO this is a real problem
17:23:58 <dustymabe> ideally incremental releases of OKD *could* bump the FCOS version too
17:24:03 <travier> So either OKD improves testing in advance, or we "lose" OKD as a "direct" FCOS user
17:24:13 <jlebon> but note OKD is already not using FCOS
17:24:18 <travier> yes
17:24:21 <jlebon> https://github.com/openshift/okd-machine-os/issues/210
17:24:23 <dustymabe> jlebon: not "directly"
17:24:39 <travier> But the messaging right now is still that it is using FCOS
17:24:45 <travier> even if that is not entirely true
17:24:54 <jlebon> yeah, agreed
17:25:13 <dustymabe> travier: thank you for trying to get OKD in more alignment so we can support that community better
17:25:24 <travier> If this changes to one Fedora release per OKD release then this should probably evolve into "based on FCOS" instead of FCOS
17:25:27 <dustymabe> I think we do want to help them if we can
17:25:33 <travier> Definitely
17:25:58 <dustymabe> is there anything concrete we can do (other than supporting a stream of FCOS on Fedora N-1)?
17:26:10 <travier> Hopefully CoreOS Layering should really help us there at it will enable them to more easily test our releases
17:26:24 <dustymabe> ok, keep us updated
17:26:36 <travier> will do :)
17:26:38 <dustymabe> any other open floor topics?
17:26:44 <nemric> Yes
17:27:01 <nemric> I've added a new PR, about relabelling
17:27:17 <nemric> I'm not sure what I have to do
17:28:29 <dustymabe> #link https://github.com/coreos/ignition/pull/1324
17:28:29 <nemric> I wanted to prevent relabeling file or paths mpultiple time, jlebon seem to say setfile handle that, but bgilbert approve my idea ...
17:28:37 <jlebon> nemric: sorry, i'll get back to you today
17:28:45 <dustymabe> jlebon++
17:28:49 <jlebon> i wasn't aware of the previous discussion on that topic, so need to read that
17:28:52 <dustymabe> follow up in #fedora-coreos ?
17:28:56 <nemric> thanks :)
17:29:09 <dustymabe> thank you nemric for sticking with and asking for advice on contributions
17:29:22 <dustymabe> any other topics for open floor?
17:29:39 <nemric> dustymabe: great OS and great team :)
17:29:45 <dustymabe> aww :)
17:29:52 <dustymabe> #endmeeting