16:30:44 <spresti> #startmeeting fedora_coreos_meeting
16:30:44 <zodbot> Meeting started Wed May 31 16:30:44 2023 UTC.
16:30:44 <zodbot> This meeting is logged and archived in a public location.
16:30:44 <zodbot> The chair is spresti. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:44 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:56 <spresti> #topic roll call
16:31:07 <dustymabe> .hi
16:31:08 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:27 <spresti> #chair dustymabe
16:31:27 <zodbot> Current chairs: dustymabe spresti
16:31:58 <travier> .hello siosm
16:31:59 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:12 <spresti> #chair siosm
16:32:12 <zodbot> Current chairs: dustymabe siosm spresti
16:33:16 <marmijo[m]> .hello marmijo
16:33:17 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:33:18 <spresti> Hey, all hope everyone is doing well, just give us a few moments as people roll in.
16:33:32 <spresti> @chair marmijo
16:33:44 <spresti> #chair marmijo
16:33:44 <zodbot> Current chairs: dustymabe marmijo siosm spresti
16:35:07 <spresti> #topic Action items from last meeting
16:35:17 <spresti> #info there were no action items from last meeting!
16:35:34 <dustymabe> 🎉
16:35:43 <spresti> #topic tracker: Fedora 39 changes considerations
16:35:45 * dustymabe makes fifofonix sit in a chair
16:35:47 <dustymabe> #chair fifofonix
16:35:47 <zodbot> Current chairs: dustymabe fifofonix marmijo siosm spresti
16:36:01 <fifofonix> .hi ha
16:36:02 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:36:06 <spresti> #link https://github.com/coreos/fedora-coreos-tracker/issues/1491
16:36:29 <spresti> We covered the system wide changes in the meeting two weeks ago. We need to cover the self contained changes now.
16:36:50 <spresti> subtopic 208. Register EC2 Cloud Images with IMDSv2-only AMI flag
16:37:04 <spresti> discuss
16:37:21 <travier> :)
16:38:04 <dustymabe> I'm thinking we should adopt this for our images
16:38:04 <travier> I think we need to look at how we configure our images and follow the same change as the one Fedora made
16:38:52 <dustymabe> travier: would we want to try to follow f39 timing or would we roll it out early?
16:38:52 <travier> From memory, users that truly want something else can always override / upload their own
16:39:22 <dustymabe> yeah the text says "by providing a flag when registering an AMI to indicate that the AMI should by default launch with IMDSv1 disabled"
16:39:36 <dustymabe> I assume that means users can override the AMI setting on launch
16:39:59 <travier> I don't think we need to wait but we're not in a hurry
16:40:02 <travier> I'd ay
16:40:04 <travier> say*
16:40:30 <travier> So instead of dropping all changes on F39 timeline, we can roll them progressively
16:40:47 <dustymabe> yeah. I'm torn
16:40:51 <travier> but both should be fine, it's up to us
16:40:52 <spresti> What is the effort level for this ?
16:41:40 <dustymabe> I'm thinking we should roll it out on f39 timeline. It has the benefit of A) matching Fedora cloud B) giving users a longer window to find/fix issues on their end.
16:41:53 <dustymabe> spresti: I'm thinking the LoE should be relatively low.
16:42:05 <dustymabe> mostly changing our API calls when we create images
16:42:10 <travier> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-new-instances.html#configure-IMDS-new-instances-ami-configuration > Looks like just adding a flag/parameter
16:42:15 <dustymabe> the unknown here is do some of our tests fail and have to be looked into
16:42:21 <spresti> Yeah that was my expectation, I agree, its a lot of value.
16:42:33 <dustymabe> and.. do we add a test or two that tests the old configuration
16:42:38 <spresti> or utility.
16:43:51 <travier> fine with rolling with F39
16:44:06 <travier> let's move to the next one?
16:44:07 <ravanell_> .hi
16:44:09 <zodbot> ravanell_: Sorry, but user 'ravanell_' does not exist
16:44:16 <spresti> #chair ravanell_
16:44:16 <zodbot> Current chairs: dustymabe fifofonix marmijo ravanell_ siosm spresti
16:44:26 <travier> It's mostly going to be the same discussion for 209 & 210
16:44:36 <dustymabe> travier: maybe first let's make a decision #proposed/#agreed
16:44:45 <dustymabe> and then maybe an #action for opening a ticket
16:44:53 <spresti> Ok, so it sounds like we agree we should include 208 for f39
16:45:19 <travier> #proposed We will update our images with the IMDSv2-only flag for F39
16:45:47 <travier> Who want to be actionned to open a ticket? :)
16:46:33 <spresti> I can take a look
16:46:43 <dustymabe> +2 got proposed
16:46:46 <dustymabe> sigh
16:46:48 <dustymabe> +1 for proposed
16:46:55 <marmijo[m]> ack proposed
16:47:01 <travier> #proposed We will update our AWS AMIs with the IMDSv2-only flag for F39
16:47:10 <bgilbert> .hi
16:47:12 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:47:18 <spresti> #chair bgilbert
16:47:18 <zodbot> Current chairs: bgilbert dustymabe fifofonix marmijo ravanell_ siosm spresti
16:47:20 <dustymabe> travier: did you mean #agreed?
16:47:31 <travier> I slightly tweaked the text
16:47:39 <travier> but a 3rd vote would be nice
16:47:41 <dustymabe> ahh - +1 to new text
16:47:53 <marmijo[m]> +1 from me on the new text as well
16:47:59 <spresti> +1 ack to the proposed
16:48:02 <travier> #agreed We will update our AWS AMIs with the IMDSv2-only flag for F39
16:48:05 <ravanell_> +1
16:48:26 <bgilbert> (late to the party; are we assuming that no app is using IMDSv1?)
16:49:04 <bgilbert> or are we doing a coreos-status post
16:49:09 <dustymabe> IIUC it's a setting on the AMI and users can override it at lauch time
16:49:24 <bgilbert> sure, but it's a compat break with default settings
16:50:01 <dustymabe> yeah I think this would be a good thing to include in the email we send out about the switch to f39 (we usually send it out around beta time and then re-link to it when stable switches over)
16:50:07 <bgilbert> +1
16:50:31 <dustymabe> existing instances shouldn't be affected - so that's nice
16:50:33 <spresti> +1, should we add that to the proposed as well and ack that its a compat break?
16:50:45 <bgilbert> should probably add it explicitly to the F39 rebase checklist so we don't forget
16:51:09 <dustymabe> bgilbert: yeah. maybe we should have a ticket specifically for the FN release communication to track stuff like this
16:51:16 <travier> hum, re-reading the description I understand the concern now
16:51:18 <bgilbert> dustymabe: that also works
16:51:58 <bgilbert> +0.7 to adding to the #agreed
16:52:06 <travier> #undo
16:52:39 <bgilbert> huh, it used to send a message for that
16:52:42 <travier> #proposed We will update our AWS AMIs with the IMDSv2-only flag for F39. We'll mention that change in the announcement for the F39 rebase and next opening
16:53:05 <dustymabe> +1
16:53:41 <bgilbert> #proposed We will update our AWS AMIs with the IMDSv2-only flag for F39.  that change could potentially break some applications, so we'll mention that change in the announcement for the F39 rebase and opening of `next`
16:54:06 <dustymabe> +1
16:54:10 <travier> We have to make sure that Ignition works with it
16:54:16 <travier> +1
16:54:17 <bgilbert> travier: it does
16:54:20 <spresti> +1
16:54:21 <travier> ah, ok
16:54:33 <marmijo[m]> +1
16:55:23 <spresti> #agreed We will update our AWS AMIs with the IMDSv2-only flag for F39.  that change could potentially break some applications, so we'll mention that change in the announcement for the F39 rebase and opening of `next`
16:55:49 <bgilbert> sorry for the last-minute redirect
16:55:56 <spresti> Ok, so now lets move on to the next subtopic
16:56:00 <spresti> No worries lol
16:56:23 <spresti> subtopic 209. EC2 AMIs default to the gp3 EBS volume type
16:56:32 <spresti> discuss
16:57:14 <travier> This one should really be "transparent"
16:57:30 <bgilbert> the Fedora Change doesn't list any downsides
16:57:50 <bgilbert> we should probably release-note it, in case someone has a dependency
16:57:59 * dustymabe tries to come up with a downside: now my storage is faster and I'm hitting a race condition!!!!
16:58:12 <dustymabe> #joke ^^
16:58:19 <travier> it could impact performance or change performance, but that's not a something we can protect users from
16:58:26 <travier> or warm about
16:58:34 <travier> any change in the kernel could
16:58:53 <travier> I'd say we release not it and do it for F39 as above
16:59:28 <dustymabe> WFM, though I'd be OK with just rolling this one out as soon as it's ready
16:59:46 <bgilbert> looking at the comparison chart: https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-gp2-to-gp3-and-save-up-to-20-on-costs/
16:59:47 <travier> +1 for rolling it out early
16:59:57 <spresti> #proposed  Set EC2 AMIs default to the gp3 EBS volume type for f39. Include it in release notes to inform users.
17:00:17 <bgilbert> looks like lower throughput for everyone, from "slightly lower" to "about half as much"
17:00:37 <bgilbert> and fewer default IOPS for volumes > 3 TB
17:01:00 <bgilbert> throughput and IOPS can both be scaled but that's a non-default paid feature
17:01:00 <dustymabe> so lower baseline throughput, higher max throughput
17:01:40 <dustymabe> haha, looks like you now need a college degree to figure out actual cost too
17:01:46 <bgilbert> hmm, the chart is confusing me wrt max throughput
17:01:57 <bgilbert> does "max" mean "burst" or does it mean "paid"?  since there's a paid option
17:02:16 <bgilbert> anyway, +1 to proposed
17:02:55 <dustymabe> if we did have a problem with this change I'd propose we take it back to this change discussion rather than opt to differ
17:02:58 <travier> https://fedoramagazine.org/new-aws-storage-type-for-fedora-linux/
17:03:31 <bgilbert> I'm not objecting; I think this is the right change.  it's just that it's not 100% transparent so we have to announce it
17:03:42 <dustymabe> bgilbert: ahh, got ya
17:03:47 <travier> Given that it is only changing the default and only for new images and users can change it back easily, I don't think we should worry
17:03:51 <dustymabe> so doing along with the other change for f39 is desirable then
17:03:56 <bgilbert> dustymabe: +1
17:04:05 <dustymabe> kk
17:04:14 <dustymabe> spresti: +1 to #proposed from my side
17:04:15 <travier> ok for batching with F39
17:04:27 <travier> +1 to proposed
17:04:36 <spresti> Ok!
17:04:48 <spresti> #agreed Set EC2 AMIs default to the gp3 EBS volume type for f39. Include it in release notes to inform users.
17:04:58 <spresti> Lets move on to the next topic
17:05:14 <spresti> subtopic 210. Register EC2 Cloud Images with uefi-preferred AMI flag
17:05:23 <spresti> discuss
17:05:56 <travier> We have to make sure we support UEFI but that should be transparent as well
17:05:59 <dustymabe> note @jlebon opened a PR for this a while back: https://github.com/coreos/coreos-assembler/pull/3402
17:06:06 <travier> batching for F39 should be fine
17:06:33 <bgilbert> +1 to batching with the other AWS changes
17:06:52 <dustymabe> yep
17:07:21 <bgilbert> we'll already be ratcheting the pipeline to roll out the changes per channel
17:08:37 <spresti> Ok, that makes sense.
17:08:49 <spresti> +1 to doing this as well.
17:09:39 <bgilbert> #proposed We'll switch AMIs to uefi-preferred for F39.
17:10:00 <spresti> +1
17:10:17 <travier> +1
17:10:33 <dustymabe> +1
17:10:39 <spresti> #agreed We'll switch AMIs to uefi-preferred for F39.
17:11:09 <spresti> Ok, lets move to the next subtopic
17:11:26 <spresti> subtopic 213. mkosi-initrd
17:11:30 <spresti> discuss
17:12:05 <travier> Once this lands in Fedora, we should probably investigate using it to build the initrds in rpm-ostree instead of using dracut
17:12:40 <bgilbert> uhh
17:12:43 <travier> We don't have much to do until the ground work is done in Fedora but the sooner we give it a try the easier it will be to make changes
17:12:57 <dustymabe> we depend on a lot of dracut modules
17:12:57 <bgilbert> if the ecosystem is moving away from dracut, we should probably consider doing it too, but it'll be a susbtantial amount of work
17:12:59 <travier> And we'll need to change all our custom dracut modeuls
17:13:00 <bgilbert> *substantial
17:13:10 <dustymabe> 👆
17:13:10 <travier> yes
17:13:17 <travier> it's indeed a lot of work
17:13:42 <dustymabe> though, I will say I'm happy to move to something better (assuming it's better)
17:13:50 <bgilbert> if the rest of Fedora switches away from dracut, are our dracut builds going to instantly bitrot?
17:14:08 <dustymabe> bgilbert: what do you mean?
17:14:14 <dustymabe> "our dracut builds" ?
17:14:22 <bgilbert> are we forced to switch on the Fedora timetable
17:14:46 <bgilbert> e.g. because the dracut package goes away, or because dracut modules we don't control will stop being maintained
17:15:00 <dustymabe> honestly I don't know.. I think we have enough control to not have to switch in the exact same release
17:15:16 <dustymabe> but yeah, "dracut modules we don't control not being maintained" is the bigger concern
17:15:38 <travier> I think there are still steps before Fedora switches away from dracut by default
17:15:41 <spresti> So from what I am hearing this sounds quite intense(and we are not even camping) and it might make sense to target a smaller; considering existing workloads.
17:15:43 <dustymabe> i wonder if the new thing could have some sort of compat mode for old modules
17:16:05 <travier> the target is to use it only for UKIs at the beginning
17:16:09 <bgilbert> "The goal of this change is to provide an alternative mechanism. If the feedback is positive, we may consider using initrds built with mkosi-initrd as default in certain scenarios. There are no plans to remove dracut in the foreseeable future."
17:16:37 <dustymabe> bgilbert: yeah. Sorry. That was in my head because I had read the proposal some time ago.
17:16:39 <travier> The transition in Fedora is going to take a while
17:17:01 <dustymabe> i imagine when the transition in Fedora happens it will be a system wide change and not self contained (like this proposal is)
17:17:07 <travier> yes
17:17:14 <travier> I think so as well
17:17:15 <dustymabe> so I think the bigger question is what (if anything) should we do now
17:17:39 <dustymabe> I think we'll probably do nothing, but if we had extra cycles (we don't really) we'd probably try to be more proactive and involved in this transition
17:17:51 <travier> If folks are interested, giving it a try could help for the future, but I don't think we need to act on it now
17:17:55 <dustymabe> i.e. trying to switch some of our stuff over to it experimentally and giving feedback to improve it
17:18:27 <bgilbert> counterargument: we can let other folks shake out the early problems this time :-)
17:18:48 <travier> works too :)
17:18:54 <spresti> +1
17:19:01 <bgilbert> I'm okay either way though.  I agree we do nothing for F39, but maybe let's file a tracker issue
17:20:15 <dustymabe> +1 for do nothing for f39 (unless someone gets motivated). I'm +0.5 for the tracker issue
17:20:22 <travier> same as dusty
17:20:30 <travier> not sure what a tracker would give us
17:20:54 <bgilbert> travier: a separate task to track in Jira and maybe assign to someone.  but we can also just wait&see for now
17:20:57 <dustymabe> travier: if I was a better project manager a tracker issue would help us plan things.. but I'm not
17:21:17 <spresti> #proposed Let mkosi-initrd land in fedora and allow it to mature before starting to adopt it into fcos. Revisit for f40 considerations.
17:21:23 <travier> there is only so much "future work" that we can track
17:21:28 <bgilbert> travier: +1
17:21:42 <dustymabe> :)
17:21:43 <travier> I have a full list if someone is interested :)
17:21:45 <dustymabe> +1 to proposed
17:21:49 <bgilbert> +1 to proposed
17:21:51 <travier> +1 to proposed
17:22:11 <spresti> #agreed  Let mkosi-initrd land in fedora and allow it to mature before starting to adopt it into fcos. Revisit for f40 considerations.
17:22:29 <travier> Should we do the last one quickly?
17:22:45 <bgilbert> travier: +1
17:22:47 <spresti> Ok, yeah lets jump to the final subtopic
17:22:53 <spresti> subtopic 215. BiggerESP`
17:22:56 <travier> it's eavily related to another issue we have
17:22:59 <spresti> discuss
17:23:06 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1465
17:23:19 <dustymabe> yep notes say: We manage our partitions independently of Anaconda, but we are discussing this change as part of https://github.com/coreos/fedora-coreos-tracker/issues/1465
17:23:36 <travier> So I'd say we'll take it into account when we fix #1465
17:23:40 <dustymabe> so I think we can just switch this from ⚠️ to ✔️
17:23:45 <travier> +1
17:23:48 <bgilbert> +1
17:23:55 <spresti> +1
17:24:02 <dustymabe> we don't need a #proposed/#agreed I don't think
17:24:10 <travier> 👍
17:24:26 <bgilbert> Fedora can't make assumptions about the larger size because of existing systems, so deferring this shouldn't cause downstream breakage for us
17:24:56 <spresti> OK well with the last few moments lets switch to open floor
17:25:10 <spresti> #topic Open Floor
17:26:29 <dustymabe> fifofonix: and I will be giving a presentation about "How do you Fedora CoreOS?" at the F38 release party on Friday
17:26:40 <dustymabe> our timeslot is at around noon eastern US time
17:26:43 <fifofonix> :o)
17:27:21 <spresti> Thats awesome I will have to check it out.
17:28:02 <travier> 👍
17:28:07 <dustymabe> one other thing
17:28:25 <dustymabe> in the next round of releases (not the ones going out today) we should have ppc64le builds getting produced
17:28:41 <dustymabe> thanks to everyone who participated in any way to make that happen
17:28:52 <spresti> Wow, good job everyone.
17:29:12 <bgilbert> 🎉
17:29:19 <spresti> Well since we are out of time lets go ahead and end the meeting.
17:29:33 <spresti> #endmeeting