16:28:21 <dustymabe> #startmeeting fedora_coreos_meeting
16:28:21 <zodbot> Meeting started Wed Mar 23 16:28:21 2022 UTC.
16:28:21 <zodbot> This meeting is logged and archived in a public location.
16:28:21 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:28:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:28:21 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:28:26 <dustymabe> #topic roll call
16:28:31 <dustymabe> .hi
16:28:32 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:29:11 <jbrooks> .hello jasonbrooks
16:29:12 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:29:19 <skunkerk> .hello sohank2602
16:29:20 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:29:39 <jlebon> .hello2
16:29:40 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:29:42 <dustymabe> #chair jbrooks skunkerk jlebon
16:29:42 <zodbot> Current chairs: dustymabe jbrooks jlebon skunkerk
16:29:56 <saqali> .hi
16:29:57 <aaradhak[m]> .hi
16:29:57 <zodbot> saqali: saqali 'Saqib Ali' <saqali@redhat.com>
16:30:00 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:30:28 <aaradhak> .hi
16:30:29 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:30:46 <bgilbert> .hi
16:30:47 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:19 <dustymabe> #chair saqali aaradhak bgilbert mnguyen
16:31:19 <zodbot> Current chairs: aaradhak bgilbert dustymabe jbrooks jlebon mnguyen saqali skunkerk
16:31:41 <miabbott> .hello miabbott
16:31:42 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:31:46 <dustymabe> #chair miabbott
16:31:46 <zodbot> Current chairs: aaradhak bgilbert dustymabe jbrooks jlebon miabbott mnguyen saqali skunkerk
16:32:42 <dustymabe> ok let's get started
16:33:21 <dustymabe> #topic Action items from last meeting
16:33:26 <dustymabe> * davdunc to put a package review in for ec2-net-utils and brainstorm on how we can use that for #601
16:34:12 <dustymabe> davdunc: around today?
16:34:39 <mnguyen> .hello mnguyen
16:34:40 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:34:52 <dustymabe> looks like he is AFK
16:34:59 <dustymabe> #action davdunc to put a package review in for ec2-net-utils and brainstorm on how we can use that for #601
16:35:07 <dustymabe> #topic F36: Fedora CoreOS Test Week
16:35:13 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1123
16:35:31 <dustymabe> beta is coming up soon (hopefully next week) and final freeze is starting on April 5th
16:36:07 <dustymabe> we're tight on time but I'd say next week is too soon. Should we try to target a test week for April 5th ?
16:36:13 <travier> .hello siosm
16:36:14 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:36:17 <dustymabe> #chair travier
16:36:17 <zodbot> Current chairs: aaradhak bgilbert dustymabe jbrooks jlebon miabbott mnguyen saqali skunkerk travier
16:37:30 <miabbott> dustymabe: week of april 5 coincides with our team vf2f
16:38:31 <dustymabe> yeah.. the concern is that earlier is "too soon" and later is getting pretty close to final
16:38:41 <dustymabe> any preferences?
16:39:41 * jlebon checks release calendar
16:39:43 <travier> do you we have anyone volunteering to help with the test week?
16:40:17 <dustymabe> travier: nothing so far - would love to have someone volunteer to organize it and run it.. I'll gladly accept delegated responsibility for some pieces
16:40:57 <jlebon> dustymabe: sounds ok to me
16:41:01 <jlebon> re. april 5
16:41:14 <dustymabe> jlebon: which release calendar? this one? https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html
16:41:29 <dustymabe> miabbott: that vf2f doesn't run the whole week does it?
16:41:37 <jlebon> yeah, it's only Wed, and Thu
16:41:40 <miabbott> dustymabe: just apr 6 + 7 right now
16:41:48 <miabbott> (and only part of the day)
16:41:49 <jlebon> dustymabe: yes was looking at that
16:41:54 <dustymabe> we usually call ours a "test week" but then we focus on a single day for most of us
16:42:15 <miabbott> dustymabe: for new people to FCOS, what is required to organize + run the test week for us?
16:42:43 <dustymabe> miabbott: i.e. what's required of someone participating in a test day/week?
16:43:04 <dustymabe> or what's required to organize it all
16:43:04 <miabbott> no, i mean...re:  "would love to have someone volunteer to organize it and run it"
16:43:08 <dustymabe> ahh
16:43:23 <dustymabe> should probably write some of this down in a checklist SOP for next time
16:43:54 <ravanelli> dustymabe: I can help with the tests as well
16:44:16 <dustymabe> but.. we basically look at test cases from last time, identify any need for new test cases (we mostly just point at our documentation) - delegate out writing those new test cases - in the process we update docs when we find cases that need it
16:44:33 <dustymabe> and then "day of" we find people to run the tests and enter test results
16:45:10 <dustymabe> on the organizer side I also try to collect info about who participated and get a new badge created a distrubuted
16:45:26 <dustymabe> and I also usually try to collate some of the outcomes of the test day in an email to send out
16:45:31 <travier> and we might have tshirts maybe to give out this time?
16:45:58 <dustymabe> travier: not sure on that one
16:46:56 <dustymabe> thanks ravanelli
16:47:15 <mnguyen> i can help out with test days too
16:47:23 <dustymabe> anyone else want to propose something other than apr 5?
16:47:25 <davdunc> .hello2
16:47:26 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:47:30 <dustymabe> #chair davdunc
16:47:30 <zodbot> Current chairs: aaradhak bgilbert davdunc dustymabe jbrooks jlebon miabbott mnguyen saqali skunkerk travier
16:47:32 <dustymabe> thanks mnguyen
16:47:44 <miabbott> dustymabe: i'll help document the SOP for future test days
16:48:08 <dustymabe> #action ravanelli and mnguyen and miabbott to work with dustymabe to organize the test day for f36. miabbott will attempt to document the process for future iterations
16:48:20 <miabbott> 👍
16:49:12 <dustymabe> i'll suggest april 5th in the ticket and move on to the next topic here
16:49:18 <dustymabe> #topic Drop libvarlink-utils from package set
16:49:24 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1130
16:49:34 <dustymabe> I opened this ticket for some discussion.
16:49:57 <dustymabe> we added libvarlink-utils a while back to support podman-remote, which used it very very early on
16:50:12 <dustymabe> anything after podman v3 doesn't use varlink at all IIUC
16:50:38 <dustymabe> my proposal is to drop the package from our package list.
16:50:56 <dustymabe> there is some concern with that because I believe you could use varlink with systemd, but not sure how common that is
16:51:46 <jlebon> yeah, it's tricky with these kinds of deps
16:51:51 <jlebon> we had the same thing happen with containerd
16:52:08 <jlebon> in that case, we decided to officialize it
16:52:26 <dustymabe> oh? /me thought docker used containerd (shows how naive he is)
16:52:53 <jlebon> by "the same thing" i mean "shipping a package because it's a dep, but turns out other things could be using it"
16:53:38 <dustymabe> ahh - yeah well there's always anecdotal evidence
16:53:38 <jlebon> i think it's probably safe to remove this though
16:54:02 <dustymabe> yeah, that's where I am - if we want to be cautious we could start by removing it from `next` for a period of time
16:54:06 <dustymabe> and then graduate that
16:54:25 <jlebon> i'm not sure what the roadmap is for varlink -- i have the impression projects are moving away from it in general, but not sure
16:54:40 <dustymabe> yeah, haven't heard much about it recently
16:55:00 <jlebon> ack, dropping it from `next` SGTM
16:55:14 <dustymabe> anybody else with strong opinions here about 1. if we should try to remove it and 2. in what way (A rip it out, B try in next first)
16:55:56 <mnguyen> +1 to dropping
16:56:27 <miabbott> +1 to removal, probably safest to graduate it from next
16:56:29 <mnguyen> +1 to try in next
16:56:47 <mnguyen> it would be like if podman dropped it from its deps in a newer version first
16:57:07 <dustymabe> mnguyen: right
16:57:13 <dustymabe> #proposed We don't know of any other common uses of libvarlink-utils on FCOS right now. We'll try removing it from `next` and send an email about it to the list to see if that gives us any new information. After a period of time with no issues we'll promote that to `testing` and `stable`.
16:57:22 <bgilbert> +1 to drop; starting with next sounds reasonable
16:57:27 <bgilbert> +1
16:57:51 <miabbott> +1 to proposed
16:57:55 <jlebon> +1
16:57:57 <dustymabe> :) - I think I have enough +1 - hoping my wording reflects the discussion well enough
16:58:05 <jlebon> (don't) ship it!
16:58:17 <dustymabe> ha
16:58:30 <dustymabe> #agreed We don't know of any other common uses of libvarlink-utils on FCOS right now. We'll try removing it from `next` and send an email about it to the list to see if that gives us any new information. After a period of time with no issues we'll promote that to `testing` and `stable`.
16:58:35 <dustymabe> look at us.. removing packages!
16:59:13 <dustymabe> #topic Update the VMware metadata to new, modern defaults
16:59:19 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1119
16:59:35 <dustymabe> miabbott: - want to give us a quick update here?
16:59:43 <miabbott> sure think
17:00:18 <travier> (I'm late but +1 for dropping)
17:00:26 <miabbott> bgilbert did fabulous work to support templating the OVF metadata in coreos-assembler, so we are able to use different values for FCOS + RHCOS
17:00:48 <mnguyen> bgilbert++
17:00:48 <zodbot> mnguyen: Karma for bgilbert changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:54 <miabbott> FCOS will remain at hw version 13 until closer to the vSphere 6.5/6.7 EOL date
17:01:45 <bgilbert> specifically, I think we should wait until after EOL :-)
17:01:49 <dustymabe> +1
17:01:52 <dustymabe> thanks miabbott
17:01:54 <dustymabe> #info because of some great work by bgilbert we now have the flexibility to use different values in the OVA for FCOS and RHCOS so each derivative can use values most appropriate. FCOS will remain at hw version 13 until closer to the vSphere 6.5/6.7 EOL date.
17:01:55 <miabbott> the defaults were changed for FCOS to use EFI firmware and Secure Boot by default
17:02:10 <dustymabe> #info the defaults were changed for FCOS to use EFI firmware and Secure Boot by default
17:02:25 <bgilbert> if we're in agreement on that, let's update that #info.  "closer to" implies we're going to pull the rug out before EOL.
17:02:27 <dustymabe> bgilbert++
17:02:31 <dustymabe> miabbott++
17:02:31 <zodbot> dustymabe: Karma for miabbott changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:02:47 <miabbott> the last thing we should probably do is send an email with the new state of the OVA and plans to change hw version after EOL date
17:02:48 <fifofonix> Wasn't there an issue with secure boot (on first boot)?
17:03:16 <bgilbert> fifofonix: yeah, with Ignition.  it's been worked around upstream and in the Fedora package
17:03:24 <dustymabe> #action miabbott to send an email to the list about planned updates to the OVA
17:03:35 <dustymabe> miabbott: see what I did there ^^ :)
17:03:36 <fifofonix> Awesome.
17:03:44 <miabbott> dustymabe: :P
17:03:53 <dustymabe> next topic?
17:03:55 <bgilbert> no
17:04:06 <bgilbert> can we address the "closer to" part please?
17:04:22 <dustymabe> bgilbert: "closer to" EOL ?
17:04:37 <bgilbert> <bgilbert> if we're in agreement on that, let's update that #info.  "closer to" implies we're going to pull the rug out before EOL.
17:04:41 <dustymabe> ahh I missed that
17:04:59 <dustymabe> hmm let me see if I can unroll this
17:05:10 <dustymabe> #undo
17:05:10 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:03:24 : miabbott to send an email to the list about planned updates to the OVA
17:05:14 <dustymabe> #undo
17:05:14 <zodbot> Removing item from minutes: INFO by dustymabe at 17:02:10 : the defaults were changed for FCOS to use EFI firmware and Secure Boot by default
17:05:17 <dustymabe> #undo
17:05:17 <zodbot> Removing item from minutes: INFO by dustymabe at 17:01:54 : because of some great work by bgilbert we now have the flexibility to use different values in the OVA for FCOS and RHCOS so each derivative can use values most appropriate. FCOS will remain at hw version 13 until closer to the vSphere 6.5/6.7 EOL date.
17:05:31 <dustymabe> #info because of some great work by bgilbert we now have the flexibility to use different values in the OVA for FCOS and RHCOS so each derivative can use values most appropriate. FCOS will remain at hw version 13 until after the vSphere 6.5/6.7 EOL date.
17:05:35 <dustymabe> better ^^ ?
17:06:04 <bgilbert> dustymabe: +1, thank you
17:06:07 <dustymabe> #info the defaults were changed for FCOS to use EFI firmware and Secure Boot by default
17:06:31 <dustymabe> miabbott: for that 2nd #info - when is that change going to land? i.e. does it need to also be communicated and planned?
17:06:50 <bgilbert> it landed
17:07:06 <bgilbert> I hope a heads-up is sufficient
17:07:19 <bgilbert> of course it'll go to stable at the same time as testing, because cosa
17:07:23 <dustymabe> yep
17:07:56 <dustymabe> the ignition workaround you mentioned.. will that land in `stable` soon enough?
17:08:12 <bgilbert> uhh
17:08:17 <travier> Where is the issue to track that so that we can have it in the release notes?
17:08:20 <jlebon> bgilbert: it's templated though, no?
17:08:33 <bgilbert> jlebon: the UEFI/Secure Boot part is not templated
17:08:38 <travier> :/
17:08:53 <jlebon> ack, fun.  maybe we should've just to ratchet this in more safely
17:08:57 <travier> Was reading the cosa change and was wondering about that
17:09:16 <bgilbert> good catch.  I think we probably just fast-track the new Ignition package into testing
17:09:22 <bgilbert> the patch has baked in RHCOS forever
17:09:25 <travier> Maybe firmware & secure boot should be templated too
17:09:51 <dustymabe> i'm OK with tagging COSA and using the one from the last time `testing` was built
17:09:57 <travier> Unless we just move everyone to the efi & sb and get rid of that
17:10:07 <bgilbert> travier: the idea is to move everyone to uefi+sb
17:10:11 <travier> ok
17:10:47 <dustymabe> :) well i'm glad I brought the topic up
17:11:11 <travier> https://github.com/coreos/coreos-assembler/pull/2762/files#diff-14140576af10bcdc1ecc6b8cc85c596338ec2b0894adb8d383f7a67ba606e5bdR83  >SB is not enabled here
17:11:21 <davdunc> travier: that will limit access in AWS to nitro instances only.
17:11:25 <dustymabe> I guess let's carry the discussion forward outside of the meeting. Either way if the UEFI+SB change is going to land we need to let people know ASAP
17:11:33 <dustymabe> davdunc: this is only VMWare
17:11:38 <miabbott> travier: https://github.com/coreos/coreos-assembler/pull/2767
17:11:38 <davdunc> ah.
17:11:43 <davdunc> nm then.
17:11:58 <travier> miabbott: +1
17:12:07 <bgilbert> dustymabe: remember to re-action miabbott unless you've changed your mind on that :-P
17:12:12 <miabbott> hehe
17:12:21 <dustymabe> #action miabbott to send an email to the list about planned updates to the OVA
17:13:11 <dustymabe> bgilbert: jlebon: miabbott: travier: let's sync about how to roll this out best after the meeting?
17:13:15 <bgilbert> sure
17:13:18 <miabbott> ack
17:13:19 <jlebon> +1
17:13:25 <dustymabe> #topic Unable to disable zincati.service using Ignition
17:13:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/392
17:13:52 <dustymabe> it's been a week, i didn't get any response to my email
17:14:04 <dustymabe> I guess we can consider the upstream systemd PR dead and move forward with workarounds
17:14:27 <dustymabe> jlebon: it seemed like you had two proposed workarounds. one was more generic and the other was more tactical
17:15:06 <jlebon> yup, one on the f-c-c side, and one in Ignition
17:15:31 <jlebon> the Ignition one would be much simpler IMO
17:15:36 <dustymabe> do we want to try to qualify which approach is preferred here?
17:16:25 * jlebon is interested in bgilbert's opinion re. the Ignition approach
17:16:29 <dustymabe> #info there was no response to last week's request for feedback on https://github.com/systemd/systemd/pull/15205 so we'll pursue a workaround solution.
17:16:48 <jlebon> bgilbert: for reference: https://github.com/coreos/fedora-coreos-config/pull/363#issuecomment-1069323399
17:17:07 <bgilbert> I haven't followed this closely.  is it an FCOS-specific problem, or is it expected to be generic?
17:17:14 <bgilbert> i.e. the way we handle presets
17:17:42 <jlebon> it'll happen to anything which ships enablement symlinks
17:17:49 <travier> generic I think
17:18:17 <bgilbert> Ignition's reliance on presets has been an ongoing source of friction
17:18:49 <jlebon> we should've shipped with no symlinks from day 1, but it's pretty tricky to change that now
17:19:05 <bgilbert> oh, how about that, https://github.com/coreos/ignition/issues/588
17:19:30 <travier> You can't really ship with 0 symlink and do all presets I think
17:20:00 <travier> AFAIK*
17:20:05 <jlebon> bgilbert: hah nice
17:20:05 <bgilbert> we could band-aid this, or maybe we should go further and do all our own symlinking
17:20:22 <bgilbert> in particular, note https://github.com/coreos/ignition/issues/587
17:20:41 <bgilbert> if presetting fails for whatever reason, we continue boot anyway, which is a pretty fundamental violation of Ignition design principles
17:21:25 <bgilbert> ("do it ourselves" might just mean "hand-run systemctl in chroot")
17:21:46 <travier> +1
17:21:51 <bgilbert> sooo
17:21:59 <bgilbert> depends on how ambitious we want to be here, but this could be a good project for someone
17:22:05 <travier> +1
17:22:48 <jlebon> hmm, i'm not 100% re. 587 FWIW
17:22:50 <dustymabe> do we want to enumerate possible solutions first and then pick one?
17:23:03 <jlebon> andrew's point there is a good one
17:23:40 <jlebon> but yes, overall agreed re. having Ignition do this more directly
17:23:51 <bgilbert> dustymabe: if we want to consider a broader solution, I don't think we're ready to do a detailed design in this meeting
17:23:54 <travier> We could make preset a new option for "generated/non-existant/runtime" units and default to doing the enablement explicitly for eveything else
17:24:09 <travier> this would potentially be a breaking change?
17:24:11 <bgilbert> "consider a broader solution" could be an action item though
17:24:32 <dustymabe> who wants to participate in the #action ?
17:25:03 <travier> +1
17:25:11 <bgilbert> travier: breaking how?
17:25:23 <jlebon> i'm happy to help with design
17:25:27 <bgilbert> dustymabe: I assume it'd be the usual Ignition folks
17:25:42 <dustymabe> bgilbert: +1
17:25:45 <dustymabe> i'll name a few
17:25:48 <dustymabe> #action jlebon bgilbert travier to discuss potential solutions for #392 and update the ticket and present the outcome at a future meeting.
17:26:03 <bgilbert> +1
17:26:04 <dustymabe> ready for open floor?
17:26:09 <travier> we would be actively disabling units that we not before (incorrectly) and some runtime units would potentially be not activated if we switched to symlinks
17:26:34 <dustymabe> #topic open floor
17:26:45 <bgilbert> travier: historically we've treated 'fixing incorrect behavior' as a bug, and we'd want to design this in a way that doesn't regress existing cases
17:27:00 <travier> This would not be a spec breaking change
17:27:04 <bgilbert> "as a bug" = routine, rather than breaking change
17:27:08 <davdunc> now that we are in open floor, I am still working on the packaging for ec2-utils / ec2-net-utils. I'll keep you updated on that in the next meeting.
17:27:10 <travier> just need a new spec version I think
17:27:20 <dustymabe> thank you davdunc
17:27:30 <bgilbert> travier: I'd very much like to avoid that.  let's discuss in #fedora-coreos afterward
17:27:36 <travier> ok
17:27:50 <travier> (new minor spec bump)
17:28:29 * dustymabe notes the 5.17 kernel landed - and jforbes applied the patch for AWS instance boot issues, waiting on a new kernel rpm build
17:28:30 <travier> (similar to how we did the mode bits fix)
17:28:47 <bgilbert> the mode bits fix was particularly security-sensitive; it's not the normal way we handle fixes
17:28:51 <dustymabe> i'll fast track it to `next-devel` when that lands
17:28:55 <bgilbert> let's discuss in #fedora-coreos
17:29:26 <dustymabe> #endmeeting