16:30:34 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:34 <zodbot> Meeting started Wed Feb 22 16:30:34 2023 UTC.
16:30:34 <zodbot> This meeting is logged and archived in a public location.
16:30:34 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:34 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:39 <dustymabe> #topic roll call
16:30:41 <travier> .hello siosm
16:30:42 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:30:44 <dustymabe> .hi
16:30:45 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:51 <apiaseck> .hi
16:30:52 <zodbot> apiaseck: Sorry, but user 'apiaseck' does not exist
16:31:20 <jlebon> .hello2
16:31:21 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:29 <travier> apiaseck: you need to give the bot your Fedora account
16:31:50 <apiaseck> travier: thanks
16:32:10 <dustymabe> #chair travier apiaseck jlebon
16:32:10 <zodbot> Current chairs: apiaseck dustymabe jlebon travier
16:32:37 <dustymabe> I think bgilbert was possibly not going to make it today
16:32:49 <dustymabe> let's get started here soon
16:33:09 <apiaseck> .hello c4rt0
16:33:10 <zodbot> apiaseck: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:33:24 * dustymabe 👋 dbelyavs
16:33:54 <jlebon> hi dbelyavs!
16:33:58 <marmijo[m]> .hi marmijo
16:33:59 <zodbot> marmijo[m]: Sorry, but user 'marmijo [m]' does not exist
16:34:01 <jlebon> #chair dbelyavs
16:34:01 <zodbot> Current chairs: apiaseck dbelyavs dustymabe jlebon travier
16:34:08 <dbelyavs> dustymabe, jlebon thanks
16:34:29 <dustymabe> #topic Action items from last meeting
16:34:32 <dustymabe> #chair marmijo[m]
16:34:32 <zodbot> Current chairs: apiaseck dbelyavs dustymabe jlebon marmijo[m] travier
16:34:46 <dustymabe> #info no action items from last meeting 🎉
16:34:58 <dustymabe> #topic rawhide: upgrades fail with new openssh-9.0p1-10.fc38.1
16:35:02 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1394
16:35:34 <dustymabe> so we have dbelyavs here to discuss the openssh change in rawhide to enforce more strict file permissions on host keys
16:35:40 <dustymabe> thank you for joining us dbelyavs
16:35:50 <dbelyavs> Thank you for the invitation!
16:36:33 <dustymabe> dbelyavs: so for full context we hit this problem because of the way OSTree works (i.e. scriptlets are always run against a pure fresh install root and never on the upgrading system directly)
16:36:34 <dbelyavs> I'm OK with the proposed patch if it covers both normal upgrade (as does mine current) and your specific one.
16:37:23 <dbelyavs> We anyway have problems with non-standard key names but I think relnotes will be sufficient
16:37:51 <dustymabe> dbelyavs: indeed - i think the suggested patch is a step in the right direction, but the discussion in https://src.fedoraproject.org/rpms/openssh/pull-request/39# has opened up more questions
16:38:14 <dustymabe> for example: https://src.fedoraproject.org/rpms/openssh/pull-request/39#comment-129116
16:38:43 <dustymabe> so I'm wondering if we shouldn't take a more conservative approach here
16:38:58 <dustymabe> which unfortunately is more work
16:39:17 <jlebon> even with the default paths, the failure mode is scary (loss of SSH access) :)
16:39:37 <dustymabe> jlebon: correct, which for some people is their only access
16:39:38 <travier> dbelyavs: Which patch are you thinking about exactly?
16:39:43 <dbelyavs> I doubt. We could reparse the configuration file, but it looks an overkill. I agree that the conseuences are scary
16:39:45 <travier> has someone made a new patch?
16:39:48 <dustymabe> travier: I think https://src.fedoraproject.org/rpms/openssh/pull-request/39#
16:40:02 <bgilbert> .hi
16:40:03 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:40:05 <dbelyavs> travier, yes, https://src.fedoraproject.org/rpms/openssh/pull-request/39#
16:40:06 <dustymabe> #chair bgilbert
16:40:06 <zodbot> Current chairs: apiaseck bgilbert dbelyavs dustymabe jlebon marmijo[m] travier
16:40:49 <dustymabe> travier: we just started chatting more in the PR discussion about the implications of the overall change
16:40:54 <travier> This patch is just doing what the current change was doing but for everyone at a different time
16:40:59 <dustymabe> and that's why we're here, to talk through the overall change some more
16:41:08 <travier> it does not handle other cases but the original change does not either
16:41:09 <dustymabe> travier: indeed, there's nothing wrong with your patch
16:41:28 <dustymabe> it just happens to be where discussion ensued
16:41:48 <travier> ok, just wanted to make sure I had not missed something
16:42:03 <jlebon> dbelyavs: has ssh in the past done migrations like this where it had to notify users somehow that action was required?
16:42:29 <dbelyavs> jlebon, don't know. Not during 2 years I maintain it.
16:42:35 <dustymabe> dbelyavs: the reason we're bringing this up is because our user's systems auto update (no human interaction), so we need to be very careful about things like t his which could lock people out of their systems
16:42:52 <dbelyavs> I'll try to ask Jakub
16:44:09 <dbelyavs> Jakub also doesn't remember
16:44:09 <jlebon> dbelyavs: do i understand that this is purely a fedora packaging issue and upstream ssh did not have to deal with this?
16:44:25 <dustymabe> jlebon: correct
16:44:36 <dustymabe> i don't think upstream ssh ever allowed the relaxed permissions
16:44:47 <dbelyavs> jlebon, yes. We deviated from upstream and found some side effects (see original proposal). Now we want to return to the upstream approach
16:45:25 <dustymabe> dbelyavs: would you be open to continuing to allow the relaxed permissions for a period of time, while somehow trying to notify users of the nonconforming state of their machine?
16:46:53 <dustymabe> we could even combine that behavior with a datestamp check on the files (i.e. if they were created after a certain date then require the new stricter permissions)
16:46:57 <dbelyavs> dustymabe, I'd prefer not to do this - we had also a special group which is also to be removed.
16:48:14 <dbelyavs> The date idea also may open some attack surface
16:49:31 <jlebon> yes, this would require keeping the special group during the window
16:50:09 <dustymabe> if we used a timeout then we could probably just implement it as a ExecStartPre script
16:50:22 <dustymabe> so wouldn't require patches to the binaries
16:50:36 <dustymabe> though, I guess the hard part there is knowing where the host's keys are
16:50:47 <dustymabe> which could have been configured to be a different location
16:51:20 <dbelyavs> dustymabe, exactly.
16:51:49 <dustymabe> which the openssh binaries know where the hosts files are, so that'd be why it'd be nice to put this logic in the binary
16:53:34 <dustymabe> dbelyavs: so delaying the new stricter permissions isn't desirable and neither is trying to notify the users through some sleep mechanism?
16:54:30 <dustymabe> obviously the files that are in the standard location will get properly updated
16:54:33 <dbelyavs> dustymabe, I'm OK with notifying users with a sleep mechanism outside the binary - but I'm not sure it works on a system startup
16:54:52 <dustymabe> dbelyavs: ok good to know
16:55:10 <dustymabe> dbelyavs: is there anyway to run the ssh binary and ask it to read the config and tell us what the values are?
16:55:46 <dbelyavs> it's not ssh but sshd
16:56:03 <dustymabe> i.e. we could run sshd to tell it to parse the config file and then find out where the host keys are then we could run our external tool on that
16:57:21 <dbelyavs> dustymabe, probably yes
16:57:57 <dustymabe> interesting.. if that's possible then maybe we could just update the PR from travier to ask where the host keys are and update those directly
16:58:13 <dustymabe> either way I think we have some things to go off and investigate
16:58:18 <dbelyavs> sudo sshd -T|grep -i hostkey\s+
16:58:35 <travier> +1
16:58:36 <dbelyavs> It will require the root permissions
16:58:47 <dbelyavs> dustymabe ^^
16:58:57 <dustymabe> dbelyavs: interesting.. cool
16:59:05 <jlebon> i mean, there's still the "scary" part of if this procedure fails, we end up with no ssh
16:59:36 <dbelyavs> jlebon, there is a chance :(
16:59:53 <dustymabe> jlebon: indeed, but I don't think we're going to get around that if there's no grace period implemented by dbelyavs and friends
17:00:04 <jlebon> i'm honestly more concerned about that than changing the default paths of the hostkeys
17:01:05 <dustymabe> in an ideal world I think we'd roll it out like this: for f38 run the migration script and make sshd sleep 10m if the keys are non-conforming -> for f39 no sleep, just fails
17:01:10 <jlebon> dbelyavs: i suspect Fedora Server may also be interested to know about this
17:01:22 <dustymabe> jlebon: I assume they reviewed the change request
17:01:39 <dustymabe> i think we knew about the change too, just didn't think through the failure scenario
17:01:42 <jlebon> dustymabe: i mean highlighting the failure mode
17:01:43 <dbelyavs> jlebon, I agree - the system-wide change is exactly for that purpose
17:02:44 <dustymabe> jlebon: if we wanted to give our users some grace period we could ship the sshd from f37 for a period of time
17:02:51 <dustymabe> and implement the sleep ourselves
17:03:11 <travier> dustymabe: that's going to be tricky real fast
17:03:19 <travier> with libraries differences, etc.
17:03:20 <dustymabe> travier: why?
17:03:26 <dustymabe> oh, yeah
17:03:35 <dustymabe> though generally libraries implement backwards compat
17:03:48 <dustymabe> if we were trying to ship f38 sshd in f37 it would be harder
17:03:50 <jlebon> maybe we could send out a coreos-status announcement for it
17:04:06 <jlebon> what makes this extra tricky is that you can't actually do the migration beforehand
17:04:07 <travier> agree that we should move forward and send an announcement
17:04:36 <dustymabe> jlebon: why not? we could ship the same systemd unit travier PRd
17:04:55 <travier> We could ship it in advance
17:04:59 <dustymabe> indeed
17:05:10 <dustymabe> we can ship the migration and the sleep in advance
17:05:19 <travier> would break host key signing now however
17:05:23 <jlebon> dustymabe: but does f37 sshd work with the older setup? i think it relies on the group user
17:05:35 <travier> only for host key signing
17:05:44 <travier> which is a niche use case
17:05:49 <dustymabe> travier: i'm not sure I understand "would break host key signing now however"
17:06:18 <travier> which is why this change is a bad change in the first place. For a niche use case, we're re-introducing a setuid binary in the system
17:06:40 <dustymabe> travier: ehh. I see value in not deviating from upstream
17:06:43 <travier> if we have users that use host key signing now and we change the permissions on the host keys, then it breaks their use case
17:06:54 <dbelyavs> dustymabe++
17:06:54 <zodbot> dbelyavs: Karma for dustymabe changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:07:45 <travier> it's not as bad as a borken sshd server but that's still breaking something
17:07:49 <dustymabe> ok - this has been a productive discussion. I think we have some things to investigate and discuss further, but for now we should maybe move on to other meeting topics. WDYT?
17:07:49 <jlebon> dbelyavs: is there a way to make host key signing still work in f37 with the newer permissions?
17:08:07 <jlebon> that would give us the window we're looking for :)
17:08:13 <jlebon> but having it in f37 instead of f38
17:08:28 <jlebon> anyway, cool to move on for now
17:08:30 <travier> jlebon: I don't think you can, it's the goal of this setgid binary change
17:08:35 <travier> ok to move on
17:08:42 <dbelyavs> jlebon, I doubt
17:09:19 <dustymabe> #topic Platform Request: Microsoft HyperV
17:09:25 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1411
17:09:30 <dustymabe> thanks dbelyavs for joining us
17:09:31 <jlebon> thanks dbelyavs for joining!
17:09:36 <dbelyavs> Thank you! Leaving
17:10:08 <dustymabe> This is a new platform request by baude
17:11:14 <dustymabe> it's good that it looks like an agent isn't required
17:11:43 <bgilbert> AFAICT the main issue is that the host-guest mechanism is odd
17:11:46 <dustymabe> haha `Are there any other platform quirks we should know about?` :)
17:11:50 <dustymabe> It's Windows
17:12:10 <bgilbert> there are key-value pairs, but values are limited to 1K characters
17:12:32 <bgilbert> and the system assumes there is a persistent daemon that the host can push values to
17:13:01 <dustymabe> and the practical requirement of zip
17:13:13 <bgilbert> we're opting not to require that daemon, so Ignition has to implement the protocol itself, and then possibly write out the KVPs so the real daemon can find them later
17:13:15 <bgilbert> also zip, yeah
17:14:22 <bgilbert> it's possible that Afterburn will need hostname support
17:15:03 <dustymabe> my thoughts: I don't really use windows too much, but it would be nice to A) enable users who want to use FCOS on windows B) help podman machine become a success on windows
17:15:16 <bgilbert> oh, and on the host side
17:15:29 <dustymabe> though, CI for something like this would probably be pretty hard
17:15:40 <bgilbert> baude and friends are implementing a program users can run to set the Ignition config
17:16:05 <bgilbert> which chunks an Ignition config into pieces to get around the 1K limit, and sets it up
17:16:19 <bgilbert> our docs would give a download link for the binary
17:16:32 <bgilbert> CI seems hard, yeah
17:16:38 <jlebon> bgilbert: should it live in the ignition repo?
17:17:04 <jlebon> since it's setting it up a precise way meant for ignition
17:17:20 <bgilbert> jlebon: possibly.  I'm not sure.  it's Windows-only of course
17:17:33 <jlebon> or in the coreos/ org i guess
17:17:39 <bgilbert> in principle, other platforms could use such tools also
17:17:53 <bgilbert> which suggests possibly a sidecar repo
17:17:59 <jlebon> anyway, we can flesh that out later
17:18:09 <bgilbert> coreos/ignition-tools or so
17:18:17 <bgilbert> (or, actually, Butane)
17:19:04 <dustymabe> anyone have any concerns about adding this as a platform?
17:20:03 <jlebon> +1 from me
17:20:14 <dustymabe> #proposed We think it's a good idea to enable Windows users who want to use FCOS and the Windows podman machine efforts by adding a hyperv FCOS platform under the `hyperv` platform ID.
17:21:10 <dustymabe> ack/nack?
17:21:16 <jlebon> the first "hyperv" -> Hyper-V. but ack as is too :)
17:21:30 <bgilbert> sgtm with Hyper-V
17:22:10 <dustymabe> #agreed We think it's a good idea to enable Windows users who want to use FCOS and the Windows podman machine efforts by adding a Hyper-V FCOS platform under the `hyperv` platform ID.
17:22:26 <dustymabe> I'll let ^^ stand if no one opposes here in the next few minutes
17:24:16 <dustymabe> #topic open floor
17:24:22 <dustymabe> anybody with anything for open floor?
17:25:43 * dustymabe notes this PR needs two reviews: https://github.com/coreos/fedora-coreos-streams/pull/657
17:25:56 <darknao> hi there o/
17:26:08 <dustymabe> darknao: 👋
17:26:28 <darknao> I might have something for you: did you had time to check the latest update on the new website?
17:26:46 <darknao> https://gitlab.com/fedora/websites-apps/fedora-websites/fedora-websites-3.0/-/issues/89#note_1285250494
17:27:00 <darknao> and https://fedora.gitlab.io/websites-apps/fedora-websites/fedora-websites-3.0/coreos/download/?stream=stable
17:28:11 <dustymabe> darknao: the lowered saturation green/orange look good
17:28:22 <dustymabe> and also the visual indicator for the arch looks good to me too
17:28:32 <dustymabe> anyone else have feedback?
17:28:54 <travier> Like the new margins!
17:29:05 <jlebon> looks great!
17:29:13 <travier> LGTM thanks for doing this!
17:29:27 <jlebon> would it make sense to replace the "Show Downloads" button with the same selector-like behaviour of the arches?
17:29:44 <darknao> so you prefer the color changing theme instead of the current single accent color?
17:30:04 <dustymabe> darknao: I think so
17:30:55 <dustymabe> jlebon: maybe, seems like a nice to have maybe
17:31:10 <jlebon> yeah, i wouldn't block on it
17:31:38 <dustymabe> though.. when I "select" something - I'm not sure I like the fact that it scrolls the page for me - I could possibly be happier without that
17:31:55 <darknao> jlebon: I can try that and we'll see how it looks
17:32:05 <dustymabe> I guess it's trying to get you to move to the next selection
17:32:13 <dustymabe> which makes sense
17:32:16 <bgilbert> I like the scrolling fwiw
17:32:23 <dustymabe> bgilbert: fair
17:32:26 <jlebon> i like it too :)
17:32:32 <darknao> yeah that was the goal here, but we can remove it if you want
17:32:33 <bgilbert> helps drive the user through the flow
17:32:52 <jlebon> i think it only seems annoying when you're just playing about with the interface like we are
17:32:53 <dustymabe> darknao: thanks for working on this. I think we're pretty much at the end
17:33:27 <darknao> thanks for the feedbacks :)
17:33:31 <jlebon> darknao++
17:33:31 <zodbot> jlebon: Karma for darknao changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:33:34 <dustymabe> darknao++
17:33:34 <zodbot> dustymabe: Karma for darknao changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:33:42 <dustymabe> anything else for this meeting for open floor?
17:34:08 <bgilbert> darknao: this looks great, thanks for all the work you've done on this!
17:34:19 <bgilbert> darknao++
17:34:19 <zodbot> bgilbert: Karma for darknao changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:35:14 <dustymabe> #endmeeting