16:30:17 <jlebon> #startmeeting fedora_coreos_meeting
16:30:17 <zodbot> Meeting started Wed Jul  7 16:30:17 2021 UTC.
16:30:17 <zodbot> This meeting is logged and archived in a public location.
16:30:17 <zodbot> The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:17 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:24 <jlebon> #topic roll call
16:30:25 <darkmuggle> .hello2
16:30:26 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:30:34 <jlebon> #chair darkmuggle
16:30:34 <zodbot> Current chairs: darkmuggle jlebon
16:30:41 <bgilbert> .hi
16:30:41 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:44 <jlebon> #chair bgilbert
16:30:44 <zodbot> Current chairs: bgilbert darkmuggle jlebon
16:31:08 <jbrooks> .hello jasonbrooks
16:31:11 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:14 <jlebon> #chair jbrooks
16:31:14 <zodbot> Current chairs: bgilbert darkmuggle jbrooks jlebon
16:31:45 <lucab> .hi
16:31:46 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:13 <jlebon> #chair lucab
16:32:13 <zodbot> Current chairs: bgilbert darkmuggle jbrooks jlebon lucab
16:32:34 <jlebon> #chair jaimelm
16:32:34 <zodbot> Current chairs: bgilbert darkmuggle jaimelm jbrooks jlebon lucab
16:32:56 <jaimelm> .hello2
16:32:57 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:33:29 <jlebon> i expect a smaller crowd today.  i'll leave off the juicier tickets for the video meeting next week, and we can tackle a few smaller ones today.
16:33:50 <jlebon> so it should be a shorter meeting
16:33:57 <lorbus> .hi
16:33:58 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:34:02 <jlebon> #chair lorbus
16:34:02 <zodbot> Current chairs: bgilbert darkmuggle jaimelm jbrooks jlebon lorbus lucab
16:34:26 * jlebon waits 30s more
16:35:05 <jlebon> #topic Action items from last meeting
16:35:26 <travier> .hello siosm
16:35:27 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:35:31 <jlebon> * bgilbert to file ticket to consider switching Ignition/Afterburn to reference GCE metadata server by IP
16:35:34 <jlebon> * jaimelm to investigate F35: CHANGE: CompilerPolicy Change #872 * darkmuggle to investigate F35: CHANGE: DNS Over TLS #873
16:35:37 <jlebon> * jlebon to investigate F35: CHANGE: "Fedora Linux" in /etc/os-release #874
16:35:40 <jlebon> * darkmuggle to investigate F35: CHANGE: More flexible use of SSSD fast cache for local users #875
16:35:43 <jlebon> * lucab to investigate F35: CHANGE: OpenSSL3.0 #876
16:35:45 <jlebon> * jlebon to investigate F35: CHANGE: RPM 4.17 #877
16:35:48 <jlebon> * dustymabe to investigate F35: CHANGE: Remove nscd #879
16:35:50 <jlebon> * walters to investigate F35: CHANGE: DNF/RPM Copy on Write enablement for all variants #878
16:35:53 <jlebon> #chair travier
16:35:53 <zodbot> Current chairs: bgilbert darkmuggle jaimelm jbrooks jlebon lorbus lucab travier
16:35:54 <jlebon> ok, my suggestion is to not tackle the CHANGE ones today and leave that for next week
16:35:59 <bgilbert> #info bgilbert filed https://github.com/coreos/fedora-coreos-tracker/issues/891
16:36:01 <jaimelm> Goo idea
16:36:05 <jaimelm> also, good idea
16:36:32 <jlebon> ehhh, instead of re-actioning them all, let's do:
16:36:59 <jlebon> #action look at the action items from the previous meeting for F35 CHANGE-related items
16:37:57 <jlebon> ok to move on?
16:38:03 <bgilbert> +1
16:38:16 <jlebon> #topic Consider referencing GCP metadata service by IP in Ignition/Afterburn
16:38:19 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/891
16:38:39 <jlebon> bgilbert filed this, but I tagged it for this meeting
16:39:02 <jlebon> from lucab's comment, it sounds like it should be safe to assume the IP will not change
16:39:21 <bgilbert> cool
16:39:39 <lucab> I did ask some GCP contacts, and they pointed out the fact that even Google is hardcoding it in a few places
16:39:48 <bgilbert> +1, thanks lucab
16:39:50 <bgilbert> the idea here is: there have already been two exploits that poison the metadata.google.internal hostname
16:39:57 <jlebon> lucab: +1
16:40:13 <jaimelm> that's helpful info lucab
16:40:24 <bgilbert> and an attack that overrides the metadata service is catastrophic for security
16:40:25 <lucab> they are still asking internally, but I doubt I'll get a more explicit answer anytime soon
16:40:47 <bgilbert> so if we can just assume 169.254.169.254, maybe we should
16:41:03 <jlebon> agree with that, yeah
16:41:18 <lucab> so I guess we can go ahead now, or wait a bunch more days to see if there are further followups
16:41:45 <lucab> but all the signs so far point to "should be fine"
16:41:48 <jlebon> i think this might've been answeredin the previous meeting but: are there any other clouds for which the question should also be asked?
16:42:04 <darkmuggle> my biggest objection is that its not documented
16:42:14 <darkmuggle> right now GCP says to use the DNS
16:42:36 <darkmuggle> and while it should be safe, I'd much rather we get the word from on high that its fine
16:43:24 <darkmuggle> plumbing the depths of my memory during days at $other_employer they use the DNS endpoints for doing image testing
16:43:31 <bgilbert> jlebon: just looked through the Ignition codebase.  the only other provider that uses a hostname is Packet, and that one uses HTTPS
16:43:39 <travier> it is documented as the links from Luca points out
16:43:41 <jlebon> darkmuggle: was lucab's comment in that ticket not convincing?
16:44:07 <darkmuggle> jlebon: well, I missed it
16:44:15 <jlebon> :)
16:44:15 <darkmuggle> nevermind
16:44:23 <darkmuggle> +1
16:44:27 <travier> +1
16:44:34 <jlebon> bgilbert: ack ok
16:44:50 <jlebon> bgilbert: want to do a #proposed?
16:45:14 <lucab> It's kind documented transitively. The metadata server is the internal name server resolver. The internal name server resolver is at 169.254.169.254.
16:45:18 <lucab> *kinda
16:45:20 <bgilbert> #proposed we will switch Ignition and Afterburn to contact the GCP metadata service by IP instead of hostname
16:45:55 <jlebon> +1
16:46:13 <lucab> can we add a "(unless further comments by $date)"?
16:46:14 <darkmuggle> +1
16:46:37 <travier> +1
16:46:47 <bgilbert> lucab: I mean, actions are always subject to revision
16:46:57 <bgilbert> it'll take a while for the PRs to percolate to the OS
16:47:04 <bgilbert> do you have a specific concern?
16:47:27 <travier> we have until this hits stable to revert it
16:47:39 <travier> and even after if this turns out a bad idea
16:47:58 <lucab> yes, this week is a bit of a short/slow one and I asked my contacts just this morning, I still have some hope for some additional feedback
16:48:15 <jaimelm> The sysadmin in my has a built-in revulsion to using hardcoded IPs in code.
16:48:21 <jaimelm> s/my/me/
16:48:49 <jaimelm> +1 on waiting
16:49:18 <jlebon> sounds fine to me too
16:49:21 <bgilbert> lucab: sure, I can slow-roll the PRs
16:49:48 <lucab> bgilbert: that'd be fine
16:50:33 <bgilbert> #action we will switch Ignition and Afterburn to contact the GCP metadata service by IP instead of hostname, unless we receive significant information to the contrary by next week
16:50:37 <bgilbert> ^ good?
16:50:42 <lucab> +1
16:50:46 <darkmuggle> +1
16:50:51 <jlebon> you mean #agreed?
16:50:55 <bgilbert> #undo
16:50:55 <zodbot> Removing item from minutes: ACTION by bgilbert at 16:50:33 : we will switch Ignition and Afterburn to contact the GCP metadata service by IP instead of hostname, unless we receive significant information to the contrary by next week
16:51:03 <bgilbert> #agreed we will switch Ignition and Afterburn to contact the GCP metadata service by IP instead of hostname, unless we receive significant information to the contrary by next week
16:51:08 <darkmuggle> +2
16:51:14 * jlebon nods
16:51:21 <jlebon> cool cool
16:51:41 <jlebon> #topic /boot/ignition/config.ign permissions 0644 to 0600
16:51:46 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/889
16:51:55 <bgilbert> good call
16:52:10 <jlebon> bgilbert: want to introduce it? :)
16:52:14 <bgilbert> sure
16:52:29 <bgilbert> coreos-installer -i or -I writes an Ignition config to /boot/ignition/config.ign
16:52:33 <bgilbert> *coreos-installer install
16:52:53 <bgilbert> Ignition reads it, and then does not clean it up afterward
16:52:55 <bgilbert> and it's world-readable.
16:53:18 <bgilbert> the Ignition config may contain secrets, so we should not leave it lying around like that
16:53:31 <jaimelm> no beuno
16:53:42 <bgilbert> but the question is, how serious is this, and how should we address it
16:53:53 <bgilbert> there's a PR up to make it mode 0600
16:53:54 <jlebon> right
16:54:08 <bgilbert> there's also been some discussion about removing the file outright after Ignition completes
16:54:30 <bgilbert> (note that this only affects coreos-installer scenarios, not cloud images)
16:54:42 <bgilbert> but there's also the question of existing installs.
16:55:11 <jlebon> for comparison on clouds: my understanding is that on some platforms, it sticks around and anyone can request it from the metadata server, right?
16:55:31 <bgilbert> jlebon: yes, and we have open docs bugs to document that fact
16:55:35 <darkmuggle> are you thinking an selinux profile?
16:56:04 <bgilbert> jlebon: but that's a common problem across OSes.  this one is specific to us.
16:56:18 <bgilbert> darkmuggle: doesn't seem worth it.  there's no real reason for us to retain that file at all
16:56:35 <bgilbert> in terms of severity: the file is not normally accessible to containers, and the default core user already has passwordless sudo
16:56:42 <jlebon> bgilbert: right ok.  so i guess the hypothetical question is: if a cloud provided a mechanism to have it removed, then there's no reason we wouldn't leverage it, right?
16:56:51 <travier> I think we should remove it. If it has a local root account password, having it there is an easy local root
16:56:53 <bgilbert> if the user creates additional unprivileged users, they will have access where they shouldn't
16:57:08 <bgilbert> jlebon: correct.  we currently do not, though.
16:57:32 <jaimelm> +1 on removing. In general, a defining a clean up process would be useful moving forward.
16:57:32 <jlebon> right, sure. just trying to establish where we want to go, and we should try to be consistent across cloud and metal
16:57:33 <bgilbert> travier: our docs encourage the use of a strong password hash, fwiw
16:57:45 <jaimelm> invariably, there will be more things to clean up down the road I bet.
16:57:50 <bgilbert> jlebon: +1
16:58:00 <travier> #link https://github.com/coreos/fedora-coreos-docs/issues/306
16:58:04 <travier> for the docs changes
16:58:41 <jlebon> IOW, looking at this at the higher level, our stance should be something like: on platforms where possible, we should scrub the Ignition config after provisioning is complete
16:58:49 <travier> +1
16:58:52 <jaimelm> +1
16:58:56 <bgilbert> that one is for removing the config from the metadata server
16:59:04 <bgilbert> this one is for firewalling the metadata server:
16:59:05 <bgilbert> #link https://github.com/coreos/fedora-coreos-docs/issues/272
16:59:14 <bgilbert> jlebon: +1
16:59:44 <bgilbert> actually that implies that docs#306 should be a code bug instead
17:00:13 <jlebon> yeah, i think you're right
17:00:14 <darkmuggle> +1
17:00:27 <travier> for Ignition? Instances will usually not have enough access to remove user-data
17:00:45 <jlebon> that's the "where possible" bit :)
17:00:56 <bgilbert> travier: ah, it's a control-plane operation?  ouch
17:00:59 <travier> but that could be for the MCO for example
17:01:05 <jaimelm> ^
17:01:18 <jlebon> oh you're talking about the firewall
17:02:00 <bgilbert> jlebon: I wasn't.  I misunderstood
17:02:04 <travier> It's 'gcloud compute instances remove-metadata' so I think you need the corresponding account access
17:02:07 <bgilbert> right
17:02:23 <bgilbert> okay, so, wrt cleaning up our messes
17:03:00 <bgilbert> my initial view is that this isn't a major fire in default configurations, since we're not leaking to anyone unprivileged
17:03:02 <travier> Creating a good 'on boot' fixup to remove all those ignition config files lying there is going to be tricky to not mess the /boot RW remount
17:03:25 <bgilbert> but that we should clean up existing systems nonetheless
17:03:51 <bgilbert> travier: I haven't followed the /boot RO saga.  we can't just remount RW in a mount namespace and do the thing?
17:03:57 <travier> Or we could simply make an announcement and mount an empty RO directory on top of the /boot/ignition instead to avoid doing the remount
17:04:19 <bgilbert> travier: we'd have to keep doing that forever though
17:05:06 <travier> bgilbert: it should work. I vaguely recall issues if something else tries to remount RW at the same tim
17:05:16 <jlebon> remounting rw in a mount namespace sounds like it should work, yeah. ignition-firstboot-complete currently dues that
17:05:38 <darkmuggle> systemd-tmpfile?
17:05:44 <bgilbert> of note, there's possible version skew between coreos-installer and the OS.  but we could fix this by changing firstboot-complete to remove the file + a barrier release that removes it on existing nodes
17:05:49 <travier> bgilbert: forever only if we find the folder in /boot
17:06:04 <bgilbert> and then fixing the perms in coreos-installer is just belt + suspenders redundancy
17:06:05 <jlebon> my strawman is to just ship a systemd service which does this gated on ConditionPathExists, but we don't make it a barrier and instead just drop it until we have a barrier for whatever other reason
17:06:17 <jlebon> darkmuggle: needs rw remount, so can't be just tmpfiles
17:06:21 <travier> systemd-tpmfiles can not remount RW and remove
17:06:27 <darkmuggle> ah
17:06:47 <bgilbert> jlebon: I like that, yeah
17:06:55 <bgilbert> travier: conditionally forever is still forever :-P
17:07:10 <travier> +1 for shipping like jlebon said until we have another barrier
17:07:12 <travier> :)
17:07:21 <jaimelm> +1
17:07:29 <lucab> travier: it can we add an ExecStart :)
17:07:33 <bgilbert> proposed incoming
17:07:38 <lucab> *if we add
17:08:07 <jlebon> lucab: to systemd-tmpfiles.service?  yuck :)
17:08:18 <travier> :)
17:08:20 <lucab> :D
17:08:45 * jlebon waits for bgilbert's proposal
17:08:54 <travier> We will have to make sure this unit is well ordered to not break systems
17:09:06 <bgilbert> #proposed we will change firstboot-complete to delete /boot/ignition/config.ign.  we will also add a systemd service that removes /boot/ignition/config.ign on existing nodes, and ship that service until after the next barrier release.  in parallel, we will change coreos-installer to write /boot/ignition/config.ign with mode 0600.
17:09:29 <jaimelm> well done sir
17:09:32 <jaimelm> ack
17:09:36 <jlebon> +1
17:09:56 <travier> +1
17:10:01 <bgilbert> travier: we have a lot of flexibility in the ordering, thankfully
17:10:32 <jlebon> this will be made easier by what i'm working on with moving firstboot-complete to f-c-c
17:10:38 <bgilbert> jlebon: yup
17:10:46 <jlebon> (i have pending code which makes it a proper script too)
17:10:53 <jaimelm> nice
17:11:12 <jaimelm> have everyone voted? Time to move on?
17:11:25 <bgilbert> #action we will change firstboot-complete to delete /boot/ignition/config.ign.  we will also add a systemd service that removes /boot/ignition/config.ign on existing nodes, and ship that service until after the next barrier release.  in parallel, we will change coreos-installer to write /boot/ignition/config.ign with mode 0600.
17:11:38 <jlebon> s/#action/#agreed/
17:11:50 <bgilbert> ....
17:11:51 <bgilbert> #undo
17:11:51 <zodbot> Removing item from minutes: ACTION by bgilbert at 17:11:25 : we will change firstboot-complete to delete /boot/ignition/config.ign.  we will also add a systemd service that removes /boot/ignition/config.ign on existing nodes, and ship that service until after the next barrier release.  in parallel, we will change coreos-installer to write /boot/ignition/config.ign with mode 0600.
17:11:56 <jaimelm> heh
17:12:00 <bgilbert> also: is there general agreement that we don't need to raise a major alarm about this one?
17:12:06 <bgilbert> #agreed we will change firstboot-complete to delete /boot/ignition/config.ign.  we will also add a systemd service that removes /boot/ignition/config.ign on existing nodes, and ship that service until after the next barrier release.  in parallel, we will change coreos-installer to write /boot/ignition/config.ign with mode 0600.
17:12:38 <bgilbert> should we have a coreos-status post?
17:12:45 <travier> I'm torned on the raising alarm side
17:12:58 <travier> This is just like the mode fixup change
17:13:08 <jlebon> also torn
17:13:08 <bgilbert> ah, right
17:13:17 <darkmuggle> I'm conflicted on raising the alarm myself
17:13:43 <darkmuggle> I can see some some admin being used to seeing the ignition config in /boot
17:13:45 <travier> I think we are really missing a place to bring news to operator in a non alarming way while bringing visibility
17:13:52 <travier> we have the coreos-status
17:13:54 <travier> ml
17:13:55 <jaimelm> It's probably best to wait until the fix is in to avoid undu panic and attempts to take advantage.
17:14:01 <jaimelm> undo*
17:14:19 <darkmuggle> but the config /boot is platform specific
17:14:49 <bgilbert> we could release-note it if we had release notes :-/
17:14:53 <jlebon> did we end up making a post/email for the /etc mode bits?
17:14:59 <jaimelm> travier: agreed.
17:15:14 <travier> no, but I think we should do one
17:15:32 <travier> but we have nowhere to put it
17:15:40 <bgilbert> coreos-status is exactly that place, isn't it?
17:15:43 <jlebon> probably worth combining it then
17:15:46 <bgilbert> jlebon: +1
17:15:50 <travier> beside the ml, which is not really a great format
17:16:06 <jaimelm> This came up last Fall for something I thought we should post
17:16:13 <travier> but it works until we have something better
17:16:18 <jlebon> i think coreos-status is OK, we just need to make sure it's worded well
17:16:25 <jaimelm> there's nothing for static informative stuff.
17:16:45 <travier> Let's do a combined coreos-status mail for those two issues
17:17:37 <bgilbert> #proposed we will post to coreos-status about /etc mode bits and config.ign mode bits, timing to be determined
17:17:48 <jaimelm> ack
17:17:49 <jlebon> +1
17:18:46 <bgilbert> travier: ^ lgty?
17:18:54 <travier> +1
17:19:11 <bgilbert> also darkmuggle
17:19:54 <bgilbert> #action bgilbert to learn the difference between #agreed and #action
17:19:59 <bgilbert> #agreed we will post to coreos-status about /etc mode bits and config.ign mode bits, timing to be determined
17:20:20 <travier> :)
17:20:24 <jlebon> heh
17:20:30 <jlebon> ok cool
17:20:53 <bgilbert> thanks for bringing this up
17:20:57 <jlebon> let's leave it there for today for the tickets since the other ones are larger
17:21:08 <jlebon> #topic Open Floor
17:21:18 <jlebon> anyone has anything to bring up?
17:21:42 <lucab> I was scrolling through some container-related posts and noticed a familiar logo in this one: https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-kubernetes-to-nomad-firecracker-and-kuma
17:21:48 <jlebon> next week *should* be a video meeting
17:22:11 <bgilbert> Ignition 3.3.0 stable spec is in `testing` and will be in the next `stable`
17:22:21 <jlebon> lucab: oh cool!
17:22:27 <bgilbert> corresponding Butane release coming soon
17:22:49 <jaimelm> Open Floor: It might be worth a discussion documentation style in the future. Several times when I've written docs, one person will sign off on it after a few suggested changes. Then someone else comes along with many suggested changes that change the tenor substantially. Perhaps a documentation style guide.
17:22:54 <jlebon> lucab: everytime that happens, it's like seeing your house on TV or something :)
17:23:08 <jlebon> bgilbert: +1
17:23:11 <bgilbert> this spec has kernel-argument support and filesystem format `none`
17:23:41 <bgilbert> jaimelm: are they usually style comments or technical comments?
17:23:42 <jlebon> jaimelm: that's a good point. we don't have a good maintainership story for docs right now
17:24:02 <jaimelm> bgilbert: mostly style
17:24:02 <jlebon> it's kind of "everyone and no one"
17:24:10 <lucab> doesn't Fedora have one?
17:24:25 <jaimelm> used to. is it still around?
17:25:16 <travier> https://github.com/travier/fedora-coreos-nomad > Nomad on top of FCOS demo here
17:25:50 <travier> From my latest FCOS talk: https://twitter.com/siosm/status/1412396467659083779
17:26:16 <jlebon> jaimelm: can you make a tracker issue for this? we can tag it with the meeting label to bring it up in a future meeting
17:26:58 <jaimelm> will do
17:27:00 <jlebon> travier: thanks for giving that talk!
17:27:16 <lucab> I only found https://fedoraproject.org/wiki/Archive:Docs_Project_Style_Guide
17:28:11 * jlebon will close in 60s
17:29:11 <jlebon> #endmeeting