16:31:37 <bgilbert> #startmeeting fedora_coreos_meeting
16:31:37 <zodbot> Meeting started Wed Jun 24 16:31:37 2020 UTC.
16:31:37 <zodbot> This meeting is logged and archived in a public location.
16:31:37 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:37 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:44 <bgilbert> #topic roll call
16:31:47 <bgilbert> .hello2
16:31:48 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:49 <dustymabe> .hello2
16:31:51 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:59 <bgilbert> .chair dustymabe
16:31:59 <zodbot> dustymabe is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
16:32:00 <cyberpear> .hello2
16:32:02 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:32:05 <bgilbert> whoops, sorry :-D
16:32:09 <dustymabe> :)
16:32:11 <bgilbert> #chair dustymabe cyberpear
16:32:11 <zodbot> Current chairs: bgilbert cyberpear dustymabe
16:32:31 <darkmuggle> .hello2
16:32:32 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:32:32 <jlebon> .hello2
16:32:34 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:44 <skunkerk> .hello sohank2602
16:32:45 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:32:56 <lucab> .hello2
16:32:57 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:16 <davdunc> .hello
16:33:16 <zodbot> davdunc: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:33:22 <davdunc> .hello davdunc
16:33:22 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:34:23 <dustymabe> #chair darkmuggle jlebon skunkerk lucab davdunc
16:34:23 <zodbot> Current chairs: bgilbert cyberpear darkmuggle davdunc dustymabe jlebon lucab skunkerk
16:35:03 <dustymabe> #topic Action items from last meeting
16:35:18 <dustymabe> * jlebon to open an RFE in upstream RPM for supporting runtime Lua
16:35:20 <dustymabe> scriptlets
16:35:41 <dustymabe> jlebon: do you have that link handy?
16:36:14 <jlebon> https://github.com/rpm-software-management/rpm/issues/1273
16:36:43 <dustymabe> #info upstream RFE for RPM supporting runtime LUA scriptlets https://github.com/rpm-software-management/rpm/issues/1273
16:36:58 <jlebon> there's some possible APIs we could expand on which would make this easier than initially thought. though waiting for feedback from maintainers
16:37:25 <dustymabe> #topic network interface name differs between Fedora CoreOS and RedHat CoreOS
16:37:31 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/484
16:38:23 <lucab> so far I put up the two PRs and tested them in isolation
16:38:33 <dustymabe> +1
16:38:35 * cverna waives
16:38:48 <dustymabe> oh hai cverna 👋
16:38:49 <lucab> tomorrow I'm planning to figure out how to test the upgrade path
16:38:51 <dustymabe> #chair cverna
16:38:51 <zodbot> Current chairs: bgilbert cverna cyberpear darkmuggle davdunc dustymabe jlebon lucab skunkerk
16:39:28 <jdoss> .hello2 jdoss
16:39:29 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:39:55 <dustymabe> a brief summary of the issue is that we're going to change the default NIC naming from what currently happens on FCOS
16:40:14 <dustymabe> i.e., a breaking change :(
16:40:25 <dustymabe> but the plan is to leave existing systems with the current behavior
16:40:48 <dustymabe> and have newly installed systems get the new behavior
16:41:26 <dustymabe> in order to keep the existing behavior for existing systems we are going to create a barrier release that all upgrades must pass through
16:41:52 <dustymabe> in that release we'll run a script that adds a karg to net.ifnames=0 to keep the current behavior
16:42:03 <cyberpear> (ideally it'd behave as if the existing systems had explicitly opted-in to the old naming scheme, via ignition)
16:42:13 <cyberpear> is this for all systems, or just metal?
16:42:25 <dustymabe> cyberpear: all
16:42:48 <cyberpear> (I think most cloud images use net.ifnames=0)
16:43:11 <lucab> and you cannot really opt-in to that via Ignition, as that logic happens before Ignition
16:43:59 <lucab> other than that, I have to start drafting an email with the details of what we plan to do
16:44:12 <dustymabe> lucab: we probably should consider how to make it easy for people who want to stay on the old scheme even for new installs of new releases
16:44:40 <lucab> dustymabe: I disagree on that
16:45:12 <dustymabe> what part?
16:45:49 <cyberpear> in my experience, "predictable" is unpredictable when you get slightly different virtual hardware
16:45:51 <lucab> making it easy to get the old behavior for new installs
16:46:05 <dustymabe> why?
16:46:14 <jlebon> can't they just `rpm-ostree kargs --append=net.ifnames=0` and reboot?
16:46:21 <cyberpear> but if there's only one interface, it's always eth0 with the net.ifnames=0
16:46:50 <jlebon> sure it won't be right on the first boot, and it costs a reboot, but it should work fine otherwise
16:46:52 <dustymabe> jlebon: they can, right - but it's not great. /me wishes we had the kargs via ignition thing worked out
16:47:09 <cyberpear> jlebon does that work? I thought it had to be set only on first boot
16:47:23 <lucab> because sure, you can do all the customization you want, but then you get a very fragile pet node which is more likely to break on updates
16:47:51 <cyberpear> as long as it was sent by ignition, it's not a pet
16:47:58 <jlebon> cyberpear: hmm, i'd think so but i didn't test it. will let lucab answer that
16:48:09 <bgilbert> +1 to not encouraging this particular customization
16:49:03 <lucab> the "network as kargs" path is already messy as it is, I wouldn't encourage more variations
16:49:11 <dustymabe> I think as a user I'd be a little upset if there wasn't at least some published way to "go back" to the way it was before the breaking change
16:50:29 <dustymabe> now - could we say that we really discourage it in the steps.. sure
16:51:38 <dustymabe> anywho let's talk about timelines
16:52:23 <dustymabe> OKD is trying to do a GA in mid july so it would be nice if we have this change in before that so OKD doesn't have to deal with migrating users either
16:52:29 <bgilbert> dustymabe: I mean, the old way was a bug
16:53:22 <dustymabe> bgilbert: or a feature? I mean the option exists on the command line for a reason. So we either disable anyone making karg changes or we accept that some people are going to make them
16:53:25 <jlebon> FWIW, i just tried this locally and it works fine to specify net.ifnames=0 even with the 99-default.link in place
16:53:42 <jlebon> and it happily switched from systemd naming to eth0 on reboot
16:53:45 <dustymabe> honestly I would encourage people to change to cgroups v2 right now so that we get a little more info on if it works good or not before we switch over
16:54:02 <dustymabe> which is a karg change they'd need to make
16:54:18 <bgilbert> dustymabe: oh, sure, if they want to change kargs using existing mechanisms, +1.  but I don't think we should e.g. add extra sugar.
16:54:33 <dustymabe> right, i'm just saying we should publish the steps
16:54:42 <bgilbert> sure
16:54:51 <dustymabe> and added a side note about how I wish we did have sugar for kargs in general (via ignition)
16:54:57 <bgilbert> +1
16:54:58 <dustymabe> cool I think we're on the same page
16:55:07 <lucab> so, next week we have a triple release
16:55:08 <dustymabe> ok here is a tentative timeline i sketched out
16:55:17 <dustymabe> [This week]
16:55:19 <dustymabe> - coreos-status post about changes incoming
16:55:21 <dustymabe> - roll new testing with fixup script to add net.ifnames=0 (barrier release)
16:55:23 <dustymabe> [Next week - approx June 30]
16:55:25 <dustymabe> - roll out stable with fixup script to add net.ifnames=0 (barrier release)
16:55:28 <dustymabe> - roll out new testing with change to use persistent NIC naming
16:55:29 <dustymabe> [3 weeks - approx July 14]
16:55:31 <dustymabe> - roll out stable with change to use persistent NIC naming
16:56:50 <lucab> it looks fine on the surface
16:56:51 <jlebon> seems reasonable to me
16:57:13 <lucab> so it means we do testing+next this week
16:57:33 <dustymabe> lucab: right - we would have to to meet the short window we're aiming at
16:57:40 <lucab> then we promote to stable next week
16:57:43 <dustymabe> the more time in `testing` the better IMHO
16:58:01 <dustymabe> correct - we promote the barrier release to stable next week
16:58:34 <lucab> more time in testing for the pinning script PR or for the new naming PR?
16:59:06 <dustymabe> pinning script should be fine with a short amount of time in testing
16:59:10 <dustymabe> it either works or it doesn't I guess
16:59:56 <jlebon> we could do the releases next week a bit later in that week to give more bake time to legacy script
17:00:29 <lucab> I prefer Dusty's approach
17:00:38 <dustymabe> #proposed the tentative schedule is to release a barrier stable release next week to convert existing systems to using net.ifnames=0 via karg and then in the following stable release (july 14) deliver the change that will cause NIC naming to get corrected to the desired behavior
17:01:22 <jlebon> lucab: it's the same approach, just delaying the next week one to wednesday or whatever
17:01:45 <lucab> jlebon: yes sorry, I mean his timeline
17:01:49 <dustymabe> yep `approx June 30` is approximate - if we delay a day or two it's OK
17:02:20 <dustymabe> ack/nack?
17:02:24 <jlebon> ack
17:02:33 * Guest24063 lurks - sorry I'm late :)
17:02:58 <lucab> we should try to be a bit more surgical than usual and not bundle a lot of bumps&changes into these releases
17:03:10 <lucab> ack on the proposed timeline
17:03:11 <dustymabe> hi Guest24063 - do you usually go by `Guest24063` ?
17:03:57 <dustymabe> speaking of bumps/changes i think the "don't bring up networking in initramfs if not needed" stuff should be landing somewhere
17:04:18 <dustymabe> is that correct?
17:04:38 <dustymabe> #chair lorbus
17:04:38 <zodbot> Current chairs: bgilbert cverna cyberpear darkmuggle davdunc dustymabe jlebon lorbus lucab skunkerk
17:05:07 <lorbus> it's that time of the again that the IRC bridge forgot my pw :D
17:05:20 <lorbus> thanks for the reminder :)
17:05:22 <bgilbert> dustymabe: I believe that needs an Ignition release?
17:05:35 <dustymabe> #agreed the tentative schedule is to release a barrier stable release next week to convert existing systems to using net.ifnames=0 via karg and then in the following stable release (july 14) deliver the change that will cause NIC naming to get corrected to the desired behavior
17:06:04 <dustymabe> bgilbert: ahh k
17:06:04 <lorbus> awesome^ :)
17:06:07 <jlebon> yes, needs an Ignition respin
17:06:20 <dustymabe> #topic Release notes
17:06:27 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/194
17:06:36 <dustymabe> I think last time we said we wanted to start breaching this topic
17:06:47 * dustymabe reads issue to see where we are
17:07:36 <jlebon> i had a suggestion in https://github.com/coreos/fedora-coreos-tracker/issues/194#issuecomment-606687194 on one way to start
17:07:42 <jlebon> but it's not fully fleshed out
17:08:07 <lucab> the package set and diff webpages are great BTW
17:08:16 <dustymabe> abai__++
17:08:21 <dustymabe> allen has done some amazing work
17:08:51 <dustymabe> jlebon: so in your suggestion we would keep it in the configs github repo
17:08:58 <jlebon> yeah, the website is looking really good now!
17:09:00 <dustymabe> and just source that content on web page load
17:09:25 <jlebon> i think it'd end up in the stream metadata
17:09:28 <lucab> jlebon: I like the -config repo approach but I'm a bit worried about continuous conflicts on the same file
17:10:29 <jlebon> lucab: it's a risk. not sure how much we'll write in there in practice
17:10:54 <dustymabe> i actually think I like it in a repo backed by git
17:11:00 <lucab> what about a `changenotes.d`?
17:11:10 <dustymabe> honestly sometimes we find out relevant information 'AFTER' we do a release
17:11:24 <dustymabe> and being able to update that info on the fly without having to re-run tooling would be nice
17:12:02 <jlebon> so you want getfedora.org to fetch from github.com?
17:12:24 <jlebon> not sure if that would work with CORS protection
17:12:26 <dustymabe> jlebon: maybe - in the worst case github doesn't respond and the page just looks like it does now
17:12:54 <lucab> my gut feeling is that I prefer it in the metadata
17:13:20 <jlebon> it could be a separate metadata which the stream metadata points to, so that we can update it more easily post-release if needed
17:13:48 <dustymabe> :) - i'm starting to run out of fingers to count the number of metadatas we have
17:14:08 <lucab> dustymabe: buy more fingers!
17:14:18 <dustymabe> lucab: maybe you could lend me a hand :)
17:14:28 <jlebon> i think having it as part of the stream metadata to start is fine
17:14:40 <dustymabe> in general I think let's just sprint to getting something - anything
17:14:48 <jlebon> let's not make it more complicated until we need it
17:15:09 <dustymabe> anyone want to list out or take any action items here?
17:15:22 <lucab> stream metadata only covers the current tip of a stream though, no?
17:16:02 <jlebon> sigh, yes you're right. so release metadata it is :)
17:16:20 <dustymabe> how is the release metadata generated ?
17:16:32 <bgilbert> that's fine for our tooling, but release metadata isn't really public
17:16:34 <jlebon> by coreos-meta-translator
17:16:35 <dustymabe> could the "generator" just look at a file backed in the config repo?
17:17:16 <jlebon> the config commit used is in the meta.json
17:17:32 <jlebon> so yes :)
17:17:58 <dustymabe> honestly I think it would be easier if it was just a yaml file on tip of the branch
17:18:23 <dustymabe> so don't worry about commits, just look at latest yaml file, find version that matches this release and use that set of text
17:18:47 <dustymabe> `latest yaml file in $branch`
17:19:06 <bgilbert> maybe we have a CI hook that JSONifies it and copies to the bucket?
17:19:24 <lucab> but we don't know the version till way later in the release process
17:19:35 <dustymabe> lucab: right so we can update it at any time
17:19:42 <dustymabe> new PR
17:20:08 <lucab> ah uhm
17:20:11 <jlebon> hmm
17:20:25 <lucab> so the flow is that we can only retrofit changenotes
17:20:38 <dustymabe> we'd still update it before we did the release
17:21:00 <dustymabe> but yes, because we rely on the versionary.py to set the version we'd need to wait
17:21:13 <dustymabe> we could have a separate fedora-coreos-release-notes repo
17:21:31 <dustymabe> but i'm not in love with that either
17:21:57 <lucab> I like keeping it in -config
17:22:01 <dustymabe> k
17:22:14 <jlebon> having it be in a single file at the tip definitely has advantages.  it's useful for users and us to grep through later on to find out when things happened
17:22:22 <dustymabe> yep
17:22:39 <lucab> what about having snippets in a .d directory, and then using git history to only include the ones that have been committed after the parent of this release?
17:23:10 <dustymabe> lucab: that would help us write release notes as we go
17:23:33 <dustymabe> but I see that as an improvement on what we're currently discussing
17:23:34 <bgilbert> lucab: could also have an "unreleased" snippet (or block in a monolithic file) that gets versioned once we have a version
17:23:50 <bgilbert> like how dch handles debian changelogs
17:23:53 <dustymabe> +1
17:24:31 * dustymabe notes time
17:24:34 <lucab> honestly, all the repos where I have to rebase PRs to fix changelog conflicts are a bit painful
17:24:38 <jlebon> hmm the .d directory avoids the awkward unreleased thing and retro-fitting the version
17:25:09 <jlebon> maybe let's discuss in the ticket?
17:25:28 <dustymabe> sure - i'll tru to summarize
17:25:31 <lucab> jlebon: because it's decoupled from the version
17:25:46 <jlebon> lucab: right, exactly
17:25:48 <dustymabe> I think we settled on at least using tip of each prod stream in the -configs repo
17:25:53 <jlebon> it's also similar to how our lockfiles work
17:26:14 <lucab> dustymabe: I'd say so yes
17:26:18 <dustymabe> cool
17:26:23 <dustymabe> ok moving to open floor soon
17:26:40 <lucab> the only point I'm not totally sold is "retro-editing changenotes"
17:27:08 <bgilbert> lucab: FWIW I've long wanted the ability to edit CL release notes after release
17:27:15 <bgilbert> to maintain a "bugs added" section
17:27:16 <dustymabe> bgilbert++
17:27:54 <jlebon> i don't think those two things are incompatible. we can figure something out
17:27:59 <dustymabe> an "on by the way systemd got bumped in this release and happened to break foo bar baz in a way we didn't discover until $now"
17:28:07 <lucab> eh, but at the same time you don't want an unbound system where doing a release today has to take care of release artifacts/docs from 5 years ago
17:28:07 <dustymabe> s/on/oh
17:28:18 <bgilbert> lucab: it'd be best-effort
17:28:27 <bgilbert> oh, you mean mechanically
17:28:30 <dustymabe> right I think we only do it where it helps us
17:28:47 <lucab> no I mean the technical part, not the human updating the notes
17:28:51 <bgilbert> right
17:29:19 <dustymabe> honestly we should be able to trim the release notes periodicially - git history would have the rest
17:29:37 <dustymabe> the website should only show X number of old releases anyway
17:29:52 <lucab> either we directly point to the live source, or I don't see that working through a publishing/processing mechanism
17:30:09 <dustymabe> let's discuss more in the ticket
17:30:13 <dustymabe> #topic open floor
17:30:24 <dustymabe> sorry for always leaving 0 time for open floor
17:30:35 <dustymabe> anybody with items for today?
17:30:36 <bgilbert> going to land https://github.com/coreos/ssh-key-dir in FCOS testing-devel soon
17:30:58 <dustymabe> bgilbert: I had a question about that - is that implementation and the one we're pursuing upstream similar ?
17:31:02 <bgilbert> so there can be arbitrary fragments ~/.ssh/authorized_keys.d, not just .../{ignition,afterburn}
17:31:14 <bgilbert> dustymabe: AFAIK
17:31:33 <bgilbert> I know lucab is excluding dotfiles, which I think is the only relevant special case
17:31:39 <dustymabe> so if/when openssh ever adopts it then we could just drop ssh-key-dir at the same time we update openssh
17:31:40 <lucab> I've reworked https://github.com/openssh/openssh-portable/pull/70 while at it, but I don't really expect any movement soon
17:31:54 <bgilbert> actually, hmm, the permission checks are slightly different
17:32:05 <cyberpear> bgilbert: can you have multiple AuthorizedKeysCommand's?
17:32:07 <bgilbert> so in a corner case, yes, we could lock someone out
17:32:23 <bgilbert> cyberpear: no.  there's an open issue for ssh-key-dir to support chaining
17:32:24 <lucab> the upstream logic is not set in stone either
17:32:34 <bgilbert> https://github.com/coreos/ssh-key-dir/issues/10
17:32:41 <dustymabe> cool - as long as we try to handle the "succession" case
17:32:51 <dustymabe> where ssh-key-dir is now obsolete
17:32:53 <bgilbert> right
17:33:04 <cyberpear> (the .keysCommand is good for having a domain-joined machine get the keys from domain)
17:33:34 <bgilbert> cyberpear: that's potentially an exciting failure mode
17:33:47 <bgilbert> if someone already has an AuthorizedKeysCommand, the upgrade locks someone out
17:33:55 <cyberpear> yeah, that'd be fun
17:33:56 <bgilbert> either their domain keys, or Ignition-provisioned keys
17:34:15 <dustymabe> `single` for the win!
17:34:18 <lucab> it's more secure that way, you are supposed to SSH into cattle nodes anyway ;)
17:34:25 <lucab> *are not
17:34:43 <dustymabe> :)
17:34:49 <cyberpear> well, there's security, then there's compliance
17:34:57 * dustymabe will close out the meeting in 60 seconds unless there are more open floor topics
17:34:59 <cyberpear> compliance would say that "central auth" is the only way :P
17:35:28 <bgilbert> cyberpear: we generally don't have any compliance
17:35:42 <cyberpear> who's "we" -- our end users will
17:35:45 <lucab> .oO(is an open floor still open if it gets closed)
17:35:59 <bgilbert> i.e., this isn't the only case where some compliance standard gets confused by FCOS
17:36:26 <dustymabe> #endmeeting