16:30:39 <bgilbert> #startmeeting fedora_coreos_meeting
16:30:39 <zodbot> Meeting started Wed Apr 29 16:30:39 2020 UTC.
16:30:39 <zodbot> This meeting is logged and archived in a public location.
16:30:39 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:42 <bgilbert> #topic roll call
16:30:44 <bgilbert> .hello2
16:30:47 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:49 <jlebon> .hello2
16:30:50 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:54 <cyberpear> .hello2
16:30:55 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:06 <jlebon> bgilbert: were you planning to run the meeting?
16:31:14 <bgilbert> jlebon: nope, just filling in
16:31:22 <jlebon> ack, thanks for getting it started! :)
16:31:26 <bgilbert> :-)
16:31:27 <bgilbert> #chair jlebon cyberpear
16:31:27 <zodbot> Current chairs: bgilbert cyberpear jlebon
16:32:47 <jlebon> dustymabe, darkmuggle: wanna say hello? :)
16:33:01 <darkmuggle> ./hello
16:33:07 <darkmuggle> .hello
16:33:07 <zodbot> darkmuggle: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:33:08 <bgilbert> or hello2 even
16:33:24 <jlebon> oh hmm... i guess that's actually not a requirement for meetbot
16:33:29 <dustymabe> .hello2
16:33:30 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:33:33 <skunkerk> .hello sohank2602
16:33:34 <jlebon> #chair dustymabe darkmuggle
16:33:34 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon
16:33:34 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:33:35 <darkmuggle> .hello2
16:33:35 <bgilbert> helpfully, hello takes one argument and hello2 takes... zero
16:33:37 <zodbot> darkmuggle: darkmuggle 'None' <me@darkmuggle.org>
16:33:43 <jlebon> #chair skunkerk
16:33:43 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon skunkerk
16:33:45 <lucab[m]> .hello lucab
16:33:46 <zodbot> lucab[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:51 <slowrie> .hello2
16:33:52 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:34:09 <jlebon> #chair lucab slowrie
16:34:09 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon lucab skunkerk slowrie
16:34:24 <jlebon> alrighty, will start in 30s
16:35:02 <jlebon> #topic Action items from last meeting
16:35:12 <jlebon> -ENOENT
16:35:14 <jlebon> \o/
16:35:33 <dustymabe> \o/
16:35:39 <jlebon> moving on to issues labelled with meeting
16:35:58 <jlebon> hmm, also -ENOENT
16:36:03 <jlebon> was that correct dustymabe ?
16:36:19 <jlebon> or did you mean to tag some things for this meeting?
16:36:59 <dustymabe> nope the ones that had meeting were from last week so I removed them. unless anyone has an issue they'd like to tag with meeting we can move to open floor OR topics related to FYI/general discussion
16:37:06 <miabbott> .hello miabbott
16:37:07 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:37:11 <jlebon> #chair miabbott
16:37:11 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon lucab miabbott skunkerk slowrie
16:37:31 <jlebon> ok cool, thanks
16:37:58 <dustymabe> so ideas for topics
16:38:00 <jlebon> does anyone have a specific ticket in mind they'd like to bring up before we move on to open floor?
16:38:28 <dustymabe> - installer rework discussion: osmet, improved interactive experience
16:38:39 <dustymabe> - next stream updates/status and 'next' steps
16:38:51 * cyberpear has "Open Floor" question about disk encryption
16:39:15 <jlebon> dustymabe: want to start with that first one?
16:39:36 <dustymabe> - summary of autit upstream discussion (if kaeso is around)
16:39:42 <dustymabe> jlebon: sure sounds good to me
16:39:55 <cyberpear> ^ yes, audit!
16:40:00 <jlebon> #topic installer rework: osmet, improved interactive experience
16:40:31 <dustymabe> so we have and are landing a lot of changes to our installer and workflow surrounding the installer
16:40:47 <dustymabe> osmet will allow for offline installs (i.e. all necessary contents are in the ISO)
16:41:05 <dustymabe> and some work I've done recently will allow for an easier interactive static networking experience
16:41:40 <dustymabe> pending work is related to making a sidechannel for configuring static networking on boot for vmware
16:42:05 <dustymabe> kaeso is working on the last item there
16:42:17 <dustymabe> and it all ties back in to https://github.com/coreos/fedora-coreos-tracker/issues/460
16:42:53 <cyberpear> osmet means taking an ostree + some extra metadata and constructing a disk image, kind of like jigdo?
16:43:05 <jlebon> cyberpear: right, exactly
16:43:15 <dustymabe> I'm wondering if we should go ahead and ship one batch of updates (in testing) that includes the osmet + interactive static networking bits
16:43:35 <dustymabe> and then let the rest of https://github.com/coreos/fedora-coreos-tracker/issues/460 land mid next week after that
16:43:53 <dustymabe> or if we should do it all together?
16:44:43 <bgilbert> I'm +1 to shipping osmet early
16:44:46 <jlebon> hmm, apart from osmet and the interactive static network bits, nothing else in theory should require a coreos-installer respin, right?
16:44:59 <dustymabe> jlebon: correct
16:45:00 <bgilbert> get some early real-world experience with it
16:45:24 <dustymabe> so maybe it's just the afterburn + default kargs stuff that we hold back
16:45:31 <dustymabe> or maybe we want it all
16:45:46 <jlebon> ok cool. then yeah, let's do a coreos-installer release for osmet (via --offline) and the networking bits
16:46:22 <miabbott> +1 to shipping osmet + interactive static networking first
16:46:43 <jlebon> i think the way we're working on things, nothing should need to get updated atomically
16:46:48 <dustymabe> lucab[m]: sorry - i've been referring to you as kaeso :)
16:47:50 <lucab[m]> dustymabe: np, I get highlights on both
16:48:41 <jlebon> bgilbert: assuming CI in https://github.com/coreos/coreos-installer/pull/187 is happy, are you good with shipping the next release with https://github.com/coreos/coreos-installer/pull/197 too?
16:48:46 <dustymabe> lucab[m]: thoughts on the current proposal ?
16:48:49 <jlebon> (i.e. osmet by default)
16:49:41 <jlebon> my current approach is having one release where it's gated via --offline first. but it's mostly going to be CI that'll know about that and not users
16:49:45 <bgilbert> jlebon: yes, that's what I was assuming
16:49:58 <bgilbert> we won't get real testing if --offline has to be specified
16:50:04 <jlebon> right
16:50:04 <bgilbert> and we do have a testing stream :-)
16:50:15 <lucab[m]> dustymabe: glueing the afterburn stuff will take several iterations anyway, it shouldn't be held as a blocker for anything
16:50:34 <jlebon> ok cool, WFM then :)  i think the rewiring to have it tested in the coreos-installer CI gives us good confidence
16:50:50 <cyberpear> does that mean --online as a fallback to the old behavior
16:50:57 <dustymabe> lucab[m]: right. I'm saying it's less of a blocker and more of let's hold it back until next testing release is done
16:51:06 <bgilbert> cyberpear: --stream stable
16:51:16 <jlebon> cyberpear: right now, if you specify --stream or --image-url it will never be offline
16:51:26 <cyberpear> I see
16:51:38 <lucab[m]> dustymabe: yep, agreed
16:51:44 <dustymabe> lucab[m]: +1
16:51:46 <jlebon> cyberpear: we might special-case the former to be partially offline if it matches what it already has
16:52:03 <cyberpear> sounds good
16:52:33 <jlebon> ok, are we ready to move on to next topic?
16:53:04 <bgilbert> +1
16:53:17 <jlebon> #topic next stream updates/status and 'next' steps
16:53:43 <dustymabe> we had a nice paragraph in the fedora 32 release annoucnement about our `next` stream
16:53:58 <dustymabe> thanks @jlebon for putting some hard work into getting that out
16:54:26 <dustymabe> we should probably do another release in the stream now that the day0 updates have landed in Fedora 32's stable repos
16:54:30 <dustymabe> and also in `next-devel`
16:54:36 <dustymabe> any other changes we should get in first?
16:54:44 <dustymabe> did the ssh key stuff land?
16:54:54 <bgilbert> not yet
16:55:10 <jlebon> dustymabe: did you mean async release, or just as part of the releases next week?
16:55:15 <dustymabe> ok let's get that in (/me makes a note to review it)
16:55:33 <dustymabe> jlebon: can probably be part of the releases next week
16:55:34 <bgilbert> yup, needs some minor tweaks
16:55:40 <bgilbert> will do that
16:56:26 <jlebon> +1
16:56:30 <dustymabe> any other things we should do in `next` ? also should we talk about timelines for actually switching `testing` to f32 content?
16:57:36 <lucab[m]> do we cover already the F31->F32 edge in CI?
16:57:40 <jlebon> yeah, we should just have a default of e.g. "2 weeks after the first next release after GA" or something. and only hold back if there are known problems
16:57:57 <cyberpear> ^ sounds good to me
16:58:16 <jlebon> lucab[m]: yeah, we need to extend CI there
16:58:28 <dustymabe> jlebon: so
16:58:36 <dustymabe> - next week: first `next` release after GA
16:58:55 <dustymabe> - 3 weeks: first `testing` release based on f32
16:59:06 <dustymabe> - 5 weeks first `stable` release based on f32
16:59:35 <jlebon> lucab[m]: it ironically used to test f31->f32, but now that we have multiple releases within next-devel, it no longer does that. so we need to tweak it so it tests both scenarios (see https://github.com/coreos/coreos-assembler/pull/1351)
16:59:52 <jlebon> dustymabe: yeah, makes sense to me
17:00:04 <dustymabe> i think that WFM
17:00:40 <dustymabe> I think we need to try to promote our next and testing streams as part of this transition
17:00:49 <dustymabe> they are only useful if people are using them and providing us feedback
17:01:10 <dustymabe> I only found the fstrim.service issue because I'm using next (fstrim.timer runs once a week)
17:01:33 <dustymabe> should we put out a coreos-status email or something ?
17:02:17 <bgilbert> normally I'd say coreos-status is only for actual real news
17:02:26 <bgilbert> but "next exists now" counts, actually
17:02:37 <jlebon> seems reasonable to me. make f32 the carrot to use non-stable streams heh
17:03:34 <jlebon> ok, ready for next topic?
17:03:36 <dustymabe> anyone want to sign up to write that email?
17:03:42 <lucab[m]> do we plan to put a barrier in place to have a single auto-update edge crossing major version?
17:03:56 <dustymabe> lucab[m]: I think that would be wise
17:04:01 <jlebon> lucab[m]: that's probably a good idea
17:04:15 <jlebon> because CI doesn't test from anything older than the latest release right now
17:04:25 <dustymabe> cincinnati + zincati ++++++++++
17:05:07 <lucab[m]> ack, I'll take a note somewhere in the tracker to do that for testing and stable
17:05:21 <bgilbert> +1
17:05:54 <dustymabe> +1. thanks all for the discussion
17:06:11 <jlebon> #topic summary of audit upstream discussion
17:06:37 <dustymabe> lucab[m]: i think there was some upstream discussion about https://github.com/coreos/fedora-coreos-tracker/issues/463 ?
17:07:55 <cyberpear> I'm not aware of the upstream discussion, but the for the initscripts tl;dr: "we must audit manual stop or reload of auditd; systemctl doesn't allow this"
17:08:07 <cyberpear> it's security theatre, of course
17:08:22 <lucab[m]> dustymabe: https://github.com/coreos/fedora-coreos-tracker/issues/461 you mean?
17:08:35 <dustymabe> oh wow. yes.. 461
17:08:40 <cyberpear> since I can just systemd-run with conflicts=auditd and it'll get shut down per the conflict
17:09:25 <lucab[m]> on the kernel-side it pretty much stalled with "linux has this API, but it shouldn't be used"
17:10:03 <cyberpear> so if I really want to stop auditd without it being traced to me, it's trivial; despite the official method being "use `service auditd stop`"
17:10:19 <lucab[m]> on the userland-side I think all our tickets are pretty much where they were initially
17:10:26 <lucab[m]> (I didn't check the initscripts story recently)
17:11:13 <dustymabe> ok +1 - i didn't know if the kernel/systemd discussion was relevant in helping us move forward or not
17:11:29 <lucab[m]> on the journald side, audit is now tunable at least: https://github.com/systemd/systemd/pull/15444
17:12:16 <lucab[m]> dustymabe: it was along the line of "can we live without auditd and still get useful info in journald"
17:13:43 <dustymabe> ack
17:14:31 <jlebon> anything else on this topic?
17:14:48 <lucab[m]> tldr is, I still don't see the light at the end of the tunnel, but in the next systemd release we can at least stop the console spammin
17:14:54 <lucab[m]> g
17:15:10 <bgilbert> the kernel folks argue that journald can miss messages
17:15:13 <bgilbert> I don't know how real that concern is?
17:15:32 <bgilbert> if it's real, it seems like we need auditd in some form, or journald needs to explicitly take over that role
17:15:37 <cyberpear> it'd be good to convince the "use service auditd stop" hardliners that since other ways are available to stop the daemon, that only good citizens' actions will be properly audited, so they should work w/ systemd to find a better solution
17:16:07 <cyberpear> they say kdbus would solve it, but it's not upstream
17:17:34 <dustymabe> if we can get initscripts out (working with upstream auditd) and the complex configuration problem solved (working with upstream auditd) I think we can just ship auditd
17:17:54 <dustymabe> so that's at least a possible path forward
17:18:04 <dustymabe> though, will take some work
17:18:14 <lucab[m]> bgilbert: I think it's a real concern, journald support is for "trace debugging" mostly
17:18:23 <lucab[m]> I don't see journald replacing auditd at any point
17:18:34 <bgilbert> yeah
17:18:35 <cyberpear> yeah, journald will never replace auditd
17:18:39 <cyberpear> (sadly)
17:19:03 <jlebon> dustymabe: in your mind, are both of those requirements before inclusion or just the first?
17:19:21 <dustymabe> jlebon: probably just the first, which is also probably the easiest to fix
17:19:24 <jlebon> IMO, the most important is that it doesn't drag in initscripts
17:19:36 <cyberpear> dustymabe: what's the "complex configuration" issue?
17:19:37 <dustymabe> if you look at what auditd uses "service" for it's literally just sending some signals to a process
17:19:46 <jlebon> right
17:20:01 <dustymabe> cyberpear: look at the 3rd bullet point in the description of https://github.com/coreos/fedora-coreos-tracker/issues/461
17:20:14 <cyberpear> is it just the lack of separate /usr and /etc rules?
17:20:42 <cyberpear> do we have rules already we'd want to ship by default?
17:20:50 <cyberpear> I know it already supprots drop-ins
17:21:41 <dustymabe> lucab[m] might be able to speak more to the details
17:21:53 <jlebon> note we have 10 mins left if we want to get to open floor
17:22:28 <lucab[m]> cyberpear: yes, https://src.fedoraproject.org/rpms/audit/blob/f30/f/audit.spec#_156
17:22:50 <lucab[m]> (that was F30, I didn't recheck recently)
17:22:53 <bgilbert> given the security theater aspect of what user sends signals, can we just... ignore that part?
17:23:25 <cyberpear> upstream's position on layered config is "there should be no vendor defaults at all"
17:23:45 <bgilbert> drop initscripts outright and tell people to use systemctl
17:23:51 <cyberpear> for that specific example, it seems an ignition drop-in could overwrite the disablement drop-in
17:24:08 <lucab[m]> cyberpear: except there are already :)
17:24:24 * cyberpear not part of upstream, just commenting
17:25:46 <dustymabe> bgilbert: regarding ignoring initscripts
17:25:55 <dustymabe> we could do that I think
17:26:21 <jlebon> right, but we still need to satisfy the rpm dep (and nuke everything in it) or drop it (much preferred)
17:26:24 <dustymabe> https://github.com/coreos/fedora-coreos-config/pull/348
17:26:48 <dustymabe> ^^ that nukes initscripts
17:27:01 <dustymabe> but I still think we should work with upstream to try to solve this problem
17:27:05 <dustymabe> more people than just us want it
17:27:11 <jlebon> +1
17:27:26 <cyberpear> agreed
17:28:01 <cyberpear> alternative to 348, make fedora-release-coreos `Provides: initscripts` or is that too dirty?
17:28:06 <jlebon> ok, i'm gonna move on to open floor while we still can
17:28:12 <dustymabe> one sec
17:28:18 <jlebon> cyberpear: yuck
17:28:28 <walters> yeah that'd be highly likely to break something
17:28:29 <dustymabe> it seems like we made more progress in here than I thought we would on that particular topic
17:28:31 <cyberpear> jlebon: agreed
17:28:42 <jlebon> #chair walters
17:28:42 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon lucab miabbott skunkerk slowrie walters
17:28:46 <dustymabe> are we at the point we'd want to make a decision about anything?
17:28:57 * dustymabe was just hoping for a summary from upstream
17:29:46 <dustymabe> #proposed we work with upstream to remove initscripts and to support complex configuration split, we don't block on complex configuration split
17:30:04 <jlebon> +1
17:30:08 <dustymabe> alternatively we don't *wait* on either and just forcibly remove files from initscripts
17:30:10 <cyberpear> cyberpear's inferred tl;dr: "no vendor should ship audit defaults", "service command must be used to stop/restart auditd to trace who sent the signal"
17:30:33 <cyberpear> ^ my understanding of upstream positions
17:30:57 <lucab[m]> I'm sad, but I don't think we can reasonably get anything more from upstream in the short term
17:31:42 <dustymabe> lucab[m]: want to make a proposal?
17:33:16 * dustymabe sorry for derailing, but if we are close to a decision I think it's worth it
17:33:39 <jlebon> dustymabe: i think we should at least try to drop initscripts first though before going with the `rm -rf` option, even if it's just making it a subpackage or something that pulls it in and not actually adding a new auditctl command
17:34:01 <cyberpear> how about create a coreos-initscripts-dummy package that has a drop-in to allow systemctl to restart auditd in a normal way
17:34:23 <cyberpear> provides initscripts, or exactly what the auditd package requires
17:34:51 <cyberpear> possible, including updating the auditd package to require exactly the parts of initscripts it needs rather than the whole package
17:35:21 <lucab[m]> dustymabe: your proposal is fine, I meant "more than that"
17:35:45 <jlebon> cyberpear: it needs `service`, which is sad to have on a modern systemd-based distro
17:36:24 <dustymabe> #proposed we work with upstream to remove initscripts and to support complex configuration split, we don't block on complex configuration split. TBD is if we will not wait on the initscripts removal and forcibly remove it ourselves.
17:36:29 <dustymabe> ack/nack?
17:36:41 <cyberpear> +1 to #proposed above
17:36:50 <dustymabe> current #proposed at least progresses us somewhat
17:37:17 <jlebon> sure, +1
17:37:23 <dustymabe> I personally think if we spend a little time with upstream we should be able to get rid of initscripts in a satisfactory way
17:37:54 <bgilbert> +1 to trying.  upstream seems pretty adamant about not sending termination signals as root though.
17:38:13 <dustymabe> right.. it would be in a different way. not through systemd
17:38:24 <dustymabe> but also at least not through initscripts :)
17:38:35 <jlebon> ok, let's wrap this up
17:38:46 <dustymabe> #agreed we work with upstream to remove initscripts and to support complex configuration split, we don't block on complex configuration split. TBD is if we will not wait on the initscripts removal and forcibly remove it ourselves.
17:38:52 <dustymabe> sorry for derailing
17:39:14 <cyberpear> a possible workaround is to wrap systemctl binary with a special case for auditd, to redirect to "service" or equivalent, but again, upstream discussion
17:39:22 <jlebon> #topic open floor
17:39:41 <jlebon> anything anyone wants to quickly bring up?
17:39:43 <cyberpear> is there a tl;dr: for disk encryption?
17:39:48 <cyberpear> otherwise, I'll bring it up next time
17:40:07 <cyberpear> RHCOS has it, what's blocking FCOS?
17:40:42 <jlebon> we want to do it in a more generic/ideal way in FCOS
17:40:51 <dustymabe> yeah I thought that was the case
17:40:53 <cyberpear> is it possible for me to do it the RHCOS way to make a custom FCOS?
17:41:01 <cyberpear> s/possible/easy/
17:41:35 <cyberpear> (if answers are complicated, I'm happy to defer until next week)
17:41:45 <jlebon> it's possible, but we have a bunch of scripts at this point that might be assuming "encrypted = RHCOS"
17:42:04 <jlebon> or to be more precise encrypted-the-way-RHCOS-does-it = RHCOS
17:42:34 <jlebon> though actually most of those scripts live in the RHCOS src config
17:43:01 <cyberpear> is that published somewhere?
17:43:28 <cyberpear> I know someone published a "CentOS Stream RHCOS config" but I think it was a one-time push over a month ago
17:43:41 <jlebon> cyberpear: could you bring it up next week too? i think slowrie could talk about the current state in more details
17:43:46 <cyberpear> sounds good
17:44:05 <slowrie> the new implementation for fcos is still a WIP
17:44:11 <slowrie> it's got a ways to go
17:44:11 <jlebon> the RHCOS src configs aren't public yet  :(
17:44:11 <cyberpear> (hard for community members to compare the two when one is behind a firewall)
17:44:24 <bgilbert> cyberpear: the current RHCOS approach was always intended as temporary
17:44:27 <jlebon> but in general, we work hard to upstream everything in FCOS, which RHCOS inherits from
17:44:29 <bgilbert> cyberpear: let's not perpetuate it :-)
17:44:37 <slowrie> Some of the Ignition work is pushed up as a draft PR but still need some dracut stuff and other config work
17:45:51 <jlebon> cyberpear: roughly, if you're curious, `luks_rootfs: true` in image.yaml is what RHCOS does. you can see the effect that has by grepping it in coreos-assembler and going from there
17:46:01 <slowrie> also for root device encryption it needs additional dracut on top of that as well as some rpm-ostree work
17:46:07 <cyberpear> great! that's exactly what I need to look further
17:46:16 <slowrie> RHCOS works around the need for those things by doing the encryption in COSA during the build
17:46:27 <slowrie> and re-encrypting later
17:46:39 <jlebon> (clarification: no actual encryption happens in cosa :)  )
17:46:48 <cyberpear> yes, null cipher, etc
17:47:11 <cyberpear> I guess that's all on this topic today. thanks for all the info
17:47:14 <jlebon> ok, cool. we're way over at this point. going to close in 30s
17:47:28 <jlebon> cyberpear: thanks for bringing it up! and sorry open floor was rushed
17:47:33 <dustymabe> +1
17:47:42 <cyberpear> jlebon++
17:47:42 <zodbot> cyberpear: Karma for jlebon changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:47:47 <dustymabe> cyberpear++
17:48:05 <jlebon> #endmeeting