16:32:00 <dustymabe> #startmeeting fedora_coreos_meeting
16:32:00 <zodbot> Meeting started Wed Apr 15 16:32:00 2020 UTC.
16:32:00 <zodbot> This meeting is logged and archived in a public location.
16:32:00 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:32:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:32:00 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:32:07 <dustymabe> #topic roll call
16:32:15 <slowrie> .hello2
16:32:16 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:32:25 <skunkerk> .hello sohank2602
16:32:26 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:32:28 <mnguyen_> kaeso[m], lol
16:32:33 <kaeso[m]> .hello lucab
16:32:33 <mnguyen_> .hello mnguyen
16:32:33 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:37 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:38 <jlebon> .hello2
16:32:40 <jdoss> .hello2
16:32:43 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:46 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:35:36 <dustymabe> welcome all
16:35:42 <dustymabe> sorry I'm just getting a few things lined up
16:36:13 <dustymabe> #chair slowrie kaeso[m] mnguyen_ jlebon jdoss
16:36:13 <zodbot> Current chairs: dustymabe jdoss jlebon kaeso[m] mnguyen_ slowrie
16:36:23 <dustymabe> #chair skunkerk
16:36:23 <zodbot> Current chairs: dustymabe jdoss jlebon kaeso[m] mnguyen_ skunkerk slowrie
16:36:29 <dustymabe> am I missing anyone ?
16:36:50 <dustymabe> #topic Action items from last meeting
16:37:30 <dustymabe> no action items were recorded from the last meeting. jlebon didn't assign any to anybody :)
16:37:46 <dustymabe> which means everyone is doing a great job!
16:38:30 <dustymabe> I'll put out a call real quick for anyone who knows of a ticket that should be discussed to please go add a meeting label to it or add a comment to the issue that you'd like to discuss it in a meeting
16:38:37 * dustymabe starts going through meeting tickets
16:38:49 <dustymabe> #topic audit messages are flooding the console
16:38:57 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/220
16:39:15 <dustymabe> let me provide some recent context on this
16:39:48 <dustymabe> I should probably have opened a new issue because the immediate problem we're trying to solve is slightly different than the problem described in 220
16:40:27 <dustymabe> the audit messages on the console are generally bad, but we need to provide better support for the interactive installer workflow using nmtui and coreos-installer
16:40:44 <dustymabe> the audit messages coming to the console make that experience really bad
16:41:24 <dustymabe> so specifically the problem we need to solve is: 'extraneous messages getting written to console during interactive live ISO experience'
16:41:57 <dustymabe> I have opened 3 PRs that are all different ways of solving the problem, linked to from https://github.com/coreos/fedora-coreos-tracker/issues/220#issuecomment-613803086
16:42:39 <dustymabe> two of them target the Live ISO (i.e. tactically fix the problem, but #220 stays open because we need a more generic fix)
16:43:03 <dustymabe> one of them is a more generic fix but it brings the audit rpm (includes auditd)
16:43:38 <dustymabe> I'd like to get some idea on the right path forward here because the work I'm doing to enhance the installer experience is useless if we don't fix this problem for at least the Live ISO case
16:44:08 * dustymabe waits
16:44:11 <kaeso[m]> I have a preference and bias on #3 for the shorter term, assuming it doesn't break other stuff that the kernel may be printing
16:44:20 <jlebon> dustymabe: i think including auditd eventually might happen. that said, i think just reducing verbosity for the live ISO is something we'd want anyway, right?
16:45:05 <jlebon> yeah, leaning towards #3 as well
16:45:10 <dustymabe> jlebon: not sure on if we'd want it anyway. I haven't seen other messages that are annoying me just yet :)
16:46:21 <dustymabe> any other opinions/thoughts here?
16:46:25 <jlebon> what i mean is, even without a TUI but just a shell prompt, it'd be a nice cosmetic cleanup to not get output/your input interspersed with kernel logs that presumably aren't important
16:46:28 <slowrie> I too would prefer not including auditd yet unless we get significant feedback that users want that included
16:46:48 <slowrie> as has been mentioned in the threads it's not something we could easily yank out later without people potentially depending on it
16:46:51 <jlebon> IIRC we did do that for `cosa run` too for example
16:46:59 <dustymabe> jlebon: yeah, we used to
16:47:04 <dustymabe> we don't anymore
16:47:21 <slowrie> s,auditd,audit rpm
16:48:38 <miabbott> this is an example of where FCOS and RHCOS would diverge
16:48:50 <dustymabe> miabbott: elaborate?
16:48:52 <miabbott> we ship auditd enabled by default in RHCOS
16:49:08 <jlebon> i think this is a good example of where FCOS could drive the fix needed to help RHCOS :)
16:49:18 <dustymabe> miabbott: i know we had talked about it but I didn't recall if we had included it
16:49:53 <dustymabe> miabbott: ahh I see it there now
16:49:55 <jlebon> we're going against the maintainer's recommendations if we just nuke `service`. and offhand it doesn't seem like it'd be too hard to add a super no frills auditcmd that just calls `kill`
16:50:23 <kaeso[m]> miabbott: I'm quite wary the auditd in RHCOS is actually integrated in the OCP flow given https://github.com/linux-audit/audit-userspace/issues/111
16:50:27 <dustymabe> jlebon: yeah I'd really like `auditcmd` upstream and have the rpm stop depending on initscripts
16:51:29 <jlebon> so i guess what i mean is, i'm not against adding auditd. though it doesn't seem like we have to accept the pitfalls just yet
16:51:40 <dustymabe> ok so maybe there is some middle ground
16:51:50 <dustymabe> let me see if I'm reading the room right
16:51:57 <kaeso[m]> I think there is a general consensus that auditd will eventually happen, but the current software+packaging is not up to par
16:52:06 <miabbott> looking over my emails regarding auditd in RHCOS, it looks like we enabled it to check a box on a list of reqs for gov agencies
16:52:21 <miabbott> i don't think we have tight integration with auditd + OCP
16:52:35 <walters> if we did https://github.com/coreos/fedora-coreos-tracker/issues/401 maybe we could kick it out again for new installs or something
16:52:46 <dustymabe> 1. the audit rpm includes initscripts and us nuking that dependency from orbit as is done in https://github.com/coreos/fedora-coreos-config/pull/348 is not ideal
16:53:11 <dustymabe> 2. there are other features of audit that would be nice to have: https://github.com/linux-audit/audit-userspace/issues/111
16:53:29 <dustymabe> 3. RHCOS has auditd in and that represents a delta with FCOS
16:54:07 <dustymabe> so maybe we go with just quieting the log level down for now and try to get the packaging/featureset of auditd up to par with what we want before including it?
16:54:31 <jlebon> +1
16:55:13 <dustymabe> slowrie: kaeso[m]: if the packaging and featureset were "ideal" would you be inclined to include auditd?
16:55:24 <kaeso[m]> ack
16:55:44 <jlebon> i mean, i wouldn't personally block on getting https://github.com/linux-audit/audit-userspace/issues/111 necessarily. definitely would be nice though
16:56:49 <slowrie> If auditd is something that people want I'm not against it; if it's just to solve a side issue that's where I'd prefer a different solution
16:56:50 <dustymabe> #proposed We have some outstanding issues that we'd like to try to push in audit upstream before we include that package in FCOS. For now we will quiet the kernel log level printed to the console of the machine and continue to push upstream for changes to make things more ideal
16:57:25 <kaeso[m]> dustymabe: I don't know auditd well enough to answer, I may have other questions coming to mind (e.g. log rotation, service monitoring)
16:58:12 <dustymabe> kaeso[m]: can you take a look soon(ish) ? the earlier we make feature requests to sooner that can possibly come to us
16:58:57 <jlebon> +1 to proposed
16:59:17 <miabbott> +1 to proposal
16:59:37 <walters> sgtm
17:00:36 <dustymabe> slowrie: kaeso[m]: thoughts? modifications to the proposal?
17:00:53 <kaeso[m]> temporarily, do we want to tweak the sysctl only on live-iso or everywhere?
17:01:15 <kaeso[m]> dustymabe: +1
17:01:58 <dustymabe> #agreed We have some outstanding issues that we'd like to try to push in audit upstream before we include that package in FCOS. For now we will quiet the kernel log level printed to the console of the machine and continue to push upstream for changes to make things more ideal
17:02:43 <dustymabe> kaeso[m]: I think I'd prefer to get other messages on the console (just not the audit ones)
17:03:01 <dustymabe> so that's why I chose to target the live ISO + autologin specifically
17:03:16 <dustymabe> but open to suggestions here
17:03:47 <kaeso[m]> I see. Yes, we can start conservative, that makes sense
17:04:07 <dustymabe> ok next topic...
17:04:22 <dustymabe> oh before that
17:04:55 <dustymabe> I think I'll open a separate issue specifically for 'include audit' which can detail the things we want to fix and links to tracker issues and such
17:05:01 <dustymabe> anyone opposed?
17:05:31 <jlebon> +1
17:05:33 <dustymabe> #action dustymabe to open an 'include audit' ticket to discuss things we want to fix and links to tracker issues
17:05:48 <dustymabe> this will make it easier to not get the problem confused with #220
17:06:17 <kaeso[m]> not opposed, but I would not rush on including it
17:06:55 <dustymabe> right. it's specifically what are all the problems that need to be fixed
17:07:31 <dustymabe> which would be easier to track in a new ticket, rather than #220. WDYT?
17:08:09 <kaeso[m]> yep
17:08:23 <dustymabe> ok next topic
17:08:28 <dustymabe> #topic next stream
17:08:38 <dustymabe> jlebon has been doing some great work here
17:09:22 <dustymabe> jlebon: can you give us an update on where we stand? Fedora 32 is coming out next week we think and it would be great to link to the next stream as part of the release announcement for F32
17:10:16 <jlebon> well, we were initially stuck on a bunch of issues related to lockfiles and the coreos pool etc... most patches are out for that now and awaiting review
17:10:41 <jlebon> however, as a tactical fix i'm suggesting just not using lockfiles for now for f32 since it's mostly frozen anyway
17:10:46 <jlebon> that's https://github.com/coreos/fedora-coreos-config/pull/351
17:11:09 <dustymabe> jlebon: nice, just reviewed that
17:11:18 <jlebon> with that, and upgrade testing enablement, we should have a next-devel pretty soon
17:11:31 <dustymabe> jlebon: can you do a local build and it spits out an artifact ?
17:11:46 <jlebon> and the next step from that is adding a next stream, which is mostly the same work we've done for next-devel but has some extras, like website tweaks
17:12:10 <dustymabe> +1
17:12:31 <jlebon> dustymabe: the previous lockfile iteration passed kola tests after some tweaks. this new build without lockfiles i've checked upgrade testing at least, but haven't done a regular kola run
17:12:41 <dustymabe> what are the chances of doing a next stream release in the next few days ?
17:12:43 <jlebon> so yes, it appears pretty functional :)
17:14:00 <jlebon> i think somewhat likely if we laser focus on it, but i definitely expect more hiccups
17:14:05 <bgilbert> .hello2
17:14:06 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
17:14:07 <kaeso[m]> I know I have already asked this but I keep forgetting: does `next` auto-update too?
17:14:09 <jlebon> just from the fact that it's a new production stream
17:14:38 <jlebon> kaeso[m]: yes, it's a bona fide stream :)
17:14:48 <dustymabe> basically we'd need to get some text to matthew miller by EOD tomorrow for the release annoucnement. We don't have to have the next stream in place by then but I'd like to be like 98% confident if we do that
17:14:58 <jlebon> kaeso[m]: do we need work on the cincinnati side for that?
17:15:13 <kaeso[m]> jlebon: what do you mean with that?
17:15:26 <jlebon> dustymabe: oh if it doesn't *have* to be ready by tuesday, i think that's definitely fine yeah
17:15:32 <kaeso[m]> jlebon: yes, once the metadata files are up we need to point Cincinnati at them
17:16:24 <jlebon> kaeso[m]: right. will we need any config/code change for it to read metadata for the next stream
17:16:41 <jlebon> which i think you just answered yes :)  i was clarifying the question
17:17:17 <dustymabe> jlebon: clarification.. If we give matthew miller text tomorrow I'd like to be 98% confident everything (all links we provide, etc) would be workable by next tuesday
17:18:01 <kaeso[m]> jlebon: yes, the service needs some tweaking
17:18:04 <dustymabe> IOW, it doesn't have to be ready tomorrow (when we hand him the text), but it would by the time the release announcement went out
17:18:09 <jlebon> dustymabe: ahh... than i revert to "somewhat likely" :)   in the worse case though seems OK to be a few days late
17:18:16 <jlebon> s/than/then/
17:18:22 <kaeso[m]> jlebon: I was asking about the "bona fide" part
17:18:35 <dustymabe> right, the problem is I don't know how easy it is to change the text of the release announcement after it's been reviewed
17:18:39 <jlebon> dustymabe: it'd be just a link to the download page, right?
17:18:56 <dustymabe> jlebon: maybe? or maybe a link to our docs that describe the next stream
17:19:05 <dustymabe> which would link to the download page
17:19:34 <jlebon> kaeso[m]: i mean, we'd match the behaviour of the other production streams, where we also auto-update. at least that was my understanding
17:20:26 <jlebon> dustymabe: i think we could work it out so the links definitely work.  e.g. docs page could happen right now even if builds aren't ready
17:20:26 <kaeso[m]> ah ok, so just another production stream following the same release checklist
17:20:39 <dustymabe> jlebon: ok let's chat more after the meeting
17:21:11 <jlebon> +1 to both of you :)
17:21:40 <dustymabe> #topic automated ostree imports into Fedora
17:22:02 <dustymabe> so we worked out the permissions issues in the ostree repos that were holding us back from automating ostree imports
17:22:22 <dustymabe> we now have the coreos-ostree-importer running and automatically importing ostree commits into our ostree repos
17:22:38 <dustymabe> this means we no longer need to open a ticket and ask releng to do something for us every time we do a release
17:22:53 <dustymabe> see https://github.com/coreos/fedora-coreos-streams/pull/82
17:23:36 <dustymabe> I should be able to modify the pipeline code now to fail if the ostree import fails to happen
17:23:37 <jlebon> woah, this is one of those notifications that's still in my backlog
17:23:47 <jlebon> awesome!
17:23:55 <miabbott> nice, this is a huge improvement.  well done dustymabe
17:23:55 <bgilbert> \o/
17:24:32 <dustymabe> also pruning.. we should be able to deploy the fedora-ostree-pruner after we get Fedora 32 out the door and infra unfreezes
17:24:41 <dustymabe> so yay :)
17:24:44 <dustymabe> #topic open floor
17:24:50 <dustymabe> anyone with anything for open floor ?
17:25:35 <kaeso[m]> Docs is getting a lot of love recently, see https://docs.fedoraproject.org/en-US/fedora-coreos/
17:25:37 <jlebon> oh that's nice too re. pruner
17:25:55 <dustymabe> thanks to all who have been contributing to the documenation
17:26:09 * dustymabe wishes he could spell today
17:26:39 <jlebon> really cool the "Provisioning Machines" section
17:27:09 <kaeso[m]> Typhoon too is adding more support FCOS platforms: https://github.com/poseidon/typhoon/pulls?q=is%3Apr+fedora+
17:27:27 <miabbott> in the OKD community meeting, diane was asking about getting someone to do a "state of FCOS" presentation that they could use during the openshift commons during RHT summit.  do we have anyone able to help with that?
17:27:33 <dustymabe> #fcostheworld
17:28:15 <dustymabe> miabbott: I should do it, but I can't get out from under some boulders that are on my shoulders right now
17:28:31 <dustymabe> hopefully I'll feel better about it next week
17:28:57 <miabbott> dustymabe: ping me after the meeting and maybe we can sort something out together
17:29:07 <dustymabe> +1
17:29:22 <dustymabe> 40 seconds til end of meeting time :)
17:30:27 <dustymabe> #endmeeting