16:30:32 <jlebon> #startmeeting fedora_coreos_meeting
16:30:33 <zodbot> Meeting started Wed Jun 17 16:30:32 2020 UTC.
16:30:33 <zodbot> This meeting is logged and archived in a public location.
16:30:33 <zodbot> The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:33 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:33 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:42 <jlebon> #topic roll call
16:30:44 <lucab> .hello2
16:30:45 <jlebon> .hello2
16:30:45 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:48 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:50 <bgilbert> .hello2
16:30:51 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:58 <skunkerk> .hello sohank2602
16:30:59 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:31:01 <darkmuggle> .hello2
16:31:02 <zodbot> darkmuggle: darkmuggle 'None' <me@darkmuggle.org>
16:31:08 <dustymabe> .hello2
16:31:09 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:15 <jlebon> #chair skunkerk lucab bgilbert darkmuggle dustymabe
16:31:15 <zodbot> Current chairs: bgilbert darkmuggle dustymabe jlebon lucab skunkerk
16:31:41 <cyberpear> .hello2
16:31:42 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:32:00 <jlebon> will give 1m08s for people to join in
16:32:12 <jlebon> #chair cyberpear
16:32:12 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jlebon lucab skunkerk
16:32:28 * dustymabe notes he needs to update https://github.com/coreos/fedora-coreos-tracker/issues/519 with the discussion from last week
16:32:42 <dustymabe> will do that as soon as the meeting is over today, removed the meeting label from that one for now
16:33:10 <jlebon> alrighty, let's begin
16:33:11 <lucab> and I should probably followup with the ticket upstream too
16:33:14 <jlebon> #topic Action items from last meeting
16:33:25 <jlebon> None!
16:33:49 <jlebon> anything anyone wants to bring up before we jump into the one issue with a meeting label?
16:34:05 <bgilbert> should we talk about the lua script issue, or do we have that under control?
16:34:19 <dustymabe> +1 for lua script issue
16:34:22 <jlebon> i'm testing a patch right now :)
16:34:40 <bgilbert> jlebon: general, or specific?  :-)
16:34:41 <dustymabe> well, there's a micro fix and a macro fix :)
16:34:43 <bgilbert> right
16:34:47 <jlebon> right
16:34:54 <bgilbert> added meeting label
16:34:54 <jlebon> short-term hack
16:35:21 <jlebon> ok, let's start with that one then
16:35:26 <dustymabe> there is some longer term work that we should discuss if we have free time today
16:35:31 <dustymabe> release notes and metrics
16:35:36 <bgilbert> +1
16:35:38 <dustymabe> there are*
16:35:48 <jlebon> #topic crypto-policies pinned to avoid Lua script
16:35:58 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/540
16:36:20 <jlebon> the TL;DR is that crypto-policies changed recently from baking in a DEFAULT config, to dynamically setting it at install time
16:36:35 <jlebon> to do this, it uses lua to avoid a dependency cycle
16:37:01 <cyberpear> the magic of lua scripts was that you didn't need lua actually installed on the target system since it'd used RPMs internal parser
16:37:01 <jlebon> we've pinned it in FCOS for now, but it's blocking other work
16:37:09 <jlebon> right
16:37:24 <jlebon> there are two issues here:
16:37:29 <jlebon> 1. rpm-ostree doesn't implement lua
16:37:44 <jlebon> 2. the crypto-policies script doesn't make sense for us
16:38:02 <cyberpear> why doesn't lua work w/ rpm-ostree? -- (isn't it using rpm itself?)
16:38:07 <jlebon> so even if we did support lua, we would want to tweak things
16:38:22 <jlebon> cyberpear: https://github.com/coreos/rpm-ostree/issues/749
16:38:26 <dustymabe> jlebon: number 2 is good info
16:38:56 <jlebon> the patch i'm working on is to hack around this for now by simply re-setting the backend to DEFAULT at compose time
16:39:05 <jlebon> and to ignore the lua script
16:39:57 <dustymabe> thanks jlebon for working on that.
16:40:09 <bgilbert> +1
16:40:09 <dustymabe> could we solve this more generically in the future?
16:40:14 <jlebon> not sure if there's more to say here other than working around it for now, and continuing discussions upstream re. (1) lua support and (2) adapting crypto-policies
16:40:33 <bgilbert> #2 is specific to this particular script, but I wanted to bring this up because of #1
16:40:44 <bgilbert> this has happened twice recently
16:40:45 <cyberpear> will this workaround allow us to still see each lua scriptlet that comes up?
16:41:27 <jlebon> cyberpear: yes, it's just special-cased for crypto-policies
16:42:05 <jlebon> the core issue IIRC is that there's no easy way to hand librpm a lua scriptlet to run
16:42:13 <dustymabe> jlebon: what happens if the crypto-policies lua script changes and it does something we do need ?
16:42:29 <dustymabe> maybe that's unlikely
16:43:06 <jlebon> dustymabe: that's a gap in all our "replacement" hacks right now.  we should indeed address that (or just work on supporting lua)
16:43:39 <dustymabe> yeah. do we know how hard supporting lua would be
16:43:39 <cyberpear> can you elaborate a bit on the underlying per script sandbox that is causing the issue? or docs link?
16:44:53 <jlebon> cyberpear: https://src.fedoraproject.org/rpms/crypto-policies/c/9b9c9f7378c3fd375b9a08d5283c530a51a5de34?branch=master
16:44:56 <dustymabe> i know asking rpm maintainers to remove lua support from their scriptlets, when they were encouraged to go there in the first place (by rpm) is a bit of a bad experience as a packager
16:46:12 <bgilbert> dustymabe: +1
16:46:33 <jlebon> yeah, we'll have to address it sooner or later
16:47:14 <cyberpear> I meant " The reason we don't implement Lua is it directly conflicts with our per-script sandboxing (using bwrap)." from https://github.com/coreos/rpm-ostree/issues/749 -- what's the sandbox? (sorry, novice question)
16:47:40 <cyberpear> and why is the sandbox needed -- is it to rewrite the paths so everything is under /usr in the end or something else?
16:48:00 <jlebon> all the scriptlets are run in a bwrap container
16:48:12 <dustymabe> with no network, right?
16:48:13 <jlebon> and we don't want to run rpm directly on the compose server
16:48:34 <jlebon> but there's no way yet to pass rpm in the container the lua script
16:48:38 <jlebon> right
16:49:43 <jlebon> running the scripts in bwrap enforces a separation between the compose server and the rootfs
16:49:49 <dustymabe> ahh, so we need some way to call rpm and just give it a lua script to run ?
16:49:54 <dustymabe> and there is no way to do that ?
16:50:49 <jlebon> and actually this is a great example. crypto-policies wants to set the FIPS backend if fips_enabled=1. but that doesn't make sense for us at compose time
16:51:05 <jlebon> dustymabe: no supported way yeah
16:51:40 <jlebon> Colin had an idea synthesizing RPMs: https://github.com/coreos/rpm-ostree/issues/749#issuecomment-644751946
16:51:50 <lucab> jlebon: my understanding is that the main blocker is librpm which should start exposing an API to receive and execute a lua scriptlet, right?
16:52:00 <dustymabe> have we even proposed with rpm upstream to support that ?
16:52:04 <cyberpear> (so is rpm-ostree re-implementing the rpm install process, then? not using rpm itself to do it?)
16:52:39 <jlebon> dustymabe: i think that's part of what the rpm-ostree upstream issue is about: https://github.com/coreos/rpm-ostree/issues/749#issuecomment-616732128
16:52:50 <jlebon> we should renew/formalize that request
16:53:28 <dustymabe> +1000
16:53:42 <dustymabe> at least if we get an answer there we can know which way we need to go
16:54:07 <jlebon> lucab: kinda, but i'm not sure if we would hand it to librpm outside the chroot anyway. i think walters was saying that having rpm support a separate installroot would be a lot of work
16:54:32 <jlebon> this is not very different from `dnf install --installroot`, though that actually does use rpm itself, so it doesn't hit this
16:55:38 <jlebon> cyberpear: yes, it's re-implementing parts of it
16:55:44 <dustymabe> so a next step would be formalizing a request for `rpm run -p` upstream
16:55:47 <dustymabe> IIUC
16:56:07 <jlebon> +1
16:56:17 <dustymabe> there is a separate topic of "does the lua script in package xyz make sense for us"
16:56:56 <jlebon> yup, exactly
16:56:57 <dustymabe> so even if we do support lua, should we be evaluating each one?
16:57:02 <jlebon> that's more of a case by case basis
16:57:08 <lucab> jlebon: ah gotcha, so librpm is too low-level and missing other primitives too, so a better target is likely to bwrap rpm or some other real command anyway
16:57:10 <cyberpear> that would be relevant for any such script, though
16:57:18 <bgilbert> cyberpear: +1
16:57:27 <bgilbert> jlebon: any reason to think lua scripts will be worse than shell in that regard?
16:57:31 <dustymabe> so basically take them all and fix things when they break?
16:57:42 <jlebon> yes, exactly
16:57:46 <dustymabe> makes sense
16:58:25 <jlebon> lucab: well i guess it could be a librpm binding if that's easier for upstream, and we call into it from a helper running in the container
16:59:20 <jlebon> #proposed we will work around the crypto-policies issue for now in rpm-ostree. meanwhile, we will also pursue a path forward with upstream rpm on supporting running lua scriptlets
17:00:00 <dustymabe> can we change the wording slightly
17:00:16 <dustymabe> supporting running lua scriptlets pass in at runtime
17:00:24 <dustymabe> or something to that effect
17:00:33 <dustymabe> since rpm already supports lua scriptlets
17:00:54 <jlebon> #proposed we will work around the crypto-policies issue for now in rpm-ostree. meanwhile, we will also pursue a path forward with upstream rpm to support passing lua scriptlets at runtime
17:01:02 <dustymabe> ack
17:02:11 <dustymabe> anyone else?
17:02:26 <bgilbert> +1
17:02:42 <lucab> ack!
17:02:49 <skunkerk> +1
17:02:51 <jlebon> #agreed we will work around the crypto-policies issue for now in rpm-ostree. meanwhile, we will also pursue a path forward with upstream rpm to support passing lua scriptlets at runtime
17:02:57 <jlebon> ok, let's move on
17:03:07 <jlebon> #topic F32 rebase tracker for changes discussion
17:03:10 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/372
17:03:27 <jlebon> so we're currently rolling out f32 on stable
17:03:46 <dustymabe> 🥳
17:03:47 <jlebon> which concludes the saga
17:04:10 <jlebon> AFAIK, no major issues reported so far
17:04:30 <dustymabe> i'm crossing my fingers and hoping all goes well
17:05:04 <jlebon> it's set to roll out over three days, so it should be almost halfway rolled out around now
17:05:15 <jlebon> thanks ksinny for doing the releases!
17:05:31 <dustymabe> ksinny++
17:05:31 <zodbot> dustymabe: Karma for sinnykumari changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:05:47 <jlebon> anyone had any other comments about this?
17:05:49 <jlebon> ksinny++
17:05:49 <zodbot> jlebon: Karma for sinnykumari changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:06:05 <dustymabe> jlebon: not stable stream related but it might be worth discussing
17:06:12 <dustymabe> https://github.com/coreos/fedora-coreos-config/pull/481#issuecomment-645038000
17:06:33 <jlebon> dustymabe: let's discuss that next?
17:06:37 <dustymabe> +1
17:06:50 <lucab> I'm surprised nobody complained yet about anything broken in the new docker
17:07:21 <dustymabe> lucab: i.e., the bump from 18.09 to 19.03
17:07:31 <jlebon> heh, maybe it's a sign lots of people are using podman?
17:08:00 <dustymabe> or maybe it went off without a hitch
17:08:04 <dustymabe> or maybe no one is using FCOS :)
17:08:07 <lucab> or that docker is finally stable across major versions!
17:08:20 <dustymabe> metrics++
17:08:20 <jlebon> 🎉
17:08:51 <jlebon> lucab: it's been better in recent times, no?
17:09:25 <lucab> I think it didn't get worse, yes
17:09:41 <jlebon> heh
17:09:49 <jlebon> ok, cool. will move on to next topic if that's all
17:10:20 <jlebon> #topic potential issues with microcode_ctl rolling out in testing/next
17:10:26 <jlebon> #link https://github.com/coreos/fedora-coreos-config/pull/481#issuecomment-645038000
17:11:11 <jlebon> to answer that question: i think it depends how severe and how widespread it is?
17:11:18 <dustymabe> right, so the microcode_ctl we shipped to testing/next might have some issues booting (should only affect bare metal machines)
17:11:41 <dustymabe> here are some reported issues: https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/31
17:11:47 <dustymabe> https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/33
17:12:28 <dustymabe> I wouldn't have shipped the new microcode_ctl update, but I did chat with the maintainer on monday and they seemed to indicate they thought they were going to be pushing the update to stable
17:12:31 <dustymabe> so I pressed ahead
17:12:54 <dustymabe> but apparently intel released a new update since users were reporting issues
17:13:02 <dustymabe> so there is a new microcode_ctl out
17:13:43 <dustymabe> we should probably push out a new `testing`/`next` at least to make sure this microcode_ctl doesn't hit stable
17:14:00 <dustymabe> but when do we do that is up for some debate
17:14:27 <dustymabe> do we wait for the new update to hit bodhi stable and then do it
17:14:52 <dustymabe> do we pause the rollouts of testing/next and do new builds now
17:15:16 <bgilbert> I don't think we're compelled to wait for bodhi stable
17:15:20 <jlebon> well, hangs are not great :(
17:15:49 <dustymabe> bgilbert: right, we aren't, but that might give us a better feeling about this new update not introducing new problems
17:16:03 <jlebon> yeah that's true
17:16:15 <dustymabe> i.e. what I didn't do last time
17:16:22 <bgilbert> faifr
17:16:23 <bgilbert> fair
17:16:51 <dustymabe> I think we either stop the rollouts and spin now - OR we wait until it hits bodhi stable (presumably later this week) and do it then
17:17:02 <jlebon> IIUC, affected machines can just reboot and rollback right?
17:17:11 <dustymabe> jlebon: correct - manual intervention
17:17:23 <lucab> let's wait unless people start reporting hangs too?
17:17:39 <dustymabe> i'm ok either way I think
17:17:42 <jlebon> yeah, i think i agree
17:18:29 <lucab> are the rollouts already at 100%?
17:18:48 <lucab> (or close)
17:18:51 <dustymabe> I think they were set for 2 days and they started like 22 hours ago
17:19:05 <dustymabe> or something like that
17:19:10 <dustymabe> should be around half done
17:19:39 <jlebon> oh yes, you're right. and stable is about a third done, not half (i said half earlier)
17:19:47 <dustymabe> stable is not affected
17:19:59 <dustymabe> which is good!
17:20:00 <jlebon> right, just correcting my earlier statement in the previous topic
17:20:03 <dustymabe> +1
17:20:24 <lucab> I'd say let them finish and start preparing for a next round soon
17:20:36 <dustymabe> ack
17:21:03 <jlebon> +1
17:21:09 <jlebon> pending the update hitting stable
17:21:10 <bgilbert> +1
17:21:33 <dustymabe> if we start to get user reports of issues we can pull the trigger sooner
17:21:45 <jlebon> shall we move to open floor?
17:22:08 <dustymabe> WFM
17:22:11 <jlebon> #topic Open Floor
17:22:32 <jlebon> anyone has anything?
17:22:38 <dustymabe> jlebon: did we want to find someone to take an action to open a feature request for LUA upstream?
17:23:29 <dustymabe> sorry for LUA in RPM upstream
17:23:29 <jlebon> #action jlebon to open an RFE in upstream RPM for supporting runtime Lua scriptlets
17:24:03 <dustymabe> we can talk about release notes and metrics next time probably
17:24:14 <cyberpear> upstream RPM is just github for issues and PRs, right?
17:24:21 <dustymabe> in short.. i think we're getting to a point where we can start to flesh out those pieces of our story
17:24:33 * cyberpear excited about metrics
17:24:48 <jlebon> for next time: i had a simple proposal to start with re. release notes in https://github.com/coreos/fedora-coreos-tracker/issues/194#issuecomment-606687194
17:24:52 <dustymabe> cyberpear: is that sarcasm? :)
17:24:58 <lucab> dustymabe: metrics == telemetry from pinger?
17:25:00 <cyberpear> not at all
17:25:18 <cyberpear> release notes are also very important
17:25:20 <dustymabe> lucab: right - some way to know if our users are keeping up with updates and how many exist
17:25:49 <dustymabe> IIUC right now we'd be able to see if the transition to F32 is going smooth on the stable stream
17:26:06 <dustymabe> cyberpear++
17:26:23 <dustymabe> lucab: is my understanding accurate?
17:26:35 <lucab> dustymabe: ack. let's try to avoid calling those metrics has it's an already overloaded term
17:26:52 <lucab> dustymabe: yes it is
17:26:54 <dustymabe> ok, so s/metrics/telemetry data/ ?
17:27:02 <ksinny> Thanks dustymabe lucab jlebon for the help with F32 release :)
17:27:18 <lucab> a bit of a bummer is that communishift is temporarily offline
17:27:23 <dustymabe> 👏👏👏
17:27:50 <dustymabe> i feel like it should be coming back soon - /me would need to dig in email to find out
17:27:51 <bgilbert> dustymabe: population statistics?
17:28:28 <dustymabe> wfm - i'll just have to try to remember what to call it :)
17:28:42 <bgilbert> I don't think we've ever really come up with a good name
17:29:12 <jlebon> "pinger data" :)
17:29:14 <lucab> indeed
17:29:17 <dustymabe> metrics seemed most appropriate in my mind, but I agree it's an overloaded term
17:29:45 <lucab> speaking of metrics though, I was hacking around this for Zincati https://github.com/coreos/zincati/pull/312
17:30:21 <cyberpear> "pinger data" is what I had assumed was referred to by "metrics" in this context
17:31:09 <jlebon> lucab: neat stuff
17:31:28 <jlebon> oh hey, looks like we're past the time already
17:31:49 <jlebon> will end meeting in 30s unless anyone wants to bring up anything else
17:32:22 <jlebon> #endmeeting