16:30:31 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:31 <zodbot> Meeting started Wed May 12 16:30:31 2021 UTC.
16:30:31 <zodbot> This meeting is logged and archived in a public location.
16:30:31 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:31 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:36 <dustymabe> #topic roll call
16:30:39 <dustymabe> .hi
16:30:39 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:40 <bgilbert> .hi
16:30:40 <travier> .hello siosm
16:30:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:45 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:30:59 <jbrooks> .hello jasonbrooks
16:31:00 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:16 <lucab> .hello2
16:31:16 <darkmuggle> .hello
16:31:16 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:31:19 <zodbot> darkmuggle: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:31:38 <darkmuggle> .hello2
16:31:38 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:32:04 <jlebon> .hello2
16:32:05 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:05 <fifofonix> .hello2
16:32:08 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:33:47 <dustymabe> #chair travier jbrooks lucab darkmuggle jlebon fifofonix
16:33:47 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab travier
16:34:05 <dustymabe> #topic Action items from last meeting
16:34:31 <dustymabe> looks like we had a clean slate. No action items from last time.
16:34:51 <skunkerk> .hello sohank2602
16:34:55 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:34:57 <dustymabe> #chair skunkerk
16:34:57 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab skunkerk travier
16:35:22 <dustymabe> I'll go to meeting tickets. Please add the meeting label or otherwise comment in a ticket if you'd like to have it brought up in a meeting.
16:35:32 <dustymabe> #topic Mounts permission denied starting in 33.20210426.3.0
16:35:38 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/818
16:35:59 <dustymabe> I wanted to bring this up to raise awareness
16:36:28 <dustymabe> the most succinct and relevant information is in this comment: https://github.com/coreos/fedora-coreos-tracker/issues/818#issuecomment-834539146
16:37:09 <dustymabe> unfortunately we didn't catch this before it hit our stable stream
16:37:25 <dustymabe> even people running `next`/`testing` woulnd't have seen this unless they deployed fresh nodes
16:37:38 <dustymabe> OR placed new containers that needed this on existing nodes
16:38:34 <dustymabe> podman upstream has landed a fix, but they're working through release candidate releases for podman 3.2 and not likely to do a backport to 3.1
16:39:02 <dustymabe> if we had caught it in `testing` we would have froze podman (probably)
16:40:08 <dustymabe> hopefully we get the podman 3.2 release into next week's `next`/`testing` builds
16:40:17 <travier> We have had a lot of podman issues recently
16:40:33 <travier> wondering how we could improve test coverage
16:40:43 <dustymabe> the pace of development is pretty high in podman land
16:41:02 <travier> which is good
16:41:21 <travier> but feels like test coverage is not there
16:41:26 <dustymabe> travier: there's nothing like actual users running `next`/`testing`
16:41:31 <fifofonix> fifofonix: (note to self, improve lower env testing by scheduling replacement of at least some nodes)
16:41:39 <walters> .hello2
16:41:40 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:41:42 <dustymabe> but in this case it was tricky
16:42:00 <dustymabe> because if it was an existing node the labels were already correct
16:42:38 <dustymabe> travier: I think we can add tests for things as they come up. like we could add a test for this now, but I did get dan to add a test upstream in podman to try to make sure it doesn't go the other way again
16:42:57 <dustymabe> #chair walters
16:42:57 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab skunkerk travier walters
16:43:01 <travier> I don't think it's on us. I've requested release testing for toolbox upstream
16:43:07 <travier> for example
16:43:11 <dustymabe> +1
16:43:48 <dustymabe> #info we hit a podman bug in our stable stream. Fix will come in the podman 3.2 release. Details in https://github.com/coreos/fedora-coreos-tracker/issues/818#issuecomment-834539146
16:43:56 <travier> #link https://github.com/containers/podman/issues/10296
16:44:17 <dustymabe> anthing else we'd like to investigate?
16:44:39 <jlebon> there's also another podman bug we're currently avoiding by pinning to 3.1.2
16:44:54 <jlebon> so whenever we unpin we need to make sure all the fixes we need are in
16:45:10 <dustymabe> jlebon: +1
16:45:17 <jlebon> #link https://github.com/coreos/fedora-coreos-config/pull/1002
16:45:26 * dustymabe moving on to next meeting item in 30s
16:45:28 <fifofonix> more broadly than podman issue, how thin is 'next', 'testing' use?  is there merit in shining a light on importance of it?
16:45:55 <dustymabe> fifofonix: that's a good question. I don't know the answer to that
16:46:08 <dustymabe> there's always merit in shining a light of importance on it IMHO
16:46:11 <bgilbert> +1
16:46:44 <fifofonix> personally i'm interested in knowing more about FCOS usage stats although that may be closely guarded?
16:46:48 <dustymabe> jlebon: can you open that PR against `next-devel` too?
16:47:11 <dustymabe> fifofonix: not closely guarded. though we don't have a lot of data right now
16:47:13 <jlebon> hmm, will the countme data give us per-stream data?
16:47:17 <dustymabe> see https://fedoramagazine.org/getting-better-at-counting-rpm-ostree-based-systems/
16:47:25 <jlebon> i guess only if it's on a different releasever
16:47:36 <jlebon> dustymabe: ack
16:48:36 <travier> We will only get numbers about how many users are running a given "Fedora release" and since how long
16:49:05 <dustymabe> ok moving on to next ticket
16:49:08 <travier> won't be including the stream they are on unfortunately
16:49:22 <travier> +1
16:49:41 <dustymabe> #topic Platform Request: PowerVS (Power Virtual Server)
16:49:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/817
16:50:18 <dustymabe> Looks like a request for a new hardware platform (Power PC architecture in IBM Cloud)
16:50:48 <dustymabe> bgilbert: seems like you've had some dialogue. Any advice how to proceed here?
16:51:43 <bgilbert> AFAICT this is a routine platform request at this point.  I think I agree that `powervs` is a reasonable platform name, and we've been told that the closed-source agent is not mandatory.
16:52:03 <bgilbert> so it should just be a matter of getting the remaining details and implementing the usual things
16:53:13 <dustymabe> yeah, including a ppc64le version of FCOS to be useful
16:53:18 <bgilbert> well, yeah
16:53:25 <dustymabe> :)
16:54:13 <jlebon> so for now this would be just about RHCOS then, right?
16:54:22 <dustymabe> Anybody have any concerns about supporting this platform in our tooling?
16:54:55 <dustymabe> jlebon: I think the goal for them is OpenShift, so yes, but I appreciate trying to be upstream first
16:55:34 <dustymabe> though, if we have some 'ppc machines running in the cloud' then we could possibly do pipeline builds on them and actually have ppc64le content
16:55:37 <lucab> I'm mildly concerned because there are no real references in that thread, and IBM Cloud Gen1 was a mess from a networking point of view
16:55:54 <jlebon> dustymabe: oh yeah, definitely. we should clarify in the ticket though that the outcoem for this does not include outputing FCOS media
16:56:44 <dustymabe> bgilbert: do you have the same concerns as lucab ?
16:57:25 <bgilbert> I'll happily defer to lucab on ibmcloud networking :-)
16:57:52 <lucab> but I guess we'll only discover going forward. If powervs actually has proper DHCP, then it's probably good.
16:58:31 <bgilbert> jlebon: wouldn't we want to produce FCOS media?
16:58:38 <dustymabe> ok. any action for us on this ticket right now? I notice they have an open PR. who can properly review that?
16:59:33 <jlebon> bgilbert: eventually yes once we're multi-arch enabled
16:59:47 <bgilbert> +1
17:00:42 <dustymabe> #info powervs as a new platform seems reasonable
17:00:54 <dustymabe> good with that ^^ ?
17:01:21 <bgilbert> +1
17:01:21 <jlebon> that #info seems reasonable
17:01:29 <lucab> dustymabe: seems reasonable
17:01:33 <dustymabe> :)
17:01:55 <dustymabe> if there is someone who would like to volunteer to work with them it would probably make things go smoother
17:02:02 <lucab> (jlebon is too fast)
17:02:24 <jlebon> :)
17:02:27 <dustymabe> they have an open PR and it's probably best if we don't just let that linger in "unknown status" land
17:03:11 * dustymabe moves on to next meeting topic soonish
17:04:06 <dustymabe> #topic Cannot upgrade from N-2 releases due to missing GPG key
17:04:12 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749
17:04:40 <dustymabe> we discussed this last week and I think the discussion lead us to a few options
17:04:53 <dustymabe> jlebon summarized in https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-832975742
17:05:15 <dustymabe> Sounds like 3. seemed reasonable enough and he added in support to rpm-ostree for that
17:05:33 <dustymabe> remaining item is: https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-834801971
17:06:13 <dustymabe> so the question is whether the logic for that bit should live in zincati or rpm-ostree
17:06:23 <jlebon> spoke to lucab about this, and he had a variation on this which was cleaner
17:06:26 <jlebon> let me make a comment
17:06:31 <dustymabe> +1
17:07:46 <jlebon> https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-839947093
17:08:13 <dustymabe> jlebon: ok, so no real issue/question where the logic should reside?
17:08:29 <travier> my concerns are pretty much as summarized by walters in https://github.com/coreos/rpm-ostree/pull/2819#pullrequestreview-656121731
17:09:40 <dustymabe> hmm. but the commit we actually deploy is gpg verified, right?
17:10:16 <travier> not necessarily afaik
17:10:35 <jlebon> this is not turning off GPG verification
17:10:35 <walters> another thing I forgot about is we have the same signing key across editions, so I think this opens up MITM rebasing (again omitting TLS) between them
17:11:28 <dustymabe> but zincati is going to do the check?
17:11:44 <travier> what check?
17:11:49 <dustymabe> as suggested here: https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-839947093
17:11:53 <travier> zincati is not checking GPG signatures
17:11:58 <walters> no, rpm-ostree does gpg verification via the remote config
17:12:09 <dustymabe> travier: the branch check
17:12:27 <lucab> to my understanding, the commits themselves are signed and there are multiple levels of rollback prevention. Zincati can check the "on same stream" condition
17:12:32 <travier> that does not resolve the gpg check issue
17:13:06 <jlebon> again, to be sure this is understood, GPG verification is still done as usual
17:13:40 <jlebon> libostree still verifies everything it pulls before continuing, and we're not changing that
17:14:23 <jlebon> all this is changing is allowing zincati to tell rpm-ostree to deploy to a specific commit without verifying that it's on the same branch the node is on
17:14:45 <dustymabe> and then zincati will do the branch verification (in a different way)
17:15:07 <travier> then I don't understand how that helps us with GPG verifications errors
17:15:10 <jlebon> but IMO branches are mostly an implementation detail, and we wouldn't be discussing this if we matched the MCO and did pinned commits
17:15:40 <dustymabe> travier: when rpm-ostree does the branch verification it walks the history from HEAD
17:15:41 <walters> i agree with that
17:16:12 <dustymabe> if HEAD is signed with an f34 key and you are on f31 then the gpg verification of the HEAD commit will fail
17:16:26 <dustymabe> (it verifies the commits as it walks the tree)
17:17:16 <dustymabe> what we're changing now is just telling it not to worry about branch verification in rpm-ostree itself, so it won't attempt to walk the tree
17:17:42 <travier> so this still means that we need release barriers for every fedora releases
17:17:46 <travier> to get the new keys
17:17:55 <dustymabe> N-2, yes
17:17:56 <jlebon> for more information, this is actually re-enabling something that rpm-ostree used to do by default: https://github.com/coreos/rpm-ostree/pull/495
17:18:11 <lucab> travier: yes, https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/
17:18:43 <travier> because we did not do that for the latest round of release for the F34 rebase
17:18:57 <dustymabe> right, we decided not to because of this very issue we're discussing
17:19:00 <dustymabe> :)
17:19:05 <travier> yes :)
17:19:15 <jlebon> it's useless right now, because the code for it won't make it into nodes which need it until we at f36 or something :)
17:19:39 <travier> so this is what I was confused about. Sounds reasonable to me now
17:19:45 <lucab> we can retro-add barriers at any time
17:19:46 <travier> oh, indeed
17:19:49 <dustymabe> ok i'll try to wind this topic down
17:19:59 <walters> though hmm actually...ostree has "ref bindings" from https://github.com/ostreedev/ostree/commit/cf16805a2f70026f24eb0b245666efcd4a7c8e44 but...I think rpm-ostree never used them
17:20:35 <walters> which is probably just an oversight
17:20:39 <dustymabe> the one thing we haven't decided (which we brought up last time) is "how old of nodes do we care about?" - will open a separate issue for that if there isn't already one
17:20:45 <walters> but they're clearly better than walking the commit history
17:21:11 <dustymabe> the outcome of that topic will help inform our decision on "what do we do about current really old nodes hitting this problem?"
17:21:19 <jlebon> walters: yeah, although it doesn't guarantee either that the commit comes lives on that branch
17:21:39 <walters> it does because the signature covers that metadata
17:21:51 <lucab> dustymabe: those need a manual intervention anyway
17:21:57 <walters> anyways i'll follow up there on GH
17:22:23 <dustymabe> lucab: yeah
17:22:53 <jlebon> walters: hmm, not following
17:23:06 <dustymabe> anything we could do on the zincati side to "deadend" them so maybe they would get a message ?
17:23:10 <jlebon> but yeah, we can chat on GH (I did look at ref-binding in https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-781453429)
17:23:36 <lucab> dustymabe: I'd say we first fix the issue and the post instructions to manually help stuck nodes
17:23:43 <dustymabe> k
17:23:54 * dustymabe flips to open floor
17:23:57 <dustymabe> #topic open floor
17:24:32 <dustymabe> Timothee is going to post an email to coreos-status soon about CountMe enablement in FCOS
17:24:45 <walters> I have one on https://github.com/coreos/fedora-coreos-tracker/issues/828 !
17:24:50 <lucab> dustymabe: we can mark deadends yes, but very ancient nodes may not yet have the logic to show that to MOTD
17:25:02 <dustymabe> lucab: yeah, we'd have to investigate if it was possible
17:25:14 <walters> So far it's going pretty well, just wanted to highlight it.  I am working on supporting the vmware ova which is the last FCOS artifact not handled (in upstream git master)
17:25:21 <dustymabe> walters: want to add a `meeting` label so we bring it up next time?
17:25:43 <walters> yeah we can talk about it next week
17:26:00 <lucab> dustymabe: I think it may be better not to mark deadends and see if "wget new GPG key to /etc" is enough to bring them back into the usual flow
17:26:08 <jlebon> walters: is this providing bit-for-bit matching of the uncompressed artifacts?
17:26:22 <walters> yes-ish
17:26:35 <jlebon> hehe
17:26:43 <bgilbert> most-bits-for-bit?
17:26:53 <walters> gcp is missing https://github.com/coreos/coreos-assembler/pull/2158 to validate
17:26:53 <bgilbert> :-)
17:27:01 <walters> the ISO for example is indeed bit-for-bit validated
17:27:18 <walters> but that's simple because we ship the squashfs which is 95% of the ISO
17:27:36 <dustymabe> i wonder if this topic would be a good candidate for a video meeting
17:27:37 <travier> lucab: agree with GPG key import
17:27:40 <walters> artifacts which have "internal compression" are harder (AWS and vmware have VMDK)
17:27:58 <dustymabe> seems like a lot of details to work through that could be good for higher bandwidth (and maybe a short presentation)
17:28:00 <travier> dustymabe: good idea, a rehydrator demo would be great!
17:28:06 <walters> yeah agree
17:28:10 <travier> (saying the guy not making the demo :D)
17:28:21 <bgilbert> #info Git repository default branches have been renamed to main; see the coreos@l.fp.o post for a list of repos and instructions on updating local checkouts
17:28:24 <bgilbert> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/RLSS4ZUMAUL7ZPZPFTZERSSBWGZFM4VU/
17:28:31 <walters> nice work on that everyone!
17:28:32 <dustymabe> we could either wait for the first meeting in June (scheduled video meeting) OR schedule one ad-hoc for next week
17:28:43 <dustymabe> bgilbert++
17:28:43 <zodbot> dustymabe: Karma for bgilbert changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:28:46 <dustymabe> travier++
17:28:51 <bgilbert> dustymabe++
17:28:51 <zodbot> bgilbert: Karma for dustymabe changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:28:51 <dustymabe> thanks for working on the branch renaming
17:28:54 <travier> dustymabe++
17:28:54 <zodbot> travier: Karma for dustymabe changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:29:08 <dustymabe> 🍪🍪🍪🍪
17:29:15 <jlebon> agreed, nice work!
17:29:18 <bgilbert> travier++
17:29:18 <zodbot> bgilbert: Karma for siosm changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:29:42 <travier> I guess making video meetings tied to a schedule was not a good idea
17:29:43 <bgilbert> and thanks to everyone else who helped!
17:29:54 <travier> let's make an ad-hoc one
17:29:58 <dustymabe> WFM
17:30:06 <travier> bgilbert++
17:30:06 <zodbot> travier: Karma for bgilbert changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:30:20 <dustymabe> travier: do you want to organize and run it next week (or even the following week)?
17:30:43 <travier> Not making any promise for the upcoming weeks (moving, etc.)
17:30:54 <dustymabe> busy busy
17:31:03 <dustymabe> ok, well we'll see what comes naturally then
17:31:16 <travier> would prefer someone else do it :)
17:31:23 <dustymabe> anything else for open floor?
17:31:48 * dustymabe goes to create a video meeting label
17:32:39 <travier> #link https://fedoramagazine.org/getting-better-at-counting-rpm-ostree-based-systems/
17:32:54 <dustymabe> thanks travier for working on that ^^
17:33:19 <bgilbert> +1
17:33:27 <dustymabe> #endmeeting