16:29:48 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:48 <zodbot> Meeting started Wed Sep 29 16:29:48 2021 UTC.
16:29:48 <zodbot> This meeting is logged and archived in a public location.
16:29:48 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:48 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:53 <dustymabe> #topic roll call
16:29:58 <dustymabe> .hi
16:30:00 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:27 <lucab> .hi
16:30:30 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:30:33 <bgilbert> .hi
16:30:34 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:41 <jlebon> .hello2
16:30:42 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:10 <dustymabe> #chair lucab bgilbert jlebon
16:31:10 <zodbot> Current chairs: bgilbert dustymabe jlebon lucab
16:32:08 <dustymabe> light attendance today.. I know there is a conference going on that some people are attending
16:32:21 <lorbus> .hi
16:32:22 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:32:30 <dustymabe> welcome lorbus :)
16:32:53 <dustymabe> #chiar lorbus
16:33:00 * lorbus waves
16:33:03 <dustymabe> will wait another minute and then get started
16:34:14 <dustymabe> #topic Action items from last meeting
16:34:20 <dustymabe> * lorbus and jlebon to reach out to the containers team to discuss what cri-o versions will be supported and how at the modular level to pull it off (context: https://github.com/coreos/fedora-coreos-tracker/issues/767)
16:34:42 <jlebon> can you re-action that one?
16:34:50 <dustymabe> #action lorbus and jlebon to reach out to the containers team to discuss what cri-o versions will be supported and how at the modular level to pull it off (context: https://github.com/coreos/fedora-coreos-tracker/issues/767)
16:34:51 <jaimelm> .hello2
16:34:52 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:34:55 <lorbus> I haven't come around to doing that. Not sure if jlebon has. Maybe re-action this one?
16:34:57 <dustymabe> #chair jaimelm
16:34:57 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab
16:35:01 <dustymabe> done :)
16:35:07 <jlebon> lorbus: let's get together at some point to chat :)
16:35:29 <dustymabe> #topic [F35] Fedora CoreOS Test Day
16:35:36 <lorbus> jlebon: yes, definitely :)
16:35:40 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/973
16:35:45 <dustymabe> jlebon++
16:35:47 <dustymabe> lorbus++
16:36:19 <dustymabe> we're going to try to have a FCOS test day for the f35 rebase here soon
16:36:31 <dustymabe> since our `next` stream is switched over it should be a good candidate for testing
16:37:08 <dustymabe> in the past we've added test cases that help test our documentation
16:37:19 <dustymabe> since most of our CI automates other testing that we care about
16:37:44 <dustymabe> does that sound reasonable for this iteration?
16:38:10 <jlebon> +1
16:38:20 <jlebon> esp. platforms we don't have any coverage on
16:38:49 <dustymabe> i think sumantro was suggesting we possibly use next week for the "test day"
16:39:08 <dustymabe> we could possibly try to go later if we think that's a bit ambitious
16:39:32 <lucab> It would be good to note down the last F34 release on next and manually test the upgrades from that
16:39:33 <dustymabe> probably need to spend some time before hand identifying where our docs have changed or adding new test cases for new docs that have been added
16:39:45 <dustymabe> lucab: indeed
16:40:47 <jaimelm> +1 on testing day, +1 on identifying docs changes beforehand. Next week sounds ambitious.
16:41:19 <dustymabe> ok, maybe we can push to the following week and schedule a day next week for syncing on test case changes
16:41:26 <jaimelm> ^^
16:41:59 <dustymabe> #action dustymabe to talk to sumantro to try to get FCOS testing on the week of oct 11
16:42:35 <dustymabe> jaimelm: we should schedule some time now to try to identify test case changes (and interested parties)
16:42:42 <dustymabe> i'm interested
16:43:00 <dustymabe> jaimelm: would love some help organizing too if you're able/willing
16:43:07 <dustymabe> I think last time we may have caught you at a bad time
16:44:07 <dustymabe> for time: next Monday (morning US) works for me - any other suggestions?
16:45:22 <dustymabe> is anyone else that's interested in looking at the docs test cases able to make it Monday (anytime)?
16:45:32 <jlebon> yup, WFM too!
16:45:35 <jlebon> i can help
16:45:54 <jaimelm> dustymabe: Monday works for me
16:46:14 * jaimelm is in two meetings at once. responses may be delayed.
16:46:18 <dustymabe> #action dustymabe to schedule some time on Monday to identify docs test cases and prepare for F35 test day
16:46:33 <dustymabe> #topic tracker: Fedora 35 changes considerations
16:46:39 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3AF35-changes
16:46:56 <dustymabe> looks like we only have two subtickets left!
16:46:58 <dustymabe> almost done
16:47:05 <jlebon> woohoo!
16:47:20 <dustymabe> looks like travier is the brave soul assigned to both of them
16:47:23 <walters> .hello2
16:47:24 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:47:38 <dustymabe> #chair walters
16:47:38 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab walters
16:47:40 <dustymabe> welcome!
16:47:58 <jlebon> travier: feels like we can drop the rpmautospec one? definitely nice to have, but no need to tie to f35
16:48:24 <jlebon> (i.e. let's keep the ticket, but drop the label)
16:48:33 <dustymabe> yeah, I think there's less discuss needed on that one
16:48:48 <dustymabe> once we close off the SSSD ticket I'll drop the meeting label and close out the top level ticket
16:48:57 <dustymabe> so we won't bring it up any longer
16:49:25 <dustymabe> I think he may be AFK (didn't see him in roll call) so we'll punt on this one for now
16:49:31 <copperi[m]1> .hello2
16:49:32 <zodbot> copperi[m]1: Sorry, but user 'copperi [m] 1' does not exist
16:49:40 <dustymabe> #chair copperi[m]1
16:49:40 <zodbot> Current chairs: bgilbert copperi[m]1 dustymabe jaimelm jlebon lucab walters
16:49:49 <dustymabe> #topic Actually move iptables to the nft backend
16:49:54 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/676
16:50:28 <dustymabe> this came up again recently because we switched to f35 and they broke it out to an iptables-legacy package
16:50:54 <dustymabe> I thought we had done this before, but let's get agreement on the proposal in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-824225675 and then execute
16:51:22 <dustymabe> migrate new nodes
16:51:23 <dustymabe> declare iptables-legacy in a deprecation period and provide instructions on how to migrate (possibly with doc links for e.g. additional OKD/k8s-specific instructions)
16:51:25 <jaimelm> is on board with jlebon's proposal. Did we define x?
16:51:25 <dustymabe> wait X months
16:51:27 <dustymabe> force migrate old nodes
16:51:29 <dustymabe> no longer consider iptables-legacy supported
16:51:33 <jlebon> was thinking about this, and it's a little trickier than the proposal makes it seem
16:51:42 <jlebon> the "migrate new nodes" part is the hard bit
16:52:03 <jlebon> i'm not very confident in the proposal at https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732514979
16:52:18 <jlebon> so my proposal now is https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968
16:53:11 <jlebon> notably, the first 2-barrier proposal is complex, and even then not foolproof
16:53:34 <jlebon> the steps would still roughly be:
16:53:43 <jlebon> - declare iptables-legacy in a deprecation period
16:53:47 <jlebon> - wait X months
16:53:58 <jlebon> - force migrate old and new nodes
16:54:07 <jlebon> so that'd be the key difference
16:54:30 <jlebon> (unless explicitly opted into not migrating)
16:55:03 <dustymabe> seems reasonable to me
16:55:22 <dustymabe> basically users have the ability to opt out if they touch the stamp file
16:55:26 <walters> I think there's two classes of user here; the "small scale non-kube" case where they may use iptables manually, or equally well may not (in cloud) versus kubernetes where it's an important part of networking setup
16:55:37 <jaimelm> ^^
16:55:40 <dustymabe> obviously requires them to be engaged and reading notices
16:55:41 <walters> the forced migration seems like a risk for Typhoon
16:55:53 <walters> (and OKD?)
16:56:11 <jaimelm> Not so much for OKD
16:56:19 <jlebon> yeah, this approach would require e.g. Typhoon to touch the stamp file
16:56:23 <jaimelm> I ran it by the team a few months ago. Not a big concern.
16:56:40 <dustymabe> jlebon: good point. the integration platform could do it
16:56:41 <jlebon> then it's in their hands when they want to move to nft
16:57:40 <dustymabe> well hmm. trying to think about how easy that would be
16:58:02 <dustymabe> not exactly sure how updates to typhoon are handled
16:58:28 <walters> i thought the stamp file was about marking use of nft?  is there another stamp file for "I want iptables-legacy"?
16:58:32 <jlebon> we can run the proposal by Dalton to see what he thinks
16:58:43 <dustymabe> yeah, +1 for running the proposal by Dalton
16:58:50 <lucab> to the best of my knowledge, the usual FCOS way through zincati
16:58:58 <jlebon> walters: in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968, the stamp file would be "i want to stay on legacy"
16:59:18 <dustymabe> lucab: but what about the platform on top? i.e. could they define new containers in a new version that put a file on the host?
16:59:21 <walters> ok
17:00:54 <lucab> dustymabe: in theory yes, but I think it would still require manual intervention as typhoon down not auto-upgrade itself
17:01:42 <dustymabe> #proposal 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes
17:01:56 <dustymabe> hmm - should have added in there talking with typhoon
17:02:07 <lucab> https://typhoon.psdn.io/topics/maintenance/#upgrades
17:02:23 <dustymabe> #proposal Talk with typhoon about proposed migration. The proposed migration is: 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes
17:02:50 <jlebon> +1
17:02:52 <dustymabe> jlebon: does that reflect your proposal?
17:02:59 <jlebon> yup
17:03:13 <dustymabe> +1 from my side
17:03:28 <copperi[m]1> +1
17:03:47 <jaimelm> +1
17:04:25 <dustymabe> walters lucab bgilbert.. look good?
17:04:29 <walters> +1
17:04:45 <dustymabe> #agreed Talk with typhoon about proposed migration. The proposed migration is: 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes
17:05:04 <walters> as a side note I'm in the "BPF is the future" camp and NFT vs iptables is...eh, not very interesting
17:05:19 * dustymabe wants to learn more about BPF - like a lot more
17:05:20 <bgilbert> wfm
17:06:04 <dustymabe> anybody want to reach out to dalton to check this plan once I add it to the ticket (jlebon might also be able to add a little more design detail about how the "opt-out" service would operate
17:06:27 <jlebon> dustymabe: was thinking mostly of pinging dalton on the ticket :)
17:06:42 <jlebon> i can do that if you'd like
17:06:45 <dustymabe> that works!!
17:07:37 <dustymabe> #action jlebon to add some design details to the proposal in the ticket and also reach out to dgubble to see if this can be handled in typhoon
17:07:51 <dustymabe> jaimelm: ^^ also worth confirming 100% with OKD this won't be a problem
17:07:59 <lucab> (my IRC client is having large latency/lagging issues)
17:07:59 <dustymabe> #undo
17:07:59 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:07:37 : jlebon to add some design details to the proposal in the ticket and also reach out to dgubble to see if this can be handled in typhoon
17:08:17 <dustymabe> #action jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dgubble to see if this can be handled in typhoon
17:08:25 <dustymabe> lucab: noted
17:08:29 <bgilbert> *dghubble
17:08:37 <dustymabe> bgilbert: boo
17:08:39 <dustymabe> #undo
17:08:39 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:08:17 : jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dgubble to see if this can be handled in typhoon
17:08:47 <dustymabe> #action jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dghubble to see if this can be handled in typhoon
17:08:53 <dustymabe> better :)
17:08:57 <jaimelm> dustymabe: straight from the man himself... https://github.com/openshift/okd/discussions/613
17:09:07 <jaimelm> but I'll ask again
17:09:18 <dustymabe> +1
17:09:23 <jaimelm> He's OOTO but will be back next week-ish
17:09:29 <dustymabe> sound sgood
17:09:31 <dustymabe> thanks jaimelm
17:09:40 <dustymabe> #topic Handle re-invocations of coreos-installer on a new disk
17:09:44 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/976
17:09:47 * sumantrom is here
17:09:53 <dustymabe> bgilbert: want to set the stage for this one?
17:10:05 <bgilbert> not my ticket, but I'll be happy to chime in
17:10:13 <dustymabe> oops - let me check
17:10:20 <bgilbert> walters?
17:10:27 <dustymabe> +1 walters?
17:10:49 <walters> We've had a lot of problems I'm pretty sure are rooted in races on multiple disks with partitions that have `LABEL=boot`
17:11:29 <walters> It is *possible* for this to happen in cloud, in theory but I think it's mostly a metal thing
17:12:12 <walters> I haven't checked other distributions but at least the way Anaconda works for example it injects a uuid for `/boot` into both the kernel and grub cfg
17:12:43 <walters> (hmm actually Fedora Cloud presumably inherits this too)
17:13:30 <walters> anyways that's the intro
17:14:06 <jlebon> hmm, i'm not sure that's the case re. Anaconda setting it in the kernel kargs.  though definitely seeing it in the grub config
17:15:35 <walters> right, sorry it's not on the kargs for Anaconda, it's in `/etc/fstab` - because most uses of Anaconda-generated-systems don't need /boot in the initramfs
17:15:38 <dustymabe> so basically the problem is when you have more than one disk attached to a system and both of them have a label of `boot` on one of the filesystems?
17:15:43 <jlebon> what's so insidious about it is that the way problems manifest it makes you question everything
17:15:48 <bgilbert> jlebon: +1
17:16:18 <bgilbert> I would tend to lean toward solutions that keep coreos-installer out of it, since that'd be another cloud vs. metal divergence
17:16:44 <walters> yeah, i worked on https://blog.verbum.org/2020/12/01/committed-to-the-integrity-of-your-root-filesystem/ partially because of this, which was a lot of fun and valuable for other reasons obviously
17:16:59 <walters> (thinking there was some race on shutdown/update around power failures)
17:17:06 <bgilbert> e.g. failing if we find multiple /boot partitions, or burning in a per-build FS UUID + adding it to kargs + replacing it with a generated UUID on first boot
17:17:32 <walters> failing in the OS boot, or failing in coreos-installer?
17:17:36 <lucab> the two cases I've seen so far are mostly human/hardware failures, and most of the pain is that it become apparent way too late (on the first upgrade)
17:17:38 <bgilbert> failing in the OS boot
17:18:03 <bgilbert> (which is fundamentally subject to races of course)
17:18:07 <dustymabe> we could fail in both places
17:18:18 <dustymabe> though coreos-installer isn't always run directly on the target hardware
17:18:26 <jlebon> i don't think we should fail in coreos-installer
17:18:30 <dustymabe> sometimes the disk is written to and then "moved"
17:18:43 <dustymabe> jlebon: what about a warning then?
17:18:44 <bgilbert> +1, we can't assume c-i is run on the target
17:18:46 <jlebon> https://github.com/coreos/coreos-installer/pull/642#issuecomment-930351108
17:19:13 <bgilbert> that warning will fire if c-i is run from inside FCOS, which feels a bit odd
17:19:26 <bgilbert> *an installed FCOS
17:19:35 <jlebon> yeah, maybe more of a "Hey, I noticed that ..."
17:20:04 <lucab> I do generally agree with the directions walters want to go (more uuids), but at the same time I find valuable that we can say "by-label/boot"
17:20:22 <dustymabe> +1 lucab
17:20:28 <walters> lucab: it'd be good to enumerate the things we know today are using the label
17:20:49 <walters> note the proposal is just *adding* and using a uuid, not dropping the label though
17:21:21 <lucab> I only did a quick grep through the initrd, and we use that for ordering units
17:21:54 <walters> (side thread: all agreed we do need boot=UUID=$x on the kernel cmdline unlike anaconda because we need access to it in the initramfs?)
17:22:14 <jlebon> kinda feels like it's going out of our way to try to be resilient something that most likely indicates something wrong that ideally should be rectified
17:22:15 <lucab> sure, but the underlying story is "we move to uuid so that there can be multiple 'boot' labels"
17:22:31 <walters> hmm side note, the RHEL FIPS stuff requires this which is why we already have https://github.com/openshift/os/blob/9cd6b20bf01aee14c4c96946b84b14dce35d5c87/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh#L58
17:22:35 <bgilbert> jlebon: +1
17:22:46 <walters> so this is basically moving that bit of FIPS to be default
17:22:46 <jlebon> we want to own the machine and we need users to know that too
17:23:23 <jlebon> if there's two boot partitions, is it possible they booted from the wrong device?
17:23:28 <bgilbert> what's the scope of the problem we're trying to solve?  I can see several options: 1. bootloader disk doesn't match boot disk, 2. root doesn't match boot, 3. user booted not from the disk they thought
17:24:08 <bgilbert> e.g. it should be easy to fail on 2
17:24:41 <bgilbert> 2 rephrased: / and /boot come from different disks
17:24:42 <walters> hmm true...
17:25:28 <walters> it'd still be racy though
17:25:33 <bgilbert> it would
17:26:32 <lucab> we reached here through reported symptoms, so we basically only know about cases of 2.
17:27:11 <dustymabe> any ways to make it less racy? is there a moment in time where we are sure all fs with label boot should be up?
17:27:28 <dustymabe> even if it's late and we've already done some $work
17:28:18 <lucab> I guess when we say racy here we mean "somebody can add another 'boot' partition at any point at runtime"
17:28:18 <bgilbert> well, no.  hotplug can happen at any time
17:28:43 <dustymabe> let's say we're up and we get a boot
17:28:52 <jlebon> hmm, we don't need to do any waiting.  we could have a stamp in /boot and / which match each other and verify they match at mount time
17:28:58 <dustymabe> and then we mount a root fs - and it doesn't match (through some heuristic)
17:29:07 <dustymabe> jlebon: +1
17:29:45 <jlebon> we dynamically generate the stamp on first boot.  so whatever devices we agreed on first boot, those are what they have to remain for the rest of its life
17:29:51 <bgilbert> jlebon: stamp isn't necessary if we check for a common backing device
17:29:59 <walters> yeah but we could also find /boot from the same block device
17:30:01 <jlebon> bgilbert: we support moving rootfs to a different device
17:30:07 <bgilbert> ...right
17:30:09 <walters> ok true
17:30:46 <dustymabe> time check
17:31:13 <walters> jlebon: can you drop the stamp proposal in the ticket for discussion point?
17:31:25 <dustymabe> sounds like we're coalescing (spelling) around trying to figure out a good way to error in this case rather than change our UUID design for this particular instance
17:31:28 <jlebon> sure thing!
17:31:55 <walters> by error do we actually fail the boot, or do we just loudly warn?
17:32:28 <dustymabe> #proposal attempt to find a good way to show the user an error and fail the boot rather than change our UUID design for this particular invalid setup.
17:32:30 <jaimelm> loudly warn
17:32:35 <lucab> fail the boot in initrd, on each boot?
17:32:43 <dustymabe> i was thinking people were leaning towards error
17:32:43 <bgilbert> walters: fail?  it's a pretty bad race
17:33:04 <dustymabe> yeah basically it's a condition people would want to rectify
17:33:12 <jlebon> +1 to fail. way nicer error than other possible failure modes
17:33:23 <jaimelm> true
17:33:24 <jlebon> because at some point, boot will fail
17:33:27 <dustymabe> could we do even more?
17:33:43 <jlebon> with an obscure error like ostree failing to find the rootfs
17:33:43 <walters> dustymabe: like, wipe out the other disk(s) partitions automatically?
17:33:45 <dustymabe> so basially we detect boot/root matching but also CLHM if we find more than one part with boot label
17:33:54 <dustymabe> walters: ^^
17:33:59 <lucab> dustymabe: format one of the boot partition, random choice :)
17:34:09 <dustymabe> let's say we get lucky and boot/root match on this boot
17:34:10 <jlebon> hehe
17:34:20 <dustymabe> but next boot it might fail to match (race)
17:34:31 <dustymabe> CLHM could help us show the user the potential problem
17:34:56 <dustymabe> just an idea, either way current proposal stands
17:34:57 <jlebon> openshift users probably don't see CLHM on every boot
17:35:02 <dustymabe> jlebon: good point
17:35:07 <dustymabe> votes?
17:35:07 <bgilbert> even if root/boot match, we should probably check for other boot partitions
17:35:15 <walters> In theory if we detect a mismatch, we could e.g. only delete/mark notable the invalid boot partition that we used
17:35:17 <bgilbert> we may not find them in time (racy) but we could fail if we did
17:35:19 <walters> and then reboot
17:35:25 <walters> that would lead to eventual reconcilation
17:35:33 <walters> without arbitrary waiting
17:35:42 <bgilbert> walters: mark ourselves invalid, or the other one?
17:35:49 <walters> ourselves, only if we detect it was invalid
17:36:08 <jlebon> unless we don't know if it's boot that's wrong or root
17:36:11 <lucab> it may be valid but still on the "other" disk
17:36:18 <dustymabe> hmm.. auto reconciliation here seems potentially problematic
17:36:22 <dustymabe> :)
17:36:30 <bgilbert> isn't it true that damage may have already been done?
17:36:37 <bgilbert> i.e. stabilizing on one outcome isn't enough
17:36:57 <walters> can you elaborate on "damage"?
17:36:59 * dustymabe is really enjoying this detailed discussion
17:36:59 <jlebon> it's mostly on first boot that we do a lot of stuff with /boot
17:37:05 <bgilbert> walters: update skew?
17:37:09 <dustymabe> sorry jaimelm for $time
17:37:11 <bgilbert> also, we don't know which disk the user intended us to use
17:37:29 <walters> agreed, this proposal is picking one arbitrarily
17:37:51 <dustymabe> #proposal attempt to find a good way to show the user an error and fail the boot rather than change our UUID design for this particular invalid setup.
17:37:55 <jlebon> can bootloader order change on each boot without explicit action?
17:37:56 <dustymabe> can we stick with this ^^ for now
17:38:08 <bgilbert> sure would be nice if we had a reliable way to get that info from the bootloader
17:38:24 <lucab> for reference, one of the reporter was happy enough to know there was a node problem coming from this duplicate partitions, and re-provision the node in a correct way
17:38:29 <jaimelm> for now, and figure out a helpful way to notify the user
17:38:35 <walters> jlebon: I think this depends, but for a server with e.g. two nvme drives both with EFI partitions, it might simply race
17:38:36 <jlebon> dustymabe: maybe let's keep chatting in the ticket for now?
17:38:48 <bgilbert> jlebon: +1
17:38:48 <dustymabe> jlebon: i.e. no agreed upon outcome?
17:38:52 <jlebon> yeah
17:38:55 <walters> yeah, continue discussion
17:39:00 <dustymabe> that's fine, but i'll atleast do a summary
17:39:04 <jaimelm> ^^
17:39:07 <jlebon> +1
17:39:14 <dustymabe> well summary of the current proposed
17:39:18 <dustymabe> #topic open floor
17:39:23 <dustymabe> welcome all to open floor
17:39:26 <dustymabe> sorry about $time
17:39:43 <dustymabe> #info help wanted to chase down a containers-common issue in rawhide https://github.com/coreos/fedora-coreos-tracker/issues/981
17:40:02 <dustymabe> #info the `next` stream has been moved to Fedora Linux 35 content - please please test!
17:40:05 <pjones> walters: it should race and then /iterate/
17:40:32 <pjones> walters: assuming you're talking about the case where no variables are configured
17:40:33 <dustymabe> any other topics for open floor?
17:41:23 <dustymabe> for the containers-common issue, don't feel like you need to be an expert.. we can help guide you
17:41:26 * jaimelm will look at 981 but is busy
17:41:43 <dustymabe> +1
17:41:45 * jaimelm isn't mediumxpert
17:41:56 <dustymabe> will close out the meeting in 60s unless new topic come up
17:42:49 <jaimelm> jlebon: reminder for the CI walkthrough - or anyone else that wants to give me the tour
17:43:09 <jlebon> jaimelm: let's chat in the channel
17:43:09 <dustymabe> #endmeeting