16:30:01 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:02 <zodbot> Meeting started Wed Feb  9 16:30:01 2022 UTC.
16:30:02 <zodbot> This meeting is logged and archived in a public location.
16:30:02 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:02 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:08 <dustymabe> #topic roll call
16:30:10 <dustymabe> .hi
16:30:11 <zodbot> dustymabe: Something blew up, please try again
16:30:14 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:15 <ravanelli> .hi
16:30:17 <zodbot> ravanelli: Something blew up, please try again
16:30:20 <zodbot> ravanelli: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:22 <dustymabe> zodbot is dead today
16:30:40 <travier> .hi siosml
16:30:41 <zodbot> travier: Something blew up, please try again
16:30:42 <travier> .hi siosm
16:30:44 <zodbot> travier: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:47 <zodbot> travier: Something blew up, please try again
16:30:48 <miabbott_> .hi miabbott
16:30:50 <zodbot> travier: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:53 <zodbot> miabbott_: Something blew up, please try again
16:30:56 <zodbot> miabbott_: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:59 <jlebon> .hello2
16:31:00 <zodbot> jlebon: Something blew up, please try again
16:31:03 <miabbott_> yikes
16:31:03 <zodbot> jlebon: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:07 <nirik> it's not dead. it's just the lookups failing.
16:31:18 * nirik can fix it, but otherwise all the rest of the meeting functions are fine.
16:33:01 <travier> :)
16:33:01 <dustymabe> nirik++
16:33:01 <jlebon> nirik: cool, thanks
16:33:01 <miabbott_> "i'm not quite dead yet"
16:33:01 <dustymabe> #chair ravanelli travier miabbott_ jlebon
16:33:01 <zodbot> Current chairs: dustymabe jlebon miabbott_ ravanelli travier
16:33:01 <fifofonix> h-e-l-l-o
16:33:01 <dustymabe> #chair fifofonix
16:33:01 <zodbot> Current chairs: dustymabe fifofonix jlebon miabbott_ ravanelli travier
16:33:01 <lucab> .hi
16:33:01 <travier> dustymabe: Let's skip 1062 again to focus on the other interesting changes instead
16:33:01 <aaradhak> .hello
16:33:01 <aaradhak> .hi
16:33:02 <dustymabe> #chair aaradhak
16:33:02 <zodbot> Current chairs: aaradhak dustymabe fifofonix jlebon miabbott_ ravanelli travier
16:33:02 <dustymabe> travier: sounds good
16:33:07 <zodbot> dustymabe: Karma for kevin changed to 21 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:33:10 <zodbot> aaradhak: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:33:13 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:33:16 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:33:39 <travier> .hi siosm
16:33:39 <zodbot> travier: Sorry, but user 'travier' does not exist
16:33:45 <travier> .hello siosm
16:33:46 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:33:52 <dustymabe> #topic Action items from last meeting
16:33:55 <dustymabe> #chair lucab
16:33:55 <zodbot> Current chairs: aaradhak dustymabe fifofonix jlebon lucab miabbott_ ravanelli travier
16:34:22 <dustymabe> I don't think we had any specific action items from the last meeting other than me re-energizing some work we have in progress
16:34:38 <dustymabe> will move on to meeting topics
16:35:08 <travier> +1
16:35:14 <dustymabe> #topic Actually move iptables to the nft backend
16:35:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/676
16:35:38 * dustymabe goes to check who added the meeting label
16:35:47 <dustymabe> jlebon: you win
16:35:53 <jlebon> ahh right heh
16:36:05 <jlebon> hmm ok, i think i meant to redrop the label on this but since we're here:
16:36:42 <jlebon> this is more or less ready to roll out now after lots of tweaking and testing. the only bit left to iron out is the schedule
16:37:09 <jlebon> since it'll line up really closely to the f36 rebase, we were suggesting tying them together
16:37:10 <lucab> (❤️ the checklist on that ticket)
16:37:34 <jlebon> i.e. it would land in next first during its rebase to f36
16:37:42 <jlebon> and in testing when that rebases to f36
16:38:15 <jlebon> this isn't just to make it easier to understand, but also to make actually rolling this out easier
16:38:40 <jlebon> because if you look at the checklist, you'll see it's non-trivial :)
16:39:06 <jlebon> anyway, that's all. it does mean though that the window between next and testing receiving it will be shorter than initially discussed
16:39:36 <dustymabe> #proposed To make it easier to understand and easier to rollout we will couple the conversion to iptables-nft with the rebase to F36. This means the notice period window will be shorter than initially discussed.
16:39:43 <dustymabe> ack/nack/discussion?
16:40:22 <travier> +1 from me
16:40:38 <dustymabe> +1
16:41:21 <travier> While this has the potential to break complex iptables setup, this is easily reverted if need be on a per-node basis or in an Ignition config, thus not a compatibility concern
16:41:39 <travier> From most "legal" iptables usage, this should be transparent
16:41:44 <travier> for*
16:41:56 <miabbott> my only comment is to flesh out the schedule/dates in the checklist, if possible.  things like `wait until scheduled migration date for testing-devel` are a bit ambiguous
16:42:12 <miabbott> otherwise +1 to the plan
16:42:15 <jlebon> miabbott: it's in the email draft :)
16:42:28 <miabbott> ah cool, thanks for the pointer
16:42:47 <jlebon> i didn't want two sources to update
16:43:14 <dustymabe> any more votes?
16:43:19 <dustymabe> jlebon I assume you're +1
16:43:40 <jlebon> dustymabe: wasn't sure if i could vote, but yes :)
16:43:54 <dustymabe> yeah it was confusing because I did the proposed
16:43:57 <mnguyen> +1 sounds like a plan
16:44:14 <dustymabe> #agreed To make it easier to understand and easier to rollout we will couple the conversion to iptables-nft with the rebase to F36. This means the notice period window will be shorter than initially discussed.
16:44:33 <dustymabe> #topic networking: consider the effects of BOOTIF kernel argument on nm-initrd-generator
16:44:40 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1048
16:45:02 <dustymabe> I updated the issue with a proposal: https://github.com/coreos/fedora-coreos-tracker/issues/1048#issuecomment-1033329183
16:45:41 <dustymabe> basically: update our "was networking config provided" logic to handle BOOTIF
16:46:59 <jlebon> i think that makes sense.
16:47:06 <jlebon> i like it better than changing default kargs
16:47:39 <travier> +1 too. Prefer to require small changes for a specific case that changes for everybody here
16:47:41 <dustymabe> yeah - the only downside is the "doesn't apply to the initramfs" part (i.e. your networking in initramfs and real root could be different)
16:48:07 <dustymabe> but if you are hitting this you are already using PXE so updating kernel arguments should be easy IMO
16:48:34 <dustymabe> IOW you can add rd.bootif=0 easily
16:49:32 <jlebon> yeah. and really it wouldn't be any different on other systems
16:49:49 <jlebon> other dracut+NM-based systems*
16:50:04 <dustymabe> #proposed We will try to address the BOOTIF issue by updated our "was networking config provided" logic to handle BOOTIF rather than blanket applying rd.bootif=0 globally
16:50:19 <dustymabe> will fix the type in the agreed
16:50:24 <dustymabe> typo
16:50:38 <jlebon> ack
16:50:54 <travier> ack
16:50:56 <miabbott> ack
16:50:58 <mnguyen> ack
16:51:18 <dustymabe> aaradhak: fifofonix: don't be too shy to vote :)
16:51:31 <aaradhak[m]> ack
16:51:58 <dustymabe> #agreed We will try to address the BOOTIF issue by updating our "was networking config provided" logic to handle BOOTIF rather than blanket applying rd.bootif=0 globally
16:52:24 <dustymabe> #topic New Package Request: qemu-user-static
16:52:31 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1088
16:53:04 <dustymabe> looks like a request for qemu-user-static to enable running aarch64 containers on x86_64 and vice versa
16:53:55 <travier> Right now this pulls in Python so this needs packaging work at a minimum
16:54:22 <dustymabe> yep
16:54:42 <dustymabe> and a bunch of other bloat we probably don't want
16:54:47 <travier> Then it could be interesting to have sub packages that provide only a specific arch support
16:54:54 <miabbott> agreed; i think the ticket sums up the current situation nicely
16:54:55 <travier> so that we can ship only that
16:55:12 <fifofonix> fyi i do that kind of x-arch build with dind gitlab-runners (on fcos of course)
16:55:23 <dustymabe> fifofonix: nice!
16:55:26 <dustymabe> fifofonix++
16:55:26 <zodbot> dustymabe: Karma for fifofonix changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:55:28 <fifofonix> ie why put it on the os.
16:55:43 <dustymabe> yeah let's game this out
16:55:54 <dustymabe> even if the packaging work was done, would we still want it in the base?
16:57:00 <jlebon> couldn't this run fine in a container?
16:57:00 <travier> If we could have just aarch64 for x86_64 and the reverse, with a reasonable size, I think it would be OK. Not a strong +1 but fine
16:57:17 <jlebon> our whole build system is centered on running VMs in containers :)
16:57:18 <lucab> I'd be a bit wary of shipping the binfmt fragments by default
16:57:49 <jlebon> or is the idea that e.g. docker transparently uses this?
16:58:10 <dustymabe> jlebon: i.e. run a aarch64 container (VM) inside a x86_64 container that has qemu-user-static ?
16:58:18 <travier> To use it from a container you need to use podman in podman
16:58:40 <dustymabe> right (at least that's my understanding)
16:58:43 <travier> Not even sure it works in fact
16:58:59 <travier> as you would not be able to do the binfmt config
16:59:03 <travier> might not even work
16:59:21 <lucab> jlebon: the latter I think
16:59:23 <fifofonix> i think this is what i'm doing no?  ie. gitlab-runner runs as an x86 docker dind container and then use qemu to build aarch64 images, but maybe i'm not getting the use case.
16:59:26 <jlebon> ok right, i misunderstood what this is trying to do
17:00:08 <travier> fifofonix: are you using a QEMU VM or the static helper?
17:00:30 <travier> And what's your host?
17:00:47 <travier> (you said fcos, sorry)
17:00:50 <fifofonix> travier: this is why i don't vote sometimes.  only qualified people should be able to vote.  host is fcos.  using qemu static (i think)
17:01:15 <travier> fifofonix: it's always ok to give your opinion
17:01:46 <dustymabe> #proposed There are obviously packaging enhancements that could be made here (removal of python dep, reduction of shipped emulators by splitting out into subpackages) that are worth making even if we don't include qemu-user-static by default. Once those packaging improvements land we'd need to further consider it for inclusion in FCOS. It's a good idea, but maybe not necessary for the
17:01:49 <dustymabe> base layer.
17:01:55 <dustymabe> there's a lot of mumbling in there
17:01:57 <lucab> AFAIU this isn't about VMs. This is about qemu doing internally translation of the machine code of a binary, which thus can run on a different arch.
17:02:25 <travier> fifofonix: would be great if you could share an example (if possible of course)
17:02:47 <travier> dustymabe: +1
17:02:53 <fifofonix> i can do that after the meeting.  my first action in one of these meetings i believe.  may go quiet now parallel processing.
17:02:54 <jlebon> ack
17:03:07 <jlebon> lucab: right, that's what i understand as well. probably worth spelling this out in the issue to make it clearer
17:04:05 <dustymabe> #agreed There are obviously packaging enhancements that could be made here (removal of python dep, reduction of shipped emulators by splitting out into subpackages) that are worth making even if we don't include qemu-user-static by default. Once those packaging improvements land we'd need to further consider it for inclusion in FCOS. It's a good idea, but maybe not necessary for the base
17:04:08 <dustymabe> layer.
17:04:13 <dustymabe> moving on to the next topic
17:04:35 <dustymabe> actually we are at the halfway point - anything that is important and needs to be discussed today?
17:04:47 <dustymabe> otherwise we might break into discussing F36 changes
17:05:05 <dustymabe> travier: CFP need to talk about today?
17:06:14 * dustymabe goes to f36 changes
17:06:27 <dustymabe> #topic tracker: Fedora 36 changes considerations
17:06:30 <travier> (good for open discussion later)
17:06:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/918
17:07:06 <dustymabe> #info some of us met earlier today to sift through and weed out changes that we don't think need discussion.
17:07:25 <dustymabe> The preliminary results are in the hackmd at https://hackmd.io/kyMDKu25T76xAsUaSjMzHQ?edit
17:07:56 <dustymabe> so we'll go through that HACKMD now for any items that don't have a check mark and see if they need a ticket opened for them for further investigation and a volunteer to look into it
17:08:20 <dustymabe> any questions before we proceed?
17:08:47 <jlebon> use ?view to track numbers since they're not on the md side
17:09:16 <dustymabe> subtopic: 102 Introduce module Obsoletes and EOL
17:09:32 <dustymabe> This one needs more investigation?
17:09:57 <dustymabe> I guess we did add support for installing modules recently
17:10:30 <jlebon> hmm ok i can take this one
17:10:47 <jlebon> we'll have to decide on the rpm-ostree side how to handle this
17:10:50 <dustymabe> #action jlebon to open issue to investigate "102 Introduce module Obsoletes and EOL"
17:10:56 <dustymabe> thanks jlebon
17:11:13 <dustymabe> subtopic: 103 DNS Over TLS
17:11:53 <jlebon> this one should mostly be transparent to us but DNS has been troublesome in the past
17:12:03 <jlebon> so i wouldn't be surprised if something breaks
17:12:22 <dustymabe> looks like it's a systemd build time flag and we use systemd-resolved
17:12:55 <jlebon> so i think no action but we should keep an eye out for potential fallout
17:13:00 <travier> +1
17:13:11 <travier> OKD might be impacted
17:13:12 <nemric> +1
17:13:13 <lucab> I was unsure if we still had some divergence from current Fedora defaults
17:13:18 <miabbott> yeah, looks like it falls back to unencrypted if the dns server doesn't support TLS
17:13:22 <dustymabe> there is also DNSOverTLS= setting in resolved.conf
17:13:32 <dustymabe> lucab: not currently
17:14:02 <dustymabe> i think I'll still open a subticket for this and record our discussion there so it's open for any potential issues that are discovered
17:14:49 <dustymabe> #action we think we can pick up DNSoverTLS changes passively but dustymabe will open a ticket to record the discussion here and provide a space for any issues that come up to be discussed.
17:15:17 <dustymabe> subtopic: 111 Drop NIS(+) support from PAM
17:15:53 <dustymabe> jlebon mentions: may affect users who use NIS+? likely not though. If so, we should direct them to e.g. LDAP or FreeIPA as the Change proposal suggests. so overall, skip.
17:16:11 <dustymabe> jlebon: safe to move on?
17:16:30 <travier> safe, NIS has been "deprecated" for a while now
17:16:34 <jlebon> i'm not familiar with NIS+ and not even sure if we ship everything needed today to support it in FCOS (though people could always layer i guess)
17:16:40 <dustymabe> #info 111 Drop NIS(+) support from PAM  may affect users who use NIS+? likely not though. If so, we should direct them to e.g. LDAP or FreeIPA as the Change proposal suggests. so overall, skip.
17:16:40 <jlebon> dustymabe: yeah i think so
17:16:45 <dustymabe> +1
17:16:46 * miabbott relives the days of relying on NIS at sun micro
17:17:09 <dustymabe> jlebon: should 112 have a ✔️ in front of it too?
17:17:31 <jlebon> dustymabe: yeah sounds good
17:17:34 <dustymabe> and 114 ?
17:17:45 <dustymabe> and 118 :)
17:17:46 <travier> +1
17:18:03 <travier> 118 might need tracking
17:18:06 <travier> to ba safe
17:18:08 <travier> be*
17:18:11 <travier> no action
17:18:13 <dustymabe> k
17:18:32 <jlebon> i think we can skip it for now though
17:18:51 <dustymabe> #action jlebon to open a tracking issue for 118 Switch GnuTLS to allowlisting to track any fallout from the change
17:19:00 <dustymabe> sorry ^^ got trigger happy
17:19:15 <jlebon> hmm, i think i'd prefer filing tickets if issues come up
17:19:20 <dustymabe> subtopic 120 Golang 1.18
17:19:26 <dustymabe> k
17:19:29 <dustymabe> #undo
17:19:29 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:18:51 : jlebon to open a tracking issue for 118 Switch GnuTLS to allowlisting to track any fallout from the change
17:20:24 <dustymabe> anything for the golang one?
17:20:38 <miabbott> maybe a tracking ticket to enable 1.18 CI on the upstream projects?
17:20:39 <jlebon> this one i think the main thing is probably to add it to upstream CIs when we can
17:20:53 <jlebon> +1
17:21:16 <dustymabe> #action miabbott to open a tracking ticket for early testing of golang 1.18 when it's available
17:21:20 <miabbott> ack
17:21:30 <travier> :)
17:21:36 <dustymabe> I guess something like this would be useful FCOS continuous stream
17:22:01 <dustymabe> subtopic 208 Retired Packages
17:22:22 <travier> #action travier file a ticket to make sure we don't ship retired packages in FCOS
17:22:58 <dustymabe> is that a problem for us though?
17:23:17 <dustymabe> I think this is more targetted at a user's system that is upgrading
17:23:20 <travier> If they are retired then it means that we probably have to take over maintenance
17:23:30 <travier> The change itself is fine for us
17:23:40 <dustymabe> no, I mean. If they are retired our builds will fail
17:23:51 <travier> true
17:23:56 <dustymabe> so there's no way for this to happen to us today
17:24:04 <travier> indeed, good catch
17:24:06 <travier> #undo
17:24:06 <zodbot> Removing item from minutes: ACTION by travier at 17:22:22 : travier file a ticket to make sure we don't ship retired packages in FCOS
17:24:31 <travier> so we can skip
17:24:42 <dustymabe> yeah i think this change is targetted at someone who has a yum based system and is upgrading and they want to get rid of old packages that aren't receiving updates
17:24:50 <travier> yep
17:24:55 <dustymabe> though I guess the same question applies
17:25:12 <dustymabe> if someone has package layered on FCOS a package that is no longer available, what happens?
17:25:33 <jlebon> if it's a local package, it stays layered. if it's from repos, it'll fail to fetch the package
17:25:33 <travier> They won't get updates anymore
17:25:47 <dustymabe> yep, and they might not know it :(
17:26:04 <dustymabe> i.e. their system auto updates, but now fails the auto update and sits there
17:26:23 <travier> Will likely break if rpm-ostree can not satisfy the dependencies (i.e. newer glibc)
17:26:33 <travier> but this is not new, this can already happen today
17:26:53 <travier> break as in: rpm-ostree/zincati will stop auto-updating
17:26:57 <dustymabe> #action dustymabe to open an issue for investigation into missing packages preventing auto-updates from working
17:27:03 <dustymabe> travier: indeed
17:27:36 <jlebon> yeah, though that's kinda part of the tradeoff when you go into hybrid mode. you're exposed to these kinds of things and there's a lot we can't shield from
17:27:51 <travier> I don't think we can do something here apart from adding a message to the admin via console-login-helper-messages for example
17:27:53 <dustymabe> jlebon: indeed. I just wonder if maybe we can make things more clear
17:27:58 <dustymabe> travier: exactly
17:28:01 <dustymabe> clhm helper
17:28:24 <travier> Zincati could write something saying "I failed to update X times"
17:28:33 <travier> and CLHM would display that
17:28:38 <dustymabe> subtopic 221 Keylime subpackaging and agent alternatives
17:28:43 <jlebon> good idea
17:29:00 <dustymabe> I wasn't sure on this one - we just generate some information using keylime during build
17:29:09 <dustymabe> we do generate*
17:29:25 <dustymabe> oh travier linked to an issue
17:29:56 <dustymabe> ok https://github.com/coreos/fedora-coreos-tracker/issues/982 was a request for us including the agent in FCOS
17:29:59 <travier> yes, there is already an issue for that so we can skip (and maybe ask for an update on the issue)
17:30:16 <jlebon> re. compose time: assuming the format itself doesn't change, i think we're ok
17:30:21 <dustymabe> so we don't currently include any keylime stuff in FCOS
17:30:34 <dustymabe> and ok.. jlebon says build time we should be good
17:30:46 <jlebon> we ship a hashlist, but it's undocumented because it's just for experimentation
17:30:54 <jlebon> not in FCOS to be clear
17:31:00 <jlebon> but it's in the builddir
17:31:22 <travier> Filed https://github.com/coreos/console-login-helper-messages/issues/107 for 208
17:31:24 <dustymabe> #info we don't currently include the keylime agent in FCOS but we do generate a hashlist at build time for experimentation. Assuming the format of the hashlist hasn't changed we should be good herer.
17:31:38 <dustymabe> #undo
17:31:38 <zodbot> Removing item from minutes: INFO by dustymabe at 17:31:24 : we don't currently include the keylime agent in FCOS but we do generate a hashlist at build time for experimentation. Assuming the format of the hashlist hasn't changed we should be good herer.
17:31:41 <dustymabe> #info we don't currently include the keylime agent in FCOS but we do generate a hashlist at build time for experimentation. Assuming the format of the hashlist hasn't changed we should be good here.
17:31:50 <dustymabe> ok thats all!
17:31:52 <dustymabe> #topic open floor
17:32:01 <dustymabe> thanks for bearing with us - and sorry we're late for open floor again
17:32:17 <dustymabe> #info f36 just branched from rawhide, so now f37 exists
17:32:37 <travier> https://github.com/coreos/fedora-coreos-tracker/issues/1093 > Container Plumbing Days 2022. Feel free to suggest a talk!
17:32:38 * dustymabe needs to go look at checklists and stuff
17:32:54 <jlebon> dustymabe: yeah, let's open the f36 rebase ticket?
17:32:58 * dustymabe craves in person conferences again
17:33:13 <dustymabe> jlebon: yeah, not yet. I think there are quite a few updates I need to make to the template first
17:33:21 <travier> How about we make a talk for "What's new and what's next in Fedora CoreOS for container users"? Anybody want to work with me on that one?
17:33:32 <travier> wants*
17:33:37 <jlebon> dustymabe: and we should close https://github.com/coreos/fedora-coreos-tracker/issues/884 :)
17:34:02 <dustymabe> :)
17:34:17 <dustymabe> any other topics for open floor?
17:34:24 <jlebon> travier: walters might be interested for the coreos layering bit
17:34:33 <travier> +1 jlebon
17:35:13 * dustymabe sets a timer for meeting end
17:36:01 <dustymabe> #endmeeting