16:30:39 <bgilbert> #startmeeting fedora_coreos_meeting
16:30:39 <zodbot> Meeting started Wed Feb 10 16:30:39 2021 UTC.
16:30:39 <zodbot> This meeting is logged and archived in a public location.
16:30:39 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:43 <bgilbert> #topic roll call
16:30:45 <bgilbert> .hello2
16:30:46 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:47 <slowrie> .hello2
16:30:49 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:56 <cyberpear> .hello2
16:30:57 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:29 <jlebon> .hello2
16:31:29 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:30 <PanGoat> .hello jaimelm
16:31:32 <zodbot> PanGoat: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:31:52 <bgilbert> #chair slowrie cyberpear jlebon PanGoat
16:31:52 <zodbot> Current chairs: PanGoat bgilbert cyberpear jlebon slowrie
16:33:56 <bgilbert> hmm, lots of topics today and not so many people
16:34:20 <bgilbert> #topic Action items from last meeting
16:34:22 <bgilbert> none!
16:34:23 <jlebon> yeah, might not be enough
16:34:42 <travier> .hello siosm
16:34:43 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:34:52 <bgilbert> some topics might be better for the smaller group than others
16:34:57 <bgilbert> #chair travier
16:34:57 <zodbot> Current chairs: PanGoat bgilbert cyberpear jlebon slowrie travier
16:35:02 <bgilbert> #topic Container Plumbing Days CFP
16:35:07 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/727
16:35:12 <bgilbert> jlebon?
16:36:02 <jlebon> that ticket is mostly relaying a message from dwalsh who wanted to see some FCOS talks there :)
16:36:19 <bgilbert> looks like the CFP closes _today_
16:36:24 <travier> We (with cverna) submitted a talk
16:36:38 <jlebon> travier: sweet!
16:36:42 <jbrooks> .hello jasonbrooks
16:36:43 <bgilbert> travier: \o/
16:36:43 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:36:48 <bgilbert> #chair jbrooks
16:36:48 <zodbot> Current chairs: PanGoat bgilbert cyberpear jbrooks jlebon slowrie travier
16:37:16 <bgilbert> cool
16:37:27 <travier> We did not get any other suggestion so we went ahead
16:38:02 <jbrooks> sounds like a cool talk
16:38:02 <bgilbert> #topic 2021-02-03: gather status update for Fedora Council
16:38:08 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/710
16:38:27 <jlebon> i guess this was technically due last week, heh
16:38:39 <jlebon> let me create a hackmd
16:38:40 <bgilbert> #link https://hackmd.io/hksDdnEMSxSrIeOJxyhAMg
16:39:17 <jlebon> nice :)
16:39:20 <bgilbert> (lmk if you can't access)
16:39:31 <bgilbert> please post recent news in the hackmd ^, or here
16:39:34 <jlebon> we don't have to fill eerything in right now. but everyone, please add things :)
16:40:28 <bgilbert> I'll wait a couple minutes for folks to fill things in, while we're on the topic
16:41:22 <travier> Should we mention the /boot RO or was it done in a previous report?
16:41:52 <bgilbert> travier: we should mention it
16:42:13 <jlebon> previous report is here: https://hackmd.io/e6uqIVaSSAqfWmpdKE784g   it wasn't mentioned
16:46:01 <bgilbert> let's see, we should probably defer cliwrap and QA cooperation until we have more people
16:46:20 <bgilbert> jlebon, should we discuss free space?
16:46:22 <jlebon> yeah, agreed
16:46:26 <jlebon> sure, let's do that
16:46:38 <bgilbert> #topic Upgrade fails due to minimum-free-space-percent exceeding 3%
16:46:45 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/731
16:47:00 <bgilbert> thanks for helping with the Council report, all!
16:48:02 <bgilbert> jlebon, do you want to start us off?
16:48:19 <jlebon> TL;DR: i'm very concerned about https://github.com/coreos/fedora-coreos-tracker/issues/731 and i think we should either implement one of the solutions mentioned in https://github.com/coreos/fedora-coreos-tracker/issues/586 or short-term have an FCCT warning and additionally break first-boot
16:49:05 <bgilbert> jlebon: are you more concerned than previously?
16:49:35 <jlebon> bgilbert: yes, because what we said would happen is now happening
16:49:40 <bgilbert> right
16:50:14 <bgilbert> FCCT warning would be great but I'm not sure how to do it.  FCCT doesn't know the device node of your boot disk, it just sees that you're adding a partition on some disk.
16:51:06 <bgilbert> breaking first boot would be feasible, and is interesting
16:51:10 <bgilbert> we could start with a warning
16:51:14 <bgilbert> journal and/or motd
16:51:20 * nirik has an item for open floor/etc later. :)
16:51:37 <bgilbert> nirik: +1
16:51:37 <jlebon> hmm, right.  guess fcct would only catch the case where the user is also reprovisioning root
16:51:49 <PanGoat> maybe both journal AND motd
16:51:57 <bgilbert> jlebon: in which case they're already specifying the size
16:52:17 <jlebon> bgilbert: right, i mean basically just sanity-checking that size they're specifying
16:52:23 <bgilbert> sure, we could do that
16:52:52 <bgilbert> (with some caveats around `rhcos` inheriting from `fcos`)
16:53:03 <jlebon> but yeah, i think doing something on the host at least is warranted
16:53:40 <jlebon> hmm, i guess warnings are OK, but they're harder to miss than a hard fail :)
16:53:51 <jlebon> *easier* to miss
16:53:57 <bgilbert> jlebon: right, I was just thinking that we need a migration period
16:54:07 <bgilbert> we don't want to start hard-failing people's configs without a heads-up
16:54:14 <jlebon> fair
16:54:28 <PanGoat> bgilbert +1
16:54:50 <jlebon> so basically, warn for a few releases, then eventually hard fail
16:55:08 <bgilbert> yeah.  and we probably need a coreos-status post before hard-failing
16:55:25 <jlebon> and those i think would still be valid to have even if we implement option 3 of https://github.com/coreos/fedora-coreos-tracker/issues/586
16:55:50 <jlebon> bgilbert: want to do a proposal?
16:55:56 <bgilbert> yeah
16:55:59 <bgilbert> hmm, re option 3
16:56:30 <bgilbert> we do have partition resizing now, but I'm not sure it'll work for all cases here
16:57:02 <bgilbert> it'd fail if the disk isn't large enough for the 'minimum size', which might not be what we want
16:57:28 <bgilbert> it might conflict with FCCT RAID sugar; need to think about that
16:57:47 <bgilbert> and more fundamentally, Ignition doesn't automatically know the boot disk
16:57:48 <bgilbert> anyway
16:58:02 <jlebon> oh yeah, that's..
16:58:32 <jlebon> well, it would reference by-* links
16:58:54 <bgilbert> that doesn't work for the disk, only for partitions
16:59:02 <bgilbert> there's no /dev/disk/by-id/boot-disk
16:59:46 <jlebon> wouldn't the ignition config just say "grow /dev/disk/by-partlabel/root to ..." ?
17:00:10 <bgilbert> Ignition configs can't reference partitions outside the context of a disk
17:00:33 <jlebon> oh heh, right right
17:00:57 <jlebon> hmm, that pretty much kills option 3 i think
17:01:07 <bgilbert> yeah, seems like it
17:01:16 <jlebon> but we can discuss long-term fixes in the ticket.  was mostly interested in getting a short-term thing in
17:01:26 <bgilbert> #proposed Have the initrd check for a root partition that is too small and is followed by another partition.  Warn for now, and eventually hard fail after a heads-up to coreos-status.  Add a warning to FCCT for the limited cases where FCCT can detect the issue.
17:01:47 <jlebon> ack
17:02:07 <PanGoat> ack
17:02:23 <bgilbert> jlebon: are you up for taking an action to work on some of this?
17:02:38 <jlebon> bgilbert: yeah, i can pick that up
17:03:16 <bgilbert> I may be able to work on the FCCT part if you don't want to dive into that code :-)
17:03:33 <jlebon> oh yes, that'd be great :)
17:03:52 <bgilbert> #agreed Have the initrd check for a root partition that is too small and is followed by another partition.  Warn for now, and eventually hard fail after a heads-up to coreos-status.  Add a warning to FCCT for the limited cases where FCCT can detect the issue.
17:04:04 <travier> ack
17:04:23 <bgilbert> #action jlebon to add initrd check and journal entry and/or motd
17:04:33 <PanGoat> It might be worth, for the time being, the 5GB suggested min be noted on more than just the storage page.
17:04:43 <PanGoat> It's kind of buried.
17:04:45 <bgilbert> #action bgilbert to investigate FCCT check
17:05:10 <bgilbert> PanGoat: hmm, any suggestions?
17:05:29 <bgilbert> that is where we tell people how to add partitions
17:06:32 <PanGoat> I'll think about it. I know folks have tried partitioning without going to that page.
17:07:27 <bgilbert> PanGoat: +1.  any suggestions would be great.  feel free to throw up a docs issue, or a PR if feeling motivated.
17:07:39 <bgilbert> should we move on?
17:07:42 <jlebon> +1
17:07:46 <PanGoat> will do
17:07:50 <bgilbert> #topic Platform Request: Proxmox
17:07:52 <PanGoat> yes
17:07:58 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/736
17:08:41 <bgilbert> okay, so this seems to be a specialized Linux distro that wraps KVM.
17:08:58 <bgilbert> including a bunch of control infrastructure
17:09:40 <bgilbert> based on the request and some very limited searching, it seems like they do have a userdata mechanism but
17:10:17 <bgilbert> the UX sort of assumes that users should fill out individual settings in the web UI, and the infrastructure should build a cloud-config behind the scenes.
17:10:34 <travier> ouch
17:10:53 <bgilbert> I don't think we want to support that part of it.  we can have a docs page that says "here's how to submit an Ignition config directly"
17:11:23 <bgilbert> instances get an ISO with a user-data file, instance metadata file, and a YAML file with network settings
17:11:26 <bgilbert> DHCP not guaranteed
17:11:43 <bgilbert> (note: all of this is based on not much research, and could be wrong)
17:12:53 <bgilbert> I have no strong opinion on the matter, just wanted to raise it
17:13:13 <PanGoat> Starting with a doc, after verifying, is fine.
17:13:14 <jlebon> could we instead try to get proxmox to support fw_cfg and then folks can just use the qemu image? :)
17:13:58 <bgilbert> I don't think fw_cfg is a winning proposition long-term, unfortunately
17:14:20 <bgilbert> (which reminds me that I should really circle back to that Ignition qemu userdata issue)
17:14:24 <jlebon> i personally feel like we're getting near the flat part of the curve of diminishing returns when it comes to number of clouds
17:14:42 <bgilbert> jlebon: the other reason we can't use the qemu image is the network config metadata
17:14:55 <walters> fw_cfg or the general virtio disk qemu?
17:15:07 <jlebon> is there a "virtualilized bare metal" workflow people can use?
17:15:07 <bgilbert> walters: sorry, what?
17:15:10 <bgilbert> jlebon: diminishing returns with respect to what?
17:15:19 <PanGoat> there will be a point of dimishing returns given limited people/time/sources.
17:15:31 <bgilbert> jlebon: sure, people can always run coreos-installer.  in this case, though, it sounds as though there's no good way to boot an instance from an ISO
17:16:21 <walters> bgilbert: why don't you think fw_cfg is good long term?
17:16:45 <bgilbert> I think it's great!  but we've failed to convince the QEMU and libvirt folks of that.
17:16:47 <jlebon> walters: the virtio disk approach hasn't gotten enough traction either sadly
17:17:18 <bgilbert> and also it doesn't scale to arbitrary qemu arches
17:17:19 <walters> isn't pushing virtio-disk adoption mostly on us?
17:17:28 <bgilbert> walters: yeah, it is
17:17:48 <walters> perhaps something that would help is convincing cloud-init to find data in the same way
17:17:58 <bgilbert> jlebon: I'd say the lack of traction is more of an internal thing (me, and one or two other folks)
17:18:07 <bgilbert> but I could be wrong
17:18:27 <jlebon> bgilbert: yeah, didn't want to spell it out :)  not that i don't think the concerns are valid
17:18:56 <bgilbert> yeah, I don't want to derail too much here, but I think it's worth a revisit
17:19:22 <walters> anecdotally it seems proxmox is pretty popular
17:19:36 <walters> in the homelab crowd, and I definitely want to appeal to them
17:19:42 <bgilbert> with respect to new platforms generally, I don't see a lot of harm in supporting them, though we shouldn't necessarily do the work ourselves
17:19:50 <PanGoat> ^^
17:19:57 <bgilbert> we don't need to carry a lot of code
17:20:07 <bgilbert> and I don't share the concern raised in a previous meeting about the number of images we ship
17:20:21 <jlebon> maybe we need to bin our clouds into support levels
17:20:47 <bgilbert> +1
17:21:03 <PanGoat> Start with a page and ask folks who use proxmox to contribute to a more substantial solution.
17:21:58 <bgilbert> I'm not sure docs are enough in this case.  we can't currently read userdata or network config.
17:21:59 <PanGoat> "here's a start, we'd like to add more support with your help"
17:22:12 <bgilbert> (and network config gets into the same problem that's blocked us on DigitalOcean so far)
17:22:18 <jlebon> i'm mostly OK adding more clouds, but we need to be clear when it's on a "best-effort" basis and people don't expect any QE happening there
17:22:25 <bgilbert> jlebon: +1
17:22:29 <PanGoat> right
17:23:00 <PanGoat> Does this group have a "Call for contributors" mechanism for projects/issues?
17:23:08 <travier> jlebon: +1
17:23:20 <bgilbert> PanGoat: not really.  some repos have a "good first issue" label, which we don't really promote.
17:23:33 <PanGoat> OK, that might be a good thing to get going.
17:23:36 <PanGoat> Happy to help.
17:23:40 <bgilbert> PanGoat: +1
17:23:47 <bgilbert> okay, anything we want to action on this topic?
17:24:04 <bgilbert> or propose
17:24:09 <jlebon> my concerns about shipping is more practical: more clouds means longer/flakier pipelines which means longer release process
17:24:21 <jlebon> but maybe we can rethink that
17:24:35 <PanGoat> #propose we create a call for contributors to build out support for proxmox
17:24:37 <bgilbert> jlebon: fair
17:24:42 <travier> We can have tiered support like Rust: Tier 1: full platform test in CI, Tier 2: generic CI test, Tier 3: best effort
17:25:06 <travier> Maybe have Tier 3 builds run after the other ones in the pipeline, independently?
17:25:08 <PanGoat> and use it as a model for future platforms
17:25:12 <bgilbert> travier: we should probably have a separate tracker ticket for that
17:25:15 <bgilbert> want to file one?
17:25:18 <travier> Sure
17:25:28 <bgilbert> #action travier to file tracker ticket to discuss platform support tiers
17:25:52 <bgilbert> PanGoat: do we need to set up a mechanism for that first?
17:26:19 <PanGoat> Well, learn as you go, but yeah
17:26:37 <bgilbert> I guess I'm saying: I don't know what that looks like.  is it just a mailing list email?
17:26:53 <PanGoat> s/proxmox/platforms
17:27:15 <PanGoat> s/call/process/
17:27:58 <PanGoat> I think mailing list, forum, twitter. Use everything at our disposal.
17:28:04 <bgilbert> PanGoat: hmm, so maybe we start with a tracker ticket to discuss?
17:28:12 <PanGoat> Yes.
17:28:16 <bgilbert> cool, want to file?
17:28:22 <PanGoat> I'll take that
17:28:23 <PanGoat> yeah
17:28:46 <bgilbert> #action PanGoat to file a tracker ticket to discuss a process for inviting contributors to platform support
17:28:55 <bgilbert> anything else on this topic before open floor?
17:30:01 <bgilbert> #topic Open Floor
17:30:05 <bgilbert> nirik, you had something?
17:30:21 <nirik> oh yeah... this is likely not the right group/place, but you might know who to ask. ;)
17:30:42 <nirik> updates pushes failed on silverblue updates yesterday. Who should I ask about silveblue stuff these days?
17:31:02 <nirik> the ultimate error is the libsolv thing. ;( DEBUG util.py:446:  error: Packages not found: libsolv-0.7.15-1.fc33
17:31:14 <nirik> https://pagure.io/releng/failed-composes/issue/2244
17:31:35 <jlebon> nirik: ahh fun.  /cc walters ^
17:31:51 <jlebon> this is likely due to pungi injecting its own repos instead of those in the git repo
17:31:55 <PanGoat> I have a quick one before we adjourn
17:32:02 <walters> nirik: thanks, let's discuss in https://pagure.io/workstation-ostree-config/pull-request/190 ?
17:32:19 <walters> but I think TL;DR is we can likely revert it
17:32:22 <nirik> walters: sure! I was just not sure where / who to bring it up to.
17:32:36 <bgilbert> PanGoat: go ahead
17:32:49 <PanGoat> I’d like to zero in on what Christian mentioned last week in regards to blockers. In particular, firewall configuration is concerning. I’ve been working to get FCOS adopted in the university environment for development and HPC, and also as a replacement for researcher lab desktops that are running Centos. One of the first vetting questions from security is going to be about firewall
17:32:52 <nirik> walters++
17:32:52 <zodbot> nirik: Karma for walters changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:32:55 <PanGoat> configuration. I’d like to keep this in everyone’s sights. #676, #467, etc.
17:33:52 <PanGoat> That's all.
17:33:59 <bgilbert> PanGoat: blockers to a Fedora rebase?  to promoting FCOS to an Edition?
17:34:05 <PanGoat> to an edition*
17:34:13 <PanGoat> and general adoption quite frankly
17:34:38 <bgilbert> is the ask that we document better (we need to), or missing functionality?
17:35:06 <PanGoat> Document to start
17:35:26 <PanGoat> which is probably going to fall on me :)
17:36:25 <travier> You should be able to setup an nftables or iptables firewall configuration without issues on FCOS. We only don't ship firewalld. Everything should work
17:36:28 <bgilbert> happy to have the help, though it shouldn't necessarily need to fall on you :-)
17:37:04 <travier> I was working on documenting that at some point. Might pick it up. Could you make an issue in the doc repo and ping me there?
17:37:13 <bgilbert> just filed https://github.com/coreos/fedora-coreos-docs/issues/247
17:37:40 <travier> 👍
17:37:53 <bgilbert> +1 to better docs, and improving/fixing papercuts as needed
17:37:55 <bgilbert> PanGoat: thanks for raising
17:38:02 <PanGoat> travier: I'll work as well
17:38:12 <travier> PanGoat: 👍
17:38:18 <bgilbert> anyone have anything else?
17:39:23 <PanGoat> thanks for leading bgilbert
17:39:32 <bgilbert> thanks all!
17:39:33 <bgilbert> #endmeeting