16:30:03 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:03 <zodbot> Meeting started Wed Jun 10 16:30:03 2020 UTC.
16:30:03 <zodbot> This meeting is logged and archived in a public location.
16:30:03 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:03 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:08 <dustymabe> #topic roll call
16:30:10 <dustymabe> .hello2
16:30:11 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:39 <lucab> .hello2
16:30:40 <slowrie> .hello2
16:30:40 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:43 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:46 <miabbott> .hello2
16:30:47 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:30:51 <skunkerk> .hello sohank2602
16:30:55 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:31:41 <jlebon> .hello2
16:31:42 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:44 <dustymabe> #chair lucab slowrie miabbott skunkerk jlebon
16:32:44 <zodbot> Current chairs: dustymabe jlebon lucab miabbott skunkerk slowrie
16:32:53 <dustymabe> #topic Action items from last meeting
16:33:09 <dustymabe> * dustymabe and jlebon (and anyone else who wants to) will review test
16:33:10 <dustymabe> cases linked from the various test day pages and try to fixup things
16:33:12 <dustymabe> that need fixing
16:33:14 <dustymabe> * dustymabe to send coreos-status post with relevant information about
16:33:16 <dustymabe> the change to fedora 32
16:33:46 <dustymabe> #info we had the FCOS test day and I did send out a coreos-status post with relevant information
16:33:58 <dustymabe> though the mail archives don't display html :(
16:34:12 <dustymabe> #topic meeting agenda items for today
16:34:36 <dustymabe> any large items that we should discuss that need immediate attention?
16:34:47 <dustymabe> otherwise we'll go through my short list and then meeting tickets
16:34:57 <jdoss> .hello2
16:34:58 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:35:02 <dustymabe> #chari jdoss
16:35:04 <dustymabe> sigh
16:35:07 <dustymabe> #chair jdoss
16:35:07 <zodbot> Current chairs: dustymabe jdoss jlebon lucab miabbott skunkerk slowrie
16:35:33 * dustymabe waits 30 seconds
16:35:33 <cyberpear> .hello2
16:35:34 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:35:39 <dustymabe> #chair cyberpear
16:35:39 <zodbot> Current chairs: cyberpear dustymabe jdoss jlebon lucab miabbott skunkerk slowrie
16:36:52 <dustymabe> #topic FCOS test day on 06/08
16:36:57 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/491
16:37:03 <dustymabe> so we had a test day!
16:37:27 <dustymabe> looks like we had quite a few people contribute and run test cases
16:37:31 <jlebon> 🎉
16:37:34 <dustymabe> #link http://testdays.fedorainfracloud.org/events/84
16:37:43 <jlebon> yeah, i was pleasantly surprised by the turnout
16:38:05 <dustymabe> looks like we might have some gaps on the cloud platforms
16:38:19 <jlebon> yeah, was going to say. no love for DO
16:38:21 <dustymabe> i'll probably try to run through the digitalocean test case sometime soon
16:38:33 <dustymabe> i think jdoss said he would run an AWS test :)
16:38:49 <jdoss> It's on my list!
16:39:02 <dustymabe> I need to go through and look at the feedback in the test day app to see what is actionable
16:39:19 <dustymabe> I know we got some feedback in the IRC channel about documentation improvements
16:39:30 <dustymabe> some stuff around static partitioning
16:39:46 <dustymabe> - need warning/docs that reconfiguring root storage doesn't work
16:40:09 <lucab> docs are not updating since last week btw, known issue on fedora-infra side
16:40:25 <jdoss> A note: I tried to fill out an entry for a test I did with QEMU and the form / site timed out yesterday. I need to go try again today.
16:40:27 <dustymabe> - need warning/docs about /home -> /var/home (fcct could spit out warning about toplevel mount points)
16:40:43 <dustymabe> jdoss: yeah, probably because of the infra datacenter move
16:40:51 <jdoss> +1
16:40:57 <jlebon> i also filed/investigated some UX issues in zincati and rpm-ostree
16:41:25 <dustymabe> yep and I found one with the displayed output from the coreos-ssh-key checker console message helper script
16:41:51 <dustymabe> one thing the test day did highlight - we have no docs for openstack
16:41:58 <cyberpear> they symlinks have always been a pain point... is a bind-mount not tenable?
16:41:59 <dustymabe> would anyone like to volunteer to add some?
16:42:31 <lucab> I tried to tag the tickets I've seen with the proper label. If I missed some, please retro-tag
16:43:43 <dustymabe> #info we need documentation for openstack deployments
16:44:15 <dustymabe> #info we had a productive test day with a lot of feedback and a few cosmetic issues found, we're still processing the feedback
16:44:22 <dustymabe> anything else for the test day topic
16:44:55 <cyberpear> (did we ever make openstack afterburn "just work" with metadata service?)
16:45:06 <cyberpear> (sorry, I didn't make it to the test day...)
16:45:26 <jlebon> i think that's still pending
16:45:27 <dustymabe> cyberpear: no worries :) - you can still participate if you find something that hasn't been done yet
16:45:40 <dustymabe> right we've decided we want to do it, but need the work to be done
16:46:03 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/422#issuecomment-607513602
16:46:24 <dustymabe> i'm not sure when it will get worked on
16:46:43 <dustymabe> anybody want to volunteer to work on ^^ ?
16:47:32 <dustymabe> If it's not a complex change then maybe someone who isn't yet familiar with the codebase could try
16:48:00 * dustymabe will move on to the next topic in 30s
16:48:55 <lucab> I don't currently have an openstack environment to check that. Once I get there, I will tackle that if nobody does first.
16:49:22 <dustymabe> #topic F32 rebase tracker for changes discussion
16:49:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/372
16:49:35 <dustymabe> we're using this ticket to discuss the rebase to f32
16:49:38 <jlebon> lucab: let's chat after
16:50:00 <dustymabe> so our `testing` stream has been on f32 for a week now
16:50:10 <dustymabe> I don't think I've seen any major issues, though some medium ones
16:50:44 <dustymabe> and those were mentioned in the last update in this issue
16:50:55 <dustymabe> so I think for now we're on schedule to ship to stable next week
16:51:22 <dustymabe> one change that we are considering making is documented in https://github.com/coreos/fedora-coreos-tracker/issues/525
16:52:47 <dustymabe> TL;DR we build FCOS with the fedora modular repos as inputs, but we have those repos disabled by default on the running FCOS systems. We're proposing to just turn off the modular repos for the FCOS builds as well. Doing this will make package layering on top of f32 more reliable and also will hopefully make rebases to f32 less likely to fail.
16:53:12 <dustymabe> #info we build FCOS with the fedora modular repos as inputs, but we have those repos disabled by default on the running FCOS systems. We're proposing to just turn off the modular repos for the FCOS builds as well. Doing this will make package layering on top of f32 more reliable and also will hopefully make rebases to f32 less likely to fail. This is documented in
16:53:13 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/525
16:54:00 <dustymabe> so we'd want to turn off those repos and respin `testing` so that we can make that change for next week's `stable`
16:54:38 <dustymabe> that's the current plan, hopefully the Fedora infra DC move will stabilize soon so we can achieve that goal.
16:54:49 <dustymabe> any other topics before we move on?
16:56:33 <cyberpear> makes sense on the modular, since there's no default modules in F32 (a big win!)
16:57:04 <dustymabe> ahh, hmm I might have to do some reading up on that
16:57:11 <dustymabe> cyberpear: link?
16:57:44 <dustymabe> and also, who is that a "big win" for? /me trying to gain context
16:57:59 <cyberpear> big win as in you don't have any headache coming from the modules
16:58:02 <cyberpear> (by default)
16:58:07 <dustymabe> kk
16:58:08 <jlebon> cyberpear: as in by default no modular packages ship on the media?
16:58:30 <jlebon> sorry, we can chat about this back in #fedora-coreos
16:58:34 <cyberpear> jlebon: I believe that is the case, yes, such as for Server media, workstation spin
16:58:36 <cyberpear> don't have it right off hand, but it was a FESCo decision, IIRC
16:58:40 <dustymabe> yeah I'll move on, but now I need to learn more :)
16:58:42 * cyberpear yields to next topic
16:58:49 <dustymabe> #topic Need dnsmasq for podman to create CNI networks
16:58:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/519
16:59:23 <lorbus> .hello2
16:59:24 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:59:27 <dustymabe> so it sounds like podman needs dnsmasq for some use cases but doesn't have a dep on it
16:59:30 <lorbus> o/ sorry I'm late :)
17:00:03 <dustymabe> #chair lorbus
17:00:03 <zodbot> Current chairs: cyberpear dustymabe jdoss jlebon lorbus lucab miabbott skunkerk slowrie
17:00:09 <dustymabe> .fire lorbus
17:00:09 <zodbot> adamw fires lorbus
17:00:16 <dustymabe> .chair lorbus
17:00:16 <zodbot> lorbus is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
17:00:22 <lorbus> :P
17:00:43 <adamw> well, dustymabe didn't respond to an @ from yesterday...
17:00:46 <adamw> .fire dustymake
17:00:46 <zodbot> adamw fires dustymake
17:00:48 <lorbus> a lake, or a plashet?
17:00:51 <adamw> close enough!
17:01:08 <dustymabe> adamw: darn :( - sorry about that - maybe ping me again and I'll try to respond today
17:01:16 <adamw> hehe, nbd
17:01:34 <dustymabe> so I guess the question is 1. should podman at least recommend dnsmasq and 2. should we include it in FCOS
17:02:13 <dustymabe> I agree and disagree with luca's statement in https://github.com/coreos/fedora-coreos-tracker/issues/519#issuecomment-639582552
17:02:16 <dustymabe> :)
17:02:49 <dustymabe> I agree that we don't users relying on dnsmasq in the host, but I do think that we want user's relying on features of the container runtime (which is in the host)
17:03:28 <dustymabe> so if it were strictly so that users could rely on dnsmasq for some application then I would be inclied to say no
17:03:40 <lucab> well, that's quite a bit of corner cutting on podman side
17:03:42 <dustymabe> but if it's for the container runtime, i lean the other way
17:04:54 <dustymabe> lucab: want to elaborate? do you think that podman should not be using dnsmasq (i.e. could do it in a different more native way)?
17:05:08 <lucab> indeed
17:05:37 <lucab> CNI bridging does not require dnsmasq
17:05:55 <jlebon> mheon's comment says it's for name resolution between containers
17:06:31 <lucab> they likely wanted to introduce some new feature, but didn't bother actually implementing it in CNI
17:06:54 <lucab> so they just shell out
17:07:07 <dustymabe> so maybe we should reach out to them to get more perspective ?
17:08:33 <lucab> to reconsider the CNI plugin design, yes
17:09:20 <dustymabe> in general i'm not opposed to shelling out to other things that have a specific job to do - we do it all the time, but in this case if it's a small part of dnsmasq they are using and it could easily be implmented in a more appropriate way somewhere else, then I'd like to explore that as an option
17:09:54 <lucab> well, yes, but those are in the category of "utilies"
17:09:58 <lucab> *utilities
17:10:11 <dustymabe> lucab: vs services ?
17:10:43 <lucab> otherwise you can shell out to httpd any time you need an HTTP support
17:11:49 <lucab> at some point I think they'll have to produce a config for dnsmasq, and have some version requirement on it
17:12:07 <dustymabe> everyone agreed to discuss it with the podman team and circle back?
17:12:32 <jlebon> WFM
17:12:32 <lucab> which means this CNI plugin now suddenly wants to be the one deciding the dnsmasq version
17:13:06 <lorbus> +1
17:15:07 <dustymabe> #topic F33 feature/change proposal SwapOnZRAM by default
17:15:13 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/509
17:15:50 <dustymabe> so currently we don't configure any swap for our instances (IIRC)
17:16:29 <dustymabe> there a few questions this topic brings up
17:16:42 <dustymabe> 1. should we be configuring swap by default
17:17:06 <dustymabe> 2. even if we don't configure swap by default, should we use zram if the user sets up swap?
17:18:14 <dustymabe> for 1. I think the institutional knowledge in me says that we don't have swap configured because if your system needs more memory then you should get a bigger system or you should scale out your fleet to handle the extra load
17:19:13 <dustymabe> anybody else have any historical knowledge for why swap is typically not enabled in cloud images?
17:19:20 <lucab> it also conflicts with most assumptions in higher-level container orchestration schedulers
17:20:28 <dustymabe> is it safe to say that we're probably not going to change and "add swap" to our disk images?
17:21:33 <dustymabe> jlebon: slowrie miabbott cyberpear jdoss ^^
17:21:53 <lucab> I think so. OKD/k8s will likely not work anymore. Even if you hammer that in place, pods QoS will behave weirdly.
17:21:58 <cyberpear> I'd take the Fedora lead and keep zram swap if Fedora makes it the default
17:22:14 <dustymabe> cyberpear: right, i think that's point 2.
17:22:18 <miabbott> +1 to not enabling swap
17:22:27 <jdoss> +1 to no swap on cloud instances
17:22:45 <dustymabe> cyberpear: so for 2. the question is - if a user enables swap should it use zram by default
17:22:55 <dustymabe> which I think is fine
17:23:47 <cyberpear> I think yes, keep zram, even if there's other swap, but again, follow Fedora.  (The proposed zram swap is higher priority than disk-swap, IIUC)
17:23:52 * cmurf says hi
17:24:13 <cmurf> about upgrades...
17:24:53 <cmurf> i'm confident it's safe but there's a good chance this is for new installations only and we back off on doing it for upgrades
17:25:06 <dustymabe> #info since fedora coreos images don't currently deliver any swap partition, it conflicts with how most people run things in cloud environments, and container orchestration platforms assuming no swap, we'll stick with what we're currently doing
17:25:15 <dustymabe> cmurf: context ^^
17:25:31 <dustymabe> cmurf: now the next question was related to "what do we do if a user configures swap"
17:26:08 <dustymabe> #proposed if a user configures swap via ignition we'll use the fedora defaults (changing in f33 to use SwapOnZRAM by default)
17:26:36 <jlebon> i think that's up to the user no? or are you thinking in the context of sugar in fcct?
17:26:37 <cyberpear> (I think swap on cloud can make sense, but maybe zram swap solves that case, so I won't argue now for adding a swap partition to cloud.)
17:26:39 <dustymabe> all - does that align with what we discussed above?
17:27:04 <jlebon> what does "user configures swap" actually mean?
17:27:15 <dustymabe> jlebon: good question
17:27:36 <dustymabe> i assume there is still some part of a hard drive somewhere that is dedicated to swap
17:27:56 <jlebon> either they use ignition to mkswap, or they drop some configs somewhere for swap-on-zram
17:27:57 <dustymabe> cmurf: you can set me straight here ^^
17:28:06 <jlebon> (or both?)
17:28:16 <lucab> https://github.com/systemd/zram-generator
17:28:48 <cyberpear> separate question: do we want to enable SwapOnZRAM early for F32-based FCOS if it's accepted for Fedora 33?
17:29:07 <dustymabe> ahh ok so I was slightly mistaken I think
17:29:36 <dustymabe> I was thinking swaponzram augmented existing swap parititions and essentially made them more effective
17:29:49 <cmurf> nope
17:29:56 <dustymabe> by balancing part of the swaped pages into ram and some in the actual swap partition
17:29:57 <cmurf> no disk based swap at all
17:30:03 <cmurf> memory block device only, using compression
17:30:27 <dustymabe> interesting
17:30:27 <lucab> I think the flow would be 1) write zram conf in Ignition 2) write a formatting + mounting in Ignition 3) let the generator create the device 4) let the unit format the swap
17:30:55 <cmurf> zram-generator being included in your image will not cause swap-on-zram to be setup, it also requires a configuration
17:31:05 <cmurf> so you can choose to opt out entirely; or you could just not ship a configuration
17:31:27 <lucab> it's basically a "formattable disk device in RAM" which can be used as a target for swap
17:31:27 <dustymabe> I see
17:31:34 <cmurf> correct
17:31:39 <cmurf> it's just a ram disk that compresses
17:31:39 <cyberpear> so I think it makes sense to include it, with the default configuration, and document having ignition masking the zram config if you don't want it
17:31:48 <cmurf> but we're gonna format it as swap
17:31:57 <dustymabe> ok good info
17:32:05 <dustymabe> thanks for bringing me up to speed
17:32:29 <dustymabe> i wonder if we can configure it such that it's only activated for some small threshold of memory
17:32:36 <cmurf> the overhead is actually less than the upstream doc suggests, if the provisioning is such that memory never gets close to full to trigger paging (swapping)
17:32:37 <lucab> it would be nice to see how it behaves with 1) mount namespaces 2) non-swap FS because it seems like a interesting alternative to loop0
17:32:50 <dustymabe> i.e., if your system has < 6G of ram then we enable it by default, else we don't
17:33:21 <dustymabe> #undo
17:33:21 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7fc74a4f9310>
17:33:34 <dustymabe> #undo
17:33:34 <zodbot> Removing item from minutes: INFO by dustymabe at 17:25:06 : since fedora coreos images don't currently deliver any swap partition, it conflicts with how most people run things in cloud environments, and container orchestration platforms assuming no swap, we'll stick with what we're currently doing
17:33:42 <cmurf> there is such an option in the config
17:33:45 <lucab> we should probably have a look at fixing their TODO note about /usr/lib defaults, because I foresee some fight on the default values
17:33:57 <cyberpear> IIUC, the proposed Fedora Change would limit the uncompressed size to 25% of RAM or 4GiB, whichever is smaller -- for a 2:1 compression ratio this would occupy 12% or 2GiB of ram, but it's only allocated as-needed
17:34:09 <cmurf> right now it's none (no limit, i.e. 128G  you still get 4G of swaponzram) but you could ship your own config with your own defaults
17:34:33 <dustymabe> cyberpear: is there a way to configure it such that zramonswap won't happen at all if you're above a certain threshold of memory?
17:34:46 <cyberpear> dustymabe: anaconda does this today, so it's possible
17:34:49 <dustymabe> i.e., "I've got 32G of memory, I don't need swap"
17:35:16 <dustymabe> very good info
17:35:27 <lucab> cmurf: which component owns the default value/algorithm/logic in vanilla Fedora?
17:35:30 <cyberpear> the "in defense of swap" article makes a very compelling argument for having some swap in all cases
17:35:37 <dustymabe> ok I think since we're over time I'll try to summarize this discussion in the ticket and we can circle back on it next time
17:35:55 <cmurf> lucab: not decided and btw i am seriously stupid when it comes to packaging
17:36:10 <cmurf> also on devel@ lennart points out that default configs shouldn't be in /etc
17:36:10 <dustymabe> seriously? :)
17:36:15 <cmurf> seriously
17:36:21 <dustymabe> haha
17:36:24 <cmurf> like impressive, not in a good way, stupid
17:36:27 <cyberpear> #url https://chrisdown.name/2018/01/02/in-defence-of-swap.html
17:36:36 <dustymabe> well thanks cmurf for fixing my misconceptions
17:37:34 <lucab> cmurf: yes, I echo Lennart
17:37:49 <cmurf> right so carving out everyone's needs? i think it can be done because i've seen it elsewhere, i just don't know exactly how it's done
17:37:50 <dustymabe> ok i'll move to open floor for now
17:38:07 <dustymabe> cmurf: don't worry we'll figure it out if/when we make that move
17:38:16 <dustymabe> just knowing it's possible helps us make the decision
17:38:23 <dustymabe> #topic open floor
17:38:28 <dustymabe> anyone have anything for open floor?
17:40:07 * dustymabe counts down to 0
17:40:55 <dustymabe> #endmeeting