16:30:34 <dustymabe> #startmeeting fedora_coreos_meeting 16:30:34 <zodbot> Meeting started Wed May 19 16:30:34 2021 UTC. 16:30:34 <zodbot> This meeting is logged and archived in a public location. 16:30:34 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:34 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:30:38 <dustymabe> #topic roll call 16:30:41 <dustymabe> .hi 16:30:42 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:30:50 <copperi_> .hello copperi 16:30:51 <zodbot> copperi_: copperi 'Jan Kuparinen' <copper_fin@hotmail.com> 16:31:31 <jdoss> .hi 16:31:32 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com> 16:31:38 <jaimelm> .hello 16:31:38 <zodbot> jaimelm: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:31:49 <cyberpear> .hi 16:31:49 <jlebon> .hello2 16:31:49 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com> 16:31:52 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:31:55 <jaimelm> .hello2 16:31:56 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu> 16:32:12 <jbrooks> .hello jasonbrooks 16:32:13 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com> 16:32:32 <lorbus> .hi 16:32:32 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com> 16:33:15 <dustymabe> #chair copperi_ jdoss jaimelm cyberpear jlebon jbrooks 16:33:15 <zodbot> Current chairs: copperi_ cyberpear dustymabe jaimelm jbrooks jdoss jlebon 16:33:15 <darkmuggle> .hello2 16:33:17 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev> 16:33:19 <dustymabe> #chair darkmuggle 16:33:19 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon 16:33:39 <dustymabe> we have a lot of representation on the 'J' letter names in this meeting 16:33:54 <dustymabe> c c d d d j j j j 16:34:07 <lucab> .hello2 16:34:08 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com> 16:34:28 <dustymabe> #chair lucab 16:34:28 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lucab 16:34:58 * lorbus is here too :) making it two L's in the list 16:34:59 <lorbus> .hi 16:35:00 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com> 16:35:27 <dustymabe> #chair lorbus 16:35:27 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab 16:35:31 <dustymabe> ok let's get started 16:35:37 <dustymabe> #topic Action items from last meeting 16:35:45 <jaimelm> #couch lorbus 16:35:50 <bgilbert> .hello2 16:35:51 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:35:58 <dustymabe> #chair bgilbert 16:35:58 <zodbot> Current chairs: bgilbert copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab 16:36:11 <dustymabe> IIUC We had no action items from last meeting. 16:36:14 <dustymabe> So nothing to discuss here 16:36:20 <lorbus> jaimelm: that's about right :D 16:36:42 <dustymabe> ok, moving on to meeting tickets 16:36:52 <dustymabe> #topic Cannot upgrade from N-2 releases due to missing GPG key 16:36:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749 16:37:15 <dustymabe> I think we've mostly discussed this and come to a conclusion on how to handle it in the future 16:37:21 <dustymabe> I posted a workaround in the bug 16:37:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-843446996 16:37:54 <dustymabe> #info posted a workaround for systems that hit this problem in the bug at https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-843446996 16:38:21 <jaimelm> pretty straight forward 16:38:29 <dustymabe> the final piece of the puzzle I think (for the future) is some state checking on zincati's part? Anything left to the discuss there? Is there an open ticket for that work? 16:38:56 <jlebon> not yet, want me to file one lucab? 16:41:10 <lucab> I think we had one but can't find it, so yes please 16:41:40 <dustymabe> jlebon: if you would link to that ticket from #749 and we'll close it when it lands in zincati 16:42:02 <jlebon> done :) 16:42:20 <jlebon> #link https://github.com/coreos/zincati/issues/558 16:42:45 <dustymabe> woot woot 16:42:48 <dustymabe> ok next topic then 16:43:01 <dustymabe> #topic systemd-oomd for Fedora CoreOS 16:43:07 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/840 16:43:32 <dustymabe> in an effort to keep more in sync with Fedora, I'm bringing up systemd-oomd 16:43:45 <dustymabe> I thought it required swap, but that's not true any longer 16:44:51 <dustymabe> any thoughts on us enabling systemd-oomd? 16:44:57 <jdoss> +1 16:45:05 <jdoss> I also think the zswap stuff is pretty great too 16:45:42 <dustymabe> jdoss: yeah, we previously had that discussion and I think we can re-visit zswap in the future. kubernetes just accepted a KEP to support swap enabled systems. 16:46:38 <dustymabe> hmm. everyone is quiet today 16:46:51 <dustymabe> let's see if this gets any action 16:47:29 <dustymabe> #proposed considering systemd-oomd can work on systems without swap we will enable it in Fedora CoreOS like other editions of Fedora do 16:47:34 <dustymabe> ack/nack? 16:47:58 <jlebon> what are the implications for OKD/k8s? 16:48:20 <jlebon> if there are any 16:48:52 <dustymabe> jaimelm: might know, i'm not sure 16:49:02 <jaimelm> I can't think of any, but will bring it to the group. 16:49:02 <lorbus> Afaik there's swap support coming to K8s soon, but it's not there yet. Idk what impact using oomd would have tho 16:49:30 <dustymabe> also I need to understand a little more to know if systemd-oomd has requirements (like cgroups v2 for some reason) 16:49:38 <jaimelm> I'll take that as a task 16:49:42 <lorbus> thanks jaimelm, raising that in the OKD WG would be great 16:49:53 <cyberpear> oomd seems nicer than the OOM Killer. I wonder what its corner cases are... 16:49:57 <jlebon> apart from that in general I don't see why we shouldn't match 16:50:02 <dustymabe> #action jaimelm to ask the OKD working group if there are any implications that systemd-oomd would have on k8s/OKD 16:50:03 <darkmuggle> In the case of FCOS, I would urge caution 16:50:08 <lorbus> jlebon: agree :100\ 16:50:12 <lorbus> * jlebon: agree :100: 16:50:44 <darkmuggle> Matching makes sense to be me, but we might need some config to handle the headless nature of FCOS 16:51:11 <dustymabe> darkmuggle: meaning the defaults might need to be tweaked? 16:51:30 <darkmuggle> Correct. 16:51:43 <darkmuggle> I'm imagining a situation where rpm-ostree or sshd is OOM'd 16:51:44 <dustymabe> i can check to see if Server has different defaults than Workstation 16:52:13 <dustymabe> darkmuggle: indeed. I think you can specify in your systemd unit if you want your process to be ignored from oomd consideration 16:52:17 <darkmuggle> That said, in general, I'm in favor. Just make sure that the defaults are sensible 16:53:00 <bgilbert> hmm, the oomd manpage recommends swap 'to give oomd enough time to react' 16:54:12 <dustymabe> bgilbert: meaning the system might run out of memory before oomd can kill something? 16:54:43 <darkmuggle> More like it gives systemd-oomd time to react 16:54:50 * dustymabe just assumes the kernel OOM would be the fallback in that case 16:54:51 <bgilbert> the manpage doesn't make it clear whether oomd livelocks or whether the kernel OOM killer takes over 16:55:04 <bgilbert> that seems... extremely hacky though 16:55:18 * bgilbert has not dug into this at all 16:55:25 <dustymabe> no worries. It's new 16:55:31 <dustymabe> let's regroup on this next week? 16:55:39 <jlebon> feels like we should have someone spike on this and play around with it 16:55:48 <darkmuggle> I think that would be the sensible course to play with it. 16:55:49 <jaimelm> might be good enable it, throw some memory tests at FCOS, and see what defaults work best under various scenarios. 16:55:51 <lucab> I don't think it really works without cgroupsv2 though 16:56:04 <jlebon> otherwise, it's more or less the blind leading the blind :) 16:56:10 <lucab> the underlying PSI primitive does not really exist on v1 16:56:17 <darkmuggle> Having dealt with OOM on a massive fleet of servers, this is topic that makes my spidey sense tingle. 16:56:19 <dustymabe> lucab: I was interested in knowing that 16:56:26 <jaimelm> dustymabe: more investigation is needed. Let's move on? 16:56:38 <travier> .hello siosm 16:56:39 <zodbot> travier: siosm 'Timothรฉe Ravier' <travier@redhat.com> 16:56:48 <dustymabe> jaimelm: correct, but one more thing 16:57:17 <dustymabe> i would really like to get some volunteers at some point. I'd like to not let things like this linger forever 16:57:27 <dustymabe> because then we diverge from Fedora because "we don't know what we don't know" 16:57:55 <dustymabe> we'll circle back in a week. hopefully there is some more information by then. 16:58:02 <darkmuggle> If someone wants to swarm with me on that, I'll throw my hat in the ring for this one. 16:58:14 <dustymabe> #dustymabe to find out how systemd-oomd acts under cgroups v1 (if at all) 16:58:19 <jaimelm> right. let's rangle some volunteers. I've been working on a template for Calls to Action. Happy to leverage it once we know more so that we know what to ask folks to do. 16:58:20 <lorbus> #chair siosm 16:58:20 <zodbot> Current chairs: bgilbert copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab siosm 16:58:25 <dustymabe> darkmuggle: that'd be great! 16:58:26 <jaimelm> wrangle* 16:58:38 <dustymabe> bgilbert: are you interested too, or you good with whatever darkmuggle finds? 16:58:47 <dustymabe> #action dustymabe to find out how systemd-oomd acts under cgroups v1 (if at all) 16:58:59 <bgilbert> I'll trust darkmuggle :-) 16:59:22 <dustymabe> #action darkmuggle to investigate systemd-oomd (in use without swap) and report back next week 16:59:39 <jlebon> darkmuggle++ dustymabe++ 16:59:41 <zodbot> jlebon: Karma for darkmuggle changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:59:43 <dustymabe> #info jaimelm dustymabe and darkmuggle will look into some open questions about systemd-oomd and report back next week 16:59:44 <zodbot> jlebon: Karma for dustymabe changed to 4 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:00:00 <jaimelm> good stuff 17:00:22 * dustymabe skipping the remaining topic (tagged for our next video meeting) 17:00:26 <dustymabe> #topic open floor 17:00:55 <dustymabe> #info the stable stream has been rebased to Fedora 34 17:01:00 <lucab> dustymabe: https://github.com/systemd/systemd/blob/v248/src/oom/oomd.c#L154 17:01:34 <jaimelm> dustymabe: that was a well-written email btw. 17:02:13 <jlebon> yay f34 ๐ 17:02:16 <dustymabe> lucab: nice. I'll still look into whether the systmed-oomd service blows up or not 17:02:27 <dustymabe> or if it just gracefully does nothing 17:02:31 <cyberpear> ๐ happy to see the rebase happen faster this time! 17:02:35 <lorbus> :tada: 17:02:36 <dustymabe> jaimelm: thanks :) 17:03:24 <dustymabe> any other topics for open floor? 17:03:31 <lorbus> That emoji now seemingly triggers a full-chat-screen-celebration on the Element client ๐ ๐ 17:03:49 * lorbus < https://matrix.org/_matrix/media/r0/download/matrix.org/cesNDFVyVGrxEgxHSYxClbJg/message.txt > 17:04:16 <jlebon> and we already have the next ticket for tracking f35 (https://github.com/coreos/fedora-coreos-tracker/issues/824) 17:04:40 <dustymabe> #info we re-enabled the rawhide mechanical stream and are opening bugs when our CI tests fail 17:04:41 <cyberpear> lorbus: IRC folks just see a paste link 17:04:52 <bgilbert> lorbus: I tend to think we should respond in some way, but consistent with what Fedora as a whole does 17:04:56 <dustymabe> this will help us track fedora changes and react much more quickly 17:05:13 <bgilbert> e.g. we don't control where meetbot runs 17:06:25 * lorbus < https://matrix.org/_matrix/media/r0/download/matrix.org/iuLYjLqKujJapGhjRkYIVvrp/message.txt > 17:06:38 <bgilbert> lorbus: still a link 17:06:56 <bgilbert> because of the long message AIUI 17:07:15 <dustymabe> yeah. I can read it though 17:07:19 <dustymabe> This is an FYI: 17:07:21 <dustymabe> You may have heard about the hostile takeover of Freenode that took place today (!) 17:07:23 <dustymabe> https://blog.bofh.it/debian/id_461 17:07:25 <lorbus> You may have heard about the hostile takeover of Freenode that took place today (!) 17:07:25 <dustymabe> Most of the freenode staff have resigned and moved to Libera.Chat: 17:07:27 <dustymabe> https://libera.chat/news/welcome-to-libera-chat 17:07:29 <dustymabe> There is already a card on the Fedora Council tracker for this as well: 17:07:31 <dustymabe> https://pagure.io/Fedora-Council/tickets/issue/371 17:07:33 <dustymabe> Does (or should) the FCOS community have a stance on this? 17:07:35 <dustymabe> Full Disclosure: I'm in favour of moving all Fedora channels from Freenode to Libera in view of these developments 17:07:41 <cyberpear> thanks 17:07:48 <lorbus> thanks dustymabe :) 17:08:06 <dustymabe> my personal opinion 17:08:10 <dustymabe> let's do whatever fedora does 17:08:29 <jlebon> dustymabe: +1 17:08:33 <dustymabe> if #fedora-meeting-1 is now on something other than freenode, then we'll move our channels to that new server platform 17:08:42 <jaimelm> Those are lots of words to throw at a person (e.g. "Trumpian"). I haven't investigated it. If true, support packing up and moving. That said, I trust but verify. 17:08:55 <bgilbert> in https://pagure.io/Fedora-Council/tickets/issue/371 it looks like a consensus is developing to move to libera 17:08:58 <dustymabe> in this community I'd like to focus on the tech.. and thankfully we have Fedora to lean on for the other stuff 17:09:41 <darkmuggle> Going with Fedora just makes sense here since its where others in the greater ecosystem will play 17:09:48 <jaimelm> true 17:09:51 <dustymabe> WFM 17:09:51 <bgilbert> darkmuggle: +1 17:10:13 * dustymabe knows there is some matrix bridging and stuff going on. I've been wanting to play with it for a long time 17:10:20 <dustymabe> but it's always been at the bottom of my priority list 17:10:35 <dustymabe> maybe now.. maybe not 17:10:42 <dustymabe> any other topics for open floor? 17:10:56 <lorbus> fwiw I agree with all the statements/concerns above, just wanted to raise awareness. Thanks all! 17:11:05 <dustymabe> lorbus: thanks for doing that! 17:11:07 <dustymabe> lorbus++ 17:11:07 <zodbot> dustymabe: Karma for lorbus changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:11:15 <copperi_> +1 17:12:17 * dustymabe will close meeting in 30 seconds if no other chatter happens 17:12:29 <cyberpear> countme should integrate with the updates polling, IMO 17:12:43 <cyberpear> but I can take that to #fedora-coreos 17:12:51 <jaimelm> we were thiiiiissss close 17:13:04 <bgilbert> lorbus: thanks for raising! 17:13:07 <dustymabe> haha.. yeah #fedora-coreos if you don't mind 17:13:10 <dustymabe> #endmeeting