16:30:35 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:35 <zodbot> Meeting started Wed Oct 14 16:30:35 2020 UTC.
16:30:35 <zodbot> This meeting is logged and archived in a public location.
16:30:35 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:35 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:40 <dustymabe> #topic roll call
16:30:43 <dustymabe> .hello2
16:30:44 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:48 <cyberpear> .hello2
16:30:49 <lucab> .hello2
16:30:49 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:30:49 <aoei> .hello2
16:30:52 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:55 <zodbot> aoei: aoei 'Joanna Doyle' <jjadoyle@gmail.com>
16:30:55 <red_beard> .hello redbeard
16:30:58 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:31:02 <bgilbert> .hello2
16:31:04 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:28 <lorbus> .hello2
16:31:31 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:31:38 <nasirhm> .hello2
16:31:39 <zodbot> nasirhm: nasirhm 'Nasir Hussain' <nasirhussainm14@gmail.com>
16:32:23 <dustymabe> #chair cyberpear lucab aoei red_beard bgilbert lorbus nasirhm
16:32:23 <zodbot> Current chairs: aoei bgilbert cyberpear dustymabe lorbus lucab nasirhm red_beard
16:32:38 <jlebon> .hello2
16:32:39 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:39 <skunkerk> .hello sohank2602
16:32:42 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:32:54 <dustymabe> #chair jlebon skunkerk
16:32:54 <zodbot> Current chairs: aoei bgilbert cyberpear dustymabe jlebon lorbus lucab nasirhm red_beard skunkerk
16:33:02 <dustymabe> nice crowd today :)
16:33:13 <dustymabe> #topic #topic Action items from last meeting
16:33:16 <dustymabe> #undo
16:33:16 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7fccda62f6d0>
16:33:23 <dustymabe> #topic Action items from last meeting
16:33:38 <dustymabe> we had one re-action from last meeting
16:33:40 <dustymabe> * jlebon to reach out to the rpm maintainers to see if the relocation of
16:33:42 <dustymabe> the rpmdb path is something they're willing to own for F34
16:34:36 <jlebon> oh shame
16:35:31 <jlebon> how about i do this right now during the meeting :)
16:35:58 <dustymabe> WFM :)
16:36:09 <dustymabe> ok moving on to topics
16:36:24 <dustymabe> #topic daylight savings time changes and meeting time
16:36:57 <dustymabe> For some countries daylight savings time is changing over the coming month
16:37:00 <nasirhm> Hmm, What'll be the next meeting time ?
16:37:45 <dustymabe> In the past we have stayed on UTC time, so it affects all locals equally. You just translate to your local time from UTC and it's up to you to handle the change (the fedora calendar also helps)
16:37:47 <nasirhm> (In UTC)
16:37:49 <red_beard> is the meeting not sticking to UTC? or is this just informational?  a few cycles ago i made sure the event in my calendar was UTC.
16:38:08 <dustymabe> red_beard: I propose we stick with UTC
16:38:15 <red_beard> seconded
16:38:22 <nasirhm> thirded
16:38:23 <red_beard> time is a constant. humans muck it up.
16:38:42 <bgilbert> +1
16:38:51 <aoei> +1
16:39:13 <dustymabe> #action dustymabe to send an email to the coreos list (not status) with a PSA about the meeting time sticking with UTC
16:39:22 <dustymabe> sound good?
16:39:36 <nasirhm> +1 :)
16:39:47 <dustymabe> ok moving on
16:39:48 <jlebon> +1 (well, really -1h for me)
16:40:26 <dustymabe> #topic tracker: Fedora 33 rebase work
16:40:43 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/609
16:40:46 <dustymabe> just a few updates here
16:41:14 <dustymabe> for the systemd-resolved migration of existing systems that upgrade we have a proposal
16:41:26 <dustymabe> outlined here: https://github.com/coreos/fedora-coreos-tracker/issues/646#issuecomment-707984576
16:42:08 <dustymabe> Thanks to a user identifying a potential issue we know some users will be affected, but we're planning a coreos-status post to detail the steps that need to be taken
16:42:34 <dustymabe> the vast majority of users will be able to migrated automatically
16:42:41 <cyberpear> does our fcct sugar not handle masking units?
16:43:07 <dustymabe> cyberpear: see the last three comments in the thread :)
16:44:15 <jlebon> dustymabe: one last question about this i had: should we just make this dumber and match what the fedora scriptlet is doing?
16:44:33 <jlebon> i.e. only care about `systemctl is-enabled` and not look at the contents of `/etc/resolv.conf` at all
16:44:57 <dustymabe> the only thing I worry about there is if somehow the user was managing their own resolv.conf
16:45:21 <dustymabe> jlebon: but I wouldn't be opposed if we think that's the right thing to do
16:46:38 * dustymabe was trying to be careful
16:46:58 <jlebon> yeah, understood. i just like the elegance of having a super simple contract for this, and also matching the rest of Fedora
16:47:21 <dustymabe> any others want to argue one way or the other?
16:47:22 <jlebon> if users want to manage their own resolv.conf they probably really want to disable resolved too
16:47:46 <jlebon> so being more strict like that will flush those use cases out, if any
16:47:55 <dustymabe> jlebon: right, but.. disabling resolved does require manual intervention at this point
16:48:23 <dustymabe> so in the past if we could auto migrate 90% maybe it drops down to 80%
16:49:09 <cyberpear> the only part of the change I don't like is replacing /etc/resolv.conf with a pointer to 127.0.0.53 by default. and especially for problematic for upgrades
16:49:43 <cyberpear> (I think the recent devel list thread was 200 messages)
16:49:53 <cyberpear> I think we should follow the rest of Fedora, but still try not to break our users
16:50:17 <dustymabe> cyberpear: i think that's what we're trying to do here, right?
16:50:21 <dustymabe> follow fedora, not break users
16:50:26 <cyberpear> yep!
16:50:40 <jlebon> dustymabe: i'm good with the current strategy i think, just wanted to explore this a bit more
16:50:53 <dustymabe> basically /etc/resolv.conf gets pointed to a file managed by systemd-resolved
16:51:13 <dustymabe> though cyberpear does bring up a good point we should try to consider
16:51:21 <dustymabe> which is, rollback
16:51:45 <dustymabe> jlebon: that change to /etc/resolv.conf would or would not get rolled back?
16:52:10 <jlebon> yup, /etc is bound to the deployment
16:52:30 <dustymabe> ok i'll run through that test just to make sure when we get this implemented
16:52:54 <dustymabe> the only other update I had for f33 is that we tracked one test failure down to a SELinux bug
16:53:13 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/643#issuecomment-708417208
16:53:30 <dustymabe> and that `next` is now based on F33 so please try to use it and give some feedback on what's broken
16:53:55 <dustymabe> ok next topic
16:54:02 <dustymabe> #topic F33 feature/change proposal SwapOnZRAM by default
16:54:12 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/509
16:54:27 <dustymabe> now that we have fedora 33 out we can consider the swaponzram change
16:55:07 <dustymabe> I think the running proposal is that we don't enable swaponzram by default because of requires of kubernetes and other clustered offerings on top that don't work by default with it enabled
16:55:25 <dustymabe> so I can up with two options for adding it but not having it actually on by default
16:55:33 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/509#issuecomment-708408426
16:56:00 <dustymabe> In one case we include both packages (including the default file) and override with a file in /etc/ that can be easily deleted
16:56:20 <dustymabe> In the other case we just include the one package and then the user can create a zram conf file on their own
16:56:38 <dustymabe> I think the 2nd option is probably most natural
16:56:38 <nasirhm> dustymabe: I would like to ask, does SwapOnZRAM has use cases on Containerized Workloads ?
16:57:06 <dustymabe> nasirhm: i would assume it has use cases in any scenario you're running low on RAM
16:57:23 <lucab> dustymabe: I share the same feeling
16:58:07 <nasirhm> dustymabe: Hmm, makes sense, We can add the one package and let the user create a zram conf file and document it in our docs.
16:58:22 <dustymabe> just anecdotally - i have a laptop with 8G RAM. it was starting to run sluggish over time and would often lock up a bit with a lot running on it. After upgrading to F33 it's pretty darn smooth and no OOMs
16:58:28 <cyberpear> I'd advotate adoping the SwapOnZRAM change in full, provided we don't break OKD, or if it would break them, see if it can be unbroken OKD-side.
16:59:45 <dustymabe> cyberpear: yeah, it would require trying to work with the various upstreams (OKD is one, typhoon another, etc..) to make sure it gets properly disabled
17:00:03 <cyberpear> ...or make them run properly with it enabled
17:00:09 <dustymabe> but I think for right now maybe we should include, but not default to on, then re-evaluate for f34 switch
17:00:23 <dustymabe> cyberpear: yeah there is an open issue against k8s to have it properly handled
17:00:28 <nasirhm> dustymabe: +1
17:00:37 <dustymabe> https://github.com/kubernetes/kubernetes/issues/53533
17:01:06 <cyberpear> dustymabe++ thanks for linking the upstream k8s swap issue!
17:01:12 <bgilbert> that issue has been open for a while
17:01:25 <bgilbert> so I wouldn't bank on it spontaneously resolving itself :-(
17:01:44 <dustymabe> bgilbert: yep. open source is the long game
17:01:47 <bgilbert> right
17:01:58 <dustymabe> so.. I'll make a proposal
17:03:10 <dustymabe> #proposed we'll include the zram-generator package for now, which will allow users to drop down a config file to enable swaponzram. In the future we'll re-evaluate if creating a swaponzram device by default, is the right thing for us to do.
17:03:22 <dustymabe> oh also, we'll add docs to show users how to do this.
17:03:34 <dustymabe> ^^ will add to the #agreed if the proposal passes.
17:04:16 <lucab> +1
17:04:17 <jlebon> makes sense to me
17:04:20 <bgilbert> +1
17:04:32 <nasirhm> SGTM
17:04:39 <red_beard> +1
17:05:25 <dustymabe> #agreed We'll include the zram-generator package for now, which will allow users to drop down a config file to enable swaponzram. Additionally we'll add docs to show users how to do this. In the future we'll re-evaluate if creating a swaponzram device by default, is the right thing for us to do.
17:05:30 <dustymabe> thanks :)
17:05:47 <dustymabe> #topic no cloud agents: gcp
17:05:51 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/67
17:06:08 <dustymabe> welcome back to one of the original issues of all time :)
17:06:47 <dustymabe> I have a periodic meeting with Google engineering and they ask about this.
17:07:04 <dustymabe> Basically OSLogin support is considered by them to be essential to their platform.
17:07:23 <dustymabe> Should we re-visit having this functionality in FCOS for GCP?
17:07:52 <dustymabe> If so what do we think would be the best way to implement it? Using Google's code, or our own minimal re-implementation?
17:08:02 <cyberpear> "Long term: Replace the platform agents, where possible, with our own minimal implementations. Ship those as part of the OS." https://github.com/coreos/fedora-coreos-tracker/issues/12#issuecomment-408112263
17:08:27 <bgilbert> there's a couple pieces
17:08:41 <bgilbert> OS Login is an NSS module and a PAM module and I think some glue
17:09:04 <bgilbert> it's pretty minimal IIRC.  the issue for us is that we'd need to conditionally swap out nsswitch.conf and pam configs
17:09:23 <bgilbert> on CL we had a firstboot systemd unit that ran a shell script that did this
17:09:24 <red_beard> ...for a single platform
17:09:31 <bgilbert> red_beard: yes, exactly
17:09:32 <jlebon> bgilbert: you mean it's not a static change?
17:10:19 <bgilbert> jlebon: on CL we let users disable it
17:10:27 <bgilbert> maybe we don't want to do that in FCOS
17:10:28 <davdunc> .hello2
17:10:29 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
17:10:39 <dustymabe> 👋 davdunc
17:10:43 <davdunc> :D
17:10:46 <bgilbert> but either way, we need to enable the NSS/PAM modules only on GCP
17:10:48 <nasirhm> davdunc 👋
17:11:05 <bgilbert> so that's either a platform-specific Ignition base config or a platform-conditional unit
17:11:33 <bgilbert> but all of the OS Login stuff is notionally separate from the question of whether we run an agent
17:11:38 <bgilbert> which IMO we should not
17:11:38 <dustymabe> hmm. so that's at least two solutions to the problem and neither sound that bad
17:11:59 <red_beard> as far as i'm concerned, unless google has provided (even a reference implementation) the server side pieces to make this generally useful for others in the future, they can pound sand.
17:12:05 <bgilbert> I know there's _some_ integration between OS Login and the agent but I think it's only around enablement of OS Login
17:12:18 <cyberpear> platform-conditional unit sounds better to me so you could have a more similar image for each platform
17:12:22 <bgilbert> red_beard: that's generally not the standard
17:12:41 <bgilbert> e.g. there's no reference implementation of the DigitalOcean metadata server but we still need to ask it for an IP address
17:12:58 <red_beard> boldly go
17:13:29 * bgilbert shrugs
17:13:41 <dustymabe> bgilbert: it sounds like (from the previous CL work) we already know what modifications we'd need to make to the system to get this to work?
17:13:47 <red_beard> it's issues like this which continue to drive platform specific behavior.  is there a line or is it a free for all?
17:14:02 <bgilbert> red_beard: we generally try to hold a line, but it's ad-hoc
17:14:03 <jlebon> from the user side, have we actually had requests to make OS login work?
17:14:20 <bgilbert> dustymabe: ajeddeloh did that work, so... not really
17:14:25 * dustymabe naively wishes most clouds were just based on openstack, then everyone can contribute to one platform and it helps them all
17:14:28 <bgilbert> (he's no longer on the project)
17:14:54 <bgilbert> dustymabe: but we can probably get some guidance from our GCP contacts
17:14:56 <dustymabe> bgilbert: right, but hopefully we could glean something from what he did
17:15:22 <bgilbert> GCP has since ported their agent to Go and it's possible that there are more/different interdependencies now
17:15:29 <dustymabe> got ya
17:15:37 <bgilbert> also on CL we shipped the agent in, I think, a rkt fly container
17:15:41 <bgilbert> so not the same situation
17:16:13 <dustymabe> but in general we think we should be able to get away with a platform dependent systemd unit that modifies a few files
17:16:30 <bgilbert> jlebon: not AFAIK, but there's been an understanding that we need to support this eventually to be in the primary list of recommended OSes
17:16:30 <dustymabe> and then magic.. oslogin works?
17:16:52 <dustymabe> there was at least one issue reported I think about OSLogin
17:17:41 <dustymabe> and my searching is failing
17:18:08 <dustymabe> I think I found it
17:18:11 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/405#issuecomment-596363000
17:18:53 <dustymabe> So several users in there. I think at least a few of them were wanting OS Login
17:19:36 <jlebon> heh, if google is pushing for that in the web UI, we'll keep having users hitting that
17:19:46 <dustymabe> ok so I think we should probable create a new issue specifically for OS Login and probably close out the gcp no agents ticket
17:20:22 <dustymabe> anything else we'd like to discuss here before moving to open floor?
17:20:58 <davdunc> dustymabe: you may have a similar request on the ec2-instance-connect at some point.
17:21:14 <davdunc> I'll help you get you what you need if that becomes a request.
17:21:16 <dustymabe> equivalent of OS Login for AWS EC2 ?
17:21:23 <davdunc> similar.
17:21:26 <dustymabe> +1
17:21:51 <dustymabe> davdunc: WDYT about rebasing EC2 on OpenStack?
17:21:59 <dustymabe> 🤣
17:22:00 <davdunc> :D
17:22:12 <dustymabe> #topic open floor
17:22:18 <davdunc> I'll hold my comments on that one. :P
17:22:18 <dustymabe> anyone with anything for open floor today?
17:22:47 <dustymabe> as always here's a reminder to add the `meeting` label to issues you'd like to discuss, or comment in the issue if you don't have permissions to add that label
17:22:56 <nasirhm> Nothing from my side, I'll be working on docs to add core user info in different Cloud providers pages.
17:23:04 <dustymabe> thanks nasirhm!
17:23:10 <dustymabe> we'll be glad to have that
17:23:26 <nasirhm> dustymabe: Welcome ^^ :)
17:24:03 <davdunc> nasirhm: if you have anything you want to collaborate/review on the AWS side, let me know.
17:24:26 <dustymabe> anybody here got some local servers?? Assuming they're not critical, switch over to `next` and let us know what breaks :)
17:24:32 <nasirhm> Sure, Thanks davdunc .
17:25:31 <dustymabe> #endmeeting