16:29:53 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:53 <zodbot> Meeting started Wed Dec  8 16:29:53 2021 UTC.
16:29:53 <zodbot> This meeting is logged and archived in a public location.
16:29:53 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:53 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:58 <dustymabe> #topic roll call
16:30:00 <dustymabe> .hi
16:30:01 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:26 <jdoss> .hi
16:30:28 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:30:37 <ravanelli> .hi
16:30:38 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:30:40 <dustymabe> well darn tootin - there's jdoss
16:30:41 <jaimelm> .hello2
16:30:42 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:30:50 <jdoss> Whats up dustymabe my good bud
16:31:02 <dustymabe> oh you know.. just linux stuff
16:31:16 <dustymabe> #chair jdoss ravanelli jaimelm
16:31:16 <zodbot> Current chairs: dustymabe jaimelm jdoss ravanelli
16:31:22 <lucab> .hi
16:31:23 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:31:27 <miabbott> .hello miabbott
16:31:28 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:31:48 <jdoss> dustymabe: yesterday evening I ended up on your blog again for your super sweet ipxe posts
16:31:57 <jlebon> .hello2
16:31:58 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:31 <travier[m]> .hello siosm
16:32:32 <zodbot> travier[m]: siosm 'Timothée Ravier' <travier@redhat.com>
16:33:09 <dustymabe> jdoss: ha - and there's so much content that never makes it to the blog either
16:33:11 <dustymabe> :(
16:33:19 <jdoss> I feel that man
16:33:23 <dustymabe> #chair lucab miabbott jlebon travier[m]
16:33:23 <zodbot> Current chairs: dustymabe jaimelm jdoss jlebon lucab miabbott ravanelli travier[m]
16:33:36 <jdoss> I can't even keep my work notes up to date.
16:34:10 <dustymabe> #chair Saffronique
16:34:10 <zodbot> Current chairs: Saffronique dustymabe jaimelm jdoss jlebon lucab miabbott ravanelli travier[m]
16:34:13 <lorbus> .hi
16:34:14 <saqali_> .hello
16:34:14 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:34:17 <zodbot> saqali_: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:34:29 <saqali_> .hi
16:34:30 <zodbot> saqali_: Sorry, but user 'saqali_' does not exist
16:34:40 <dustymabe> #chair lorbus
16:34:40 <zodbot> Current chairs: Saffronique dustymabe jaimelm jdoss jlebon lorbus lucab miabbott ravanelli travier[m]
16:35:04 <saqali_> .hello saqali
16:35:05 <zodbot> saqali_: saqali 'Saqib Ali' <saqali@redhat.com>
16:35:34 <dustymabe> #topic Action items from last meeting
16:35:43 <dustymabe> Look at that
16:35:47 <dustymabe> No action items from last meeting 🤷‍♂️
16:36:03 <jlebon> 🎉
16:36:11 <jdoss> too much Turkey
16:36:22 <jdoss> at least for us in the US
16:36:41 <miabbott> #chair saqali
16:36:41 <zodbot> Current chairs: Saffronique dustymabe jaimelm jdoss jlebon lorbus lucab miabbott ravanelli saqali travier[m]
16:36:43 <dustymabe> 🦃
16:37:19 <dustymabe> miabbott: haha - I thought I had chaired him already but just realized my tab complete completed to Saffronique instead
16:37:41 <dustymabe> #topic New Package Request: google-authenticator-libpam
16:37:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1037
16:38:21 <dustymabe> travier[m]: and lucab have been chatting back and forth.. either one of you want to intro this one?
16:41:21 * dustymabe picks it up
16:41:50 <dustymabe> looks like a request for  google-authenticator-libpam to enable use of OTP for SSH login
16:42:21 <dustymabe> #chair aaradhak
16:42:21 <zodbot> Current chairs: Saffronique aaradhak dustymabe jaimelm jdoss jlebon lorbus lucab miabbott ravanelli saqali travier[m]
16:42:40 <travier[m]> I can take it
16:42:46 <dustymabe> travier[m]: +1
16:42:56 <jmarrero> .hi
16:42:57 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:43:03 <aaradhak> .hi
16:43:04 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:43:14 <dustymabe> #chair jmarrero
16:43:14 <zodbot> Current chairs: Saffronique aaradhak dustymabe jaimelm jdoss jlebon jmarrero lorbus lucab miabbott ravanelli saqali travier[m]
16:43:16 <travier[m]> Summary: Add a PAM module to enable 2FA authentication over SSH
16:44:02 <lucab> (I implicitly asked for the usecase and when I saw the answer I regretted asking :)
16:44:25 <dustymabe> lucab: you're referring to the last comment (reply) in the ticket?
16:44:38 <travier[m]> What makes it not really great for by-default inclusion: it needs setup (a shared secret between the host and client) and this is a per-node secret
16:45:09 <nemric> hi, sorru
16:45:11 <travier[m]> Nothing prevents this from being first-boot overlayed afaik
16:45:16 <nemric> sorry for being late
16:45:29 <jaimelm> There will be a penalty
16:46:16 <dustymabe> could someone generate a secret once and share it with their fleet (i.e. same OTP for many nodes)?
16:46:33 <dustymabe> jaimelm: explain?
16:46:48 <jaimelm> I was joking about someone being late, not the topic.
16:47:15 <jaimelm> I'm still reading the ticket
16:47:20 <jlebon> travier[m]: to be fair, they replied they do include the shared secret in their config. I presume they have a secure provisioning mechanism
16:47:36 <lucab> I'm not very familiar with SSSD but I think that one as well can be configured to do PAM 2FA
16:47:50 <dustymabe> jaimelm: ha
16:48:10 <jaimelm> yes, it can
16:48:17 <travier[m]> And yes, the use case is not great. Personally I'm moving my servers behind Wireguard instead to avoid SSH issues
16:48:32 <travier> There might be some lag between IRC & Matrix
16:48:51 <jlebon> but yeah, in general I'm in agreement with lucab's comment re. trying to de-emphasize SSH
16:49:05 <travier> Matrix -> IRC is instant but IRC -> Matrix is lagging
16:49:34 <jaimelm> I should switch to Matrix
16:49:54 <dustymabe> de-emphasize is fine, but I really don't think we're going to get rid of it as an escape hatch
16:50:14 <travier> If you use the same secret for your whole fleet then it's no longer really 2FA
16:50:15 <dustymabe> and I commend the effort here by the requestor to try to enhance security
16:50:21 <travier> but this would work :)
16:50:39 <dustymabe> travier: maybe i'm confused.. do you use a different SSH key for every machine you log in to?
16:50:51 <travier> dustymabe: mostly yes :)
16:51:10 <jaimelm> ^^
16:51:13 <jlebon> dustymabe: right, I don't think we should actively try to hinder it
16:51:38 <dustymabe> i've got like 10+ machines here at home. def don't have a different key for each one
16:51:43 <travier> In this case, if you have 100 systems, as OTP is time based, if you have the password or the key, then you get 100*3 tries for the OTP code
16:51:51 <travier> if you have only a single shared secret
16:52:10 <jdoss> I don't think we should try to influence how people consume FCOS too heavily. I feel SSH is fine and I use it a lot across my fleet of FCOS servers.
16:52:23 <travier> I keep the key separated by usage & platform/providers
16:52:32 <travier> OVH has a different one than DO, etc.
16:52:35 <dustymabe> yeah I have a different key per provided
16:52:37 <dustymabe> provider
16:52:49 <lucab> for sure it's staying around, but the less it gets abused to transport other production services, the better
16:53:06 <travier> I'm not against adding it but the experience won't really be better then overlaying on first boot
16:53:49 <dustymabe> anywho - i guess what I'm trying to say is I'm personally a little more on the fence. Though I guess travier has a good point about reduced security with reusing the OTP secret
16:54:27 <lucab> dustymabe: the SSH key is asymmetric (pub+priv). The TOTP seed is a shared secret.
16:54:47 <travier> I feel ssh is fine too and I use it. I don't think we should say either how folks should use FCOS
16:54:55 <travier> I use it too*
16:55:36 <travier> Would adding it improve the experience? Not by that much
16:56:04 <dustymabe> well, if we added it and also provided good documentation for how to use it, might be nice
16:56:13 <dustymabe> but yeah, the docs could just add the layer as well
16:56:28 <jaimelm> SSH is necessary for Ansible if there is config which can't be provided by ignition or unit files (that that it relates to the 2FA aspect of this discussion)
16:56:31 <jdoss> I want to get smallstep.com working on FCOS in the next few months. I think that is going to require some layering however and I accept that on my end. I hope maybe the container layering that is in the works for rpm-ostree makes this a better end user exp.
16:56:33 <lucab> my bottom line is: this is not a common setup nor one we want to recommend users to do. RPM overlaying covers exactly that for people who need to.
16:56:40 <travier> We don't even have an example proving that it will work with a Butane config (I don't know the module storage format)
16:56:43 <dustymabe> also might be nice to investigate what lucab suggested to see if SSSD can handle this use case
16:57:06 <jaimelm> dustymabe: it can
16:57:33 <dustymabe> oh nice
16:57:45 <dustymabe> maybe one of the 8 sssd packages we include could handle this use case then?
16:57:55 <jaimelm> Maybe update the ticket that this is still under discussion and research is being done.
16:58:10 <lucab> there is also an implicit time dependency there. So if NTP on the node is screwed then you also can't access SSH anymore.
16:58:12 <dustymabe> would be nice if we had a volunteer for that "research" part
16:58:23 * jaimelm looks around
16:58:40 <dustymabe> 👀
16:58:44 <travier> jaimelm: are you sure this is handled by SSSD?
16:58:57 <jaimelm> Pretty sure, but I can verify.
16:59:02 <dustymabe> if there was a doc page for SSSD we could point the original requestor to they might be able to do the research
16:59:11 <lucab> which goes back to: let's not mess with SSH, it's an admin/recovery tool, not a general transport for other services
16:59:29 <jdoss> IDK lucab I order lunch over SSH...
16:59:40 <dustymabe> I'm talking to you right now over SSH :)
16:59:49 <jaimelm> same here
16:59:55 <copperi[m]> :)
16:59:56 <jaimelm> SSH forever
17:00:16 <dustymabe> it's hard to get away from it for a single node use case
17:00:20 <nemric> :D do you mean ssh is the future ?
17:00:28 <jaimelm> indeed
17:00:31 <jdoss> and the past
17:00:32 <dustymabe> if you've got a platform like OKD, it's easier to let it fade away
17:01:05 <jaimelm> right, but if you're using nodes without that layer of control, it's necessary (again, Ansible)
17:01:24 <lucab> I do not disagree, but by that line it's also easier to mess and install with a bazillion of random RPMs compared to containers
17:01:44 <travier> While I agree that we should not *emphasize* SSH usage, to me this is more about what adding this will get us
17:01:59 <jaimelm> for example, fifofonix uses many nodes without kubernetes.
17:02:04 <travier> Right now we are not sure this will significantly improve things, so we need the research first
17:02:15 <jaimelm> yep
17:03:01 <dustymabe> #proposed It was suggested that SSSD, which we already ship, can handle 2FA in a similar. We'd like to investigate more to see if the SSSD stack we already ship is a possible solution.
17:03:17 <dustymabe> similar fashion.
17:04:00 <dustymabe> does that reflect the sentiment in the room?
17:04:05 <jdoss> +1
17:04:08 <jaimelm> We use SSSD with Duo on RHEL. So... should be cool.
17:04:11 <lucab> but even if it turns out it isn't, my answer would still be "overlay and customize as you wish"
17:04:23 <travier> would prefer we mention overlaying
17:04:42 <dustymabe> travier: I thought the jury was out on that
17:04:55 <dustymabe> do we want to do a hypothetical scenario?
17:05:09 <jdoss> I think the bigger question is how do we make layering easier and better so users default to that route. That is a topic for a later time I think tho.
17:05:14 <dustymabe> Even if SSSD can't handle 2FA/OTP....
17:05:28 <jaimelm> +1 jdoss
17:05:33 <dustymabe> jdoss: there's open tickets for that
17:05:40 <jdoss> Oh I know :)
17:05:44 <travier> #proposed: We will not add google-authenticator-libpam for now. Overlaying the package on first-boot is the currently recommended approach. It should be investigated if SSSD would be able to perform 2FA.
17:05:47 <jdoss> It's a tough one.
17:06:25 <dustymabe> votes?
17:06:29 <jaimelm> ack
17:06:33 <jdoss> +1
17:06:41 <dustymabe> i'm -0+
17:06:42 <copperi[m]> +1
17:07:12 <nemric> I give my vote to someone who lnow what exactly is sssd :D
17:07:18 <jdoss> lol
17:07:27 <dustymabe> more votes? jlebon lucab others?
17:07:39 <jlebon> +0.75
17:07:42 <lorbus> -1 on this
17:07:56 <lucab> +1 from my side
17:08:02 <travier> lorbus: what are your thoughts?
17:08:13 <jaimelm> lorbus: what are you thinking?
17:08:52 <jlebon> if the SSSD line fails and a lot of people want this, it could make sense to just bake it in
17:09:00 <lucab> also, I'd really like to see an actual Butane configuration of this working on N>1 nodes, either via SSSD or this PAM plugin, before discussing package inclusion
17:09:25 <lorbus> if SSSD works with 2FA, I'm +1 :)
17:09:52 <travier> +1 for lucab's suggestion. I think this should work as layering so we need examples that adding this to the image would significantly improve things
17:10:05 <dustymabe> #agreed We will not add google-authenticator-libpam for now. Overlaying the package on first-boot is the currently recommended approach. It should be investigated if SSSD would be able to perform 2FA.
17:10:12 <dustymabe> ok I'll try to write this up in the ticket
17:10:31 <lucab> also, this isn't the first time we get "I'd like to get package X in FCOS" and it's unclear whether they actual tried to use the proposed software first
17:10:41 <dustymabe> #topic Make it easier to change crypto policy
17:10:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/607
17:10:55 <jaimelm> this will be a fun one.
17:10:58 <dustymabe> jlebon: you opened this a while ago..
17:11:19 <jdoss> BTC miners by default in FCOS???
17:11:21 <dustymabe> I'm starting to try to go through old tickets and push forward discussion/action or close out as irrelevant.
17:11:26 <jdoss> I'll see myself out
17:11:35 <dustymabe> 🚪
17:11:52 <travier> jdoss: :)
17:11:55 <jlebon> dustymabe: blast from the past :)
17:11:57 <lorbus> I'd like to have the 2FA feature. the pam module is small so I feel it's OK to include. If it works without that natively with SSSD, even better.
17:12:26 <dustymabe> lorbus: you can help answer our open questions in the ticket then.. more discussion will be had there for sure
17:12:27 <travier> lorbus: note that Matrix is lagging behind today
17:12:40 <jaimelm> lorbus: thanks
17:13:18 <travier> For this one, Butane sugar might not be that appropriate as it would change meaning depending on the config actually available on the system
17:13:26 <dustymabe> jlebon: so the current suggestion was to add Butane sugar to set the policy rather than someone running `update-crypto-policies`
17:13:29 <dustymabe> travier: correct
17:13:43 <travier> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening
17:14:36 <jaimelm> mmmmm FIPS
17:14:41 <travier> But it would only set the symlinks to a named version so this would still be the same independent of any version
17:14:49 <travier> FIPS is not supported on FCOS
17:14:56 <jaimelm> I know. Believe me I know.
17:15:14 <jaimelm> It's why I can't use it in some of my work environments.
17:15:14 <dustymabe> the alternative was to try to get someone to rewrite `update-crypto-policies`
17:15:43 <dustymabe> jlebon: what's the suggested path forward now
17:16:12 <jlebon> travier: hmm, if e.g. FUTURE changes meaning, that's common to all systems that ship crypto-policies, no?
17:16:22 <jlebon> do we need to insulate FCOS users specifically from that?
17:16:22 <travier> This is the same thing as the alternatives issue. We should write a short workaround Butane config with the symlinks while we wait for a rewrite of the tool
17:16:57 <travier> jlebon: no, you're right.
17:17:01 <jlebon> but yeah, the annoying thing is that there's no butane-time way to know what is a valid policy
17:17:08 <travier> yes
17:17:37 <jlebon> mostly agreed re. just documenting Butane snippets for now
17:17:41 <travier> Just like alternatives, we need a Rust/Go rewrite of this small tool.
17:17:42 <dustymabe> could we use toolbox for this?
17:18:05 <travier> dustymabe: how would that help?
17:18:06 <jlebon> dustymabe: this was discussed in the thread IIRC
17:18:35 <dustymabe> run `update-crypto-policies` from inside a toolbox container (target the host)
17:18:37 <lorbus> wow Matrix is way behind on my end today, it's arriving here all out of order. My vote doesn't make a lot of sense now. +1 on supporting 2FA in SSSD :)
17:18:43 <jlebon> it works, but the problem is version skew
17:18:53 <travier> https://discussion.fedoraproject.org/t/strong-crypto-settings-how-best-to-define-apply/22600/16?u=siosm
17:19:12 <dustymabe> travier: sorry - oh man - there was a lot of context there
17:19:24 <travier[m]> lorbus: I dropped from Matrix and re-joined from IRC
17:19:42 <jaimelm> IRC and SSH, the past and the future.
17:20:18 <dustymabe> jlebon: "version skew" - I need to understand that more
17:20:29 <nemric> No ssh here, just an alpine on WSL windows machine
17:20:36 <dustymabe> what if you had the same version packages in the container as on the host?
17:20:46 <travier> dustymabe: if you define custom policies, they will drift WRT the default ones
17:20:51 <jlebon> then it'd work, yeah
17:21:04 <travier> but that happens no matter what
17:21:06 <jlebon> but the UX is super dubious and I wouldn't want to document it
17:21:13 <dustymabe> we have the updates-archive repo now, we could make sure we get the same packages in the container
17:21:28 <dustymabe> jlebon: agree, it's a bit awkward
17:21:41 <dustymabe> just trying to get something out of these lemons
17:21:53 <jlebon> plus, you usually want it right from the start, not after e.g. pulling containers
17:22:04 <jlebon> dustymabe: hehe
17:22:31 <dustymabe> it would be really nice if we didn't have to run a dynamic tool `update-crypto-policies`
17:22:39 <jaimelm> ^^
17:23:22 <dustymabe> anyone want to put forth a #proposed on this one?
17:24:20 <jlebon> #proposed we will recommend using Butane to set symlinks for now and approach upstream on whether there are any plans to rewrite the tool
17:24:30 <jaimelm> are folks generally in agreement on documenting a Butane solution first?
17:24:44 <jaimelm> there we go
17:24:45 <jaimelm> ack
17:24:48 <dustymabe> seems reasonable as it's the quickest path to getting something
17:25:05 <dustymabe> ack / +1
17:25:29 <travier> This could be an outreachy project
17:25:31 <jlebon> there's a strongly related item here, which is that FCOS isn't fully wired for FIPS mode
17:25:32 <travier> https://gitlab.com/redhat-crypto/fedora-crypto-policies
17:25:46 <jaimelm> jlebon++
17:25:46 <zodbot> jaimelm: Karma for jlebon changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:26:29 <dustymabe> at least update-crypto-policies is less than 500 lines of code
17:26:37 <dustymabe> any other votes on the current proposed?
17:26:38 <jlebon> in the case where what the users actually want is FIPS, we should have better sugar for that
17:26:56 <jaimelm> Indeed
17:27:08 <travier> +1 for proposed
17:27:34 <dustymabe> ok we have 3 +1
17:27:41 <jlebon> #agreed we will recommend using Butane to set symlinks for now and approach upstream on whether there are any plans to rewrite the tool
17:28:18 <dustymabe> #topic open floor
17:28:27 <dustymabe> we're close to time so I'll skip other topics
17:28:33 <dustymabe> anybody have anything for open floor today?
17:28:59 <travier> Should we cancel the meetings during the holidays?
17:29:04 <dustymabe> #info we'll probably skip the community meeting on 12/22 and 12/29 for the holidays.
17:29:04 <jlebon> yes, we have kubernetes tests running in the pipeline now!
17:29:05 <travier> (I won't be there)
17:29:07 <jlebon> https://github.com/coreos/fedora-coreos-pipeline/blob/main/jobs/kola-kubernetes.Jenkinsfile
17:29:13 <dustymabe> travier: :)
17:29:16 <travier> :)
17:29:17 <nemric> Is there someine working on user level units ?
17:29:21 <dustymabe> jlebon: nice! is it passing yet?
17:29:25 <jaimelm> jlebon: cool
17:29:26 <jlebon> dustymabe: we'll find out soon :)
17:29:34 <travier> jlebon: nice!
17:29:36 <dustymabe> nemric: you mean Ignition support for user systemd units?
17:29:45 <dustymabe> jlebon++
17:29:52 <travier> jlebon++
17:30:04 <nemric> @dustymabe yes
17:30:15 <dustymabe> I don't know, which means probably not
17:30:21 <dustymabe> would you want to work on it with some guidance?
17:30:24 <nemric> :D
17:30:51 <nemric> oh ! not sur being able to do that
17:31:02 <dustymabe> #info I just learned you can actually make API calls and get an s390x cloud instance (with /dev/kvm) in IBM Cloud
17:31:15 <lucab> woah
17:31:30 <dustymabe> nemric: it's OK, just let me know if you want to and we can try to find people to help
17:31:55 <dustymabe> https://cloud.ibm.com/docs/vpc?topic=vpc-profiles&interface=ui#balanced-s390x-profiles
17:32:02 <dustymabe> only in the Tokyo region ^^
17:32:27 <dustymabe> any other topics for open floor?
17:32:33 <nemric> ok, just let me know later how to contribute in a staightforward way and only a poor windows laptop ^^
17:32:57 <nemric> I'd like to try
17:33:19 <dustymabe> +1
17:33:23 * dustymabe closes meeting soon
17:33:30 <lucab> dustymabe: does the last part of your sentence mean that they have nested virtualization?
17:34:06 <dustymabe> lucab: I assume so. The device exists in this instance I'm playing with but haven't tried anything
17:34:21 <dustymabe> so it might exist and not work
17:34:23 <jaimelm> dustymabe: loop me in on that. Having written that unit documentation a while back, it's still in my head.
17:34:43 <dustymabe> jaimelm: i.e. you want to help add the feature to Ignition?
17:34:49 <jaimelm> yessir
17:34:54 <dustymabe> wow, that's awesome
17:35:04 <dustymabe> will chat with you all more over in #fedora-coreos
17:35:06 <dustymabe> #endmeeting