16:00:53 <petebuffon[m]> #startmeeting Infrastructure (2021-09-02)
16:00:53 <zodbot> Meeting started Thu Sep  2 16:00:53 2021 UTC.
16:00:53 <zodbot> This meeting is logged and archived in a public location.
16:00:53 <zodbot> The chair is petebuffon[m]. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:53 <zodbot> The meeting name has been set to 'infrastructure_(2021-09-02)'
16:00:58 <petebuffon[m]> #meetingname infrastructure
16:00:58 <zodbot> The meeting name has been set to 'infrastructure'
16:01:04 <petebuffon[m]> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:01:04 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik petebuffon[m] pingou siddharthvipul zlopez
16:01:26 <petebuffon[m]> #info Agenda is at: https://board.net/p/fedora-infra
16:01:27 <petebuffon[m]> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:01:33 <petebuffon[m]> #topic greetings!
16:01:41 <petebuffon[m]> .hello petebuffon
16:01:42 <zodbot> petebuffon[m]: Something blew up, please try again
16:01:43 <nirik> morning everyone.
16:01:43 <siddharthvipul> \o
16:01:44 <zodbot> petebuffon[m]: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:47 <siddharthvipul> .hello siddharthvipul1
16:01:48 <zodbot> siddharthvipul: Something blew up, please try again
16:01:51 <zodbot> siddharthvipul: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:52 * nirik has to go get a cup of coffee, back in a few.
16:01:56 <siddharthvipul> zodbot: there there
16:02:06 <petebuffon[m]> How's zodbot doing?
16:02:10 <darknao> .hi
16:02:12 <zodbot> darknao: Something blew up, please try again
16:02:15 <zodbot> darknao: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:02:31 <darknao> :/
16:02:42 <siddharthvipul> we still love you zodbot
16:02:47 <dtometzki> hello together
16:03:03 <petebuffon[m]> 😆
16:03:08 <dtometzki> .hello dtometzki
16:03:09 <zodbot> dtometzki: Something blew up, please try again
16:03:15 <zodbot> dtometzki: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:03:22 <t0xic0der> .hello2
16:03:23 <zodbot> t0xic0der: Something blew up, please try again
16:03:26 <zodbot> t0xic0der: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:03:31 <mobrien> .hi
16:03:32 <zodbot> mobrien: Something blew up, please try again
16:03:34 <zodbot> mobrien: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:03:44 <ekidney> o/ hi all
16:03:45 <dtometzki> zodbot isnt our friend :-)
16:04:02 <t0xic0der> Hello folks!
16:04:31 <petebuffon[m]> Well zodbot seems to be having some trouble, I guess we'll just move on
16:04:39 <petebuffon[m]> #topic New folks introductions
16:04:39 <petebuffon[m]> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:39 <petebuffon[m]> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:03 <petebuffon[m]> Any new folks today?
16:05:33 <siddharthvipul> me me
16:05:39 <siddharthvipul> well, after some time for sure
16:05:47 <lenkaseg> .hi
16:05:48 <zodbot> lenkaseg: Something blew up, please try again
16:05:51 <zodbot> lenkaseg: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:05:54 <t0xic0der> siddharthvipul: Let me open you a welcome ticket :)
16:05:58 <siddharthvipul> :V
16:07:33 <petebuffon[m]> tricks I tell you, tricks
16:07:33 <nirik> siddharthvipul: any relation to siddharthvipul1 ? :)
16:07:33 <siddharthvipul> nirik: that's my evil cousin
16:07:33 <siddharthvipul> oh oh, twin I meant
16:07:53 <nirik> .hello kevin
16:07:54 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
16:07:54 <petebuffon[m]> #topic Next chair... (full message at https://libera.ems.host/_matrix/media/r0/download/libera.chat/a22c8c5d2a7a485e0808cb6dee7167d3efe7e251)
16:08:25 <petebuffon[m]> looks like we have chairs for the next two meetings.
16:08:28 <siddharthvipul> petebuffon[m]: didn't get all the messages this side (from matrix I guess) it puts messages in link if they are multiline
16:08:33 <dtometzki> anything got wrong
16:08:53 <petebuffon[m]> siddharthvipul: hmm okay thanks
16:09:44 <petebuffon[m]> siddharthvipul: which messages did you get?
16:09:54 <siddharthvipul> let's see a nice way to share
16:09:56 <dtometzki> then you need to use the old irc hexchat client :-)
16:10:36 <petebuffon[m]> right I am using the matrix element client
16:11:22 <petebuffon[m]> #topic Next chair
16:11:23 <siddharthvipul> petebuffon[m]: blob:https://imgur.com/9f7b9037-9c24-4123-9b02-3394c4c0b73c
16:11:46 <petebuffon[m]> #info magic eight ball says:
16:11:52 <petebuffon[m]> #info chair 2021-09-02 - petebuffon
16:12:03 <petebuffon[m]> #info chair 2021-09-09 - eddiejennings
16:12:10 <petebuffon[m]> #info chair 2021-09-16 - dtometzki
16:12:15 <petebuffon[m]> #info chair 2021-09-23 - ??
16:12:26 <siddharthvipul> petebuffon[m]: hmm, let's try this if the last link didn't work: https://imgur.com/gsd38ZL petebuffon[m] ^ this is what I saw
16:13:04 <petebuffon[m]> Got it okay noted, I'll either put in one line at a time or next time I'll use hexchat
16:13:51 <petebuffon[m]> #topic announcements and information
16:13:55 <petebuffon[m]> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:14:00 <petebuffon[m]> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:14:05 <petebuffon[m]> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:14:09 <petebuffon[m]> #info Nest with Fedora is over, videos to be posted in sept
16:14:14 <petebuffon[m]> #info Fedora 35 Beta freeze is now in effect 2021-08-24 -> 2021-09-14
16:14:20 <petebuffon[m]> #info monday (2021-09-06) is a holiday in the US
16:15:39 <dtometzki> long weekend for US
16:16:15 <petebuffon[m]> just trying to avoid the smoke in California
16:17:06 <nirik> not at a national park tho. ;(
16:17:22 <nirik> (they closed all the national parks in ca, us due to fire danger)
16:17:57 <petebuffon[m]> hah, yeah trees are just fires waiting to happen
16:17:59 <petebuffon[m]> anyways on call
16:18:02 <petebuffon[m]> #topic Oncall
16:18:07 <petebuffon[m]> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:18:10 <petebuffon[m]> #info nirik on call for 2021-08-26 to 2021-09-02
16:18:17 <petebuffon[m]> #info dtometzki on call for 2021-09-02 to 2021-09-09
16:18:21 <petebuffon[m]> ## .oncalltakeeu .oncalltakeus
16:18:39 <dtometzki> .oncalltakeeu
16:18:39 <zodbot> dtometzki: Kneel before zod!
16:19:24 <petebuffon[m]> great
16:19:35 <nb> hi
16:19:36 <petebuffon[m]> any pings this week?
16:20:11 <nirik> I had 2 oncall pings... one was when I was asleep and I never found out what they wanted... the other was a user registration problem which I fixed. ;)
16:20:43 <petebuffon[m]> #info Summary of last week: (from current oncall )
16:20:51 <petebuffon[m]> #topic Monitoring discussion [nirik]
16:20:52 <petebuffon[m]> #info https://nagios.fedoraproject.org/nagios
16:21:03 <petebuffon[m]> #info Go over existing out items and fix
16:21:04 <petebuffon[m]> nice doesn't sound too bad
16:21:28 <nirik> yeah, been pretty quet.
16:21:31 <nirik> anyhow, on monitoring
16:22:11 <nirik> we have one machine down... mobrien was working on bringing it back
16:22:23 <nirik> it had a cpu/motherboard replacement and I don't think it's coming up right
16:22:52 <nirik> There's a PR to fix hopefully all the ocp (openshift 4 cluster) hosts showing alerts.
16:23:16 * nirik looks for it
16:24:02 <nirik> now I can't find it.
16:24:12 <nirik> anyhow, mostly we are looking ok...
16:24:14 <mobrien> This one https://pagure.io/fedora-infra/ansible/pull-request/765
16:24:15 <nirik> we can move on
16:24:26 <petebuffon[m]> got it
16:24:50 <petebuffon[m]> and thanks all for bearing with me while I'm still figuring out irc and chairing these meetings
16:25:19 <petebuffon[m]> ready for the learning topic nirik ?
16:25:42 <nirik> mobrien: that was merged?
16:26:02 <nirik> the one I was thinking of was from smooge...
16:26:40 <mobrien> Oh yes this one https://pagure.io/fedora-infra/ansible/pull-request/762
16:27:07 <mobrien> not sure if it was run or not
16:27:09 <nirik> thats it. I wonder if I just didn't run the playbook after that or it did not work
16:27:15 * nirik can run it now and see
16:28:26 <nirik> anyhow, we can move on
16:28:47 <petebuffon[m]> #topic Learning topic
16:28:52 <petebuffon[m]> #info 2021-09-02 - ssh host keys signing and ansible ssh interactions [nirik]
16:29:27 <nirik> ok. :) lets dive in. Feel free to ask questions anytime
16:30:07 <nirik> So, most of you may know ssh (secure shell). It's a way to securely login to machines and get a shell/command line and do things. It's used by everyone.
16:30:51 <nirik> Normally, ssh uses keys for it's operations. There's host keys generated when you first start sshd on a rhel/fedora/centos host.
16:31:10 <nirik> and on the user side users generate a user key (public and private pair).
16:31:56 <nirik> so when you connect to a server with your client, your client first gets the server key, checks if it's connected before and if it's the same key
16:32:21 <nirik> this is when you can get that anoying prompt if the host key has changed or you haven't connected before.
16:33:53 <nirik> However, as fewer people know, ssh can also use certs instead of keys for things.
16:33:54 <petebuffon[m]> are server keys normally rotated at all?
16:34:28 <nirik> They are not, except by newer key types. There's a server key of each type you tell it to make... and over time the older ones are not user and newer ones are
16:34:59 <nirik> in our infra we use a certificate signing setup for host keys. This has a number of advantages.
16:35:00 <petebuffon[m]> got it thanks :)
16:35:55 <nirik> so when we install a new host/vm... one of the first thigns ansible runs on it is the 'basessh' role.
16:36:06 <nirik> roles/basessh/ in ansible repo.
16:36:54 <nirik> what this does is gathers the host keys, signs them with out CA (certificate authority) and puts those signatures back on the host.
16:37:02 <nirik> then configures it to use them etc.
16:37:24 <nirik> This means when you connect the host offers you the signature too.. and you can tell it was signed with our CA and can just trust it.
16:37:42 <nirik> https://admin.fedoraproject.org/ssh_known_hosts
16:37:57 <nirik> this is what goes in your ~/.ssh/known_hosts to trust our CA.
16:38:58 <petebuffon[m]> So if they weren't signed then the keys could have been modified without your knowledge?
16:40:02 <nirik> well, no? I mean if someone modifies a host key... if you have never connected to that host, you wouldn't know it. If you have connected to that host you would get a warning that the host key you have stored for it has changed.
16:40:03 <dtometzki> but i have no such entry as described in the link
16:40:30 <nirik> the cert allows you to trust the CA instead of saying 'yes' to each host key
16:41:05 <nirik> dtometzki: you can download it and use if you like. Mainly it's in use by our ansible control host.
16:41:26 <petebuffon[m]> hmm okay that makes sense
16:41:34 <dtometzki> ahh ok understand
16:43:00 <nirik> so, our ansible control host (batcave01) has this setup, so it can always know the host keys it's seeing are signed by our CA (except for that very first time we gather and sign them)
16:43:27 * nirik thinks what else to go over here. Anyone have questions so far, or related questions?
16:44:02 <dtometzki> no
16:44:15 <petebuffon[m]> what software is used as the CA?
16:44:36 <nirik> you can also use certs as a replacement for user keys. ;) There's a interesting section about this in ssh-keygen man page.
16:45:21 <nirik> petebuffon[m]: it's part of ssh-keygen I think. ssh certs are similar to, but not 100% identical to regular certs.
16:45:48 <petebuffon[m]> and a cert is just a key with metadata? in the x509 format right?
16:46:27 <nirik> "Note that OpenSSH certificates are a different, and much simpler, format to the X.509 certificates used in ssl(8)"
16:46:44 <petebuffon[m]> right right
16:47:10 <petebuffon[m]> i'll have to do some reading about that
16:47:28 <darknao> nirik: how does this compare to a SSHFP dns record ? is there any advantages to use certs instead ?
16:47:44 <nirik> You can also do some pretty cool things with the user certs.. like make them valid only for specific hosts or expire at specific times.
16:48:07 <nirik> darknao: it's just another method. I guess it depends on how you want to distribute that trust.
16:48:31 <nirik> oh, finally, I want to mention another fun ssh thing that I don't think many people know about:
16:48:49 <mobrien> I guess this way its all controlled in the one place in ansible without having to worry about dns pushes
16:49:16 <nirik> mobrien: yeah, for us, dns would be much more pain, as we would have to add records everytime something was installed.
16:49:32 <nirik> ed25519-sk and edcsa-sk keys are new key times in new openssh.
16:50:14 <nirik> They use FIDO2 keys (yubikey, etc). So you can make ssh ask you to touch the key when using them and using the key requires you to have the auth token
16:50:45 <nirik> ie, you get a: "Confirm user presence for key ECDSA-SK SHA256:..."
16:50:52 <mobrien> Ah so built in 2fa? Nice!
16:51:08 <nirik> with this your ssh private key can never be taken/used without the 2nd factor
16:51:32 <nirik> Sadly, this is only supported in fedora currently. No rhel7/8 support.
16:52:18 <nirik> (otherwise I would use it in our infrastructure for sure)
16:52:22 <petebuffon[m]> that would be great to be able to use FIDO2
16:52:52 <nirik> anyhow, I think thats everything, unless there's more questions?
16:53:32 <darknao> i'll definitely look into this for my homelab
16:54:25 <mobrien> I have one related question. Are the accepted keys for key signing changed on each fedora release?
16:54:43 <mobrien> Or the ca rather
16:55:00 <nirik> mobrien: for rpm packages you mean?
16:55:17 <mobrien> I had some previously accepted but since I upgraded to F34 it no longer works
16:55:30 <nirik> yeah, each new fedora release gets a new key.
16:55:44 <nirik> that key is used for the life of the release tho (barring catastrophy)
16:55:46 <mobrien> Ah OK that makes sense
16:56:23 <petebuffon[m]> nice
16:56:30 <petebuffon[m]> Four minute for openfloor?
16:56:36 <petebuffon[m]> #topic Open Floor
16:58:03 <dtometzki> thanks nirik
16:58:38 <petebuffon[m]> yeah thanks that gave definitely gave me some things to look up/ponder
16:58:38 <nirik> anytime.
17:00:34 <petebuffon[m]> Thanks everyone, have a great rest of the week, and if applicable a great long weekend!
17:00:41 <petebuffon[m]> #endmeeting