18:01:10 <smooge> #startmeeting Infrastructure (2017-04-13)
18:01:10 <zodbot> Meeting started Thu Apr 13 18:01:10 2017 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:01:10 <zodbot> The meeting name has been set to 'infrastructure_(2017-04-13)'
18:01:10 <smooge> #meetingname infrastructure
18:01:10 <zodbot> The meeting name has been set to 'infrastructure'
18:01:10 <smooge> #topic aloha
18:01:10 <smooge> #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson
18:01:10 <zodbot> Current chairs: abadger1999 dgilmore nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:01:18 <smooge> hello everyone
18:01:22 <clime> hey
18:01:24 <puiterwijk> Hello
18:01:24 <Skeer> morn
18:01:27 <nirik> morning
18:01:30 <icole> hello
18:01:35 * pingou around but not really here
18:02:22 <bowlofeggs> .hello bowlofeggs
18:02:23 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:02:37 * cverna here
18:02:43 <ExoUNX> afternoon
18:02:52 <athos> hello :)
18:02:55 <jcline> .hello jcline
18:02:56 <zodbot> jcline: jcline 'Jeremy Cline' <jeremy@jcline.org>
18:02:58 <smooge> I am glad to see a lot of 'new' faces this week
18:03:58 <smooge> ok moving to that
18:04:11 <smooge> #topic New folks introductions
18:04:32 <marc84> hi everyone
18:04:54 <smooge> hello any new people who wish to introduce themselves?
18:05:32 <icole> I am still pretty new. My name is Ian. Last week was my first meeting
18:05:57 <icole> Still looking for some Python projects, so if anyone has anything in particular feel free to send it my way :)
18:06:11 <icole> otherwie am still just getting in to a flow of looking throw "easyfix" tickets
18:06:17 * pbrobinson is sort of here
18:06:59 <smooge> ok I opened up a non-programming ticket which has several 'easy-fix' steps in it today  https://pagure.io/fedora-infrastructure/issue/5931
18:07:06 <Skeer> welcome icole
18:07:36 <icole> great, thanks @smooge, will take a look
18:07:51 <smooge> OK next section
18:07:54 <smooge> #topic announcements and information
18:07:54 <smooge> #info hosted03 has had final systems removed
18:07:54 <smooge> #info beta freeze will start 2017-05-17
18:07:54 <smooge> #info 2017-04-14 is a Red Hat holiday
18:07:55 <smooge> #info fmn-1.2.0 is in staging, plan to push to prod Monday or Tuesday. - jcline
18:07:55 <smooge> #info FMN design proposal (https://github.com/fedora-infra/fmn/issues/170#issuecomment-293699913) feedback welcome
18:08:22 <puiterwijk> So, Fedora Hosted is finally fully turned off \o/
18:08:30 <linuxmodder> .fas linuxmodder
18:08:31 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org>
18:08:38 <linuxmodder> hard stop for bus in about 20 mins for me
18:08:53 <smooge> its ghost to come back and haunt us with RHEL-5
18:09:11 <smooge> any other announcements from the group?
18:09:54 <smooge> icole, the FMN might be something you and other python coders want to at least look at and see if there are obvious questions needing to be answered
18:10:08 <smooge> s/python coders/coders in general/
18:10:49 <smooge> Anything else? If not its time to start the next section
18:11:00 <smooge> #topic Upcoming Hackathon
18:11:00 <smooge> #info https://fedoraproject.org/wiki/CI_and_Infrastructure_Hackathon_2017
18:11:00 <smooge> #info core members will be less available.
18:11:00 <smooge> #info focus on modularity
18:11:32 * nirik notes he hasn't actually powered off hosted03 yet. But I intend to soon.
18:11:58 <smooge> nirik, its already in the notes.. you are making me look bad :P
18:13:03 <nirik> sorry
18:13:12 <nirik> nothing to see here, move on. :)
18:13:30 <puiterwijk> It's turned off, and that's the end of that fight! :)
18:13:43 <smooge> ok this topic is more informational. This will be working on some core infrastructure changes to make modularity clearer
18:14:43 <smooge> I do not know if we will have a meeting that week or not as most of us will hve been meetinged for a bit
18:15:22 <smooge> nirik, is there anything else to say on this do you think?
18:15:57 <nirik> hopefully we will have irc and possibly some video thing so people can participate remotely.
18:16:05 <nirik> and will of course be sending info to the list
18:17:02 <smooge> So the next item was something I would like to get some idea on...
18:17:06 <smooge> #topic Clean out the backlog of tickets
18:17:06 <smooge> #info Put in a day for going through old tickets
18:17:07 <smooge> #info Triage and clean out old tickets
18:17:07 <smooge> #info Goal bring it down to 40 tickets
18:17:18 <athos> +1 to video thing :)
18:17:33 <nirik> I think that cleaning tickets is good, but getting to 40 will be hard
18:17:45 * nirik cleans things all the time. We have been hovering around 100.
18:17:53 <smooge> nirik had said this wistfully in irc a while ago and I thought hey we should have a merciless day
18:18:04 <nirik> we could definitely give it a try.
18:18:25 <puiterwijk> Well, if we're going to be merciless, we could just close all tickets as "if you still need this, and there's less than 40 tickets, reopen"
18:18:47 <smooge> basically go from the oldest and if there hasn't been any move on it and not going to be in a year.. just close it
18:19:05 <smooge> and we have a pretty good idea about what we will really get to versus "well if miracles occur"
18:19:05 <nirik> well, there's things we still want to do that are just old/hard.
18:19:26 <threebean> +1 for Spring Cleaning
18:20:02 <nirik> There are also a few tickets people kept reopening
18:20:17 <nirik> for example... our oldest ticket is ssl fas trafic.
18:20:39 <nirik> HTTP strict transport security (thats been a long road)
18:20:40 <nirik> etc
18:20:59 <nirik> But yeah... I'm game to try and move some along or refile them or do something with them
18:22:19 <nirik> we just need to pick a day and spend the time discussing.
18:22:21 <smooge> I figure that its not like we are deleting them. If we get to them later we can reopen
18:22:55 <smooge> Do you want to do it before RDU or after?
18:24:34 <smooge> ok I am going to say May 1st :)
18:25:16 <smooge> (IE I am going to bring it up on the list for us to think about :))
18:25:17 <nirik> either way. sure
18:25:24 <smooge> #topic Apprentice Open office hours
18:25:42 <smooge> hello apprentices. do you have any questions on tickets ?
18:26:14 <Skeer> i hit up a few folks last night about the bodho template one.
18:26:23 <Skeer> *bodhi.
18:26:37 <Skeer> ill end up picking more brains before its over ;)
18:26:54 <icole> @smooge, am good for now. Thanks for pointing to the IP ticket and FMN stuff
18:27:19 <smooge> thanks for the feedback. anyone else?
18:27:34 <ExoUNX> yah
18:27:47 <smooge> hi ExoUNX what can we help with
18:27:56 <ExoUNX> I've done the basics for the fi-app, are there any tasks I can start with?
18:28:25 <smooge> well generally it would be to go through the easyfix tickets. Skeer do you have the link you used?
18:28:56 <puiterwijk> https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix
18:29:10 <smooge> thanks puiterwijk
18:29:12 <ExoUNX> ok, and yah I've been looking through easyfix on the pgaure repo, but that's a different story
18:29:14 <ExoUNX> thank you
18:29:19 <Skeer> for exasyfix specifically? or the git repo?
18:29:34 <smooge> Skeer, nm it was the one puiterwijk gave me :)
18:29:39 <Skeer> gotcha
18:29:49 <ExoUNX> pagure*
18:30:08 <ExoUNX> thanks again
18:30:30 <smooge> no problem. if you have specific questions and irc isn't a good mechanism to ask them, please send me an email
18:30:39 <smooge> smooge@fedoraproject.org
18:30:41 <Skeer> ticket question if I may..
18:31:01 <smooge> Skeer, yep go ahead
18:31:03 <Skeer> verbiage-wise, should I keep urls out of comments?  I see kevin removed mine, just wanna make sure whats kosher or not
18:31:40 <nirik> hum? I didn't remove anything that I know of...
18:32:14 <Skeer> I pasted the url to the ansible.git repo just more as a note to myself so I didnt forget.
18:32:55 <Skeer> Ah.. ok for some reason chrome was only showing my original comment then your followup from last night/today.  My bad.  nothing to see here.
18:33:03 <Skeer> ;)
18:33:10 <smooge> ok well then.. I am going on to the last topic of the day.
18:33:20 <smooge> #topic Learn about: SSH and where its going.
18:33:57 <puiterwijk> So, I was going to give a short summary of things I've done recently and will be doing in the near future for SSH connections to infra
18:34:43 <puiterwijk> First of all, as you all might've seen in the infra list (I sent this yesterday), we now allow more SSH key types to be added to FAS accounts. However, Infra members need to be very careful with this, as RHEL6 boxes will *not* recognize them.
18:35:11 <puiterwijk> If you ask me for help about not being able to SSH into a box, and it's because of this, I will publicly reply to you that you've been silly.
18:35:12 <linuxmodder> what hosts are on rhel6 tho still?
18:35:18 <puiterwijk> linuxmodder: list is linked to in my email
18:35:30 <puiterwijk> It's only Infra-internal boxes, and no boxes with non-infra SSH access
18:35:34 <linuxmodder> ah haven't seen it yet then will read the email
18:36:22 <Skeer> Ive got a semi-related ssh question once you are done.
18:36:23 <nirik> its not many
18:36:33 <ExoUNX> yah, RSA4096 is probably has the best security:compatibility
18:37:08 <puiterwijk> So, the second subtopic is on our SSH servers again
18:37:18 <puiterwijk> As of a few days, I've deployed SSH host certificates to all of them.
18:37:51 <puiterwijk> This means that you can now verify their fingerprints by a set of certificates (one staging, one production) that I will publish later today on the infrastructure website.
18:38:10 <ExoUNX> nice work
18:38:22 <linuxmodder> puiterwijk,  are they still gonna be on the SSH fingerprint link on the webui for them ?
18:38:25 <puiterwijk> this means you no longer will have to maintain a long known_hosts file, since the server will just present a certificate to you, and even if we change the SSH keys (due to reinstall, or something), it'll still work
18:38:35 <Skeer> Nice
18:38:49 <puiterwijk> linuxmodder: I don't know that we have a webui. But I will replace the public known_hosts file with our cert lines
18:39:02 <linuxmodder> ack
18:39:24 <puiterwijk> I really want everyone to move to these certificates, so that we get rid of the leap-of-faith many people do instead of updating the known_hosts, and get rid of any issues because of reinstalled machines
18:40:12 <puiterwijk> Do note that because the certificate authorities are ed25519 keys, you will *no longer* be able to connect from a RHEL6 or equally old distribution's  SSH client. They will just error out
18:40:19 <linuxmodder> will gladly nuke my long hostfile for all of infra :P
18:40:27 <puiterwijk> Any supported Fedora release and RHEL7 will work perfectly fine
18:40:36 <smooge> I need to see how to do this for my own systems
18:40:41 <smooge> cool
18:40:46 <puiterwijk> smooge: RHEL7 works. RHEL6 doesn't.
18:40:47 <linuxmodder> smooge,  same here
18:40:56 <smooge> ah man.. all my boxes are RHEL6 and RHEL5
18:40:58 <puiterwijk> If you're still on RHEL6, this is your reminder to upgrade
18:41:06 <puiterwijk> smooge: then you'll be locked out of infra.
18:41:19 <smooge> you guys will be safer
18:41:21 * nirik sshes in from his RHL 3.0.3 box
18:41:31 <smooge> good times good times
18:41:31 <bowlofeggs> puiterwijk: is there/will there be docs on how to use this new key? (btw, sounds great)
18:41:32 * puiterwijk shudders. And revokes nirik's keys
18:41:35 <nirik> :)
18:41:38 <linuxmodder> nirik,  tell me how well that works:P
18:41:54 <puiterwijk> bowlofeggs: yes. As said, I'll update the known_hosts file on the infrastructure.fedoraproject.org page, and you can just copy-paste that in there
18:42:06 <nirik> I personally just wiped my known_hosts here and added the 2 cert authorities.
18:42:16 <puiterwijk> REMINDER: do *not* trust SSH fingerprints given to you on IRC. That's just silly :(
18:42:21 <linuxmodder> so only hte two one prod and onestg now ?
18:42:27 <puiterwijk> Yes
18:42:38 <bowlofeggs> puiterwijk: excellent
18:42:47 <puiterwijk> So also if people ask for SSH fingerprint validation, just link them to the official page for that.
18:42:48 <linuxmodder> puiterwijk,  we all use tls connects here so what's the risk ? :P
18:42:55 <nirik> we need to do pagure and fedorainfracloud.
18:43:08 <puiterwijk> nirik: they've got keys signed already.
18:43:13 <puiterwijk> If it's in ansible, it got certificates.
18:43:21 <puiterwijk> For Pagure, I just need to add the additional public hostname
18:43:26 <linuxmodder> nirik,  please ping me when you do infracloud so danofsatx and I can make sure we are solid for the respins host
18:43:27 <nirik> great. I guess pkgs and pagure will need to fall back tho for all the people that don't add the CA
18:43:29 <Skeer> puiterwijk: what url will that be on for those of us w/o a favorite there ;)?
18:43:40 <linuxmodder> nm
18:43:44 <puiterwijk> Skeer: the url that's in the sshaccess SOP :)
18:43:53 <Skeer> *hides*
18:43:54 <puiterwijk> I don't have it around at this particular moment, will send it in a few
18:44:10 <puiterwijk> nirik: every SSH client will automatically fall back
18:44:15 <Skeer> thx
18:44:24 <puiterwijk> Which is what everyone here will have noticed the last few days if they connected even once :)
18:44:42 <nirik> right, but those ones will need to keep their old host keys for... a long time.
18:44:45 <smooge> ... hmmms
18:44:51 <puiterwijk> nirik: ah, right. Yeah
18:45:07 <puiterwijk> Any further questions regarding SSH host certs?
18:45:13 <smooge> not from me
18:45:36 <smooge> thank you puiterwijk
18:45:39 <puiterwijk> If nothing, I have one more subtopic on SSH
18:45:40 <linuxmodder> Skeer,  https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=b29f5a76be5dd8c6dbe6f6a141dd9e482aa71340 for the actual commit that did this
18:45:49 <smooge> oops
18:45:53 <smooge> go ahead
18:46:04 <puiterwijk> Skeer: https://admin.fedoraproject.org/ssh_known_hosts
18:46:16 <puiterwijk> (that's the known_hosts file, and I will add the SSH certs there later today)
18:46:26 <Skeer> thx guys. I did just try the sshaccess.rst and it errored out on me.
18:47:05 <linuxmodder> what link Skeer
18:47:13 <Skeer> https://fedoraproject.org/wiki/SSH_Access_Infrastructure_SOP
18:47:38 <puiterwijk> So, the last SSH subtopic is for the future, but will have a bigger impact to infra folks. I am currently finishing up the code for the new SSH client certs system. When that is done, you will from then on require a special Fedora SSH client to SSH into infra-internal boxes. That client will validate a 2-factor-auth token on connecting
18:48:23 <puiterwijk> I will give more details on the full internal workings and how to use it next week, but the short summary is that you will be required to use two factor auth to SSH into any infra internal boxes
18:48:35 <puiterwijk> (aka, everything not pagure.io or fedorapeople.org)
18:48:46 <linuxmodder> https://pagure.io/infra-docs/blob/master/f/docs/sysadmin-guide/sops/sshaccess.rst
18:48:58 <linuxmodder> Skeer,  it moved to pagure and infra-docs
18:49:43 <linuxmodder> puiterwijk,  nice any 2fa token or yubikeys will work?
18:50:53 <puiterwijk> linuxmodder: the exact policy is still up for draft, but it's likely that the tokens we will accept depends on the specific hosts you're trying to reach
18:51:10 <smooge> oh boy token keychain
18:51:22 <puiterwijk> For high security hosts, I want to enforce non-phishable tokens, which basically means U2F or something else that's in the works and I will announce with this next week
18:51:37 <ExoUNX> smooge we can do a token ring instead ;)
18:51:38 <Skeer> puiterwijk: Is this to address security issues, or just to head off potential future security incidents?
18:51:39 <puiterwijk> But the specifics on this will all come in the coming week(s).
18:51:44 <puiterwijk> Skeer: just proactive
18:51:48 <linuxmodder> nice yubikeys do u2f too so that is promising
18:51:49 <Skeer> *thumbsup*
18:52:05 <smooge> ok anything else?
18:52:08 <linuxmodder> we do secure early Skeer  :P
18:52:14 <puiterwijk> Skeer: I have no indication there has been any active attacks against the FEdora Infra, but I would like to keep us ahead :)
18:52:15 <linuxmodder> nfm
18:52:44 * linuxmodder heads to bus see everyone later
18:52:53 <smooge> bye linuxmodder
18:53:00 <smooge> #topic Open Floor
18:53:01 <marc84> bye linuxmodder
18:53:02 <puiterwijk> This was your quick update on the past, present and future of Fedora Infra SSH access by your trusty (am I trusty?) Infra Security Officer. If you have any further questions, please don't hesitate to ping me on IRC or send me an email
18:53:17 <smooge> thank you again puiterwijk . that was very informative
18:53:27 <Skeer> yessir thanks very much
18:53:46 <puiterwijk> Also, if you have any doubts regarding Fedora Infra security, please do get in touch with me
18:54:01 <puiterwijk> For open floor, I think pbrobinson has some nice news to share about armv7? :)
18:54:46 <pingou> it got upgraded to v8?
18:54:56 <puiterwijk> pingou: well, there is armv8, but that's aarch64 :)
18:55:18 <pbrobinson> I'm also there for ARMv7 virt builderrs
18:55:31 <pbrobinson> I'm just glueing together bits of ansible
18:55:48 <nirik> hurray. Will be nice to have those finally online. ;)
18:56:01 <pbrobinson> yes, it's been a long time coming :)
18:56:13 <smooge> well good I expect we are one power supply accident from no arm7
18:56:39 <smooge> ok more than one
18:56:43 <smooge> but I had an axe
18:57:11 <smooge> pbrobinson, congratulations!
18:57:25 <smooge> ok is there anything else for open floor or should I call it?
18:57:40 <pbrobinson> smooge: save that until after you've seen my terrible ansible hacks ;-)
18:57:41 <pingou> pbrobinson++
18:57:41 <zodbot> pingou: Karma for pbrobinson changed to 7 (for the f25 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:57:53 <Rhea> !
18:57:59 <Rhea> .hello rhea
18:58:00 <zodbot> Rhea: rhea 'Radka Janek' <radka.janek@redhat.com>
18:58:11 <Rhea> Hey guys, I've sent an email about GSoC Mentorship to the list a few days ago, please, do take a look and if you would be able to help get in touch with me. https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/LTEKPZZVJ22L2KFSMYFHGDFMCSTLITM7/
18:58:13 <Rhea> <eof>
18:59:03 <smooge> thanks Rhea
18:59:37 <smooge> ok I am going to close this meeting down and kick all of you out. I don't care where you sleep, but you can't sleep in this meeting tonight.
18:59:41 <smooge> #endmeeting