18:00:04 <nirik> #startmeeting Infrastructure (2017-03-02) 18:00:04 <zodbot> Meeting started Thu Mar 2 18:00:04 2017 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:04 <zodbot> The meeting name has been set to 'infrastructure_(2017-03-02)' 18:00:04 <nirik> #meetingname infrastructure 18:00:04 <zodbot> The meeting name has been set to 'infrastructure' 18:00:04 <nirik> #topic aloha 18:00:04 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:04 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:04 <nirik> #topic New folks introductions 18:00:09 <nirik> morning everyone. 18:00:18 <smooge> .hello smooge 18:00:18 * cverna is here 18:00:19 <zodbot> smooge: smooge 'Stephen J Smoogen' <smooge@gmail.com> 18:00:22 <sayan> .hello sayanchowdhury 18:00:22 * doteast present 18:00:23 <marc84> hi everyone 18:00:23 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com> 18:00:23 <clime> Hello 18:00:34 <puiterwijk> Hello 18:00:45 * doteast waves 18:01:04 * threebean waves 18:01:23 <x3mboy> .hello x3mboy 18:01:25 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com> 18:01:33 * x3mboy just listening 18:01:59 <nirik> Any new folks today that would like to give a short introduction of themselves? 18:02:17 <x3mboy> ! 18:02:27 <nirik> go ahead x3mboy. :) 18:02:36 <harrisonbrock> Hello, I'm starting back 18:03:00 <bowlofeggs> .hello bowlofeggs 18:03:03 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com> 18:03:22 <nirik> harrisonbrock: welcome back again. :) 18:03:45 <x3mboy> Eduard Lucena, from Venezuela, living in Chile, trying to make the world more open. Working with Ambassadors, Marketing, Magazine, Hubs, trying to work a little bit with Hubs. I work for telecom industries where everything is managed with Linux/Unix, and currently sysadmin if CentOS VM that runs an MVNO 18:04:09 <x3mboy> Just curious about admin team, being that my job is mostly sysadmin and DB-admin 18:04:16 <nirik> welcome. Thats quite a lot of things... great you are so active. :) 18:04:33 <nirik> do ask questions anytime and again welcome. 18:05:20 <nirik> ok, lets go ahead and dive on in... 18:05:23 <x3mboy> Thanks 18:05:26 <nirik> #topic announcements and information 18:05:26 <nirik> #info koji prod -> stg sync done. The script(s) need a lot more work - kevin 18:05:26 <nirik> #info magazine and communityblog are now backed up - kevin and patrick 18:05:26 <nirik> #info package maintainer instancees are moved to the fedorainfracloud - kevin 18:05:27 <nirik> #info fedorahosted.org retired! - kevin 18:05:28 <nirik> #info lots of redirects and migration work to finish off some projects - kevin 18:05:29 <nirik> #info el5 signing key expired, patrick fixed it - patrick 18:05:30 <nirik> #info March apprentice nag email out, respond today! - kevin 18:05:39 <nirik> anything anyone would like to add to those status/info or talk about? 18:06:20 <bt0> .hello bt0dotninja 18:06:21 <zodbot> bt0: bt0dotninja 'Alberto Rodriguez Sanchez' <hotgalan@gmail.com> 18:06:23 <nirik> it's been a busy week. ;) There were some pagure releases too I think. 18:06:52 <puiterwijk> #info FAS to IPA sync bug finally analyzed and should be fixed - Patrick 18:07:35 <nirik> puiterwijk: cool. Does that mean we shouldn't see any more users with that issue? or are there some existing ones that would still have it, but none moving forward... 18:07:40 <nirik> (if that makes sense) 18:08:00 <puiterwijk> nirik: there will be people that hit it before. I'm going to go look at everyone that's hit it, and clear their status 18:08:16 <puiterwijk> So they should get reattempted on their next login 18:08:42 <nirik> ok, great. 18:09:40 <nirik> well, I don't have any discussion items listed... 18:09:50 <puiterwijk> Also, after Wiki has moved to OpenID, we will be disabling the feature where you can login with your email address 18:09:50 <nirik> #topic Apprentice Open office hours 18:10:08 <nirik> any apprentices with questions or comments or looking for things to work on? 18:10:29 <harrisonbrock> I'm looking for something to do 18:10:30 <nirik> hopefully things will be a bit quieter this next week and we can help answer more questions, etc. 18:11:07 <nirik> harrisonbrock: cool. Whats your background? any types of things appeal more than others? 18:11:07 <harrisonbrock> ok 18:11:12 * nirik looks at the easyfix list 18:11:48 <nirik> if pagure would load 18:12:19 <harrisonbrock> I have mostly did C++ application dev, DBA, and web application, I also take care of 3 Redhat servers for a radio station 18:12:36 <puiterwijk> nirik: I'll give it a kick 18:12:45 <clime> (easyfix bug list contains bugs that nobody could fix for a few years :)) 18:13:16 <puiterwijk> nirik: PAgure should be back 18:13:16 <nirik> yeah, we need to add more... it's hard tho, as if it's too easy I tend to just fix it as thats faster than filing the bug. ;) 18:13:47 <nirik> https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix 18:13:54 <nirik> theres a bunch of SOP writing... 18:15:10 <nirik> I'll try and add some new ones. I might have a good one about something that sends me email and would be nice to track down and fix. 18:16:02 <nirik> https://pagure.io/fedora-infrastructure/issue/5841 could be an interesting one for someone new... it's pretty complex to know where you need to make changes tho... so not sure it's really easyfix 18:17:25 <nirik> https://pagure.io/fedora-infrastructure/issue/5750 might be an interesting one too... basically need to figure out a way to get nagios/nrpe to query postgres on some db servers 18:18:01 <nirik> https://pagure.io/fedora-infrastructure/issue/5640 (checking koji fedmsg plugin) might be something not too hard to do... 18:18:40 <nirik> any of those look interesting harrisonbrock ? if not I can try and file some new ones... 18:18:43 <bt0> the 5841 looks fine :D 18:19:02 <harrisonbrock> I would like to look into 5750 because I have worked with many databases in the past 18:20:05 <nirik> cool. I don't know the state of nagios plugins for db's, but you can look and update the ticket with your findings. 18:20:43 <nirik> bt0: it's not too hard if you know where to change, thats the hard part. Basically 'git grep' some host that already goes to our proxies and you can see the playbooks where thats set... 18:21:49 <nirik> I think we lost some of our easyfix tagged issues in the pagure move 18:22:33 <nirik> I'll go thru later today and update some. :) so, do look later tonight/tomorrow for easyfix tag 18:22:37 <x3mboy> ! 18:22:49 <nirik> x3mboy: go ahead. :) Just speak up anytime... 18:22:51 <x3mboy> It's ok to start reading here: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 18:22:57 <x3mboy> ? 18:23:19 <nirik> absolutely. :) 18:23:34 <nirik> and if you see anything out of date or confusing do let us know 18:23:53 <x3mboy> Ok! 18:24:22 <nirik> ok, any other apprentice questions? 18:24:50 <nirik> #topic Learn about - autosign and sigul - kevin and puiterwijk 18:25:13 <nirik> So, I thought I would talk today about our signing setup and give an overview.... 18:25:35 <nirik> and puiterwijk is here for any more technical details and to keep me from being too wrong. :) 18:25:55 <sayan> :-) 18:25:58 <nirik> I've talked about sigul (our signing setup) in the past. Basically: 18:26:42 <nirik> There is a pair of servers 'vaults' that are not running ssh or easily accessable. They are now running Fedora 25. They have encrypted disks and yubikeys plugged in. 18:27:17 <nirik> when one is started (one is running and the other is a hot spare) an admin gives the vault password + the yubikey pin. 18:27:51 <nirik> it then if all looks good, connects to the next machine: the bridge 18:28:20 <harrisonbrock> Is it okay that I'm running Redhat Workstation 7.3 ? 18:28:23 <nirik> The bridge runs a service that the vault connects to and that clients connect to, and it basically works as a middleman between the two sides 18:28:39 <nirik> harrisonbrock: sure. 18:29:27 <nirik> We not have a service called robosign that runs on autosign01 (another box that has ssh off and is not easily accessable) that autosigns builds for us as they are done. 18:29:49 <nirik> koji tags packages into a tag that robosign watches and it signs them and moves them to another final tag. 18:30:28 <nirik> (except for epel5 and epel6, where it just tries to sign them but they land in the tag that they were built into) 18:31:06 <nirik> You can see the "queue" of things waiting to be signed in the f27-pending, f26-pending, f25-signing-pending and f24-signing-pending tags: koji list-tagged f27-pending 18:31:41 <nirik> when packages are signed there's also a fedmsg for it.. you can see those in #fedora-fedmsg or any of the ways you look at fedmsgs 18:32:04 <nirik> puiterwijk: anything I missed or got wrong there? :) 18:32:13 <nirik> thats basically the overview I think. 18:32:20 <puiterwijk> Nope. That's a good overview 18:32:54 <clime> Cool 18:33:32 <nirik> With this setup you need a person with access to a key and their passphrase + vault with access to it's yubikey to sign something. 18:34:05 <nirik> the vaults only ever reach out to the bridge, never accept anything incoming. 18:34:33 <nirik> So, barring specialized hardware it's a pretty nifty little setup. 18:34:52 <nirik> It does from time to time get behind... 18:35:04 <nirik> When we were signing all the new f27 packages after branching. 18:35:13 <nirik> Or when one or more texlive builds land. ;( 18:35:49 <nirik> It also currently uses gpg1. 18:36:29 <nirik> But I think puiterwijk has plans to fix that. 18:36:48 <puiterwijk> Yes. I have a patch to make it work with GPGv2. The problem there is just GPG v2.0 18:37:04 <nirik> Anyhow, thats all I had. Any questions or comments? 18:38:02 <nirik> #topic Open Floor 18:38:07 <sayan> nirik: who keeps the yubikeys? 18:38:08 <nirik> anyone have anything to bring up? 18:38:23 <nirik> sayan: they stay in the machines. :) 18:38:28 <nirik> at the datacenter 18:38:29 <puiterwijk> sayan: the yubikeys are physically inserted in the vault hardware in the datacenter 18:38:45 <sayan> nirik: oh ok 18:38:56 <doteast> like hsm? 18:38:59 <nirik> it's basically just a way to tie it to the hardware. 18:39:13 <nirik> so, if someone stole all the data from a vault they couldn't use it anywhere else. 18:39:29 <nirik> it would refuse to start without the yubikey and pin. 18:39:32 <clime> Will It be possible at all to fix that cloud networking issue 18:40:10 <nirik> clime: I hope so. It's just been crazy busy... and it's not hit the top of the list yet for me. ;) 18:40:23 <nirik> I was wondering if there was some kind of DOS going on. 18:40:24 <doteast> so the yubikey stores the signing keys? 18:40:35 <nirik> or a cloud machine sending out too much traffic. 18:41:32 <clime> Well, not sure There are domě ssh attacks bit no that string imho 18:42:09 <clime> I never to restart copr-keygen all the zimě :( 18:42:30 <clime> Copr-backend sry 18:42:44 <nirik> clime: oh? because it cannot talk to the builders? or ? 18:43:03 <clime> Yep 18:43:33 <nirik> doteast: no, just a binding (at least thats my understanding). The "signing keys" are encrypted and need the vault decryption + user... neither the user nor the vault can sign anything without the other 18:43:46 <nirik> clime: but thats internal, not external right? 18:44:20 <nirik> they don't even have external ips... 18:44:34 <clime> Fedorainfracloud.org is outside od openstack network 18:45:00 <nirik> copr talks to builders via a 172.x.x.x net I thought? not a 209.x.x.x 18:45:05 <clime> And cloud controller IS there 18:45:29 <nirik> fedorainfracloud.org ip is just a external ip on the controller node. 18:45:51 <clime> I think maybe ONLY cloud controller činnosti IS failing 18:46:19 <clime> Sorry for uppercase Ian on phone 18:46:55 <nirik> ok, we can take this to fedora-admin... I can dig thru logs... 18:47:37 <clime> Would be cool 18:47:56 <clime> thank you 18:48:11 <clime> ! 18:48:42 <nirik> ok, if nothing else, will wrape up in a minute or so 18:49:41 <nirik> ok. Thanks for coming everyone. 18:49:42 <x3mboy> Thanks, I will try to work with the docs in the wiki 18:49:49 <nirik> #endmeeting