18:00:04 <nirik> #startmeeting Infrastructure (2017-03-02)
18:00:04 <zodbot> Meeting started Thu Mar  2 18:00:04 2017 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:04 <zodbot> The meeting name has been set to 'infrastructure_(2017-03-02)'
18:00:04 <nirik> #meetingname infrastructure
18:00:04 <zodbot> The meeting name has been set to 'infrastructure'
18:00:04 <nirik> #topic aloha
18:00:04 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:04 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:04 <nirik> #topic New folks introductions
18:00:09 <nirik> morning everyone.
18:00:18 <smooge> .hello smooge
18:00:18 * cverna is here
18:00:19 <zodbot> smooge: smooge 'Stephen J Smoogen' <smooge@gmail.com>
18:00:22 <sayan> .hello sayanchowdhury
18:00:22 * doteast present
18:00:23 <marc84> hi everyone
18:00:23 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
18:00:23 <clime> Hello
18:00:34 <puiterwijk> Hello
18:00:45 * doteast waves
18:01:04 * threebean waves
18:01:23 <x3mboy> .hello x3mboy
18:01:25 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
18:01:33 * x3mboy just listening
18:01:59 <nirik> Any new folks today that would like to give a short introduction of themselves?
18:02:17 <x3mboy> !
18:02:27 <nirik> go ahead x3mboy. :)
18:02:36 <harrisonbrock> Hello, I'm starting back
18:03:00 <bowlofeggs> .hello bowlofeggs
18:03:03 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
18:03:22 <nirik> harrisonbrock: welcome back again. :)
18:03:45 <x3mboy> Eduard Lucena, from Venezuela, living in Chile, trying to make the world more open. Working with Ambassadors, Marketing, Magazine, Hubs, trying to work a little bit with Hubs. I work for telecom industries where everything is managed with Linux/Unix, and currently sysadmin if CentOS VM that runs an MVNO
18:04:09 <x3mboy> Just curious about admin team, being that my job is mostly sysadmin and DB-admin
18:04:16 <nirik> welcome. Thats quite a lot of things... great you are so active. :)
18:04:33 <nirik> do ask questions anytime and again welcome.
18:05:20 <nirik> ok, lets go ahead and dive on in...
18:05:23 <x3mboy> Thanks
18:05:26 <nirik> #topic announcements and information
18:05:26 <nirik> #info koji prod -> stg sync done. The script(s) need a lot more work - kevin
18:05:26 <nirik> #info magazine and communityblog are now backed up - kevin and patrick
18:05:26 <nirik> #info package maintainer instancees are moved to the fedorainfracloud - kevin
18:05:27 <nirik> #info fedorahosted.org retired! - kevin
18:05:28 <nirik> #info lots of redirects and migration work to finish off some projects - kevin
18:05:29 <nirik> #info el5 signing key expired, patrick fixed it - patrick
18:05:30 <nirik> #info March apprentice nag email out, respond today! - kevin
18:05:39 <nirik> anything anyone would like to add to those status/info or talk about?
18:06:20 <bt0> .hello  bt0dotninja
18:06:21 <zodbot> bt0: bt0dotninja 'Alberto Rodriguez Sanchez' <hotgalan@gmail.com>
18:06:23 <nirik> it's been a busy week. ;) There were some pagure releases too I think.
18:06:52 <puiterwijk> #info FAS to IPA sync bug finally analyzed and should be fixed - Patrick
18:07:35 <nirik> puiterwijk: cool. Does that mean we shouldn't see any more users with that issue? or are there some existing ones that would still have it, but none moving forward...
18:07:40 <nirik> (if that makes sense)
18:08:00 <puiterwijk> nirik: there will be people that hit it before. I'm going to go look at everyone that's hit it, and clear their status
18:08:16 <puiterwijk> So they should get reattempted on their next login
18:08:42 <nirik> ok, great.
18:09:40 <nirik> well, I don't have any discussion items listed...
18:09:50 <puiterwijk> Also, after Wiki has moved to OpenID, we will be disabling the feature where you can login with your email address
18:09:50 <nirik> #topic Apprentice Open office hours
18:10:08 <nirik> any apprentices with questions or comments or looking for things to work on?
18:10:29 <harrisonbrock> I'm looking for something to do
18:10:30 <nirik> hopefully things will be a bit quieter this next week and we can help answer more questions, etc.
18:11:07 <nirik> harrisonbrock: cool. Whats your background? any types of things appeal more than others?
18:11:07 <harrisonbrock> ok
18:11:12 * nirik looks at the easyfix list
18:11:48 <nirik> if pagure would load
18:12:19 <harrisonbrock> I have mostly did C++ application dev, DBA, and web application, I also take care of 3 Redhat servers for a radio station
18:12:36 <puiterwijk> nirik: I'll give it a kick
18:12:45 <clime> (easyfix bug list contains bugs that nobody could fix for a few years :))
18:13:16 <puiterwijk> nirik: PAgure should be back
18:13:16 <nirik> yeah, we need to add more... it's hard tho, as if it's too easy I tend to just fix it as thats faster than filing the bug. ;)
18:13:47 <nirik> https://pagure.io/fedora-infrastructure/issues?status=Open&tags=easyfix
18:13:54 <nirik> theres a bunch of SOP writing...
18:15:10 <nirik> I'll try and add some new ones. I might have a good one about something that sends me email and would be nice to track down and fix.
18:16:02 <nirik> https://pagure.io/fedora-infrastructure/issue/5841 could be an interesting one for someone new... it's pretty complex to know where you need to make changes tho... so not sure it's really easyfix
18:17:25 <nirik> https://pagure.io/fedora-infrastructure/issue/5750 might be an interesting one too... basically need to figure out a way to get nagios/nrpe to query postgres on some db servers
18:18:01 <nirik> https://pagure.io/fedora-infrastructure/issue/5640 (checking koji fedmsg plugin) might be something not too hard to do...
18:18:40 <nirik> any of those look interesting harrisonbrock ? if not I can try and file some new ones...
18:18:43 <bt0> the 5841 looks fine :D
18:19:02 <harrisonbrock> I would like to look into 5750 because I have worked with many databases in the past
18:20:05 <nirik> cool. I don't know the state of nagios plugins for db's, but you can look and update the ticket with your findings.
18:20:43 <nirik> bt0: it's not too hard if you know where to change, thats the hard part. Basically 'git grep' some host that already goes to our proxies and you can see the playbooks where thats set...
18:21:49 <nirik> I think we lost some of our easyfix tagged issues in the pagure move
18:22:33 <nirik> I'll go thru later today and update some. :) so, do look later tonight/tomorrow for easyfix tag
18:22:37 <x3mboy> !
18:22:49 <nirik> x3mboy: go ahead. :) Just speak up anytime...
18:22:51 <x3mboy> It's ok to start reading here: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
18:22:57 <x3mboy> ?
18:23:19 <nirik> absolutely. :)
18:23:34 <nirik> and if you see anything out of date or confusing do let us know
18:23:53 <x3mboy> Ok!
18:24:22 <nirik> ok, any other apprentice questions?
18:24:50 <nirik> #topic Learn about - autosign and sigul - kevin and puiterwijk
18:25:13 <nirik> So, I thought I would talk today about our signing setup and give an overview....
18:25:35 <nirik> and puiterwijk is here for any more technical details and to keep me from being too wrong. :)
18:25:55 <sayan> :-)
18:25:58 <nirik> I've talked about sigul (our signing setup) in the past. Basically:
18:26:42 <nirik> There is a pair of servers 'vaults' that are not running ssh or easily accessable. They are now running Fedora 25. They have encrypted disks and yubikeys plugged in.
18:27:17 <nirik> when one is started (one is running and the other is a hot spare) an admin gives the vault password + the yubikey pin.
18:27:51 <nirik> it then if all looks good, connects to the next machine: the bridge
18:28:20 <harrisonbrock> Is it okay that I'm running Redhat Workstation 7.3 ?
18:28:23 <nirik> The bridge runs a service that the vault connects to and that clients connect to, and it basically works as a middleman between the two sides
18:28:39 <nirik> harrisonbrock: sure.
18:29:27 <nirik> We not have a service called robosign that runs on autosign01 (another box that has ssh off and is not easily accessable) that autosigns builds for us as they are done.
18:29:49 <nirik> koji tags packages into a tag that robosign watches and it signs them and moves them to another final tag.
18:30:28 <nirik> (except for epel5 and epel6, where it just tries to sign them but they land in the tag that they were built into)
18:31:06 <nirik> You can see the "queue" of things waiting to be signed in the f27-pending, f26-pending, f25-signing-pending and f24-signing-pending tags: koji list-tagged f27-pending
18:31:41 <nirik> when packages are signed there's also a fedmsg for it.. you can see those in #fedora-fedmsg or any of the ways you look at fedmsgs
18:32:04 <nirik> puiterwijk: anything I missed or got wrong there? :)
18:32:13 <nirik> thats basically the overview I think.
18:32:20 <puiterwijk> Nope. That's a good overview
18:32:54 <clime> Cool
18:33:32 <nirik> With this setup you need a person with access to a key and their passphrase + vault with access to it's yubikey to sign something.
18:34:05 <nirik> the vaults only ever reach out to the bridge, never accept anything incoming.
18:34:33 <nirik> So, barring specialized hardware it's a pretty nifty little setup.
18:34:52 <nirik> It does from time to time get behind...
18:35:04 <nirik> When we were signing all the new f27 packages after branching.
18:35:13 <nirik> Or when one or more texlive builds land. ;(
18:35:49 <nirik> It also currently uses gpg1.
18:36:29 <nirik> But I think puiterwijk has plans to fix that.
18:36:48 <puiterwijk> Yes. I have a patch to make it work with GPGv2. The problem there is just GPG v2.0
18:37:04 <nirik> Anyhow, thats all I had. Any questions or comments?
18:38:02 <nirik> #topic Open Floor
18:38:07 <sayan> nirik: who keeps the yubikeys?
18:38:08 <nirik> anyone have anything to bring up?
18:38:23 <nirik> sayan: they stay in the machines. :)
18:38:28 <nirik> at the datacenter
18:38:29 <puiterwijk> sayan: the yubikeys are physically inserted in the vault hardware in the datacenter
18:38:45 <sayan> nirik: oh ok
18:38:56 <doteast> like hsm?
18:38:59 <nirik> it's basically just a way to tie it to the hardware.
18:39:13 <nirik> so, if someone stole all the data from a vault they couldn't use it anywhere else.
18:39:29 <nirik> it would refuse to start without the yubikey and pin.
18:39:32 <clime> Will It be possible at all to fix that cloud networking issue
18:40:10 <nirik> clime: I hope so. It's just been crazy busy... and it's not hit the top of the list yet for me. ;)
18:40:23 <nirik> I was wondering if there was some kind of DOS going on.
18:40:24 <doteast> so the yubikey stores the signing keys?
18:40:35 <nirik> or a cloud machine sending out too much traffic.
18:41:32 <clime> Well, not sure There are domě ssh attacks bit no that string imho
18:42:09 <clime> I never to restart copr-keygen all the zimě :(
18:42:30 <clime> Copr-backend sry
18:42:44 <nirik> clime: oh? because it cannot talk to the builders? or ?
18:43:03 <clime> Yep
18:43:33 <nirik> doteast: no, just a binding (at least thats my understanding). The "signing keys" are encrypted and need the vault decryption + user... neither the user nor the vault can sign anything without the other
18:43:46 <nirik> clime: but thats internal, not external right?
18:44:20 <nirik> they don't even have external ips...
18:44:34 <clime> Fedorainfracloud.org is outside od openstack network
18:45:00 <nirik> copr talks to builders via a 172.x.x.x net I thought? not a 209.x.x.x
18:45:05 <clime> And cloud controller IS there
18:45:29 <nirik> fedorainfracloud.org ip is just a external ip on the controller node.
18:45:51 <clime> I think maybe ONLY cloud controller činnosti IS failing
18:46:19 <clime> Sorry for uppercase Ian on phone
18:46:55 <nirik> ok, we can take this to fedora-admin... I can dig thru logs...
18:47:37 <clime> Would be cool
18:47:56 <clime> thank you
18:48:11 <clime> !
18:48:42 <nirik> ok, if nothing else, will wrape up in a minute or so
18:49:41 <nirik> ok. Thanks for coming everyone.
18:49:42 <x3mboy> Thanks, I will try to work with the docs in the wiki
18:49:49 <nirik> #endmeeting