18:12:35 #startmeeting Infrastructure (2014-06-19) 18:12:35 Meeting started Thu Jun 19 18:12:35 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:12:35 Useful Commands: #action #agreed #help #info #idea #link #topic. 18:12:35 #meetingname infrastructure 18:12:35 The meeting name has been set to 'infrastructure' 18:12:36 #topic aloha 18:12:36 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:12:36 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:12:46 woohoo 18:12:52 who all is around for a late starting, net splitting meeting? ;) 18:12:53 * oddshocks present and accounted for 18:13:09 * ausmarton here 18:13:15 * puiterwijk is around but busy 18:13:17 Greetings 18:13:27 #topic New folks introductions and Apprentice tasks. 18:13:28 * threebean is here 18:13:32 * danofsatx-work is nuking sketty 18:13:38 * lmacken 18:13:44 and there's the beep... 18:13:46 any new folks around who want to introduce themselves? or apprentices with questions, comments or items to note? 18:14:10 * roshi is here 18:14:18 hi 18:14:20 * randomuser is milling around 18:14:30 just lurking trying to get a better understanding of our infra :) 18:14:48 learn what I can about how we do things on the infra side 18:15:06 oh god 18:15:07 why 18:15:09 :) 18:15:20 2x netsplit combo 18:15:44 lost kevin and patrick again 18:15:46 * roshi thought you were meaning "oh god why?" to checking out our infra 18:15:56 roshi: oh no, it's not that bad, I promise ;) 18:16:21 we just lost about 30 connections to netsplit for the second time in 5 mins 18:16:22 :D 18:16:28 same ones 18:16:39 ursabot: ping 18:16:39 pong 18:16:39 and they're back 18:16:39 yeah... 18:16:42 freenode isn't too staable today 18:16:56 #topic Applications status / discussion 18:17:00 * tflink shows up late, apologies 18:17:01 any applications news? 18:17:07 https://apps.fedoraproject.org/github2fedmsg/ 18:17:20 pushed that out this past week. 18:17:21 #info fedoauth is updated in stg. Please test it against as many sites as you can... it's going to production this weekend. 18:17:42 threebean: pretty cool. Is there any kind of overview of projects connected to it? or it's all per user? 18:18:03 all per user, but we could make an overview dashboard thing if it would be useful 18:18:21 just my curiosity, dunno how important that is. 18:18:44 * threebean nods 18:18:58 Fedimg is making its strides! Working out some libcloud issues with #libcloud, but things are looking good! Got all the fedmsg meta code merged in to the fedmsg meta infra repo... docs started on RTFD... makin' progress. that's all for me 18:19:20 oddshocks: cool. 18:19:49 oddshocks: your fedmsg.meta stuff is released too and on its way to the updates-testing yum repos 18:19:52 oddshocks: oh, I wonder... do we want to look at having it upload to the infrastructure private cloud too? dunno how useful it would be, but we could look into that if it is. 18:20:23 threebean: awesom! 18:20:34 nirik: I 100% do 18:20:58 should just need an account that can upload I would suspect. 18:22:10 any other app news? 18:22:10 nirik: our goal is definitely automatic uploads to external AND internal cloud providers. I was originally uploading to our internal Fedora FTP server too but I was told that is The Wrong Thing To Do and that I should just have nightly builds uploaded to the internal fedora FTP server, but I am 100% down for internal cloud image uploads 18:22:35 nirik: but any info anyone can get me on getting the images on our internal cloud server is super-helpful 18:22:38 and can be done with ease 18:22:42 :) 18:23:04 oddshocks: sure, we can talk out of meeting on it sometime... if we do nightlies we should also expire/delete older ones (ie, I don't think we want to keep them forever) 18:23:17 Definitely. Sounds good. 18:23:56 #info copr upgrade also coming up tomorrow. There will be a short outage. 18:25:03 * tflink doesn't know if this counts as application status (or if it's too far out) but we're hoping to get taskotron farther deployed this week and next 18:25:10 cool. 18:25:57 tflink: oh, there's an issue with resultsdb-stg01... I tried to put it in the stg group, but it can't talk to stg fas. ;( I have a RHIT ticket to fix that... 18:26:08 we do want the *stg* qa stuff to be in infra stg right? 18:26:20 (for fas, and talking to proxy01.stg and koji01.stg, etc?) 18:27:30 nirik: resultsdb doesn't have auth right now, so fas isn't a huge issue 18:27:49 well, for local accounts/sudo 18:27:57 yeah, I wasn't thinking about that 18:28:13 I'm fine eihter way, stg means that my yubikey doesn't work but I can deal with that 18:28:13 I can get it sorted out. I would expect we want them to interact with stg 18:28:22 ok 18:28:39 unless the 2fa issue with stg fas was worked out 18:28:41 hopefully the proxy stuff works for you now. 18:28:49 er 2fa with yubikey and stg fas 18:29:02 I'm not sure which issue? wanting one yubikey to work with the same slot for both? 18:29:15 for a while, yubikey couldn't be used with fas stg for 2fa 18:29:28 I was told to use google authenticator instead 18:29:40 huh, I haven't tried, but I can... 18:30:07 I hooked the load testing setup up to resultsdb-stg for a test 18:30:21 so it'll get tested out more shortly, even if the names don't quite match :) 18:30:38 ie, taskotron-dev.fp.o working with taskotron.stg.fp.o/resultsdb 18:30:42 ok. 18:31:25 #topic Sysadmin status / discussion 18:31:33 ok, lets see on the sysadmin side... 18:31:58 I have all our ducks in a row finally for rhel7 setup... so we will be setting up kickstarts and moving to rhel7 instances soon where we can. 18:32:32 We will have to see where we are application stack wise before moving any of the application instances over. 18:32:44 and we also talked about python3 sometime. 18:32:45 * oddshocks nods 18:33:04 \me supee 18:33:34 but a lot of simpler stuff can move to 7 sooner... like download servers, virthosts, the mailman instances (so we can deploy hyperkitty with rhel7 instead of f19) 18:33:55 * mpduty is here 18:34:01 nirik: does it look like things are back on track? 18:34:04 I'm still hanging on 18:34:04 roshi: I'm starting work on getting that documentation created 18:34:11 \ still watching over the meeting attempting to start 18:34:28 ootbro: we started and have had a few minutes with no net split. ;) 18:34:45 which documentation ootbro? 18:34:51 * roshi doesn't recognize your nick 18:35:07 nirik: thanks.. I've been watching the lines scroll past here 18:35:16 I've gotten ansible on lockbox using controlpersist and ssh pipelining, so playbook runs should be much faster. This is using a openssh scratch build with that support. 18:35:25 roshi: documentation on how the Fedora infrastructure is laid out 18:35:43 ah, sweet 18:36:01 sorry, I ask a lot of people for documentation on things and can't keep it all straight in my head :) 18:36:01 #info rhel7 setup almost in place to start moving machines to it. Will work on kickstarts next week. 18:36:18 #info ansible using ssh pipeline/controlpersist on lockbox, should be much faste.r 18:36:24 not a problem..... I picked up the project a few weeks ago, with the announcement on here 18:36:44 #info smooge and nirik will be heading out to phx2 datacenter in late july... 22nd to 25th 18:37:48 #topic nagios recap 18:38:19 ?tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25 18:38:20 nirik: http://tinyurl.com/q8j48o9 18:38:32 our bug alerter is virthost-comm02 this week. 18:38:46 it's got some kind of disk issue going on thats making it slow. 18:38:59 we are waiting for it to finish raid check and then likely will try a reboot. 18:39:36 nirik: when were you thinking the reboot would be? 18:39:47 tflink: not sure yet... this raid check is taking forever. ;( 18:40:04 [==================>..] check = 91.9% (447933184/487298048) finish=653.5min speed=1003K/sec 18:40:04 I think we will be rebooting it before the raid check is done 18:40:12 oh wait 18:40:18 it's getting close 18:40:18 thats different from when I looked 18:40:28 tflink: so, perhaps tomorrow? 18:40:36 that is 1 MB/sec on a bunch of 3 GB/sec disks 18:40:45 of course they are busy 18:40:56 ok, I think I have some stuff on there that's not set up to rebound on reboot so please let me know when you reboot it 18:40:56 yeah, it's definitely not right. ;) 18:41:20 tflink: can do... we also need to coordinate with arm folks as the arm koji instance is on that virthost too 18:41:39 #topic Upcoming Tasks/Items 18:41:39 https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:41:56 anyhing upcoming anyone would like to note or schedule? 18:42:47 #topic Open Floor 18:42:53 anyone have items for open floor? 18:44:16 ok, if nothing else will end the meeting in a minute, hopefully before the next net.split. ;) 18:44:24 :) 18:44:47 my email issues are fixed, and I'll be RHCE certified next Friday ;) 18:44:59 danofsatx-work: congrats 18:46:00 ok, thanks for coming everyone. Do continue in #fedora-admin, #fedora-apps and #fedora-noc. ;) 18:46:03 #endmeeting