18:00:11 <nirik> #startmeeting Infrastructure (2014-05-22)
18:00:11 <zodbot> Meeting started Thu May 22 18:00:11 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:11 <nirik> #meetingname infrastructure
18:00:11 <nirik> #topic aloha
18:00:11 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:11 <zodbot> The meeting name has been set to 'infrastructure'
18:00:11 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:45 <dgilmore> hola
18:01:01 <janeznemanic> hi
18:01:08 <lorddemon> Buenas tardes
18:01:10 <hammad> Hello !
18:01:16 <charul> Hi :)
18:01:19 * bwood09 is here
18:01:23 <mhaynes> Good afternoon.
18:01:29 <mapyth> Hello everybody! :)
18:01:29 <smooge> hllo
18:01:31 <nirik> hello everyone.
18:01:32 <ootbro> hi all
18:01:37 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:49 <nirik> any new folks like to introduce themselves in a line or two?
18:02:00 <nirik> or apprentices with questions or comments?
18:02:11 <ootbro> (waiting for new folks first)
18:02:17 <mapyth> Hello everyone, I am Mayank from India, would be hacking on bugspad this summer!
18:02:24 <charul> Hi everyone. I am Charul and have started working on Shumgrepper project.
18:02:27 * relrod here
18:02:40 * danofsatx-work is here
18:02:45 <hammad> Hello, This is Hammad, Working on fedora-college, that inherently comes under infra.
18:03:00 <lorddemon> Hizo everyone i am Gonzalo from Bolivia
18:03:20 <nirik> great! welcome everyone.
18:03:51 <nirik> Those of you doing summer coding, would you have links to your projects for us to read up on?
18:04:04 <brnzi> Hello guys, My name is Bruno! I am from Brazil, but living in Los Angeles,Ca!
18:04:44 <mapyth> https://github.com/kushaldas/bugspad the project I would be working on!
18:04:51 <mapyth> making a UI for it.
18:04:55 <hammad> https://github.com/hammadhaleem/fedora-college    Fedora College.
18:05:24 <charul> https://github.com/fedora-infra/shumgrepper for Shumgrepper
18:05:47 <nirik> great. :)
18:06:10 <mpduty> hello everyone
18:06:34 <nirik> good luck in your coding. ;)
18:06:51 <ootbro> ready for the apprentices ?
18:06:56 <mapyth> I and my mentor kushal, were discussing about some revisions in the timeline of the project, to include the suggestions received from the infra team, for
18:06:57 <charul> thanks nirik :)
18:07:14 <nirik> ootbro: sure, fire away...
18:07:20 <ootbro> following up from the map/landscape/overview "new project" item last week (and the e-mail I sent to the mailing list).....   I haven't gotten any additional source material, so I'll start with what I listed in the e-mail.
18:07:26 <mapyth> and will be discussing tomorrow!
18:07:34 <nirik> mapyth: ok. Might be good to post to the list on it and that way we can get replies from anyone intrested...
18:07:58 <nirik> ootbro: sounds good.
18:08:59 <ootbro> nirik has already said that the first two steps sounded good -- an overall list of servers with a location, function, production status (prod, staging, testing, etc.)
18:09:10 <ootbro> which starts to put the pieces on the table.
18:09:15 * nirik nods.
18:09:32 <ootbro> and add a basic description for each general function that tells what that family of servers does.
18:09:32 <mapyth> nirik: okay sure! shall I put it on summer-coding mailing list?
18:09:57 <nirik> mapyth: I'd say the infrastructure list if you want infrastructure input into things...
18:10:50 <ootbro> In getting my ssh access fixed (thanks, again, nirik :) )...  I found an update that needs to be done to the sshaccess.txt file
18:10:55 <nirik> For any other new folks, do see: http://fedoraproject.org/wiki/Infrastructure/GettingStarted if you haven't already, and we can point you in the right direction in #fedora-admin and/or #fedora-apps agter the meeting.
18:11:37 <nirik> ootbro: those docs are in the 'infra-docs' git repo... which actually apprentices do have write access to. ;) Just go to lockbox01 and 'git clone /git/infra-docs' and modify it and commit and push.
18:12:04 <mapyth> nirik: okay. will finalize with my mentor tomorrow and post.
18:12:06 <ootbro> ok.  thanks.  that was my next question -- how to get an update posted.  :D
18:12:26 <nirik> ootbro: easy peasy. ;) (Hopefully)
18:12:50 <ootbro> I'll dig into the documentation for the "how" part of commit and push
18:13:09 <nirik> yep. should be lots of git docs out there... possibly too many. :)
18:13:16 <ootbro> :)
18:13:22 <ootbro> (done)
18:13:31 <nirik> ok, any other new folks or apprentices with questions ?
18:13:36 <nirik> welcome again to all new folks.
18:14:02 <nirik> #topic Applications status / discussion
18:14:18 * nirik sees most if not all our applications folks aren't around. ;(
18:14:29 <nirik> There continues to be some fallout from the pkgdb2 rollout.
18:14:46 <nirik> #info process-epel-requests script being worked on to work with pkgdb2
18:14:51 <brnzi> hi @<nirik> I am new, I am actually writing my introduction right now! :-)
18:15:04 <nirik> #info bugzilla component sync is also not working right, still need to investigate.
18:15:09 <nirik> brnzi: cool. ;)
18:15:14 <brnzi> I am looking for a sponsor.. :-)
18:15:42 <nirik> look for something interesting to you to work on first. ;)
18:15:52 <brnzi> :-)
18:16:21 <nirik> any other applications news today?
18:16:51 <nirik> #topic Sysadmin status / discussion
18:16:59 <nirik> on the sysadmin side...
18:17:11 <nirik> we did a mass reboot tuesday, everything seems to have gone just fine.
18:17:43 <nirik> Our build system is 100% working for the first time in a while... all our arm SOC's, buildvm's, buildhw are all up and running along nicely.
18:17:55 <danofsatx-work> yay!
18:17:58 <nirik> the ansible migration is rolling along
18:18:18 <nirik> I'm hoping to move the last things off our old app servers soon and retire them all.
18:18:35 <nirik> that will be nice.
18:18:49 <smooge> yay
18:19:02 <lorddemon> Sound good
18:19:58 <nirik> our backup server almost ran out of inodes last night... will be trying to clean up what we can there.
18:20:14 * threebean arrives late
18:20:15 <smooge> what was that a recursive backup?
18:20:24 <smooge> someone not playing nice?
18:20:37 <nirik> smooge: there was one gnome backup taking up a lot, it's now been fixed...
18:20:53 <nirik> but there's still a lot of inodes taken up. If my find ever finishes I can see what dirs have a lot.
18:21:36 <nirik> #info mass reboot last tuesday, went fine.
18:21:50 <smooge> for (i=0; i<infinity; i++); do ln -s foo $foo.$i done
18:21:51 <nirik> #info buildsys is 100% up and operational. All arm, buildvm, buildhw boxes working
18:22:09 <nirik> #info smooge and relrod got all the new download servers in place and working
18:22:35 <smooge> now I am dealing with hardware problems on the RDU download servers
18:22:38 <relrod> that was mostly smooge
18:22:47 <smooge> relrod, you did the ansible stuff
18:23:18 <nirik> ok, any other sysadmin side items to mention?
18:23:21 <abadger1999> nirik: question re: app servers  ; are the last things simply shifting as a group to new hosts or are they shifting to separate hosts?
18:23:47 <nirik> abadger1999: most of the last things are moving to the sundries servers... the big apps already moved to their own things
18:23:53 <abadger1999> <nod>
18:23:57 <relrod> smooge: doing ansible stuff is easier than looking at a screen and trying to hit f12 in a vnc window within 5 seconds ;)
18:23:58 <nirik> the last thing is freemedia. which is just a php/cgi
18:24:24 <nirik> I was thinking once I get that moved to power them off for a few days... see if anything breaks  or still depends on them.
18:24:36 <abadger1999> nirik: Cool.  One thought about that -- we may want to upgrade to rhel7 before mirrormanager is ported away from tg1.
18:24:48 <smooge> current items on my task list: fix download RDU server hardware issues, rebuild RDU servers to be ansible, build new log server, get stuff off old log server, and move virthost box over to cloud
18:25:05 <abadger1999> nirik: So we may want to split that away from the other sundries stuff i nthe future.
18:25:36 <abadger1999> (we can cross that bridge when we start thinking about rhel7 migration, though :-)
18:26:08 <nirik> abadger1999: ok. yeah. MM has 3 parts: mirrorlists (already moved to their own instances), mirrormanager adminwebapp (moved to sundries) and backend/cron stuff thats still on bapp02... still need to move that.
18:26:19 <henderbj> Hello. Sorry to be late!
18:26:33 <nirik> henderbj: no worries. welcome.
18:26:38 <abadger1999> (for those who haven't followed -- I'm not planning on maintaining TurboGears1 on EPEL7.  So mirrormanager nad FAS will be stuck on RHEL6 until we port them to a newer framework).
18:27:08 * oddshocks here late
18:27:13 * oddshocks roommate troubles
18:27:20 <nirik> abadger1999: we will have lots of other things to migrate, so we can save those for last.
18:27:23 <nirik> welcome oddshocks
18:27:29 <abadger1999> <nod>
18:27:36 * mapyth is pissed by my troublesome internet connection
18:28:22 <ootbro> (went business-class for my home connection and it's usually very stable)
18:28:46 <nirik> ok, lets see how nagios treated us this last week...
18:28:53 <nirik> #topic nagios/alerts recap
18:28:54 * threebean cringes
18:29:01 <nirik> https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25
18:29:09 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25
18:29:10 <zodbot> nirik: http://tinyurl.com/q8j48o9
18:29:33 <nirik> yeah, the new fedmsg monitoring was a bit shouty. ;)
18:30:04 <nirik> but I think we have that mostly tuned better now?
18:30:10 <henderbj> mirrorlist-serverbeach is always swaping?
18:30:20 <threebean> yeah.. there were actual problems it was reporting in the beginning.. those seem mostly worked out, but the periodic UNKNOWNs from badges-backend01 is still a mystery.
18:30:23 <nirik> ha. Just got another feedmsg alert. ;)
18:30:57 <nirik> henderbj: yeah, it's proving troublesome. I am not sure why it's having trouble where the other instances aren't. ;( I guess I could just destroy it and make one somewhere else.
18:31:23 <nirik> I've tried various things to make it happier. (Fewer threads, etc)
18:32:10 <henderbj> nirik: Maybe some processes are running there that are not present on those others
18:32:14 <nirik> and the telia stuff is typical. Our phx2 main datacenter to telia often has network issues.
18:32:33 <henderbj> nirik: maybe some backups?
18:32:40 <nirik> henderbj: very unlikely. They are configured from the same ansible playbook, so they should be pretty much identical.
18:32:49 <henderbj> nirik: ok
18:33:24 <nirik> The main difference is that is in another datacenter on different hardware.
18:33:33 <henderbj> Ah... i finished (at least i think) work for ticket #4325: https://fedorahosted.org/fedora-infrastructure/ticket/4325
18:33:37 <smooge> and the serverbeach hardware can at times get odd
18:33:39 <nirik> so it could be that that hw/network just sucks. ;(
18:33:57 <nirik> henderbj: cool. ;) I saw, but haven't had time to look yet.
18:34:35 <henderbj> nirik: Please check it and if it works, then we are done and can close the ticket
18:35:10 <nirik> so, on nagios: tune fedmsg alerts more and figure out badges-backend timeout, move or do something with mirrorlist-serverbeach, and sigh at telia. ;)
18:35:16 <nirik> it's on my list, yep.
18:35:38 <nirik> #topic Upcoming Tasks/Items
18:35:38 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:35:47 <nirik> anyone have upcoming items they want to note or schedule?
18:36:27 <threebean> Oh, I'll be out on vacation tomorrow.
18:36:27 <nirik> we have our FAD coming up in just a few weeks...
18:36:36 <nirik> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
18:36:38 <threebean> also Monday is a US holiday, so things might be quiet around then too.
18:37:04 <nirik> oh yeah, true...
18:37:36 <threebean> i'm not actually travelling, so I might poke my head into channel here and there.. :)
18:38:00 * nirik should be around, but might be playing video games or watching movies or whatever. ;)
18:38:04 <nirik> #topic Open Floor
18:38:24 <nirik> anyone have any items for open floor? questions? comments?
18:38:34 <henderbj> nirik has time to play video games???
18:38:40 <ootbro> (crickets chirping)
18:38:42 <nirik> sometimes. ;)
18:38:59 <nirik> on weekends. If I get to the gaming system before my GF does. ;)
18:38:59 <henderbj> ansible, and nagios wonderfull ;)
18:39:17 <ootbro> maybe you need two gaming systems?
18:39:41 <nirik> that would need 2 tv's...
18:39:55 <ootbro> space would be a problem with that
18:40:01 <henderbj> ho... or use multiseat consoles ;)
18:40:10 <nirik> yeah. Where's our virtual reality headsets!
18:40:22 <henderbj> i have a multiseat fedora machine... that's great thing to have ;)
18:40:42 <nirik> anyhow, lets continue over in #fedora-admin, #fedora-apps and #fedora-noc... Thanks for coming everyone!
18:40:44 <nirik> #endmeeting