18:00:25 <nirik> #startmeeting Infrastructure (2014-07-10)
18:00:25 <zodbot> Meeting started Thu Jul 10 18:00:25 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:26 <nirik> #meetingname infrastructure
18:00:26 <nirik> #topic aloha
18:00:26 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:26 <zodbot> The meeting name has been set to 'infrastructure'
18:00:26 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:01:15 <tflink> .hellomynameis tflink
18:01:16 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
18:01:29 * threebean 
18:01:29 <relrod> here
18:01:31 * lmacken 
18:01:32 <janeznemanic> hi
18:01:36 <selvakumarm> Hi
18:01:37 * hammad_ is here
18:01:47 * lalit_ here
18:01:50 <nirik> morning everyone.
18:01:54 * charul_ here
18:02:00 <ootbro> or good afternoon?   (or evening?)
18:02:02 <dgilmore> afternoon
18:02:03 * pingou 
18:02:06 <pingou> evening
18:02:07 <nirik> #topic New folks introductions and Apprentice tasks
18:02:08 <relrod> sitting at the park tethering - it's a beautiful day, 80F :D
18:02:13 <nirik> it's always morning on irc. ;)
18:02:31 <nirik> any new folks like to introduce themselves or apprentices with questions, comments or ideas?
18:02:41 * lanica is here for the meeting.
18:03:30 <ootbro> nirik: any (final?) comments on my last e-mail about the map?   I plan to start digging through the zone files for proxy info after the meeting.
18:04:07 <nirik> ootbro: I don't think I had any more... I've been very swamped of late tho. ;)
18:04:27 <oddshocks> Here
18:04:28 <ootbro> I understand ....  no problem.....
18:04:53 <threebean> ootbro: thanks for working on it.
18:05:01 <threebean> it will be useful to have
18:05:13 <nirik> indeed. ;)
18:05:15 <ootbro> you're welcome ......   yeah.....  it will answer some of my still-newbie questions
18:05:30 <smooge> good day to you all
18:06:40 <nirik> ok, any other new folks or apprentice questions? moving on in a few...
18:06:58 <ootbro> for everyone else....   the first part of the work is a basic (pictoral) diagram that shows all the data centers (with location notations), with the 10.5 and 192 networks tagged.
18:07:39 <ootbro> .... and an indication of which which centers have a proxy and what services are forwarded through each proxy
18:08:04 <ootbro> (It's the last part that will have me digging through the DNS zone files.)
18:08:10 <ootbro> <end>
18:08:31 <selvakumarm> ootbro: where will i find the diagram ??
18:08:47 <ootbro> I plan to put it somewhere on the fp.o wiki
18:09:47 <selvakumarm> please do the same and circulate the location.. thanks
18:09:48 <nirik> #topic Applications status / discussion
18:09:55 <nirik> any application news this week or upcoming?
18:10:01 <pingou> new elections in prod
18:10:07 <pingou> new pkgdb2 in prod
18:10:29 <pingou> cnucnuweb has been renamed anitya, dev instance updated: http://209.132.184.188/anitya/
18:10:30 <lmacken> new bodhi went to prod last week I think, minor bug fixing around python-bugzilla
18:10:37 <tflink> taskotron staging is mostly deployed, waiting on a firewall rule change ticket before it's done
18:10:45 <pingou> (the backend was pretty much completely re-written)
18:10:52 <threebean> I've got two stuck in staging at the moment.
18:10:59 <pingou> busy week :)
18:11:02 <threebean> adimania's datagrepper work is good -> https://fedorahosted.org/fedora-infrastructure/ticket/4393
18:11:13 <threebean> but I'm having a hrad time getting it all set oh rhel7
18:11:15 <nirik> cool. lots of stuff.
18:11:28 <nirik> tflink: whats left for prod? that vpn access... and ?
18:11:32 <threebean> also, fedora-tagger.  new release in staging.  one of the features broke on postgres so we're holding off for the moment.
18:11:43 <oddshocks> Fedimg is almost production ready. fedimg01 and fedimg01.stg machines were set up + playbooks thanks to threebean.
18:11:50 <lmacken> threebean: related to fedmsg (wrt rhel7 issues) ?
18:12:04 <threebean> lmacken: nah, just lots of stuff that never got branched.
18:12:09 <lmacken> ah
18:12:11 <tflink> nirik: client setup, mostly. so a few more rhel7 installs, vm creations and hostname changes
18:12:21 * webpigeon sneaks in
18:12:26 <lmacken> threebean: you've gotta pave the way for us ;)
18:12:40 <pingou> +1, it's nice :)
18:12:53 <nirik> tflink: cool. Hopefully not too bad to get done.
18:13:25 <tflink> nirik: yeah, all stuff that has been done before in dev+stg so hopefully not too bad :)
18:13:41 <nirik> we also have a new fedoauth version in prod
18:14:02 <nirik> #info application updates this week: elections, pkgdb2, fedoauth, bodhi (prod)
18:14:33 <nirik> #info in stg: fedora-tagger, datagrepper, taskotron (almost)
18:14:49 <nirik> #info development, cnucnu web renamed to anitya
18:15:28 <nirik> anything else on the application front?
18:15:35 <pingou> I've been going through the pkgdb1 tickets and closing what could be, so we have 32 tickets left
18:15:44 <nirik> tflink: is there a deadline to get prod taskotron replacing autoqa?
18:15:45 <abadger1999> Yay!
18:15:50 <pingou> which are either still valid or which needs input from someone else :)
18:15:53 <nirik> excellent.
18:15:56 * pingou glances at abadger1999 ;-)
18:16:00 <abadger1999> pingou: Are we going to just migrate whatever is left at the end of hte purge?
18:16:07 <nirik> pingou: oh, we still need to orphan that one inactive maintainers packages.
18:16:18 <pingou> nirik: should work now from the pkgdb-cli in git
18:16:28 <pingou> server side it should work
18:16:31 * pingou crosses his fingers
18:16:43 <nirik> pingou: cool. I can try after the meeting?
18:17:00 <pingou> nirik: either that or tomorrow, I won't last long after the meeting
18:17:17 <tflink> nirik: I had wanted it to be by branch, honestly
18:17:23 <pingou> nirik: if you try, run it with --debug and if it fails I can look at fixing it tomorrow morning
18:17:28 <nirik> pingou: ok
18:18:01 <pingou> tflink: is there a target branch to get taskotron in prod and replacing autoqa? :)
18:18:17 <nirik> tflink: well, hopefully soon then.
18:18:25 <tflink> nirik: but realistically, I leave for vacation 2 weeks from tomorrow. I'd like to have it done with a week to spare before then if possible
18:18:32 <tflink> pingou: target branch?
18:18:37 <pingou> tflink: f21, f22?
18:18:48 <pingou> f19? (we're late for that one ^^)
18:18:50 <nirik> tflink: yeah, we go into freeze on the 22nd too, so ideally before then would be very good.
18:18:51 <tflink> pingou: f21 branch :-P
18:19:32 <nirik> ok... anything else on the application side?
18:19:43 <nirik> #topic Sysadmin status / discussion
18:19:54 <nirik> we have been moving more instances to rhel7 and ansible.
18:20:20 <nirik> we are down to 95 hosts in puppet.
18:20:30 <pingou> cool
18:20:34 <nirik> #info smooge working on log01 migration
18:20:41 <nirik> #info relrod working on noc migration
18:20:47 <relrod> I almost have nagios moved into ansible, will work on it more later today
18:20:48 <relrod> yes that
18:21:12 <nirik> cool.
18:21:21 <nirik> relrod: could you also at some point move noc03 off virthost02?
18:21:30 <nirik> vh02 is one of the ones we are going to be retiring
18:21:53 <nirik> perhaps vh17 would be a good place?
18:21:56 <relrod> nirik: hah. yeah, I was just using it for testing - it should hopefully be easy to just change the host and rerun the playbook
18:22:04 <nirik> yep
18:22:32 <relrod> I only picked vh02 because noc01 is currently there
18:22:52 <tflink> smooge: any news on racking for virthost-comm03.qa and qa09.qa?
18:22:53 <nirik> I moved/redid our rawhide and branched compose machines. After a few hiccups they are working fine now.
18:23:08 <nirik> tflink: they have been racked. ;) we just need to get them networked and installed.
18:23:10 <smooge> they are racked. I need to set them u
18:23:29 <nirik> smooge: oh, what happened to those retrace machines? where were we on them?
18:23:51 <smooge> I have asked michael friedman for tracking numbers
18:24:05 <nirik> might ping again, it's been a while since we heard.
18:25:41 <nirik> #topic nagios/alerts recap
18:25:56 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25
18:25:56 <zodbot> nirik: http://tinyurl.com/q8j48o9
18:26:26 <threebean> I like this meeting routine ;p
18:26:28 <nirik> so, typical telia stuff. I am working on moving everything off telia.
18:26:37 <nirik> threebean: should we adjust the notifs backlog?
18:26:46 <nirik> it seems to get hammered by big clumps?
18:26:55 <threebean> it does.  and we could.
18:27:04 <pingou> .tiny https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=notifs-backend01.phx2.fedoraproject.org;plugin=fedmsg;plugin_instance=hub;type=queue_length;type_instance=FMNConsumer_backlog;begin=-3000000
18:27:04 <zodbot> pingou: http://tinyurl.com/ocdbwf7
18:27:20 <threebean> I think the big one was the mass branch this past week.
18:27:39 <threebean> pingou: that goes too far back.  it includes peaks that wash out the latest activity.
18:27:39 <nirik> yeah, then the secondary arches following that
18:27:58 <pingou> .tiny https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=notifs-backend01.phx2.fedoraproject.org;plugin=fedmsg;plugin_instance=hub;type=queue_length;type_instance=FMNConsumer_backlog;begin=-900000
18:28:00 <zodbot> pingou: http://tinyurl.com/pjgob9a
18:28:01 <pingou> over 10 days
18:28:06 <nirik> yeah, our collectd has been acting up.
18:28:12 <threebean> hm, :/
18:28:16 <nirik> I think it's due to new rhel7 hosts using a different collectd version
18:28:43 <nirik> we need to get a log01 up with the new server (which smooge is working on)
18:29:14 <nirik> the other thing I see is packages03 hitting oom lately. Not sure what the story is there.
18:30:26 <nirik> it's been a bit more quiet the last few days, so we can wait and see.
18:30:30 <nirik> #topic Upcoming Tasks/Items
18:30:30 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:30:38 <nirik> anyone have upcoming items they would like to note or schedule?
18:31:08 <nirik> #info smooge and nirik will be out at the phx2 datacenter july 22 to 24th
18:31:34 <nirik> Our freeze for alpha is supposed to start the 22nd, but I might make it start the 24th and give us time to get things down out there.
18:31:48 <lanica> What kind of tasks will you be working on?  Just auditing or...?
18:31:48 <nirik> not sure how much distruptive we have pending tho
18:32:03 <threebean> #info threebean on vacation july 26 through august 2nd
18:32:19 <nirik> lanica: pulling old machines out, setting up some new storage, possibly doing firmware updates, checking whats connected where.
18:32:48 <nirik> we may be renetworking things if we get some new network space by then
18:33:27 <nirik> lanica: basically anything that we can do that needs hands on
18:34:14 <nirik> threebean: just in time to rest up before flock. ;)
18:34:23 <relrod> #info relrod away at a conference July 22-23
18:34:51 <relrod> Will have my phone though
18:34:54 <nirik> tis the season of summer travel. ;)
18:35:12 <threebean> :)
18:35:14 <nirik> #topic Open Floor
18:35:24 <nirik> anyone have anything for open floor? questions, comments, ideas?
18:36:06 <ootbro> one general question to the experienced admins .....   what are some good (free) resources for learning shell scripting and python?
18:36:39 <nirik> for python there's a 'dive into python' book thats supposed to be pretty good.
18:36:56 <oddshocks> A lot of folks love Learn Python The Hard Way which is free online
18:37:02 <hammad_> Hello ,I am hammad, working on fedora college. We are almost done with it, and pushed the code to fedora-infra.
18:37:09 <hammad_> Working on GsoC
18:37:27 <pingou> hammad_: btw, you may want to see with your mentor if you want to keep the repo being a 'fork'
18:37:39 <pingou> in the future, for pull-request it can quickly become annoying
18:37:55 <hammad_> yeah, let me talk with him ,maybe we can move it permenantly
18:37:56 <ootbro> oddshocks:  the price is certainly right -- and I wouldn't have another book jammed into the home office.
18:37:58 <hammad_> to fedoraa infra
18:38:17 * pingou has the fedora-tagger example in mind
18:38:23 <tflink> ootbro: "learn python the hard way" is a good python resource
18:38:41 <tflink> or "the hitchiker's guide to python"
18:38:58 <tflink> but that's more of a reference, lpthw is designed to teach
18:39:11 <threebean> oo, I recommend http://pyvideo.org/
18:39:22 <ootbro> tflink:  oddshocks:  with both of you recommending lpthw, I'll probably go that route at first.
18:39:42 <tflink> ah, I missed that it was already mentioned :)
18:39:48 <nirik> I can't think of any shell books off hand.
18:39:49 <ootbro> not a problem... :)
18:40:00 * nirik doesn't learn well from books... I learn better by doing.
18:40:00 <tflink> there's a 'learn shell the hard way' as well
18:40:38 <oddshocks> tflink: great minds
18:40:45 <nirik> anyhow, if nothing else, will close out in a minute or so.
18:40:56 <oddshocks> ootbro: IMO the best resource for shell scripting is sometimes Stack Exchange :P
18:40:58 <ootbro> nirik:  I do, too, but I need to do a little reading first....   plus I'm probably several months away from needing to learn either
18:41:21 <oddshocks> ootbro: just search "how do I do the thing in <language>" and it's probably already been answered, and if not, you can ask the question yourself
18:41:30 <nirik> that too
18:41:31 <tflink> nvm, not a shell book - more of a cli crash course
18:41:51 <ootbro> my thanks to all for the suggestions.  I'll cull the URLs and titles from the meeting minutes.
18:42:05 <tflink> I've always liked http://www.tldp.org/LDP/abs/html/ for bash, though
18:42:20 <nirik> ah, tldp... there's something I have not seen in a while.
18:42:29 <ootbro> I can program in other languages, but know now shell or python at this point.
18:42:34 <ootbro> "know no"
18:43:58 <nirik> alright, thanks for coming everyone!
18:44:01 <nirik> #endmeeting