19:00:05 <nirik> #startmeeting Infrastructure (2012-12-13)
19:00:05 <zodbot> Meeting started Thu Dec 13 19:00:05 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:06 <nirik> #meetingname infrastructure
19:00:06 <nirik> #topic welcome to the thunderdome
19:00:06 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:06 <zodbot> The meeting name has been set to 'infrastructure'
19:00:06 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:13 <nirik> who all is around for meeting?
19:00:17 * threebean is here
19:00:19 * samkottler is here
19:00:20 * lmacken 
19:01:01 <smooge> I am here
19:01:51 * nirik will wait a bit more for folks to wander in
19:02:02 * skvidal is here
19:02:05 * maayke is here
19:02:20 * marcdeop is here
19:02:55 * SmootherFrOgZ is around
19:03:50 <nirik> ok, lets go ahead and get started.
19:03:58 <nirik> #topic New folks introductions and Apprentice tasks.
19:03:58 * abadger1999 here
19:04:06 <nirik> any new folks, or apprentices with questions?
19:04:16 <maayke> hi guys, i'm new here. I'm  Linux System Admin and i would like to help with some tasks. I've been using Linux: RedHat(on servers) and Fedora since 2006, and other for a few years before that. I'm interested in sysadmin tasks.
19:04:39 <nirik> excellent. welcome maayke.
19:04:51 <maayke> thanks
19:04:53 <nirik> please see me in #fedora-admin after the meeting and I can get you setup in our apprentice group.
19:05:06 <maayke> ok, thanks
19:05:22 <nirik> any other new folks?
19:06:07 <nirik> ok, lets move on then... please do chime in anytime with questions or comments. ;)
19:06:09 <nirik> #topic Applications status / discussion
19:06:17 <nirik> any applications news this week or upcoming?
19:06:30 <nirik> we are in final freeze, but staging work is always open. ;)
19:07:25 <threebean> I've been trying to speed up the fedora-packages app and have some success with a new lib called dogpile.cache (python-dogpile-cache).
19:07:31 <nirik> I know threebean and lmacken have been working on the packages app and in particular it's bug handling page
19:07:36 * threebean nods
19:07:37 <nirik> jinx. ;)
19:08:01 <threebean> nirik: we applied it to the whole app; now everything's sped up, not just bugs.  :)
19:08:06 <nirik> oh, I can share some info about askbot here... we are behind on versions, so I am trying to update it to the newest upstream.
19:08:13 <threebean> we'll probably deploy to stg soon so people can poke at it.
19:08:19 <SmootherFrOgZ> abadger1999: do you have things to add to fas otherwise I think we can start push 0.8.16 to stg
19:08:28 <nirik> sadly, the new version requires 2 new packages. One of them I just branched for epel and should be fine.
19:08:51 <nirik> the other even more sadly, has 0 license info. I mailed the author, but have had no response.
19:09:11 <nirik> threebean: cool! could this apply to some of our other apps too?
19:09:29 <threebean> nirik: definitely.
19:09:37 <abadger1999> SmootherFrOgZ: I'd like to get some changes in surrounding the openid problem we had yesterday.
19:09:39 <nirik> nice.
19:09:56 <abadger1999> SmootherFrOgZ: But I can finish those today if you want to work on getting new fas to stg tomorrow/this weekend.
19:10:52 <SmootherFrOgZ> abadger1999: nods. I can wait this weekend or even next week if needed.
19:10:53 <abadger1999> threebean: how outdated does the bugz cache get using dogpile?
19:11:09 <threebean> abadger1999: we can configure it from puppet, but I have it set to 300 seconds locally right now.
19:11:35 <abadger1999> threebean: thinking about the workflow of, open bugz, open a page per open bug.  work on some set of bugs.  Close.  Refresh bugz page to see overview of what's left.
19:11:46 <abadger1999> threebean: sounds like that will work fine.
19:12:00 <threebean> cool
19:12:12 <nirik> should we add that it's cached to the page?
19:12:30 <threebean> nirik: good idea.. we can stick that in the footer.
19:12:33 <abadger1999> nirik: that would be a ggod idea.
19:12:40 <nirik> ie, 'this page is refreshed every 300 seconds' or something.
19:13:23 <nirik> also in app news, pingou had: "I hope to release fedocal 0.1.0 alpha this week-end :)"
19:13:34 <threebean> \o/
19:14:35 <smooge> yay
19:14:52 <nirik> lmacken: you around? have you had any chance to get anywhere with bodhi 2 recently? I wonder if it it's in enough state we could setup a test instance for people to see it? (in sight, in mind and all that). :)
19:15:26 <nirik> just a thought.
19:15:33 * nirik looks for any other application side news.
19:16:19 <nirik> oh, we have a test instance setup for a gallery3. I've not gotten to doing anything on it... if there are folks with cycles that want to work on it, please let me know.
19:16:22 <lmacken> nirik: there's not much to see right now other than bare-bones grids and json apis, but I'll try and get something staging ready shortly
19:16:30 <nirik> I think we were waiting a bit for relrod to come up with a php fas plugin for it.
19:17:03 <nirik> lmacken: it would be cool if we had at least something ready for fudcon... and could get off the 'keep tweaking bodhi1' treadmill. ;)
19:17:12 <lmacken> nirik: yeah, exactly
19:17:59 <smooge> threebean, lmacken, the swap problem on packages... do we need to add more ram?
19:18:16 <lmacken> smooge: it doesn't seem like the swap is being used because it is out of memory
19:18:18 <nirik> I don't think that would help, it's consuming all memory+swap
19:18:24 <lmacken> someone suggested tweaking the swapiness
19:18:31 <nirik> I thought it was using it all.
19:18:51 <lmacken> nirik: it had a bunch free when I was poking at it yesterday
19:18:58 <lmacken> not sure... collectd would know?
19:18:58 <smooge> when I got onto the box memory was full and swap was full.. but there are probably multiple issues
19:18:59 <nirik> huh.
19:19:02 * nirik looks at collectd
19:19:10 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=packages01&hostname=packages02&plugin=swap&timespan=86400&action=show_selection&ok_button=OK
19:19:59 <nirik> so I think it's def just leaking.
19:20:04 <nirik> or not freeing up things.
19:20:13 <nirik> is that wsgi?
19:20:23 <threebean> on packages01, it looks like it starts a midnight..
19:20:27 <threebean> it that when the cronjob runs?
19:20:32 <threebean> cronjobs
19:21:14 <nirik> daily should be 4am
19:21:36 <nirik> anyhow, we should fix that... restarting it is anoying. ;)
19:21:47 <nirik> any other apps news?
19:22:25 <nirik> #topic Sysadmin status / discussion
19:22:33 <nirik> ok, on sysadmin side of things... lets see.
19:22:47 <nirik> we have some new boxes coming in, but will need to get them racked before we can do much with them.
19:22:58 <lmacken> threebean: yeah, I think the cron jobs are probably pushing it over.
19:23:04 <lmacken> giving packages01 a little more ram couldn't hurt :)
19:23:21 <nirik> we still have 3 bits of storage to migrate. I am hoping to do that after f18 is final...
19:23:35 <nirik> our wiki attachments nfs (should be a very short downtime for wiki)
19:24:01 <nirik> and our 2 iscsi volumes (should be migrating 2 machines (shutdown, bring up, shutdown again, move back))
19:24:29 <nirik> there's also a network outage scheduled for phx2 for early jan... (at least I think it still is, haven't heard)
19:24:37 <smooge> Jan 5
19:24:41 <nirik> yeah
19:24:42 <smooge> I believe
19:25:03 <nirik> thats the last I heard.
19:26:07 <nirik> I did some work on making trac faster. I found one nasty bottleneck. I need to adjust all the trac instances for that (it was syncing git repos on every page load)
19:26:34 <threebean> on every HTTP request?
19:26:43 <nirik> smooge: have you heard anything yet about the i2 download move stuff?
19:26:55 <smooge> no :/
19:27:05 <smooge> sometime soon ish.. maybe
19:27:06 <nirik> threebean: every one to trac. It would sync the git repo, then process the sql or whatever for the page load...
19:27:19 <threebean> man.  wow.
19:27:25 <nirik> yeah, crazy.
19:27:40 <nirik> I guess to keep it's source browser up to date...
19:27:49 <nirik> you can disable it with a simple ini add.
19:28:14 <nirik> makes things about 1/2 as much loading time.
19:28:18 * mdomsch lurks
19:28:47 <nirik> anything else sysadmin side?
19:28:59 <nirik> #topic Private Cloud status update / discussion
19:29:13 <nirik> we continue to work with the cloud setup and ansible repo... ;)
19:29:47 <nirik> I suspect we cna look at things at fudcon and see if we can knock out the rest to make them 'production' ready
19:30:28 <SmootherFrOgZ> nirik: sure thing ;)
19:30:38 <nirik> SmootherFrOgZ: you gonna make it to fudcon?
19:30:44 <SmootherFrOgZ> nirik: yep
19:30:47 <mdomsch> network outage 3 days before release?
19:30:47 <nirik> awesome.
19:30:47 <mdomsch> ugh
19:30:51 <mdomsch> lousy timing
19:30:54 <nirik> mdomsch: yeah, I am not happy about that.
19:31:02 <nirik> mdomsch: they claim it will be <15min for us
19:31:08 <mdomsch> k
19:31:17 <nirik> but you know how things go...
19:31:39 <nirik> ok, moving on to upcoming items:
19:31:48 <nirik> #topic Upcoming Tasks/Items
19:31:48 <nirik> #info 2012-12-11 to 2013-01-08 F18 Final Freeze
19:31:49 <nirik> #info 2012-12-20 colorado RH holiday lunch
19:31:49 <nirik> #info 2012-12-24 to 2013-01-01 Red Hat Shutdown for holidays.
19:31:49 <nirik> #info 2012-01-05 PHX2 outage.
19:31:50 <nirik> #info 2013-01-08 F18 release.
19:31:51 <nirik> #info 2013-01-18 to 2013-01-20 FUDCON Lawrence
19:31:53 <nirik> #info 2013-01-29 f19 feature submission deadline.
19:31:56 <nirik> anything we need to note or schedule?
19:32:38 <mdomsch> oooh, I want in on the colorado lunch. :-)
19:32:41 * nirik notes the colorado rh lunch is on there because I will be away from keyboard for a while that day driving down to it and back. ;)
19:32:52 <nirik> mdomsch: you gonna be in co then? :)
19:33:02 * mdomsch passes through Denver 12/29
19:33:33 <nirik> I'll be around if you want to grab a beer somewhere. ;)
19:33:35 <mdomsch> how confident is rel-eng / QA on hitting 01/08 release?
19:33:55 <nirik> There's a lot of blockers right now, but they are knocking them out... hard to say
19:34:24 <nirik> #topic Open Floor
19:34:36 <nirik> anyone have anything for open floor? ideas? suggestions? rants?
19:35:07 <mdomsch> MM grew a new set of repository pointers this afternoon: https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-install-18&arch=x86_64
19:35:11 <mdomsch> for fedup's use
19:35:23 <nirik> cool. ;)
19:35:24 <mdomsch> just DB items, no code changes, so it doesn't break freeze
19:35:46 <mdomsch> is there a list of stuff to accomplish at FUDCon?
19:36:04 <nirik> there's not, but perhaps we should make one.
19:36:27 * mdomsch also raises 3 cheers for the 2FA team
19:36:51 <nirik> yeah, I'm glad we have that in place now.
19:37:32 <nirik> and we will probibly want to discuss where we want to expand it to down the road.
19:37:39 <nirik> (if anywhere)
19:37:46 <abadger1999> <nod>
19:38:20 <abadger1999> Due to how well the fad worked, one thought was to use fudcon to plan for some fads that we should have over the course of hte year.
19:38:26 <smooge> RH lunch in CO?
19:38:36 <nirik> smooge: yep. You could drive up! :)
19:38:41 <smooge> next you will get an office!!!
19:38:57 <nirik> abadger1999: yeah, agreed...
19:39:16 <SmootherFrOgZ> abadger1999: +1
19:40:20 <nirik> where was that wiki page we had about things to do from the last fudcon?
19:40:28 <nirik> I wanted to make a new version of it for the next one.
19:41:09 <nirik> oh well, will just make a new page.
19:41:31 * mdomsch wants some real-world examples of using the new message bus, so I can hook MM up to it
19:41:58 <nirik> we can def provide that.
19:43:02 <nirik> anyhow, I will make a fudcon planning page of some kind and we can gather ideas and discuss them next week... hash out the big sessions we want to have.
19:43:45 <nirik> ok, anything else? or shall we call it a meeting ?
19:43:59 <nirik> mdomsch: drop by #fedora-apps and threebean can point you at docs for fedmsg
19:44:15 <threebean> mdomsch: I missed your comment.  yeah, I can show you.  :)
19:45:38 <nirik> ok, thanks for coming everyone.
19:45:45 <smooge> okie dokie
19:45:49 <nirik> do continue in #fedora-admin, #fedora-noc and #fedora-apps!
19:45:55 <nirik> #endmeeting