19:00:00 <nirik> #startmeeting Infrastructure (2012-11-15)
19:00:00 <zodbot> Meeting started Thu Nov 15 19:00:00 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic Welcome starfighters!
19:00:01 <nirik> #chair smooge skvidal relrod ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik relrod ricky skvidal smooge threebean
19:00:10 * skvidal is here
19:00:14 <smooge> here
19:00:21 * pingou 
19:00:36 * SmootherFrOgZ waves
19:01:14 * miguelcnf yeallow
19:01:17 * relrod_school here
19:01:43 <frankly3d> lurking
19:01:50 * lmacken 
19:02:09 <nirik> ok, cool. lets go ahead and get started.
19:02:11 <nirik> #topic New folks introductions and Apprentice tasks
19:02:26 <nirik> any new folks like to introduce themselves, or any apprentices want to discuss anything?
19:02:49 <threebean> (here)
19:03:07 * samkottler is here
19:03:11 <abadger1999> óla
19:03:47 <nirik> ok, lets move along then... ;)
19:03:53 <nirik> #topic Applications status / discussion
19:04:03 <nirik> Any application/devel news this week or upcoming?
19:04:13 <pingou> I wrote some docs for fedocal
19:04:20 <skvidal> coprs work is moving along
19:04:25 <lmacken> I pushed a new bodhi bugfix release out this week.
19:04:33 <skvidal> we have code in the git repo now and I'm doing tests on the backend in euca now
19:04:50 <skvidal> hopefully slavek will have some good stuff for us next week for the front end, he's been sick :(
19:05:11 <nirik> #info docs update for fedocal - pingou
19:05:21 * relrod_school is still working on php-fas though this week has been pretty hectic school-wise, and I've been offline more than I'd like.
19:05:22 <nirik> #info coprs work is moving forward
19:05:32 <pingou> https://fedocal.readthedocs.org/en/latest/
19:05:33 <nirik> #info lmacken pushed out a bodhi bugfix update.
19:05:35 <skvidal> http://git.fedorahosted.org/cgit/copr.git
19:05:53 <nirik> skvidal: whats the frontend in? flask?
19:05:53 * jds2001 here
19:06:02 <skvidal> nirik: yep and its on openshift atm
19:06:17 <nirik> ah, ok.
19:06:17 <SmootherFrOgZ> I did work on couple defect task from FAS preparing a new release but I don't think it will hit prod before security FAD
19:06:38 <SmootherFrOgZ> there are some already pushed to master, etc
19:06:46 <nirik> SmootherFrOgZ: will you be able to help out remote on the 2 factor fas work at the fad?
19:07:07 <SmootherFrOgZ> sure, I be there remotely
19:07:11 <SmootherFrOgZ> will*
19:07:16 <nirik> excellent.
19:07:21 <threebean> I never got that zeromq3 stuff into production like I said I would last week.  Gonna wait now until the freeze is up again.
19:07:30 <nirik> fair enough.
19:07:43 <nirik> #info zeromq3 will go to production after freeze.
19:07:57 <threebean> But i released fedmsg-0.6.0 today with a bunch of goodies.  A fedmsg-announce command for admins, a fedmsg-twitter daemon, and some new authz stuff.
19:08:08 <nirik> FYI, I pushed a security update for cgit to all our cgit using hosts this morning. Thanks to lmacken for seeing it. ;)
19:08:09 <SmootherFrOgZ> ls
19:08:11 <threebean> gonna test it all out in stg in the coming weeks
19:08:33 <nirik> #info cgit update today for security issue.
19:09:01 <lmacken> .bug 870713
19:09:03 <zodbot> lmacken: Bug 870713 CVE-2012-4548 cgit: syntax-highlighting.sh command injection - https://bugzilla.redhat.com/show_bug.cgi?id=870713
19:09:04 <lmacken> (for reference)
19:09:19 <nirik> ok, any other application / development news ?
19:10:12 <nirik> lmacken / threebean: You want to mention the outreach for women internship thing? Is that mostly devel tasks?
19:10:54 <threebean> oh right.
19:11:00 <lmacken> http://gnome.org/opw
19:11:09 <lmacken> some devel, some devops tasks
19:11:20 <threebean> https://fedoraproject.org/wiki/Outreach_Program_For_Women_2013?rd=Outreach_Program_For_Women_2012
19:11:23 <nirik> cool.
19:11:32 * nirik hopes it gets some good uptake.
19:11:48 <threebean> Already got one person asking to volunteer on cleaning up the packages webapp
19:11:58 <lmacken> and fedbages
19:12:06 <threebean> cool
19:12:17 * marcdeop is sorry is late
19:12:18 <skvidal> what's a fedbage?
19:12:23 <skvidal> oh badge!
19:12:24 <skvidal> got it
19:12:25 <skvidal> sorry
19:12:36 <lmacken> sorry, tyop
19:12:52 <nirik> excellent.
19:13:10 <nirik> #info https://fedoraproject.org/wiki/Outreach_Program_For_Women_2013 program coming up.
19:13:17 * jds2001 buys lmacken some typing lessons :D
19:13:23 <nirik> #topic Sysadmin status / discussion
19:13:33 <nirik> Any exciting sysadmin news this week or upcoming?
19:13:54 <pingou> cgit update you said :)
19:14:04 <nirik> I'm considering moving collab03 to collab01/02 pair on thanksgiving. ;) It takes about 10 hours to sync...
19:14:08 <smooge> budgets
19:14:14 <skvidal> nirik: umm. seriously?
19:14:14 <nirik> and really no one should be posting on thanksgiving to lists. ;)
19:14:21 <pingou> skvidal: exciting ?
19:14:23 <skvidal> nirik: you're going to move them during a holiday?
19:14:28 <pingou> err, I meant smooge
19:14:36 <skvidal> nirik: let me know in advance so I can turn off my phone
19:14:53 <nirik> skvidal: sure, since it should just be 'start rsync, go eat and relax, change dns, profit' ?
19:15:04 <skvidal> nirik: what is the key word in your sentence there?
19:15:04 <nirik> skvidal: I'm open to ideas on how to make the sync shorter.
19:15:16 <skvidal> nirik: 'should'
19:15:17 <skvidal> :)
19:15:32 <nirik> it's 82GB of mail archives and exploded mail posts.
19:15:50 <skvidal> okie doke
19:15:57 <smooge> hmmm I wonder how long it would take to cut it to a USB key run the key to the other system and copy off :)
19:15:59 * SmootherFrOgZ notices some scripts we're using on hostedxx are not up-to-date. will take care of them
19:16:01 <nirik> sure. I guess I can do it some other time, I just thought it might be good to get out of the way then
19:16:34 <nirik> SmootherFrOgZ: oh? what scripts?
19:17:16 <SmootherFrOgZ> scripts to setup scm repo, hosted project. there are some obsolete links (such as gnome-git-mail-blalbla)
19:17:45 <nirik> smooge: yeah, I think we just keep plugging away on budgets. There's 2 outstanding things I think now... a) quote for a bc02 replacement and b) I need to talk to qa and find out how many machines they really need.
19:18:15 <nirik> SmootherFrOgZ: we never had a script to setup scm/project. ;) Or you mean the one that adds trac? relrod was working on a app to make hosted projects.
19:18:42 <SmootherFrOgZ> ha, cool then.
19:19:00 <SmootherFrOgZ> well, we did use script one time.
19:20:06 <nirik> a script/app would be great... if you want to check with relrod you can help work on it I'm sure. ;)
19:20:26 <SmootherFrOgZ> like, gitsetup.sh (kinda old)
19:20:42 <SmootherFrOgZ> sure thing.
19:20:44 <threebean> https://fedorahosted.org/fedorahosted/
19:20:44 <pingou> https://github.com/CodeBlock/fedorahosted
19:21:00 <nirik> :)
19:21:21 <relrod_school> yep ^ that has both the flask app and the CLI to it.
19:22:07 <nirik> any other sysadmin news? We do have a few new machines on the way (hopefully)... a pair to replace sign-vault02 with (we have had hardware problems with it in the past), and a virthost. Also, at some point we will be getting some arm secondary arch stuff to add to the qa/community network.
19:22:36 * relrod_school sighs and just noticed a spam ticket on that fedorahosted/fedorahosted trac.
19:23:01 <nirik> #topic Private Cloud status update
19:23:17 <nirik> ok, so I decided to try cloudstack on our evens cloudlet this week...
19:23:17 <skvidal> euca cloudlet has a number of active instances on it
19:23:21 <skvidal> sorry
19:23:23 <skvidal> go ahead
19:23:45 <nirik> it's setup, but network is not working. I am exchanging emails with some cloudstack folks to try and track down the issue.
19:24:20 <nirik> I'll give it until next week, then probibly redo openstack on it...
19:24:28 <nirik> or just decide to drop it all into euca and get further on toward production.
19:24:35 <skvidal> nirik: so here's an idea
19:24:49 <skvidal> nirik: what if you setup euca on the evens cloudlet?
19:24:57 <skvidal> nirik: and compared the setup
19:25:06 <skvidal> nirik: also as a verification of what I setup on the odds?
19:25:06 <nirik> as a seperate instance/cloud?
19:25:13 <nirik> sure, could do that.
19:25:24 <skvidal> is there any thing in folsom
19:25:30 <skvidal> which is lacking for us to take that to production?
19:25:37 <skvidal> vlans?
19:25:46 <nirik> yeah, I need to mess with the new networking/vlans.
19:25:56 <skvidal> that's cinder, right?
19:26:04 <nirik> quantum. ;)
19:26:08 <skvidal> ah, sorry
19:26:10 <skvidal> what's cinder?
19:26:13 <nirik> cinder is the new storage thingie.
19:26:20 <skvidal> ah - sorry
19:26:28 <skvidal> I get the names confused
19:26:28 <nirik> no worries, there's too many codenames. ;)
19:27:01 <smooge> and they will change next release too
19:27:03 <skvidal> so if you'd rather not setup euca - then alternatively more openstack testing and enhancing for what we would need to do to make it production
19:27:17 <skvidal> the reason I was thinking of the euca setup
19:27:27 <skvidal> is to maybe spend more time on bfebs
19:27:30 <nirik> I kinda like the idea of diversity, and I think openstack will work for our production needs with some more work
19:27:38 <skvidal> sounds fair
19:28:00 <skvidal> speaking of production.
19:28:11 <skvidal> is there anyone who needs more instructions/help for using the euca cloudlet?
19:28:20 <skvidal> or for setting up an instance?
19:28:28 <skvidal> I know I have to reconnect with pingou
19:28:30 <skvidal> on the jenkins master
19:28:47 <skvidal> and to making fedocal a persistent ip
19:29:09 <nirik> I should make a persistent one just for grins to make sure, but I thought it looked pretty easy
19:29:09 * pingou likes
19:29:27 <SmootherFrOgZ> skvidal: we're using ansible to deal with new instance, right?
19:29:33 * herlo likes too
19:29:41 <nirik> so, do we want to discuss any of https://fedoraproject.org/wiki/Infrastructure_private_cloud#Moving_to_.22production.22
19:29:42 <smooge> I still need to go through the instructions you put out. I have it at #5 on my todo list
19:29:54 <skvidal> SmootherFrOgZ: we are
19:30:14 <skvidal> SmootherFrOgZ: and I've written instructions on doing that on an adhoc basis in the ansible repo
19:30:25 <skvidal> if anyone wants a relatively trivial programming task
19:30:36 <skvidal> the cron-runner for ansible needs to be fleshed out
19:30:37 <abadger1999> skvidal: Are there instructions/do we have persistent storage for our euca cloud?
19:30:41 <skvidal> so it will handle playbooks nicely
19:30:41 <SmootherFrOgZ> nod. anyone from syadmin has acces to it or?
19:30:43 <skvidal> abadger1999: I believe so.
19:30:45 <SmootherFrOgZ> skvidal: awesome
19:30:47 <abadger1999> Cool.
19:30:52 <skvidal> abadger1999: let me verify
19:31:01 <SmootherFrOgZ> nirik: we can
19:31:07 <abadger1999> skvidal: I'll look for it later to setup an elections dev box for fchiulli
19:31:29 <pingou> abadger1999: I had flask elections running on the fedocal box at some point
19:31:34 <skvidal> abadger1999: so.... okay - the instructions are in the euca notes - but not in the ansible repo
19:31:46 <pingou> abadger1999: I can do that again (unless you want to play w/ ansible of course :))
19:31:47 <skvidal> abadger1999: I will augment them with the proper sudo-ish ness etc
19:31:59 <skvidal> abadger1999: and an example of auto-attaching drives is in the copr-be playbook
19:32:07 <abadger1999> skvidal: Thanks!
19:32:19 <abadger1999> pingou: I want to play with ansible :-)
19:32:30 <pingou> abadger1999: fair enough :)
19:33:44 <nirik> skvidal: so, say I setup openstack again and get it to where euca2ools work against it nicely (which I already did have last time I set it up :) how do we determine which cloudlet we should run something on? random? based on capacity?
19:34:04 <skvidal> nirik: sure...
19:34:13 <skvidal> nirik: we could make any sort of arbitrary rule
19:34:25 <skvidal> we could make euca disposable-instances - using instance-backed
19:34:36 <skvidal> and openstack persistent instances - using volume-backed
19:34:43 <skvidal> or we could make them based on projects or users?
19:34:55 <skvidal> nirik: I'm open to whatever
19:34:58 <nirik> ok. just something to think about, we don't need to decide right away.
19:35:02 <skvidal> and if we decide we don't need diversity anymore
19:35:11 <skvidal> I'm also fine with just reducing us down to a single cloud software
19:35:25 <nirik> yeah, that also has some advantages sure.
19:35:31 <SmootherFrOgZ> +1
19:35:33 <skvidal> afaict the code I've written for ansible or the euca2ools is portable to both cloud instances
19:35:40 <skvidal> so it shouldn't require any porting
19:35:51 <skvidal> important words of note here
19:35:55 <skvidal> 'SHOULD NOT'
19:35:58 <skvidal> doesn't mean it won't ;)
19:36:14 <skvidal> but I've tried to be careful to not rely on any specific apis
19:36:24 <nirik> yeah, I think it should just be plug and play.
19:36:36 <nirik> but we will surely have to see what the world has in store for us
19:36:37 <skvidal> if, however, we opt to go to openstack wholey
19:36:46 <skvidal> then I'd think moving to nova-client and the api might be helpful
19:37:01 <skvidal> I've done some work against nova's client too - and it's not complicated to code against
19:37:38 <nirik> yeah, might give us better/different info.
19:37:49 <nirik> but we can burn that cloud when we come to it.
19:38:05 <nirik> any other cloud news/comments/ideas?
19:38:09 <skvidal> nirik: :)
19:38:28 <skvidal> nirik: one more comment
19:38:47 <skvidal> a good reason to stick with ec2-compat api is that as of right now fedora/red hat doesn't have a rax account
19:38:55 <skvidal> so if we ever wanted to move to public cloud for any reason
19:39:01 <nirik> yeah, good point.
19:39:02 <skvidal> and we were bound up in openstack's api
19:39:07 <skvidal> it would be difficult
19:39:23 <skvidal> that's all I had
19:39:36 <nirik> ok.
19:39:38 <nirik> #topic Security FAD update
19:39:48 <nirik> FAD is coming up soonly.
19:39:57 <nirik> any more planning or prelim stuff we can get done before then?
19:40:13 <abadger1999> the fas prelim could definitely use more work.
19:40:35 <abadger1999> I haven't had time in the past week but I'll see if I can do something if no one else does.
19:40:51 <skvidal> abadger1999: what sort of prelim do we need?
19:41:06 <nirik> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/FAS_plan
19:41:13 <abadger1999> Just making the https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/FAS_plan   more organized, broken down.
19:41:25 <skvidal> ah
19:41:35 <abadger1999> make some separate solid tasks that we can hand to different people independently.
19:41:37 <abadger1999> things like that.
19:41:38 <smooge> People going should start making blog posts about it starting next week
19:41:54 <smooge> I will be emailing reminders about that today
19:42:55 <nirik> yeah, I figure we get setup, we appoint/find someone to be IRC interface for remote folks, we do a bit of overview from mricon / others, we discuss policy decisions, then perhaps break into smaller teams... implement the pam side, the fas side, packaging if we need any, docs, etc.
19:43:09 <nirik> we can implement in stg first.
19:43:35 <nirik> also, beta release is scheduled for that morning, so we need to make sure we keep watch on that.
19:43:47 <smooge> I figure I could do those parts
19:43:52 <skvidal> abadger1999: are we planning on doing it against staging first - or do we want to dummy-up some systems in the cloudlets?
19:43:59 <smooge> an eye and IRC interface
19:44:02 <nirik> do we have idea on airport->hotel transport?
19:44:14 <abadger1999> I'd be fine doing it against staging.
19:44:22 <abadger1999> all te people that normally touch stg will be there I think.
19:44:32 <abadger1999> so it's okay if we make that environment unusable.
19:44:41 <smooge> nirik, as much as I would like ot believe we will be out of freeze that morning... I expect staging will be frozen
19:44:50 <nirik> well, threebean and lmacken won't be... but hopefully we will not disrupt them too much.
19:45:04 <nirik> smooge: staging is never frozen. ;)
19:45:04 <skvidal> nirik: it looks like it is only a few pickups
19:45:16 <nirik> but yeah, we could well slip again...
19:45:18 <abadger1999> dummying up systems always has the problem of "where do we get our fake data"?
19:45:21 <nirik> we will just see and handle it...
19:45:37 <abadger1999> since it's fas and we want to protect the production data a little better.
19:45:42 <skvidal> nirik: I'll be adding everyone's flight to my calendar and I'll see about making it all work :)
19:46:09 * skvidal wonders if nb went out of his way to find an inconvenient flight
19:46:14 <skvidal> 9:45am :(
19:46:16 <nirik> skvidal: cool. I think smooge will be there with car too?
19:46:20 <smooge> nirik, about travel.. I plan to drive up and be in RDU by noon on Monday to get my ID and such.
19:46:48 <nirik> oh, do we want to assign roomies? or just do it as people arrive/checkin?
19:46:49 <smooge> I can get there earlier if needed but it is a 4->5 hour drive from SC to RDU
19:47:03 <skvidal> herlo: when are you getting in?
19:47:28 <skvidal> nirik: I think I can just ferry people  from the airport w/o too much trouble starting around 2 or so on monday
19:47:38 <nirik> skvidal: ok, great!
19:47:45 <abadger1999> I get in late so I should either room with a heavy sleeper or someone who plans on staying up late despite jet lag :-)
19:47:46 <skvidal> eunice won't need the car on monday anyway, I don't think
19:47:49 <nirik> #info skvidal to pick up folks from airport.
19:48:07 <skvidal> I'll just email everyone directly
19:48:14 <skvidal> and confirm times and if they are not planning on renting a car
19:48:16 <abadger1999> Will we need more cars on thursday to ferry people back to the airport?
19:48:32 <nirik> abadger1999: I'm a deep sleeper, but I snore... (or so I have been told, I'm asleep at the time)
19:48:33 <skvidal> which hotel is it again?
19:49:19 <nirik> Marriott ?
19:49:22 <skvidal> the Raleigh MArriott
19:49:25 * skvidal calls them
19:49:32 <SmootherFrOgZ> what's time should we jump in for remote people?
19:50:09 <abadger1999> I was going to rent a car on thursday... Could go in with the first airport trip and rent one if we need more vehicles.. otherwise I'll go in with the last trip and just leave town from RDU.
19:50:10 <nirik> SmootherFrOgZ: I'd say we can hang out in #fedora-fad anytime starting monday ?
19:51:02 <nirik> (not this coming monday, the one after)
19:51:08 <SmootherFrOgZ> k.
19:51:14 <SmootherFrOgZ> yeah, i know
19:51:17 <skvidal> so thursday will need to be orchestrated a bit
19:51:29 <skvidal> and I think some people will need to find their own way back to the airport
19:51:31 * SmootherFrOgZ will have time to do to gym :)
19:51:34 <SmootherFrOgZ> go*
19:52:47 <nirik> ok, I'm sure we will figure it out. ;)
19:52:54 <smooge> I figured I would be ferrying people on Thursday
19:52:55 <nirik> anything else on fad planning?
19:53:47 <nirik> #topic Upcoming Tasks/Items
19:53:49 <relrod_school> abadger1999: poke me next week sometime and I'll help with organizing the FAS Plan
19:54:04 <nirik> infodump ahead:
19:54:06 <nirik> #info 2012-11-13 to 2012-11-27 F18 Beta Freeze
19:54:06 <nirik> #info 2012-11-20 FY2014 budget due
19:54:06 <nirik> #info 2012-11-27 F18 Beta release
19:54:06 <nirik> #info 2012-11-22 to 2012-11-23 Thanksgiving holiday
19:54:06 <nirik> #info 2012-11-26 to 2012-11-29 Security FAD
19:54:07 <nirik> #info 2012-12-18 to 2013-01-08 F18 Final Freeze
19:54:08 <abadger1999> relrod_school: Cool.  Will do
19:54:09 <nirik> #info 2012-11-30 end of 3nd quarter
19:54:11 <nirik> #info 2012-12-01 nag fi-apprentices
19:54:13 <nirik> #info 2012-12-24 to 2013-01-01 Red Hat Shutdown for holidays.
19:54:15 <nirik> #info 2013-01-08 F18 release.
19:54:17 <nirik> #info 2013-01-18 to 2013-01-20 FUDCON Lawrence
19:54:19 <nirik> anything else we need to schedule or note?
19:54:29 <nirik> oh yeah, I need to add the phx2 outage on dec 15th
19:55:06 <nirik> they are saying it should be 5min or so now of outage.
19:55:22 <nirik> so, not a big deal, but we should have a longer window and be ready in case it goes long
19:56:00 <nirik> also, need to schedule the collab move if I am going to do that...
19:56:19 <relrod_school> should just be a matter of shut up nagios and pull phx from dns for that block of time, right? Or are there other things to do to prepare for that?
19:56:55 <nirik> relrod_school: yeah for a short outage... longer term we might need to copy off wiki data or setup a fas db somewhere or other more difficult things.
19:57:04 <smooge> well I would just shut up nagios. Pulling phx out of dns is not easy since everything kind of talks to it
19:57:18 <nirik> well, we can pull proxy01 out so it gets no direct hits.
19:57:28 <relrod_school> smooge: yeah, just from the roundrobin rotation
19:57:50 <nirik> anyhow, if it's just 5min it won't be too big a deal.
19:57:50 * LoKoMurdoK here
19:57:57 <relrod_school> cool.
19:58:26 <nirik> ok, any more scheduling stuff?
19:58:35 <nirik> #topic Open Floor
19:58:36 <smooge> not fo rme
19:58:42 <nirik> anything for lovely open floor?
19:58:43 * pingou will be there in Lawrence
19:58:51 <abadger1999> pingou: yay!
19:58:58 <pingou> abadger1999: yup :)
19:58:58 <nirik> excellent. :)
19:59:16 <pingou> SmootherFrOgZ: as well, but you already know him :)
20:00:43 <nirik> after holidays we should make sure there's no infra stuff we need to do for fudcon... ie, help with streaming or whatever
20:00:55 <nirik> anyhow, thanks for coming everyone!
20:00:59 <nirik> #endmeeting