18:00:43 <threebean> #startmeeting Infrastructure
18:00:43 <zodbot> Meeting started Thu Jul 24 18:00:43 2014 UTC.  The chair is threebean. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:50 <threebean> #meetingname infrastructure
18:00:50 <zodbot> The meeting name has been set to 'infrastructure'
18:00:57 * lanica is here for the infrastructure meeting.
18:00:59 <threebean> #topic greetings
18:01:07 <threebean> Hi lanica
18:01:14 <threebean> anyone else around for the meeting today?
18:01:14 * michel_slm is here
18:01:14 <christianh814> o/
18:01:18 * lmacken 
18:01:31 * nirik is kinda sort of around, but running back and forth to racks, so distracted.
18:01:37 * puiterwijk is here.
18:01:38 * pingou 
18:01:50 <puiterwijk> threebean: just curious, you didn't set the chairs. any reason for that?
18:02:01 <threebean> oh, lazy.
18:02:11 <puiterwijk> hah, okay
18:02:14 * bochecha is here
18:02:17 <threebean> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:02:17 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:02:29 <ootbro> I'm here
18:03:03 <threebean> cool - welcome, everyone.
18:03:12 <threebean> #topic Any new people?  Apprentice tasks!
18:03:21 <threebean> Anybody new to the meeting today?
18:03:28 <threebean> if so, could you introduce yourself?
18:03:47 <christianh814> Well I'm new-ish...this is my second meeting.
18:03:55 * msuchy-hm is here
18:03:57 * relrod is kinda sort of around too
18:04:32 <threebean> christianh814: welcome back :)
18:04:54 <threebean> christianh814: do you have any questions based on your first week hanging around?
18:04:59 <threebean> anything we can try to answer or help with?
18:06:01 <christianh814> Just been looking through the tickets (easy fix) and working on learning ansible as I got (came from a chef background). No questions yet. I was able to get ssh-ed in
18:06:16 <christianh814> to lockbox and such
18:06:28 <threebean> cool.  good stuff.
18:06:42 <ootbro> apprentice news ....   after getting a layout preference from nirik last night, started to build the final version of the fp.o network diagram  (very high-level picture) ....  discovering OpenOffice draw has some quirks..... :p
18:07:40 <threebean> ootbro: cool.  i'm looking forwards to it :)
18:08:21 <ootbro> I've almost got it ready to post to a temp spot (my own web server) and will send the URL to the infra list ...   then wait for the peanut gallery
18:08:38 <threebean> ha!  great.  previews are good :)
18:08:45 <michel_slm> ootbro: you could use fedorapeople.org too
18:09:17 <ootbro> second picture to create is roughly how the proxies work  (from info I got from nirik)
18:09:19 <lanica> ootbro: Sounds very cool!  Can we see a preview?
18:10:31 <ootbro> lanica: I'm battling Draw now, trying to get a quality .png out of it....  will sent the URL to the infra list as soon as I have the .png posted for review / preview
18:11:16 <ootbro> michel_slm: I'll take a look at fedorapeople.org in a little bit, see how easliy I can get the .png posted
18:11:29 <lanica> cool, thx!
18:11:40 <threebean> #info progress made on the infrastructure map.  a preview will be available in the coming weeks.
18:11:46 <threebean> thanks, ootbro
18:11:49 <pingou> ootbro: if you are cla+1 it's as easy as scp ;-)
18:11:52 * webpigeon sneaks in
18:12:03 <threebean> unless there's anything else in this sections, we can move on...?
18:12:33 <threebean> #topic Applications status / discussion
18:12:54 <threebean> applications.  development, maintenance, deployments.. any news?
18:13:09 <pingou> 2 new pkgdb2, 1 new fedocal (fixing login)
18:13:12 <puiterwijk> an unintended FedOAuth upgrade in prod broke auth for a few hours today, but I downgraded it and am working on a new release now
18:13:23 <msuchy-hm> I deployed new version of copr
18:13:32 <pingou> and with abadger1999 we're working on the group maintainership in pkgdb2
18:14:20 * Xirrin is late but present.
18:14:34 <msuchy-hm> I am having lot of problems with OpenStack - 70 failures to spin up VM in just an hour.
18:14:45 <threebean> oof :/
18:14:52 <webpigeon> ouch
18:15:11 <pingou> msuchy-hm: something changed ?
18:15:21 <msuchy-hm> installin new open stack is my priority now
18:15:26 <threebean> this is the new, secondary cloud setup?
18:15:34 <threebean> parallel to our existing private cloud?
18:16:18 <msuchy-hm> pingou: nope, it have better days and worse, today it was one of very bad one, but it behaved similary in past, but it seem that every week it became little bit worse
18:16:49 <msuchy-hm> new cloud, and we will slowly migrate one by one
18:17:56 <threebean> #info new openstack instance is coming together, but still lots of problems
18:17:57 <threebean> https://lists.fedoraproject.org/pipermail/infrastructure/2014-July/014621.html
18:18:04 <threebean> #info 2 new pkgdb2 releases this week
18:18:08 <threebean> #info 1 new fedocal release this week
18:18:24 <threebean> #info accidental upgrades of both fedocal and fedmsg broke some production services, but were downgraded and restored this morning.
18:18:43 <pingou> threebean: fedocal?
18:18:51 <threebean> oh, whoops.
18:19:06 <threebean> fedoauth.
18:19:08 <pingou> #undo
18:19:08 <zodbot> Removing item from minutes: INFO by threebean at 18:18:24 : accidental upgrades of both fedocal and fedmsg broke some production services, but were downgraded and restored this morning.
18:19:14 <threebean> thanks ;)
18:19:22 <threebean> #info accidental upgrades of both fedoauth and fedmsg broke some production services, but were downgraded and restored this morning.
18:19:27 <pingou> :)
18:19:31 <nirik> so are they good to re-upgrade? or should they stay at the old versions?
18:19:53 <threebean> nirik: not yet.  the hopefully last batch of fedmsg rpms are cooking now
18:20:32 <nirik> ok. will that be soon then? or just hold off upgrading them today?
18:20:46 <threebean> yeah, hopefully right after the meeting.  I'll let ya know.
18:21:16 <threebean> any other items or news on the apps side of things?
18:21:57 <threebean> #topic Sysadmin status / discussion
18:22:21 <threebean> we're in the middle of an outage right now.. nirik, relrod, and smooge are working on it.  so there might not be much to discuss here.
18:23:11 <threebean> heh, anyone want to bring anything up?
18:23:38 <bochecha> not sure it counts here, but I'm waiting for review of the migration of distgit to ansible, nirik said he'd have a look later, but feedback from anybody else is also welcome: https://fedorahosted.org/fedora-infrastructure/ticket/4452
18:25:42 <threebean> cool, hopefully someone can take a look at it soon.
18:26:01 <threebean> we got a handful more migrations done in the last week.. so, slowly making progress.
18:26:06 <relrod> oh I have two things
18:26:18 <threebean> I don't have access to the OS scorecard stats, so, we'll have to wait until next week for those.
18:26:20 <threebean> relrod: yeah?
18:26:32 * nirik will try and work on that next week, yeah.
18:27:13 <relrod> 1) Nagios is moved into ansible now as of last week some time, and uses mod_auth_openid and rhel7 - we're still working out some glitches with mod_auth_openid but it seems to be working in general, please report any issues.
18:27:59 <relrod> 2) During this outage window I am moving fedorapeople web stuff (except planet) to SSL, so if you have anything that depends on non-ssl fedorapeople sites, make sure they handle redirects to ssl :)
18:28:09 <relrod> EOF
18:28:15 <threebean> cool, thanks relrod.
18:28:20 <pingou> relrod: nice :)
18:28:25 <pingou> valid cert I assume?
18:28:30 <relrod> yeah we have one now
18:28:59 <pingou> cool
18:30:09 <threebean> all right, moving along
18:30:11 <threebean> #topic Upcoming Tasks/Items
18:30:17 <threebean> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:30:34 <threebean> anything coming up to plan or schedule?
18:30:46 <msuchy-hm> I am on vacation next week
18:30:54 <threebean> nirik and smooge should head back from phx2 tomorrow.
18:31:02 <msuchy-hm> asamalik will take care of copr in that time
18:31:47 <threebean> regarding taskotron:  tflink is going to be on pto from tomorrow until flock, mkrizek will be the best person to ask questions
18:31:49 <threebean> of. kparal and roshi will know how to get ahold of me if anything really urgent comes up
18:32:08 * bwood09 reeeeallly late
18:32:47 <threebean> #topic Open Floor
18:32:59 <ootbro> one quick follow-up.....  I just won the battle with Draw (getting a good .png) and got the preview network pic posted to a temporary spot -- http://www.jt-sw.com/docs/index.nsf/docs/fpo-diag  ....  also included a link on the page for sending comments back to me.
18:33:38 <ootbro> I'll got look at fedorapeople.org as an alternate temp spot in a little bit
18:33:55 <ootbro> "go"
18:34:03 <threebean> cool :)
18:34:58 <dne0> I would like to update you all on the status of waartaa(GSoC project).
18:35:14 <threebean> dne0: yes, go ahead.
18:35:29 <dne0> Currently, I am working on admin dashboard(my last task in GSoC). First thing on the subtask list is create IRC nick status page where admin can see all active nicks using waartaa. Admin can connect/disconnect any nick any time.
18:35:45 <dne0> I have added that functionality with a decent interface
18:35:59 <dne0> I am hoping to finish other sub task by this weekend :)
18:36:42 <pingou> nice
18:36:50 <pingou> and nice time-management :)
18:37:02 <dne0> thanks pingou :)
18:37:43 <threebean> any other open floor items before I close out the meeting?
18:37:56 <christianh814> do we have a standard on how to build playbooks? looking to tackle https://fedorahosted.org/fedora-infrastructure/ticket/4290 and just want to know if there is a directory structure for the playbooks/variables/etc
18:39:20 <threebean> christianh814: afaik, we don't have a written standard for how to build playbooks
18:39:28 <threebean> but if you look around in the ansible git repo, you'll see the playbooks directory
18:39:52 <christianh814> Okay I'll check it out and make a "best effort" when building it.
18:40:01 <threebean> most things in the playbooks/groups/ dir have a common pattern.  they each take a group of hosts from non-existant to functional.
18:40:27 <threebean> the things in the topmost playbooks/ dir, though, are each kind of different.
18:42:10 <threebean> ok, if nothing else.. thanks for coming everyone.
18:42:16 <threebean> #endmeeting