19:00:01 <nirik> #startmeeting Infrastructure (2013-01-03)
19:00:01 <zodbot> Meeting started Thu Jan  3 19:00:01 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:02 <nirik> #topic welcome to the new bactun
19:00:02 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:02 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:13 <skvidal> hi
19:00:13 <nirik> who's around for a nice infra meeting? :)
19:00:21 * skvidal is
19:00:21 <dgilmore> hola
19:00:26 * puiterwijk is
19:00:31 <rdesanno> here!
19:00:40 <_xtrashot_> hello
19:00:48 <fchiulli> fchiulli is listening in
19:01:06 * lmacken 
19:01:11 * threebean threebean threebean
19:01:41 <nirik> #topic New folks introductions and Apprentice tasks
19:01:41 <skvidal> threebean: shouldn't that just be beanbeanbean?
19:01:45 <nirik> ha.
19:01:48 <threebean> point taken
19:02:03 * relrod is here
19:02:03 <nirik> any new folks like to introduce themselves? or have questions/comments?
19:02:06 <pjones> skvidal: some irc networks have name character limits. sometimes it's nice to pick a shorter version.
19:02:12 <pjones> (for consistency)
19:02:24 <_xtrashot_> Hi all! I'm xtrashot ( or alex ), I come from a solaris background, I run my own linux server @home and I would like to lend a hand to fedora on the sysadmin side of the things.
19:02:25 <nirik> it could also be 'bean^3' ?
19:02:36 <nirik> _xtrashot_: welcome!
19:02:53 <_xtrashot_> thanks nirik!
19:03:01 <nirik> _xtrashot_: see me after the meeting over in #fedora-admin and we can get you added to our apprentice group and point you to things to look at.
19:03:04 <rdesanno> Hi. My name is Rob and I am a Linux admin here trying to lend a hand where possible too
19:03:10 <Adran> hi
19:03:13 <nirik> welcome rdesanno also. :)
19:04:02 <nirik> and Adran.
19:04:10 <Adran> Hi nirik!
19:04:32 <nirik> happy to help point you all to resources after the meeting... and do feel free to chime in with questions in the meeting here, or #fedora-admin at any point.
19:04:41 <_xtrashot_> thanks. will do
19:04:46 <rdesanno> Will do and thanks! Looking forward to helping
19:04:49 <Adran> Will do!
19:05:02 <nirik> great. Glad to see new folks. ;)
19:05:24 <nirik> mixing things up a bit... lets mention the freeze first:
19:05:26 <nirik> #topic Freeze reminder
19:05:37 <nirik> we are still in final freeze until f18 goes out the door.
19:05:47 * skvidal has noticed - it's REALLY cold in here!
19:05:53 <nirik> there's a go/no-go meeting in #fedora-meeting-1, but it looks like it will be no-go. ;)
19:06:09 <smooge> well I expected as such since there isn't an RC yet
19:06:10 <skvidal> alright! fedora 18 during fudcon?
19:06:16 <skvidal> or maybe after?
19:06:25 <nirik> just before probibly...
19:06:34 <smooge> I am voting for 2018
19:06:35 <nirik> or we might go for just a few days slip this time.
19:06:45 <smooge> we can then do once per year releases afterwords
19:06:51 <nirik> anyhow, the upshot is that we are frozen, don't make any bug changes without approvals, etc.
19:07:12 <nirik> I also have filed out release tickets...
19:07:29 <nirik> most of which need something to stage before we can do anything. :)
19:07:53 <nirik> #topic Applications status / discussion
19:08:04 <nirik> ok, any applications status or things to note?
19:08:17 <nirik> given the holidays and freeze, I suspect not much...
19:08:29 <nirik> do we have anything we want to look at setting up to demo at fudcon?
19:08:34 <skvidal> yes
19:08:46 <skvidal> I have 3 talks in mind for fudcon
19:09:01 <skvidal> 1. private cloud
19:09:02 <skvidal> 2. ansible
19:09:04 <skvidal> 3. copr
19:09:43 <nirik> #topic Fudcon
19:09:50 <nirik> might as well go to the fudcon topic. ;)
19:09:53 <threebean> cool :)
19:09:55 <skvidal> <shrug>
19:10:07 <nirik> https://fedoraproject.org/wiki/Fedora_infrastructure_tasks_2013
19:10:16 <nirik> is an ideas container/setup for our fudcon plans.
19:10:26 <nirik> I added some more things there too.
19:10:39 <nirik> how do we want to try and organize things?
19:10:53 <skvidal> I suspect to meet during one of the hackfest times?
19:10:57 <skvidal> and in the evenings
19:11:19 <nirik> I'm fine with a generic Infrastructure hackfest and try and cover a bunch of things that need high BW, but might be nice to have a checklist of things we should try and address.
19:11:54 <skvidal> I see the bits you've added
19:12:06 <skvidal> some of those have been on the desired-list for quite a while
19:12:10 <nirik> yeah.
19:12:27 <nirik> never enough hours in the day.
19:12:53 <skvidal> indeed
19:13:00 <skvidal> and some of them are not overly difficult
19:13:08 <skvidal> but they involve a bunch of people and moving parts
19:13:16 <skvidal> so the inertia is high (fedorahosted as an example)
19:13:23 <skvidal> something maybe for fudcon
19:13:23 <nirik> yep.
19:13:26 <skvidal> maybe for before
19:13:39 <skvidal> is to go through all the [cr]apps we're supporting
19:13:43 <skvidal> and get rid of those unused
19:13:56 <skvidal> we killed off the old fedora community before the break
19:13:58 <skvidal> and that was great
19:14:05 <dgilmore> id like to talk about replacing the ca infrastructure
19:14:05 <skvidal> but I have an item on my list to ask about insight now
19:14:22 <skvidal> dgilmore: what ca infrastructure? it's just easy-pki :)
19:14:24 <nirik> so, the general fudcon setup is barcamp first day... then hackfests next two
19:14:35 <dgilmore> skvidal: what we use for koji and users certs
19:14:47 <skvidal> dgilmore: ah, gotcha
19:14:51 <dgilmore> skvidal: its some make files
19:14:54 <skvidal> nod
19:14:58 <dgilmore> and a python script
19:15:37 <nirik> dgilmore: what would you like to change it to?
19:15:52 <dgilmore> nirik: im thinking dogtag
19:16:16 <nirik> that seems like a lot of overhead...
19:16:16 <skvidal> dgilmore: what's the selling point?
19:16:26 <nirik> though I admit I haven't looked at it recently
19:16:54 <dgilmore> skvidal: that we could have two nodes, and fas can talk to it over the network, so all fas boxes could handle cert requests,
19:17:04 <dgilmore> eliminate spof
19:17:37 <dgilmore> we would also not have a static crl and have real time checking on revoked certs
19:17:49 <nirik> could nfs/gluster mount also fix that?
19:18:11 <dgilmore> nirik: not without major locking checking
19:18:25 <dgilmore> and it still doesnt give us real time checking
19:18:39 <nirik> well, we can discuss it. dogtag has always seemed very heavy to me...
19:18:41 <dgilmore> not that its a huge deal to fetch the crl every hour
19:18:55 <dgilmore> nirik: maybe its a bit too heavy for us
19:19:23 <nirik> we can see what makes sense...
19:19:45 <dgilmore> nirik: skvidal: id like a config management talk also
19:19:54 <dgilmore> so we can go over things we have changed
19:19:58 <skvidal> dgilmore: that's the ansible talk
19:20:06 <nirik> yeah...
19:20:07 <dgilmore> id like to get arm setup exactly the same as primary
19:20:20 <skvidal> and I think it's less about what's changeD and more about what is still changing
19:20:29 <skvidal> but the point is still the same
19:20:31 <dgilmore> skvidal: sure
19:20:52 * nirik wonders if we should try and have a generic '2013 planning for infra' barcamp session... but that sounds boring. Perhaps we just try and do that at the start of hackfests.
19:21:41 <nirik> anyone else looking at doing barcamp sessions?
19:21:59 <threebean> Hmm.. I'd like to do a fedmsg talk; how should I aim it?  More towards what's been done or more towards where things should go?
19:22:01 <lmacken> fedmsg session!
19:22:12 <lmacken> both
19:22:15 <nirik> yeah...
19:22:27 <nirik> go over whats been done and talk some about upcoming?
19:22:33 <threebean> cool, sure.
19:23:03 <nirik> lmacken: any chance we could stand up a mockup of bodhi 2.0 for people to look at? or still too early?
19:23:54 <lmacken> nirik:  might be too early for that, but hopefully a bodhi2 hackfest will happen
19:24:09 <nirik> ok.
19:25:03 * nirik might pitch a 'make rawhide nicer to use' session, but thats not really much infra related. ;)
19:25:28 <dgilmore> nirik: maybe i could run a using enterprise arm hardware session
19:25:35 <nirik> I'd also like us to see if we can decide a fad topic... kock something out in 2013 thats difficult to get done remote.
19:25:37 <dgilmore> not sure what the iterest would be
19:26:18 <nirik> dgilmore: yeah. I bet there will be a few arm sessions... it would be cool if we could show off some server arm stuff (if even only remote)
19:26:30 <bconoboy> nirik: we expect to have an arm server on site
19:27:00 <nirik> bconoboy: sweet. ;)
19:27:09 <bconoboy> arm barcamp?
19:27:18 <skvidal> barmcamp?
19:27:20 <skvidal> no sorry
19:27:23 <nirik> :)
19:27:43 <nirik> ok, so, any folks doing barcamp sessions, please do go add them to the fudcon page now...
19:27:49 <dgilmore> nirik: hopefully we can show some stuff off
19:27:55 * bconoboy does so
19:28:21 <nirik> I will add a infra hackfest for sat/sun and we can use the first part of sat to try and plan out what we want to try and get done and pull in anyone we need for those things.
19:28:31 <nirik> unless we can plan that in advance?
19:28:59 * nirik thinks it might be better higher BW
19:29:11 <skvidal> nod
19:29:26 <nirik> I'll try and organize a bit more if I can...
19:29:31 <skvidal> theoreticaly we should have everyone nearby-ish to put a bullet in insight*
19:29:46 <skvidal> do we have anything else on our 'should die' list?
19:30:05 <nirik> yeah, do we want to mail people now about that? I guess lists and asrob and stickster?
19:30:10 * skvidal hmms - maybe I should check last login on all systems by a user other than root
19:31:37 <nirik> I don't have anything on that list off hand...
19:32:02 <nirik> any other fudcon business?
19:32:30 <nirik> #topic Upcoming items
19:32:34 <nirik> #topic Upcoming Tasks/Items
19:32:34 <nirik> #info 2012-12-11 to 2013-01-08 F18 Final Freeze
19:32:34 <nirik> #info 2012-01-05 PHX2 outage.
19:32:34 <nirik> #info 2013-01-08 F18 release.
19:32:35 <nirik> #info 2013-01-18 to 2013-01-20 FUDCON Lawrence
19:32:36 <nirik> #info 2013-01-29 f19 feature submission deadline.
19:32:46 <nirik> we have a phx2 network outage this weekend.
19:32:53 <nirik> it's supposed to be very short, but we will see.
19:33:02 <smooge> I will be around for it
19:33:10 * relrod has a note in his calendar to nuke proxy01 from dns a few hours ahead of time.
19:33:14 * nirik will be too.
19:33:23 <nirik> #topic Open Floor
19:33:31 <nirik> any general topics or questions?
19:34:03 <skvidal> so
19:34:07 <skvidal> about our web apps/etc
19:34:27 <skvidal> I was talking with nirik yesterday about thinking of ways of making our apps easier to debug when things go wrong
19:34:49 <skvidal> as it is right now we end up trying to figure out if an app is misbehaving, being hit too much from the outside or if it is being impacted by SOMETHING ELSE
19:35:04 <skvidal> and it means a lot of our fixes are somewhat imprecise (restart httpd)
19:35:25 <skvidal> I was debating some ways of dealing with that - it's a subject I suggested to discuss at fudcon
19:35:27 <nirik> I think moving to a model of 'one app per virtual' might be good for that.
19:35:35 <smooge> I like that one
19:35:37 <skvidal> but I wanted to toss it out here to see if anyone wanted to scream murder
19:35:41 <nirik> but containers, etc might also work, not sure.
19:36:12 <smooge> I remember we wanted to look at that back a couple of fudcons ago but ran into "how will we manage a wave of them"
19:36:25 <skvidal> I think we know how to manage them
19:36:27 <smooge> and now we have the answer.. nirikseth cloud
19:36:37 <threebean> not murder.. but.. the same problem might be solved just as well by fixing our logging setup instead.
19:36:37 <smooge> and ansible
19:36:45 <skvidal> right.... so..
19:36:53 <skvidal> a consideration, definitely.
19:37:50 <smooge> threebean, well one of the issues we run into is that we have 8 different apps on one box.. and it has turned into that A impacts B but B complains out because it is a delicate flower of an app. then C and D get into a fight and E goes on marxist strike.
19:38:11 <nirik> well, A OOMs apache, and BCDE all also die.
19:38:33 * threebean nods
19:38:49 <threebean> More separation is better, then.  :)
19:39:13 <nirik> part of this issue is the roles our smaller sites play.
19:39:17 <smooge> then I figure logging will help on the A needs B but B doesn't want to play with A without C
19:39:52 <nirik> I think on the new apps we have deployed to their own app servers it's worked out ok... packages, ask, paste...
19:40:02 <nirik> but some of the 'core' ones might be different
19:41:11 <skvidal> anyway - I just wanted to float it
19:41:21 <skvidal> it also might have the advantage of letting us get to know the apps a bit better
19:41:24 <relrod> well it's the same thing with the db servers, many app dbs per server -- we started solving that by moving the fas one separate of the others, I guess
19:41:36 <skvidal> since we'd spend less time learning how the apps interact w/each other
19:41:39 <skvidal> and more how they act on their own
19:41:57 <skvidal> relrod: agreed  - but same thing -we've moved some apps with special db needs to their own isolation
19:42:17 <skvidal> relrod: fas was just moved to keep from having a looping SPOF in the shape of the fas and db servers
19:42:33 <nirik> right, and can avoid stack issues like TG1 / TG2 / pyramid in the same app server...
19:42:46 <nirik> anyhow, yeah, I think it's a good plan to discuss more at fudcon
19:43:01 <relrod> agreed
19:43:09 <skvidal> okay cool
19:43:10 <skvidal> thanks!
19:44:23 <skvidal> that's all I had
19:44:30 <skvidal> other than to say
19:44:31 <skvidal> die insight
19:44:32 <skvidal> die
19:44:36 <skvidal> unless it is being used
19:44:38 <skvidal> then that's great
19:45:54 <nirik> ha. ;)
19:46:00 <nirik> it's really not
19:46:07 <nirik> anyhow...
19:46:12 <nirik> anything else general of note?
19:46:28 <nirik> or shall we close out and go back to our frozen existance?
19:46:29 <smooge> not for me
19:46:49 <threebean> yeah, good here.  I'm pumped for fudcon
19:47:20 <relrod> threebean: yeah, should be fun :)
19:47:50 <nirik> yeah, fudcons are always great.
19:48:10 <Adran> I hear they are fun. Never been my self though. :\
19:48:10 * skvidal wonders if spot is going to have a copy of cards against humanity for boardgames
19:48:25 * nirik needs to come up with an agenda tho of what he wants to get done. otherwise it's very easy to get sidetracked into doing other things. ;)
19:48:26 <spot> skvidal: yep. plus expansion pack 1.
19:48:28 <skvidal> I played it over the break and.... well let's just say I'm well suited to the game
19:48:31 <skvidal> spot: ooooo
19:48:39 <nirik> yeah, it's an awesome game. :)
19:48:45 <relrod> :D
19:48:50 <skvidal> it's a horrible game for terrible people
19:48:56 <lmacken> whoa, nice.
19:48:56 <skvidal> like mytself
19:49:05 <nirik> heh. :)
19:49:11 <nirik> ok, thanks everyone...
19:49:15 <nirik> #endmeeting