20:00:15 <smooge> #startmeeting Infrastructure
20:00:15 <zodbot> Meeting started Thu Feb  3 20:00:15 2011 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:15 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:22 <smooge> #meetingname Infrastructure
20:00:22 <zodbot> The meeting name has been set to 'infrastructure'
20:00:35 <smooge> #chair skvidal ricky smooge
20:00:35 <zodbot> Current chairs: ricky skvidal smooge
20:00:42 * CodeBlock is here
20:00:43 <smooge> #topic roll call
20:00:47 * nirik is around.
20:00:50 * CodeBlock is still here.
20:00:56 * skvidal is here
20:01:52 * sijis is here
20:02:00 <skvidal> #topic fudcon discussions and decisions
20:02:13 * mdomsch 
20:02:16 <skvidal> So - we had a lot of whining and complaining at fudcon
20:02:21 <skvidal> and we came up with some decisions to limit that
20:02:39 <skvidal> the whining and complaining was mostly b/c we're all a bit over-served on things to do
20:02:55 <skvidal> and we are not in a good place as far as maintenance and sustainability goes
20:03:06 <skvidal> we came up with a list of all the horribly unfinished/ugly/icky stuff we need done
20:03:08 <skvidal> https://fedoraproject.org/wiki/Infrastructure_Cleanup_Tasks_2011
20:03:27 <skvidal> and we are also pursuing a direction which is new(ish) for fedora infrastructure.
20:03:57 * ricky 
20:04:06 <skvidal> the gist I took away is this: we are growingly comfortable with outsourcing some of our systems/services to other people provided they are 1. not evil and 2. more open or completely open
20:04:29 <skvidal> that should be and/or not just 'and'
20:04:42 * nirik notes perhaps we could tie this in with the fedora vision.
20:04:55 <skvidal> okay, lemme see if I can spin this well
20:05:07 <skvidal> 1. fedora infrastructure does not have to bootstrap itself.
20:05:23 <skvidal> 2. fedora infrastructure is here to help fedora achieve its goals
20:06:11 * nirik thinks outsourcing is great as long as the places outsourced to are aligned with our vision...
20:06:17 <skvidal> agreed
20:06:21 <skvidal> to a point
20:06:23 <skvidal> hear me out
20:06:31 <skvidal> we are using servers at serverbeach
20:06:35 <skvidal> and at internetx01
20:06:36 <nirik> http://fedoraproject.org/wiki/Vision_statement (just for reference)
20:06:51 <skvidal> and in many cases the software we use to access those servers' consoles is not open source at all
20:07:12 <skvidal> we're just using the systems, we're not buying into the perspective of the folks who own the systems
20:07:40 <skvidal> so for example
20:07:56 <skvidal> rackspace's backend is not completely open
20:08:06 <skvidal> they are pursuing MORE openness but it's not 100%
20:08:11 <skvidal> this does not bother me at all
20:08:25 <skvidal> no more than using the netapp in phx2 bothers me
20:08:29 <skvidal> and for the same reasons
20:08:46 <skvidal> wordpress.com is an example in the other direction
20:08:53 <skvidal> there's absolutely nothing to be bothered about there AT ALL
20:09:14 <skvidal> if we bought services from them to host our blogs for us
20:09:35 <skvidal> we're actually helping pay for the development and maintenance of a widely used piece of open source software
20:10:04 <skvidal> anyone not like the general idea?
20:10:07 <ricky> Win :-)
20:10:25 <CodeBlock> :)
20:10:45 <mdomsch> so far so good
20:10:46 * skvidal likes the echo-y silence
20:10:51 * nirik has no problems at all with wordpress.com.
20:11:02 <skvidal> nirik: any problems with rackspace or even ec2?
20:11:06 <smooge> what would you have problems with?
20:11:17 <mdomsch> azure
20:11:25 * skvidal googles
20:11:29 <smooge> gogols
20:11:42 <skvidal> mdomsch: heh
20:11:47 <skvidal> mdomsch: fair enough.
20:12:25 <skvidal> so one thing that has kept us hosting our own everything
20:12:28 <skvidal> has been a FI policy which is
20:12:34 <ricky> Would that be for image reasons, technical reasons (which is what I see it as)
20:12:35 <skvidal> "we won't use any software that's not in fedora"
20:12:36 <nirik> not off hand. I think I would have problems with a place that didn't allow our users to control or move their content, etc.
20:12:45 <ricky> Like it'd be good to have a set of criteria we look at when considering hosted services
20:12:59 <ricky> I think image and technical reasons are valid to be on that list
20:13:10 <skvidal> ricky: I have no problem with that - maybe we can refine that list on the mailing list?
20:13:17 <ricky> Sounds good.
20:13:36 <skvidal> ricky: if you feel like starting that discussion I'd appreciate it
20:13:46 <skvidal> so - here's a thought
20:13:59 <skvidal> let's say tomorrow we get an RFR for a new webapp
20:14:06 * sijis has no problem with moving some services to other suited locations
20:14:07 <skvidal> it's free software
20:14:20 <skvidal> it's using ruby or php or haskell or what-not
20:14:26 <skvidal> something that we don't have a lot of on-hand experience with
20:14:32 <skvidal> it's NOT packaged in fedora
20:14:41 <skvidal> but it will help our fedora contributors get their tasks done
20:14:55 <skvidal> and we can spin up an instant-host of it running on centos at rackspace.
20:15:03 <skvidal> (just as an example)
20:15:13 <skvidal> should we not do this?
20:15:37 <skvidal> centos is clearly in the family of downstream linux distros of fedora
20:15:44 <skvidal> the app is freesoftware
20:16:00 <skvidal> it won't be running on fedora but most/all of our apps now don't run on fedora in production, either.
20:16:08 <skvidal> does the pkg need to be in fedora at all?
20:16:28 <skvidal> that's not a rhetorical question
20:16:49 <skvidal> anyone object to the above?
20:16:59 <CodeBlock> nope
20:17:01 * nirik ponders.
20:17:04 <ricky> I think having it packaged served two purpose - making sure it was free, and making sure it was maintained, got security updates, etc. when we ran it
20:17:18 <ricky> I agree that the second purpose becomes less important if we move to hosted stuff.
20:17:36 <nirik> yeah, so would updates there be done by rackspace? or ?
20:17:39 <skvidal> ricky: well it's not LESS important it just may not be as much our problem - we still don't want our services being cracked
20:17:48 <skvidal> nirik: that's a good question to ask - I don't know
20:18:05 <nirik> also, it means we are pushing upstreams less about getting them packaged so any centos/rhel/fedora people could use them...
20:18:20 <nirik> but I don't know how much leverage we really have or how much good we have done for that.
20:18:27 <ricky> the rackspace thing - is that spinning up a fully managed install of an app, or is it spinning up a machine we have root on to run the app?
20:18:38 <skvidal> ricky: could be either, potentially.
20:18:57 <skvidal> ricky: there are a number of fully-ready-to-go imgs out there in both rackspace and ec2 space
20:19:05 <nirik> a example app here might be diaspora... to run a fedora pod. It's ruby on rails tho, and not packaged (at least that I know of yet). :)
20:19:41 <skvidal> that's a good example, sure.
20:20:05 <skvidal> the reason I'm asking about all this is simple
20:20:09 <skvidal> if we pursue this path
20:20:23 <skvidal> it would be best if we don't piss off/lose anyone in the process
20:20:55 <skvidal> which is why I want to hear from folks who don't like this or feel it is a violation of something we believe in
20:21:10 <skvidal> I don't feel it is but I admit my pragmatism has been growing over the years
20:21:11 <nirik> I think with many things in life it's tradeoffs... so we may have to weigh those on a case by case basis.
20:21:20 <skvidal> agreed.
20:22:10 <skvidal> so let's say we move in that general direction.
20:22:13 <smooge> the main issues would be "our charter" so to speak
20:22:23 <skvidal> smooge: ?
20:22:28 <skvidal> "our charter"?
20:22:45 <smooge> well we are here to run fedora's stuff. but are there other things that have been assumed by others
20:23:03 <smooge> as in "infrastructure" is a place to grow new sysadmins who know RH/Fedora stuff
20:23:49 <skvidal> so that's a good point
20:23:56 <smooge> or something else
20:23:57 <skvidal> the services we have been talking about so far
20:24:09 <skvidal> are not ones that help us grow sysadmins using rhel/fedora
20:24:18 <skvidal> they are services which, ultimately a sysadmin might SUPPORT
20:24:21 <skvidal> but not develop
20:24:26 <skvidal> and we're developing and maintaining them
20:25:35 <skvidal> when I think of a sysadmin task I think of setting up and maintaining the authn/authz infrastructure.
20:25:44 <skvidal> but I don't think of WRITING the entire thing, for example
20:25:50 <smooge> and a lot of our stuff "works" when we have full-time people versus 1hour/week volunteers
20:27:23 <smooge> so we end up with feeling burnt out and/or not feeling we can give anyone anyting because its too much time unless they do it
20:27:46 <ricky> smooge: I definitely feel that second part some times ^
20:27:56 <skvidal> right
20:28:03 <skvidal> which turns all of us into 'captain no'
20:28:08 <ke4qqq> skvidal: I think there'd be no complaint once rackspace actually starts using openstack for their cloud stuff. (/me knows you have moved on)
20:28:13 <skvidal> b/c that's what we find ourselves saying all the time
20:28:17 <abadger1999> Re: the packaging portion... having it packaged helps other people set up a similar structure to test outside of fedora infra.
20:28:43 <abadger1999> Not demanding packaging would certainly be easier... otoh, some thing that turn up in packaging are actually problems that should be resolved.
20:28:59 <smooge> so here are the issues off the top of my head:
20:29:05 <skvidal> abadger1999: that's my question - we block on packaging a lot from what I've seen
20:29:10 <abadger1999> <nod>
20:29:19 <smooge> 1) configuration management
20:29:23 <smooge> 2) getting data out later
20:29:27 <smooge> 3) authn
20:29:30 <smooge> 4) authz
20:29:32 <ke4qqq> abadger1999: but how would you effectively manage many non packaged apps with puppet
20:29:33 <sijis> i always that that was so 'everyone' could benefit from using that software/tool/etc
20:29:35 <abadger1999> I'm wondering if not packaging simply shifts the burden around though..
20:29:40 <ricky> Hm, that's a very good point about packaging.  Can packaging still get done if we don't block on it?
20:29:48 <smooge> 5) backups
20:30:01 <ricky> Because having infra get more useful packages out there is a nice side effect
20:30:20 <ricky> Although it seems that packages can languish once we stop using them (remember all the zikula orphaning?)
20:30:55 <skvidal> ricky: and we can get bogged in packaging: plone/zope :(
20:31:00 <ricky> With the hosted services I'm thinking of, the only config we touch is stored in a db somewhere.
20:31:16 <ricky> Which is the same problem as backups
20:31:27 <sijis> i think a packager packages because he/she was interested in it. once that dies, so will the package maintennace (likely)
20:31:38 * ricky shudders at the mention.  abadger1999's point of developers needing to test *definitely* applied in the plone/zope case.
20:33:09 <skvidal> so
20:33:38 <skvidal> let's take an example case of wordpress.com
20:33:44 <skvidal> and run it through smooge's set of 5 items
20:34:03 <smooge> there may be others those were just the ones I could think of
20:34:09 <skvidal> 1. config mgmt - it's a config-via-website thing aiui and I doubt it makes much sense for that to be in puppet
20:34:10 <ricky> configuration management and backups are both available through wordpress's export features, I think.
20:34:29 <skvidal> 2. data out later is exporting xml of the content which wordpress does promise
20:34:43 <skvidal> 3. authn - good question - probably not fas unless we do something magical in openid
20:34:49 <ricky> authn/authz we currently lose out on.
20:34:51 <skvidal> 4. authz - same as above
20:35:01 <skvidal> 5. backups - that doesn't feel difficult to me
20:36:15 <sijis> for wordpress.com it would be on the blogger to backup, no?
20:36:30 <skvidal> quite possibly, yes - but for the central service
20:36:34 <skvidal> it would make sense for us
20:37:23 <skvidal> to do it, I mean
20:37:45 <smooge> for me it is a no-brainer to move it out to them.. if the website team does not have problems and can get their jobs done with it
20:38:09 * sijis has no problem to move it
20:38:59 <abadger1999> wordpress.com makes sense to me.
20:39:23 <abadger1999> Managing of the app moves out to someone else.
20:40:06 <sijis> not noly that.. but maintenance too
20:40:33 <smooge> ok so blogs makes sense.
20:40:33 <sijis> i'm sure they are releasing bug fixed faster than we currently are.
20:41:02 <abadger1999> Which  makes it distinct from the case of "we spin up an instance on rackspace and let a contributor manage our blogs there using software that's free but not packaged in Fedora"
20:41:49 <smooge> correct.
20:42:36 <ricky> One other thing that migh be affected
20:42:54 <ricky> In the past, we never stopped contributors who wanted a service and were willing to put in the time to maintain it on our infrastructure
20:43:05 <ricky> But now there's the issue of "who pays for the service" that's a lot easier to say "no" to
20:44:05 <ricky> (Of course, I can't think of that many services we have now that started that way, but there definitely are some)
20:44:34 <skvidal> so that comes to another discussion we had
20:44:44 <skvidal> publictest instance restrictions
20:44:50 <skvidal> 1. limit time available
20:44:54 <skvidal> 2. limit who can login to them
20:44:54 <ricky> smolt, our review stats, freemedia stuff (which they're already not too happy about how full-featured we've allowed their setup to get, apparently)
20:45:05 <mdomsch> wrt wordpress.com - why is it necessary for Fedora to provide a blog platform for our members at all?  Why not just have them each sign up for WP accounts (or blogger.com, or wheverver) themselves?
20:45:11 <mdomsch> why are we in the middle?
20:45:18 <ke4qqq> mdomsch: +1
20:45:33 <ricky> For publictest instances, the money is probably small - what about for stuff that we commit to run for a long time?
20:45:57 <ricky> mdomsch: Agreed.  Wordpress was brought up at the birth of insight - there was probably some thought back then that it could be useful for that)
20:46:11 <ricky> Then it grew into a general blog service that we got stuck maintaining
20:46:15 <skvidal> mdomsch: I don't disagree - I think there was some drive for a single css and style
20:46:23 <mdomsch> ehh
20:46:48 <ricky> Which is why I'm pushing for us not to run wordpress while it's still small and very lightly used
20:47:09 <skvidal> ricky: can weidentify the set of wordpress users on our system now?
20:47:19 <ricky> sijis did some data mining on that yesterday
20:47:23 <mdomsch> FWIW, Dell is trying to figure out how to give employees the ability to blog, w/o having to go through corp communications for each post.  /me suggested tell people to simply use wordpress, why are we in the middle?
20:47:31 <ricky> I believe there were something like 16 blogs with more than 6 posts or something?
20:47:47 <sijis> yeah, it was something like that.
20:47:57 <skvidal> ricky: how many of them are something specific for fedora?
20:48:04 <ricky> The conclusion was that we have only 3 or 4 real big users
20:48:06 <sijis> the top 5 posts count was like 220, 22, 15, 8, 7
20:48:07 <skvidal> like a fedora SERVICE blog as opposed to a personal one?
20:48:56 <ricky> I didn't cross-reference them exactly, but consider this:
20:49:08 <sijis> i think that providing blogs was a good idea but something like planet makes it moot (i think)
20:49:13 <ricky> grep blogs.fedoraproject.org /home/fedora/*/.planet* | wc -l on people01 gives: 32
20:49:35 <ricky> Out of more than double that many blogs
20:49:36 <smooge> I think that blogs were thought of being a "hey come join fedora, make things happen and get a blog"
20:50:25 <abadger1999> So for wordpress -- does this seem like a good plan: identify who's using it. Send them a message that we'd like to push it to being managed by wordpress.com. Ask what features of blogs.fp.o make it more desirable to use than having an individual account on wordpress.com.  See which of those features we can get by paying some amount of money to wordpress.com.
20:50:46 * abadger1999 would like to move on to other aspects of this conversation... like publictest.
20:51:18 <ricky> Yes - although there will probably be a line we draw, like not paying for custom CSS for all 32 blogs on planet
20:51:19 <mdomsch> ricky: let's get out of the middle entirely
20:51:22 * ricky is fine with moving in
20:51:23 <mdomsch> abadger1999: go
20:51:26 <ricky> **moving on
20:51:29 <skvidal> abadger1999: +1
20:51:42 <skvidal> #agreed find out how to kill wordpress with fire
20:51:54 <skvidal> #agree find out how to kill wordpress with fire
20:52:02 <skvidal> #halp
20:52:08 * skvidal boggles
20:52:21 <ricky> Don't think zodbot gives feedback
20:52:27 <skvidal> okie doke
20:52:32 <skvidal> publictest
20:52:39 <smooge> #topic publictest
20:52:39 <skvidal> here's the general plan
20:52:54 <skvidal> 1. you request a public test instance you get it for X amount of time
20:53:13 <skvidal> 2. you request a public test instance and it is YOURS and for whomever else you specify it should be for
20:53:26 <skvidal> 3. no one else (outside of the sysadmin-$group  groups can get in
20:53:37 <skvidal> at the end of X amount of time you can request an extension
20:53:41 <skvidal> if you don't request it
20:53:43 <skvidal> we nuke it from orbit
20:53:58 <skvidal> how does everyone feel about that?
20:54:04 <ke4qqq> skvidal: does this imply purging sysadmin-test as well?
20:54:19 <skvidal> ke4qqq: I'm generally in favor of purging groups, yes
20:54:31 <skvidal> if you req the box, you get root, if you screw it up we purge it with fire
20:54:34 <ricky> Sounds great to me.
20:54:41 <skvidal> does that sound ok?\
20:54:42 <ke4qqq> makes sense to me
20:54:54 <ricky> Do we want to require an RFR for publictest machines?
20:54:55 <smooge> I would like to say " pt boxes get their own root password
20:55:09 <ricky> Like RFR a description of what service you want to run + a commitment to support it if it moves out of pt
20:55:16 <mdomsch> ricky: yes
20:55:19 <ricky> Or is that too early to require that?  I'm not sure
20:55:20 <skvidal> smooge: that sounds good to me
20:55:22 <abadger1999> All sounds good so far.
20:55:26 <mdomsch> smooge: good idea
20:55:29 <skvidal> ricky: also fine
20:55:35 <skvidal> so here's how I figure we maintain it
20:55:38 <smooge> and it is assumed that the boxes are "rooted" and people in sysadmin-main are sudo no passwd
20:55:40 <ricky> smooge: Definitely - and passwordless sudo, all that good stuff.
20:55:45 * mdomsch filed an RFR for an ssh bouncehost for use during FUDCons
20:55:45 <skvidal> smooge: okay
20:55:48 <skvidal> I'm with you
20:55:55 <mdomsch> perfect for build it / nuke it a week later
20:56:08 <abadger1999> Also, set them up more like nirik's test boxes -- with NOPASSWD sudo and using a different account than systems to pull data from fas.
20:56:23 <skvidal> mdomsch: also an excellent case for set it up at ec2/rackspace and let it run until we kill it
20:56:33 <skvidal> so - I'm all for involved and interesting systems for notifications, etc
20:56:46 <skvidal> but right now I'm thinking simple db run off of puppet1
20:56:59 <skvidal> when an rfr is filled we add the entry and the users
20:57:01 <smooge> brb have to get the door
20:57:10 <skvidal> and put the timeout on it
20:57:20 <skvidal> by simple db - I don't necessarily mean 'DATABASE'
20:57:29 <skvidal> I mean 'thing in which we stuff data and can query it'
20:57:35 <skvidal> so 'a file' also works
20:57:38 <ricky> One way to do this is to maintain a file on each publictest machine with a timestamp
20:57:47 <ricky> Yeah, or on puppet01 works too, cool.
20:57:50 <mdomsch> puppet
20:57:51 <skvidal> ricky: owner has root - they could screw with it
20:58:02 <skvidal> ricky: puppet seems like a safe place to me
20:58:11 <ricky> Sure, but owner could request an extension too.  puppet01's fine with me
20:58:20 <ricky> Human review is always nice to have
20:58:49 <skvidal> nod
21:00:10 <skvidal> okay
21:00:12 <skvidal> we're out of time
21:00:19 <skvidal> the cloud people will be barging in here any minute
21:00:49 <skvidal> do we want to continue in #fedora-admin?
21:00:52 * jgreguske lights a torch
21:00:53 <skvidal> or do we wnat to call it?
21:01:13 <skvidal> jgreguske: pitchfork-bearing meanie
21:01:19 * abadger1999 fine with #fedora-admin
21:01:23 <smooge> wait
21:01:23 <jgreguske> ;p
21:01:34 <smooge> is today the cloud/infra sync meeting?
21:01:41 <skvidal> I thought that was last week
21:01:42 <rbergeron> yes
21:01:44 <rbergeron> oh
21:01:52 * rbergeron barges in
21:02:08 * rbergeron doesn't know what the cloud/infra sync meeting status is - gholms?
21:02:25 <bmahe> there has been an email on the ml
21:02:28 <bmahe> Thursday, 2010/02/03 @2100 UTC (In NA - East coast,
21:02:28 <bmahe> 4pm; West coast, 1pm.)
21:02:52 <skvidal> okay
21:02:56 <skvidal> then with that in mind
21:02:58 <rbergeron> bmahe: yeah, but there was talks of a joint meeting.
21:02:59 <smooge> ok lets hold off on this til after this meeting is over
21:03:02 <smooge> #endmeeting