19:00:01 <nirik> #startmeeting Infrastructure (2013-04-04)
19:00:01 <zodbot> Meeting started Thu Apr  4 19:00:01 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:02 <nirik> #topic welcome y'all
19:00:02 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:02 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:19 * skvidal is here
19:00:25 * pingou here
19:00:26 * lmacken here for 15m
19:00:33 * relrod here
19:00:37 * threebean is here
19:00:52 * fchiulli is listening
19:01:18 <nirik> #topic New folks introductions and Apprentice tasks.
19:01:19 <smooge> is here
19:01:29 <nirik> any new folks or apprentices with questions or comments?
19:01:41 <nirik> oddshocks: you wanted to introduce yourself? ;)
19:01:44 * oddshocks is new!
19:01:52 <threebean> :P
19:02:09 <oddshocks> Hey everyone, I just sent out a mailing list introduction, I'm an IT student and free software hacker from RIT hoping to get involved with things around here :)
19:02:17 <oddshocks> potential Red Hat intern for the Summer
19:02:19 <skvidal> oddshocks: welcome
19:02:21 <oddshocks> nice to meet you all :)
19:02:23 <lmacken> welcome :)
19:02:28 <rossdylan> hye oddshocks :p
19:02:28 <nirik> welcome oddshocks
19:02:30 <pingou> nice, welcome :)
19:02:38 <nirik> you more interested in sysadmin or software devel?
19:03:35 <oddshocks> I'm still kinda on the fence, but systems administration catches my eye a bit more
19:03:51 <nirik> sure, you could also help out in both time and focus permitting. ;)
19:04:16 <oddshocks> I like writing code and I also like working with systems, so yeah
19:04:20 <nirik> anyhow, welcome and do see us after the meeting in #fedora-admin / #fedora-apps and we can point you at where to get started.
19:04:27 <oddshocks> Perfect, thanks!
19:04:33 <nirik> #topic Applications status / discussion
19:04:46 <nirik> any applications news? Note that we are in alpha freeze right now.
19:04:55 <abadger1999> wanna talk about gsoc here or open floor?
19:05:18 * pingou worked on pkgdb2 still
19:05:21 <nirik> either way. ;)
19:05:24 <pingou> I let threebean mention tagger :)
19:05:32 * threebean mentions tagger
19:05:47 * nirik will have a seperate topic about freeze discussions after apps/sysadmin ones.
19:06:01 <threebean> :P
19:06:11 <pingou> that was easy ;)
19:06:28 <abadger1999> GSoC is Google Summer of Code. It's coming up soon and Fedora has applied.  We've already had students interested in working on infra come to find us in irc.  What we really need to pull this off, though, is mentors.
19:06:30 <threebean> tagger dev work is getting close to something we could put in staging.  early next week at the earliest.
19:06:52 <nirik> cool.
19:07:04 <nirik> abadger1999: yeah, most of us are pretty strapped for time tho. ;(
19:07:13 <abadger1999> Most of the usual suspects are busy this year (lmacken, myself, nirik, skvidal, pingou, etc).
19:07:15 <pingou> threebean: btw we should update the dev instance with the python-fedora from abadger1999 so we can start checking it
19:07:27 <skvidal> abadger1999: additionally i'm a terrible mentor
19:07:33 <threebean> pingou: oh yeah, let's set that up in #fedora-apps post meeting
19:07:45 <pingou> threebean: cool
19:07:49 <pingou> skvidal: really? :)
19:07:59 <nirik> abadger1999: some of the items are not really directly needing mentors from us... the gitlab thing could well be mostly packaging.
19:08:01 <abadger1999> I sent a request to the infra list for mentors to raise their hands if they want to work on something but so far no replies.
19:08:02 <skvidal> pingou: no need for you to mock
19:08:07 <abadger1999> <nod>
19:08:18 <threebean> ...chain.py!
19:08:47 <abadger1999> nirik: although... being useful to infra would make it more likely to be accepted.
19:08:51 <pingou> wrt GSoC: tbh, there has been some people that looked very motivated, it kind of makes me want to help them
19:08:58 <abadger1999> <nod>
19:09:01 <nirik> it's a bit difficult in that ideas on the page are submitted by anyone without infra really looking or saying they would like them. Perhaps next year we could be more proactive and prune ideas to ones we really would like and that might get more mentors involved.
19:09:21 <abadger1999> Yeah, biker has showed up a number of times and been pretty enthusiastic.
19:09:40 <pingou> skvidal: honestly, I wasn't
19:09:46 <nirik> yeah, agreed.
19:09:49 <abadger1999> He wondered about doing a unittesting GSoC... I'd be willing to partially mentor that.
19:10:03 <nirik> biker was also interested in any other ideas too... wasn't set on the ideas page ones.
19:10:07 <skvidal> pingou: SSSSSSSSSSSSSURRE you weren't :)
19:10:08 <abadger1999> <nod>
19:10:13 <pingou> skvidal: no really :)
19:10:32 <pingou> GSoC: pkgdb2? Elections?
19:10:32 <nirik> I've seen a lot of interest in gitlab and unit testing for fedpkg...
19:10:48 <pingou> start porting to ansible?
19:11:02 <pingou> MirrorManager?
19:11:02 <abadger1999> elections, fchiulli is managing... pretty far along -- but from the hours that he's available I don't htink he can mentor.
19:11:31 <nirik> I don't think mdomsch has time to mentor for mm... and not sure anyone else is up to speed enough yet.
19:11:33 * fchiulli is not available much during the day.  Nights vary
19:11:49 <abadger1999> mirrormanager needs a new driver since mdomsch has less time.  Without a driver, gsoc contributions are likely to not get merged.
19:11:58 <abadger1999> <nod>
19:12:19 <nirik> would anyone be willing to co-mentor with abadger1999 on unittesting?
19:12:21 <abadger1999> ansible would be good -- but that would need a sysadmin mentor... and you guys are all pretty tapped, right?
19:12:23 <nirik> perhaps thats one we could do?
19:13:21 <pingou> abadger1999: well I did work quite a bit trying to port MM to TG2
19:13:23 <nirik> the problem with the ansible side is that a lot of it is just deciding how we want to organize things... I'm not sure what GSoC folks could do there until we have more things in place and then not sure how exciting it would be. and yeah, time...
19:13:30 <abadger1999> <<nod>
19:13:40 <abadger1999> pingou: Want to take ownership?
19:14:01 <nirik> we still need to get mm1.4 into production.
19:14:06 <nirik> and fix issues with it.
19:14:06 <abadger1999> pingou: heh -- if you aren't careful, you'll end up with ownership of pkgdb as well  ;-)
19:14:18 <pingou> abadger1999: I can deal w/ pkgdb more than MM :)
19:14:36 <pingou> abadger1999: but don't think you're out of the loop ;-)
19:14:50 <abadger1999> darn! ;-)
19:15:25 <nirik> so, where does that leave us? ;)
19:15:30 <abadger1999> anyhow -- if anyone wants to step up to mentor, I can co-mentor unittesting python-fedora and some other things.
19:15:50 <abadger1999> can't do it alone.
19:15:57 <pingou> pkgdb2 is an option?
19:16:08 <dgilmore> nirik: it would be awesome for some mm love
19:16:31 <nirik> sure, but without a driver on it, not sure GSoC would help us any
19:16:46 <dgilmore> yeah
19:16:53 <abadger1999> pingou: yes and no.  If you want to mentor that I can help you --- but I wouldn't want to be primary mentor for that this summer.
19:16:54 * Smoother1rOgZ is around
19:17:26 <pingou> abadger1999: and I shouldn't (not that I wouldn't)
19:17:30 <abadger1999> hehe :-)
19:17:34 <abadger1999> understood :-)
19:17:50 <nirik> ok, well, lets keep discussing and see if we can offer anything...
19:17:54 <nirik> any other application news?
19:18:10 <abadger1999> python-fedora alpha is in infra; surprisingly stable.
19:18:21 <threebean> cool :)  stg or prod?
19:18:22 <abadger1999> I just sent a change request to fix a few bugs.
19:18:28 <abadger1999> threebean: prod
19:18:30 <threebean> awesome
19:18:50 <abadger1999> working on a pkgdb update (pingou's slimming of the api and db)
19:18:51 <nirik> cool.
19:18:58 <abadger1999> will go to stg next week I hope
19:19:00 <nirik> that would be after alpha?
19:19:04 <nirik> (for prod
19:19:05 <nirik> )
19:19:16 <abadger1999> and figuring out how to make otp fit into the python-fedora api
19:19:22 <abadger1999> nirik: correct.
19:19:25 <nirik> fun times. ;)
19:19:41 <threebean> good news: I've heard no negative feedback after tuesday's freeze break request - http://lists.fedoraproject.org/pipermail/infrastructure/2013-April/012724.html
19:19:42 <nirik> ok, moving along to sysadmin...
19:19:53 <nirik> threebean: yeah, seems ok, but the None's are weird.
19:19:57 * threebean nods
19:20:14 <threebean> those are omitted from the ircbot messages in git... coming down the pipe.
19:20:15 <nirik> #topic Sysadmin status / discussion
19:20:35 <nirik> threebean: yeah
19:20:41 <nirik> so, on the sysadmin side of things...
19:21:08 <nirik> we have 3/4ths of our arm SOC's on line and I will be redoing the other one soon when switch ports are changed on it.
19:21:18 <nirik> I created a sop for dealing with the arm socs:
19:21:27 <nirik> http://infrastructure.fedoraproject.org/infra/docs/arm.txt
19:21:56 <dgilmore> nirik: :) many thanks
19:22:11 <nirik> no problem. Nice to see them finally cranking away.
19:22:36 <nirik> I will likely request a freeze break in a bit for updating postgresql... might need a short outage for it.
19:23:21 <nirik> bnfs01 (our backup storage box) is hosed again. smooge was looking into it.
19:23:41 <nirik> we are finalizing our new netapp storage now, hopefully will have some in a while.
19:23:54 <smooge> it is not looking good
19:24:01 <nirik> smooge: ;(
19:24:22 <nirik> any other sysadmin stuff?
19:24:29 * skvidal has been working on ansible-y stuff
19:24:38 <skvidal> the cron-runner
19:24:40 <abadger1999> postgres update -- no dump and reload needed right?
19:24:41 <nirik> oh, I was thinking we should add monitoring for the arm boxes at some point. I might file a ticket on that and let someone else do it. ;)
19:24:46 <skvidal> the freezes thing (which uncovered a bug)
19:24:51 <nirik> abadger1999: correct. just update and restart
19:24:56 <abadger1999> cool.
19:25:34 <nirik> skvidal: so, were we going to do cron there? or triggered by commit? (or jury still out)?
19:25:45 <skvidal> nirik: well the script is the same, ultimately
19:25:52 <skvidal> so...
19:25:58 <nirik> yeah.
19:25:59 <skvidal> one way or the other we need to be able to say
19:26:09 <skvidal> 'these hosts or groups' - run the playbook for them and don't overlap
19:26:14 <nirik> I don't know if we can get ansible to say 'this commit affects these hosts'
19:26:17 <skvidal> run them forked off - multiple in parallel at a time
19:26:25 <nirik> yeah.
19:26:26 <skvidal> nirik: we can for playbooks, we can't, easily, for files
19:26:46 <nirik> ah. ok.
19:26:48 <skvidal> nirik: for tasks it is.... theoretically... possible - but there will be a lot of grepping involved
19:26:56 <pingou> speaking of ansible, do we finally have an idea how we want to handle stg/prod?
19:27:08 <nirik> yeah, we will just need to look at where we want to draw the line probibly.
19:27:36 <nirik> pingou: we did come up with a plan. ;) But it's not sure yet that it will work, so we need to test it...
19:27:41 <skvidal> :)
19:27:50 <nirik> basically similar to how we do it in puppet... a path for things.
19:28:05 <skvidal> that plugin should work - right now the issue is this
19:28:16 <nirik> skvidal: that was pulled into 1.1? or no?
19:28:20 <skvidal> nirik: pretty sure
19:28:22 <skvidal> lemme verify
19:28:37 <skvidal> yep
19:28:40 <skvidal> first_found lookup plugin
19:28:58 <skvidal> so - there are 2 things here, really
19:29:08 * nirik did update lockbox01 to 1.1
19:29:23 <skvidal> the playlets idea that I came up with is.... not very attractive unless we can make it easy to use in an existing playbook
19:29:42 <skvidal> in many case we can make-do w/o needing those playlets - just with included tasks
19:29:55 <skvidal> but I'm trying to figure out which cases where tasks will make it more confusing to use
19:29:56 <nirik> yeah.
19:30:11 <skvidal> in most of the cases I can think of
19:30:11 <nirik> yeah, I figure we will need to try things and adjust.
19:30:20 <skvidal> it's just where the variable includes would be ugly in every playbook
19:30:24 <skvidal> and you really just want to say
19:30:37 <skvidal> "you know that last group of hosts you ran against?"
19:30:48 <skvidal> "run this playbook over here, against them and only them"
19:31:38 <skvidal> if anyone wants to talk about this with me sometime
19:31:47 <skvidal> I think I could really use someone to bounce this back and forth with
19:32:04 <skvidal> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playlets/README
19:32:06 <skvidal> this is the idea
19:32:18 <threebean> cool -- I'll queue it up for reading
19:32:24 <skvidal> and then
19:32:41 <nirik> yeah, lots of ways to do things... ;)
19:33:20 <skvidal> http://paste.fedoraproject.org/6712/10399613/
19:33:36 <nirik> ok, lets keep poking at it. ;) I'd like to add arm-releng to ansible repo soon so I can try and work out in reality how things will look
19:33:45 <skvidal> yah
19:33:59 <skvidal> did we already do cloud stuff?
19:34:00 <nirik> any other sysadmin stuff anyone has?
19:34:03 <nirik> not yet. ;)
19:34:05 <skvidal> ok
19:34:10 <nirik> #topic Private Cloud status update / discussion
19:34:14 <nirik> now. go! :)
19:34:18 <skvidal> heh
19:34:28 <skvidal> so I'd like to add a new ip range to the openstack cloudlet
19:34:42 <nirik> sounds good. are we ready to add more compute nodes too?
19:34:42 <skvidal> I'm almost positive that the command I want to run won't break the existing network ip range ;)
19:34:47 <skvidal> nirik: we are
19:35:02 <nirik> if it does... oh well, we do the best we can. ;)
19:35:02 <skvidal> fed-cloud05 and 07 need to be reformatted
19:35:16 <skvidal> but they are not doing anything now
19:35:33 <skvidal> so anyone in sysadmin-main who would like to kick them  should feel free
19:35:47 <nirik> cool.
19:35:59 <skvidal> also
19:36:02 <nirik> adding compute nodes is super easy too... nova.conf and a few packages and away you go.
19:36:16 <skvidal> 01 and 03 should be ready to be turned over to grizzly if anyone would like to do that
19:36:29 <nirik> excellent.
19:36:32 <skvidal> this is sorta-cloud
19:36:52 <skvidal> I played with systemd-nspawn a bit and what it can do
19:37:12 <nirik> oh yeah. f18?
19:37:16 <skvidal> and I think if anyone would like to tinker with infrastructure/scripting/etc for that on a big fedora 18 cloud instance
19:37:27 <skvidal> it would be something that will eventually be useful to us
19:37:31 <skvidal> us == fedora infra
19:37:37 <nirik> yeah, I agree.
19:37:45 <skvidal> it is also completely bluesky afaict
19:37:51 <skvidal> so if you want to noodle around with it
19:37:54 * nirik nods.
19:37:58 <threebean> bluesky?
19:38:07 <skvidal> threebean: go, play, doesn't matter if nothing comes of it
19:38:10 <skvidal> threebean: no constraints
19:38:15 <nirik> not ready for real use right now, by someday... perhaps...
19:38:24 <threebean> ok, thanks
19:38:29 <skvidal> so a nspawn is a container
19:38:35 <skvidal> forming one up is easy
19:38:47 <nirik> I think the big things around it would be: managing them, and networking.
19:38:51 <skvidal> yum --installroot=/some/path install @core
19:39:09 <skvidal> systemd-nspawn -D /some/path -b
19:39:17 <skvidal> and then you're in the chroot, lickety-split
19:39:51 <nirik> yep.
19:40:00 <skvidal> anyway - if anyone wants to do that
19:40:04 <skvidal> yell at one of us
19:40:10 <skvidal> and we can spin up an f18 instance quickly
19:40:20 <nirik> sounds good.
19:40:26 <nirik> #topic Freezes
19:40:29 <skvidal> brrr
19:40:35 <nirik> so, we are in a freeze for alpha now...
19:40:43 <nirik> and the topic came up how we determine whats frozen, etc.
19:40:47 <nirik> (see post on list)
19:41:29 <nirik> would anyone like to make a case for more hosts NOT being frozen than dev/stg ? or some way to decide those?
19:41:48 <skvidal> nirik: there's one example of a host-specifically marked  as freezes:false
19:41:51 <skvidal> people03.fedoraproject.org
19:42:01 <skvidal> adding additional ones like that to the inventory is really easy
19:42:22 <skvidal> nirik: actually - this brings up a question
19:42:32 <skvidal> nirik: right now we have the inventory inside the public ansible repo
19:42:33 <nirik> sure... but I was meaning from a standpoint of critera.
19:42:38 <abadger1999> for alpha/beta or final or both?
19:42:45 <skvidal> abadger1999: I think for all
19:42:50 <nirik> abadger1999: both, either... ;)
19:43:07 <nirik> I'd like to have a critera we can hold a host up to and see if it should be frozen or not.
19:43:26 <nirik> a very simple one would be: is the host *stg* or *dev* ?
19:43:35 <nirik> but that means a lot more things are frozen than were before.
19:43:55 <abadger1999> yeah... I don't really like mor ethings frozen.
19:44:10 <skvidal> nirik: I think there is room for more non-distro-releasing-impacting things being unfrozen
19:44:29 <skvidal> people, hosted, hosted-lists,
19:44:31 <nirik> yeah, but what critera can we use. ;)
19:44:34 <skvidal> the items you mentioned in your email
19:44:38 <nirik> I posted some other attempts...
19:44:42 <skvidal> nirik: 'does it impact the ability to release the distro'
19:44:54 <nirik> well, hosted actually does impact the distro. ;)
19:45:07 <skvidal> nirik: how?
19:45:07 <nirik> comps is pulled from there.
19:45:40 <skvidal> sigh
19:45:43 <abadger1999> comps isn't mirrored into a package?
19:45:50 <nirik> anyhow, we could continue discussion on list too. :)
19:45:57 <abadger1999> i guess its repodata so no.. :-(
19:46:02 <nirik> abadger1999: no, it's pulled from git by composes...
19:46:09 <nirik> also spin-kickstarts is there.
19:46:25 <nirik> now, we could look at moving those, but right now they are there. ;0
19:46:42 <nirik> brb
19:47:22 <skvidal> yah.... I see
19:47:31 * skvidal grumbles...
19:47:34 <skvidal> okay fine
19:47:40 <skvidal> hosted-lists!
19:47:42 <dgilmore> nirik: and branched is pulled from git on fedorahosted nightly
19:48:06 <nirik> yeah, rel-eng repo too...
19:48:27 <skvidal> so - back to my earlier question
19:48:35 <nirik> anyhow, lets continue that discussion on list... for purposes of this alpha freeze, we just keep using the picture. ;)
19:48:37 <skvidal> do we want to split out the inventory subdir of the ansible repo - into its own repo
19:48:47 <nirik> skvidal: what does that get us?
19:48:51 <skvidal> so we can have people editing the inventory who might not be able to commit to ansible itself
19:49:13 <nirik> not sure thats a very common thing is it?
19:49:18 <skvidal> true enough
19:49:28 <skvidal> this is why I ask these things ;)
19:49:30 <nirik> I'd prefer to keep number of repos down if we can. ;)
19:50:05 <nirik> commit on ansible is going to be the same as commit on puppet right?
19:50:15 <nirik> which is what... sysadmin?
19:50:20 <skvidal> lemme look
19:50:34 <skvidal> yes
19:50:34 <skvidal> sysadmin
19:50:36 <skvidal> yah
19:50:38 <skvidal> you're right
19:50:43 <skvidal> I withdraw the suggestions
19:51:12 <nirik> :) of course they must be in a group that has shell on lockbox01 too...
19:51:21 <nirik> #topic Upcoming Tasks/Items
19:51:33 <nirik> here's what I have for upcoming:
19:51:35 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze
19:51:36 <nirik> #info 2013-04-15 - remove legacy-openid
19:51:36 <nirik> #info 2013-04-15 - announce openid change on hosted.
19:51:36 <nirik> #info 2013-04-16 F19 alpha release
19:51:36 <nirik> #info 2013-04-17 - switch to new sigul servers.
19:51:37 <nirik> #info 2013-04-19 - switch hosted to openid
19:51:38 <nirik> #info 2013-05-07 to 2013-05-21 BETA infrastructure freeze
19:51:40 <nirik> #info 2013-05-21 F19 beta release
19:51:42 <nirik> #info 2013-05-31 end of 1st quarter
19:51:44 <nirik> #info 2013-06-11 to 2013-06-25 FINAL infrastructure freeze.
19:51:46 <nirik> #info 2013-06-25 F19 FINAL release
19:51:48 <nirik> any other items people would like to schedule or note?
19:52:34 <nirik> #topic Open Floor
19:52:41 <nirik> anyone have items for open floor?
19:52:58 <skvidal> nirik: one question
19:53:24 <skvidal> nirik: what were the future points for determining what to do with hosted
19:53:31 <skvidal> nirik: and making it more-better
19:53:44 <skvidal> as in - it's running right now
19:53:46 <nirik> yeah... so I think we decided on at least a short term plan...
19:53:51 <skvidal> do we just sit on it?
19:53:57 <skvidal> ah - okay -
19:54:02 <nirik> well, we could do that too I guess...
19:54:10 <nirik> I was thinking we could move it to osuosl02...
19:54:28 <nirik> (or at least all the git ones)
19:54:31 <skvidal> okay - so that's something we do plan to do.
19:54:40 <skvidal> did you decide if you wanted osuosl02 to be cloud-y or not?
19:55:15 <nirik> I was really on the fence there. I think cloud would be nice from a 'figure more stuff out' with it, but on the other hand, I don't know how much time we want to spend on hosted really...
19:55:40 <nirik> Ideally we can get it to a stablish thing and not have to deal with it again for a while.
19:56:11 <nirik> let me try and write up a plan and we can tweak it and see what all of it we want to do.
19:56:11 <skvidal> nirik: so... what does that mean?
19:56:13 <skvidal> ok
19:56:18 <skvidal> sounds round
19:56:19 <nirik> I can put it on that hosted wiki page with all the ideas.
19:56:21 <skvidal> thank you
19:56:45 <nirik> hosted02 is running ok now, but I think I'd like to move it anyhow, even if we do nothing else but move it.
19:57:19 <skvidal> cool
19:57:32 * nirik adds to his list. ;)
19:57:37 <nirik> ok, anything else ?
19:58:08 <nirik> ok, thanks for coming everyone!
19:58:09 <threebean> not from me :)
19:58:14 <nirik> #endmeeting