19:00:01 <nirik> #startmeeting Infrastructure (2013-04-11)
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> Meeting started Thu Apr 11 19:00:01 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic welcome y'all
19:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:11 <nirik> who all is around for a nice infra meeting?
19:00:13 * skvidal is here but grumbly about food
19:00:13 * samkottler is here
19:00:15 * oddshocks is here
19:00:19 <jerzyr> here
19:00:23 * lmacken here
19:00:27 * pingou here
19:00:31 <frankly3d> here
19:00:46 * threebean here
19:01:01 * abadger1999 here
19:01:30 <nirik> cool.
19:01:34 <nirik> lets go ahead and dive in
19:01:36 <nirik> #topic New folks introductions and Apprentice tasks.
19:01:43 <nirik> any new folks? or apprentices with questions?
19:01:53 <oddshocks> things are going well on my end
19:01:54 <smooge> here
19:02:03 <skvidal> smooge: you're not new!
19:02:18 <skvidal> smooge: :-D
19:02:25 <nirik> oddshocks: glad to hear. :)
19:02:41 <nirik> #topic Applications status / discussion
19:02:50 <nirik> any apps news? note that we are still in freeze.
19:02:58 * pingou mainly bothered threebean
19:03:09 * threebean bothered pingou
19:03:11 <nirik> #info threebean fixed email notifications in askbot. hurray!
19:03:16 * biker here
19:03:21 <pingou> threebean: I bothered you more :-p
19:03:30 <threebean> nuh-uh! :P
19:03:47 <frankly3d> Behave Children!
19:03:51 <threebean> on the askbot front, it's an issue with python-lxml on 32bit rhel6 boxes.
19:04:01 <nirik> we have fedocal and blockerbugs in stg... so possibly they will go to prod after this alpha freeze is over.
19:04:09 <nirik> threebean: really? crazy
19:04:29 <threebean> I filed an bug about it against the rhel package, but we should reinstall ask01 as 64bit as soon as is convenient.
19:04:47 <pingou> abadger1999: btw, would you mind to test a bit fedocal, I'm not sure how it will behave with non-ascii chars
19:04:57 <nirik> threebean: so, we are just working around it now?
19:05:01 * threebean nods
19:05:17 <threebean> I patched askbot to not use lxml.
19:05:17 <nirik> threebean: I think we did 32bit due to the python memory issues...
19:05:18 <abadger1999> Just a note that I'm behind on the pkgdb release -- dove into a few Fedora packaging messes and porting python-fedora away from paver instead.
19:05:26 <nirik> but I guess we can try 64 and see how it is.
19:05:42 <threebean> nirik: interesting.  I didn't know that.  ask01.stg is 64bit which made things a little confusing.
19:06:00 * relrod is here, a bit late.
19:06:06 <abadger1999> pingou: https://fedocal.dev.fedoraproject.org/ ?  I can browse aroundand see if I get 500s :-)
19:06:12 * nirik is confused then, which box is 32bit?
19:06:18 <pingou> abadger1999: that one or the .stg. one
19:06:24 <nirik> oh, prod vs stg. got it.
19:06:37 <abadger1999> pingou: What's the stg url?
19:06:45 <threebean> :) yeah :)
19:06:54 <pingou> abadger1999: admin.stg.fedoraproject.org/calendar
19:06:54 * nirik adds 'reinstall ask01 as 64bit' to list.
19:07:04 <abadger1999> pingou: Cool.  I'll browse around there.
19:07:49 <nirik> ok, any other applications news?
19:08:04 <nirik> #topic Sysadmin status / discussion
19:08:04 <threebean> Oh, there's talk of debian picking up fedmsg -> http://lists.debian.org/debian-qa/2013/04/msg00010.html
19:08:11 <nirik> threebean: oh yeah? wow.
19:08:40 <threebean> yeah, nothing super concrete yet, but olasd in #fedora-apps is looking to mentor someone on it.
19:08:42 <nirik> awesome.
19:08:46 <threebean> I'll be keeping in touch with them :)
19:08:58 <nirik> I wonder if we could setup some kind of meta fedmsg of all the fedmsg using places...
19:09:07 * threebean nods
19:09:14 <nirik> cool.
19:09:15 <threebean> it would be trivial to subscribe to both streams
19:09:22 <nirik> so, on sysadmin side of things...
19:10:07 <nirik> #info hopefully will see new netapp space at some point for /mnt/koji and hopefully backups
19:10:30 <nirik> #info we have all our arm boxes installed, but are going to look at testing replacing disk on one or two with ssd to see how much that helps builds.
19:10:47 <nirik> #info some ansible poking ongoing
19:10:54 <skvidal> so much poking
19:11:02 <nirik> I whipped up a fas-client ansible module, but I haven't tested it yet.
19:11:12 <nirik> but hopefully soon.
19:11:20 <skvidal> and the new roles mechanism in ansible 1.2/git is going to make us restructure a bunch of things
19:11:21 <nirik> skvidal: whats the status on the cron/trigger stuff?
19:11:22 <skvidal> for beauty
19:11:40 <lmacken> AnsibleFest 2013 in Boston Jun 12th
19:11:40 <skvidal> nirik: cron is actually looking ok
19:11:56 <skvidal> nirik: it breaks out the set of hosts and knows the playbooks to run
19:12:01 <nirik> and then we need to work on moving the builder stuff over.
19:12:03 <nirik> nice!
19:12:20 <skvidal> nirik: I have to 1. parallelize it so we are running them roughly all at the same time (up to a limit)
19:12:27 <skvidal> the trigger thing is hell on wheels
19:12:37 <nirik> yeah, I feared it might be. ;(
19:12:44 <skvidal> it's more or less 'if this is from a task or a playbook directly, great - I can figure that out'
19:12:46 <skvidal> else 'screwed'
19:12:54 <nirik> so, perhaps this:
19:12:55 <skvidal> also - commits to things like vhost-reboot?
19:13:04 <skvidal> not something we want to trigger on :)
19:13:21 <nirik> if its a task/playbook we can easily see what is affected, trigger run
19:13:35 <nirik> if it's a playbook we don't want to do like vhost-reboot, don't do anything.
19:13:46 <skvidal> so isolating the playbooks is easy enough - they are all in a subdir
19:13:46 <nirik> if it's something and we can't tell, touch a file to run cron next time?
19:14:02 <skvidal> nirik: yah - it's what we discussed before
19:14:15 <skvidal> nirik: I was looking at what it would take to include the roles in this whole thing
19:14:17 <nirik> perhaps move the one-time type playbooks to scripts?
19:14:17 <skvidal> like tasks
19:14:24 <nirik> ah yeah, need to get roles in there.
19:14:36 <skvidal> and that's where I kinda meandered down a rathole
19:14:48 <skvidal> nirik: I'm sure you've experienced that. :)
19:14:58 <nirik> yeah.
19:15:05 <nirik> there's many ways to do things... which is great, but...
19:15:09 <skvidal> hah
19:15:27 <nirik> at some point we just need to pick one, do it and then if it sucks redo it
19:15:36 <skvidal> yah  - sorry
19:15:41 <skvidal> oh on the builders merge
19:15:48 <skvidal> I put in some group_vars and host_vars for the builders
19:16:05 <skvidal> that are going to be the basis for porting the builders_reinstall playbook
19:16:17 <nirik> cool.
19:16:21 <skvidal> all of it is pretty tame - but I've not been able to test it b/c of the freeze
19:16:34 <skvidal> I really just need one, removed, builder to beat up
19:16:35 <skvidal> :)
19:16:47 <nirik> BTW, I think it might be pretty easy to make a 'arm-reinstall' playbook too...
19:17:01 <skvidal> nirik: can we trigger reboots/ks from lockbox?
19:17:07 <skvidal> nirik: or do we need to delegate to some other box?
19:17:09 <skvidal> (which is fine)
19:17:12 <nirik> yep. or from noc01...
19:17:21 <nirik> via ipmitool commands
19:17:23 <skvidal> how reliable is it?
19:17:37 <nirik> very. I've not had it fail.
19:17:42 <skvidal> cool
19:17:58 <nirik> tell to pxe boot next time, power cycle, wait to finish install
19:18:12 <skvidal> got an average wait?
19:18:17 <skvidal> what's the reinstall time take?
19:18:42 <nirik> a few minutes... but then it takes another 3-4 for sshd to get enough entropy to get ssh host keys
19:18:49 <skvidal> ummm
19:18:52 <skvidal> 3-4 minutes for entropy?
19:18:54 <skvidal> really?
19:18:56 <nirik> or something.
19:18:58 <skvidal> wow
19:19:12 <nirik> I can time it. I might be thinking it's longer than it was
19:19:34 <nirik> they def are up and working but sshd isn't yet for a bit
19:20:03 <nirik> anyhow, lets keep plugging at ansible and try and get some things setup in it/in use...
19:20:12 <nirik> any other sysadmin stuff?
19:20:34 <skvidal> speaking of tihs stuff
19:20:44 <skvidal> at some point I will need to spend more time back on the copr-be
19:20:47 <smooge> need to install entropy network :)
19:21:07 <nirik> skvidal: ok.
19:21:12 <nirik> #topic Private Cloud status update / discussion
19:21:12 <smooge> what does the be stand for skvidal ?
19:21:29 <skvidal> backend
19:21:38 <nirik> so, just a bit of news here, we have been working to reinstall our odd cloudlet and get some of the nodes added to openstack.
19:21:52 <skvidal> 07,05 are done
19:21:53 <nirik> should have that before too long and 2 nodes we can use to test new stuff
19:21:56 <skvidal> 03 is wacked out
19:22:01 <skvidal> 01 is just sitting there
19:22:21 <skvidal> so right now
19:22:29 <nirik> ok. shall we add 05/07 in?
19:22:31 <skvidal> we just need to turn 05/07 into compute nodes
19:22:32 <skvidal> yes
19:22:44 <skvidal> do we have a playbook/processed setup for that?
19:23:03 <nirik> there is one on 02 in /tmp I think, or I have it locally here...
19:23:15 <skvidal> nirik: hmm - might be good to get that somewhere quasi-official
19:23:18 <nirik> it's basically install openstack-nova-compute and copy over the nova.conf file and start it.
19:23:19 <smooge> do the ansible stuff for them run off 01/02 versus lockbox? [silly question I know]
19:23:22 <skvidal> even if that is just in /root on 02
19:23:28 <nirik> skvidal: agreed.
19:23:55 <nirik> I can put it wherever...
19:24:09 <nirik> if someone wants to poke at it, or merge it into our main repo or run it or whatever.
19:25:06 <nirik> ok, any other cloud news?
19:25:12 * skvidal listens to the cricket storm of volunteers
19:25:23 <smooge> sorry you needed something?
19:25:33 <smooge> oh you want it merged with the main ansible
19:25:36 <smooge> I can look at that.
19:25:47 <smooge> sorry
19:25:47 <nirik> cool.
19:25:52 <nirik> #topic Upcoming Tasks/Items
19:26:03 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze
19:26:03 <nirik> #info 2013-04-15 - remove legacy-openid
19:26:03 <nirik> #info 2013-04-15 - announce openid change on hosted.
19:26:03 <nirik> #info 2013-04-15 - drop inactive apprentices
19:26:03 <nirik> #info 2013-04-16 F19 alpha release
19:26:04 <nirik> #info 2013-04-17 - switch to new sigul servers.
19:26:05 <nirik> #info 2013-04-19 - switch hosted to openid
19:26:13 <nirik> anything else people would like to schedule or note or change?
19:26:21 <nirik> oh, alpha has slipped a week, I need to adjust.
19:26:31 <nirik> so, probibly those things after alpha will push out a week
19:27:23 <nirik> #topic Open Floor
19:27:27 <nirik> any items for open floor?
19:28:37 <nirik> Oh... I did whip up some plans on hosted:
19:28:55 <nirik> https://fedoraproject.org/wiki/Infrastructure_FedoraHosted_Version2_Notes
19:29:52 * nirik is sure there's other stuff, but I can't think of it right now. :)
19:30:37 <nirik> ok, if nothing else will close out in a minute.
19:31:36 <smooge> not from me
19:31:51 <nirik> thanks for coming everyone!
19:31:54 <nirik> #endmeeting