18:00:00 <nirik> #startmeeting Infrastructure (2012-04-19)
18:00:00 <zodbot> Meeting started Thu Apr 19 18:00:00 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic Robot Roll Call
18:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:11 <nirik> who all is around for a infrastructure meeting?
18:00:15 * lmacken 
18:00:19 * skvidal is here
18:00:23 * threebean is here
18:00:27 * athmane is here
18:00:31 * ianweller is last-minuting a physics lab report but will pay attention
18:01:18 * pingou 
18:01:24 <dgilmore> buenos
18:02:06 <nirik> ok, lets go ahead and dive in.
18:02:13 <nirik> #topic New folks introductions and Apprentice tasks.
18:02:14 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:02:14 <nirik> would like to ask general questions, they can do so here.
18:02:32 <nirik> any new folks? or questions on easyfix tickets or general getting started questions?
18:03:54 <nirik> ok, moving on then...
18:04:02 <nirik> #topic two factor auth status
18:04:03 <skvidal> no changes on 2fa - pto the beginning of this week.
18:04:18 <nirik> ok. We can ping it next week...
18:04:29 <nirik> #topic Staging re-work status
18:04:34 <nirik> I've worked on this some.
18:04:40 <nirik> I've commited docs on the new setup.
18:04:56 <nirik> please read http://infrastructure.fedoraproject.org/infra/docs/staging.txt and let me know if anything needs fixing.
18:05:18 <nirik> I'm going to try and get the bapp01->02 move done, and easyfix moved to production, then we can do the staging branch killing. ;)
18:05:53 <nirik> any questions on that ?
18:06:35 <nirik> ok, I'll schedule the actual staging killing after those other 2 things are done.
18:06:38 <nirik> #topic Applications status / discussion
18:06:44 <nirik> any news on applications this week?
18:07:08 <nirik> abadger1999 / threebean / lmacken / pingou / CodeBlock / anyone else who works on apps. ;)
18:07:33 <abadger1999> New python-fedora this week that should resolve some issues clients were seeing.
18:07:45 <nirik> yeah, thanks for that fix.
18:07:54 <lmacken> abadger1999++
18:08:12 * nirik still doesn't fully understand it, but thats ok
18:08:16 <abadger1999> I'm not aware of any of our scripts that were running into this but it won't hurt them to update on the infra boxes.
18:08:58 <threebean> fedmsg progress - got a mediawiki plugin done and tested locally.  waiting still on lots of package reviews.
18:09:13 <ianweller> he's alive!
18:09:44 <nirik> threebean: could you perhaps drop a list to the infra list? or a wiki page link or something? then we can try and divide them up...
18:09:54 <threebean> sure thing
18:10:08 * threebean writes that now
18:10:09 <nirik> #info reviews needed to move fedmsg ahead.
18:10:20 <nirik> #action threebean to post list
18:11:08 <nirik> ok, any other app news?
18:11:22 <dgilmore> where did we ever get with re-writing fpaste?
18:12:15 <nirik> dgilmore: oh yeah, have some news on that...
18:12:32 <nirik> we are looking at possibly using sticky-notes instead for the server side.
18:12:46 <nirik> http://gitorious.org/sticky-notes
18:12:47 <abadger1999> mmello, I believe was interested in helping out with reviews, too.
18:13:01 <nirik> it's under review, pending some fixes from upstream.
18:13:13 <nirik> athmane has found some issues and submitted patches to it.
18:13:24 <mmello> abadger1999: yes..I'm
18:13:32 <dgilmore> nirik: cool
18:13:44 <dgilmore> mmello: get to work ;)
18:13:50 <mmello> abadger1999: threebean is helping to understand the process and I'm currently review my first package
18:13:53 <mmello> dgilmore: :)
18:14:03 <nirik> #info looking at sticky-notes for a pastebin server
18:14:11 <dgilmore> nirik: would the current fpaste client work with the new server? or need rewritten?
18:14:18 <nirik> there's a command line 'pastebinit' already in fedora.
18:14:25 <dgilmore> cool
18:14:35 <nirik> so, we could look at making it obsolete fpaste and perhaps provide a compat link.
18:14:50 <nirik> it misses a few things like the --sysinfo tho.
18:15:18 <nirik> anyhow, it looks promising.
18:15:34 <codemaniac> hello nirik and everyone
18:15:44 <nirik> morning codemaniac. welcome.
18:16:01 <codemaniac> good morning to you too nirik
18:16:03 <dgilmore> nirik: :) figured id ask
18:16:09 <pingou> but php
18:16:15 <ctria> hi all. did i get lost timezones again?
18:16:17 <nirik> pingou: yeah, its php.
18:16:27 <nirik> ctria: meeting just started 15m ago. ;)
18:16:35 <nirik> any further news on applications?
18:16:35 <ctria> :(
18:16:49 <codemaniac> sorry for joining late team
18:16:56 <nirik> no problem at all.
18:17:24 <codemaniac> CodeBlock: you around ?
18:17:39 <nirik> ok, will move along then...
18:17:44 <nirik> #topic Meeting time
18:17:54 <nirik> is everyone ok with this meeting time? 18UTC?
18:17:59 <skvidal> +1
18:18:00 <pingou> +1
18:18:03 <ianweller> +1
18:18:07 <abadger1999> +1
18:18:25 <nirik> ok then... seems popular.
18:18:47 <codemaniac> yea niriki , i am very much comfortable with this time , its +5:30 UTC here
18:19:07 <codemaniac> +1024
18:19:13 <nirik> cool.
18:19:34 <nirik> #topic Upcoming Tasks/Items
18:19:43 <nirik> #info 2012-05-01 to 2012-05-15 - F17 Final Freeze.
18:19:43 <nirik> #info 2012-05-01 - nag fi-apprentices.
18:19:43 <nirik> #info 2011-05-03 - gitweb-cache removal day.
18:19:43 <nirik> #info 2012-05-09 - Check if puppet works on f17 yet.
18:19:43 <nirik> #info 2012-05-10 - drop inactive fi-apprentices
18:19:44 <nirik> #info 2012-05-15 - F17 release
18:19:53 <nirik> so final freeze is coming up.
18:20:25 <nirik> One thing I will probibly schedule soon is a migration of our hosted machine... hosted03 -> hosted01/02. Should be a pretty easy migration.
18:20:37 <nirik> anyone have other items they are working on they would like to schedule or shout out?
18:21:46 <nirik> ok, I'll take that as a no. ;)
18:21:55 <dgilmore> nothing here
18:22:06 <nirik> #topic Private Cloud
18:22:14 <nirik> just a short mention of our cloud plans....
18:22:14 <skvidal> we have new progress?
18:22:35 <nirik> not really... but I made a wiki page: https://fedoraproject.org/wiki/Infrastructure_private_cloud
18:22:58 <nirik> so, if people have suggestions/ideas, post them on list.
18:23:01 * skvidal counts backward from 5
18:23:16 <nirik> we are currently waiting for hardware still, but I think we have close to the config we want speced out.
18:23:32 <codemaniac> nirik: you have already finalized the budget and initial designs right , t
18:24:02 <codemaniac> what exactly the cloud is going to host ?
18:24:05 <nirik> codemaniac: yeah, for the initial part of it. We can hopefully expand it next year if it works out. ;)
18:24:11 <skvidal> codemaniac: read the page - it explains
18:24:12 <nirik> see the wiki page...
18:24:52 <nirik> so, I think it will be very handy once we have it in place.
18:24:56 <ctria> just curiosity, can't we have any cloud solution from red hat or this is what red hat proposes?
18:25:13 <nirik> as we get closer to deploying it, we will no doubt want lots of people to help test it and such.
18:25:22 <herlo> nirik: who's doing the work? I'd love to help. I should have some time in the next couple of weeks to ramp up
18:25:31 <nirik> ctria: we evaluated a number of cloud tech, and found eucalyptus to meet our needs.
18:25:52 <nirik> herlo: skvidal is point on it... we are kinda holding on hardware tho.
18:25:59 <skvidal> hi
18:26:13 <herlo> nirik: skvidal: okay, just ping me when you are ready
18:26:14 <skvidal> we have a test cluster running. it doesn't do a lot b/c of the lack of ips to use w/it
18:26:27 <herlo> nod
18:26:44 <nirik> yeah. hard to use when you can't get to it from outside easily.
18:26:51 <codemaniac> me too in , would like to offer whatever i can
18:27:23 <nirik> #info waiting for hardware, will move forward once thats done.
18:27:24 <herlo> what will it take to get more IPs? Or do we need to free some up elsewhere?
18:27:53 <nirik> the new hardware will have a pile of external ips for us to use.
18:28:01 <herlo> okay, great
18:28:14 * herlo is done asking questions now
18:29:25 <nirik> cool...
18:29:28 <nirik> #topic Open floor
18:29:42 <nirik> anyone have items for open floor? general questions? etc
18:29:59 <skvidal> global updates
18:30:14 <skvidal> should I go ahead and just shoot a yum update on most everything?
18:30:59 <nirik> yeah, I think so.
18:31:02 <abadger1999> +1
18:31:08 <nirik> catch back up after the freeze.
18:31:10 <skvidal> ok
18:31:37 <nirik> I don't think we need reboots tho. The last update kernel fixes several security issues, but not sure how much they apply to our setup
18:32:04 <skvidal> second thing
18:32:12 <skvidal> builders and koji
18:32:25 <skvidal> I'd like to take buildvm01 and 02 out of koji for a while
18:32:31 <dgilmore> skvidal: ok
18:32:31 <skvidal> so I can mess with them
18:32:44 <nirik> sounds fine to me. Not much build activity right now anyhow.
18:32:56 <skvidal> and I'd like to find out about more migration to having more builders be vm-based
18:33:32 <nirik> what were the next steps in that plan?
18:33:53 <skvidal> 1. we need a way of telling koji that some pkgs need A LOT of ram
18:34:13 <skvidal> 2. we need to convert more of the x86-## boxes to buildvmhost
18:34:33 <dgilmore> skvidal: we have no way today of knowing that
18:34:50 <codemaniac> A lame question , what is koji ?
18:34:52 <nirik> dgilmore: won't we need that for arm anyhow? and seed with their list?
18:34:58 <misc> codemaniac: the build hub
18:35:00 <nirik> codemaniac: not lame at all. ;) Our buildsystem...
18:35:11 <dgilmore> skvidal: for arm we have a seperate channel called heavybuilder, and it has a list of packages in the koji hub policy that it sends to that channel
18:35:13 <ctria> is there a list of packages to blame for LOT ram?
18:35:16 <dgilmore> but its not an automatic thing
18:35:23 <dgilmore> its a manually maintained list
18:35:25 <skvidal> dgilmore: oh - so we can't do that?
18:35:28 <skvidal> ctria: more or less
18:35:40 <nirik> http://fedorahosted.org/koji and http://koji.fedoraproject.org/koji/
18:35:41 <ctria> then we could probably create a new channel and redirect them there
18:35:44 <misc> codemaniac: https://fedoraproject.org/wiki/Using_the_Koji_build_system should answer the questions do not hesitate to ask for clarification
18:35:48 <dgilmore> skvidal: we can but its prone to failure  and will need constant tweaking
18:35:56 <ctria> in the new channel to have better specs for builders
18:36:07 <skvidal> dgilmore: having all of our hosts have 8GB+ ram seems.... like poor resource use
18:36:13 <nirik> dgilmore: does changing that need a hub restart?
18:36:16 <codemaniac> thanks misc, nirik .Sure i will ask
18:36:20 <dgilmore> nirik: it does
18:36:20 <skvidal> dgilmore: since 95% of our pkgs don't need even 2GB of ram
18:36:25 <nirik> thats a pain.
18:36:25 <dgilmore> skvidal: right
18:36:39 <herlo> skvidal: +1 to not using a hammer for all builders
18:36:41 <skvidal> I suspect the largermem pkgs are probably under 50
18:36:46 <nirik> wonder if we could ask upstream koji for a better fix... some on the fly thing.
18:37:06 <misc> and I think the list would not grow much over time
18:37:19 <dgilmore> nirik: well we could probably write the policy so that if the task load is above say 4  where  6 is the max it goes to a bigger builder
18:37:28 <herlo> I've run into similar issues, I think it's possible to have a koji setting for ram requirements.
18:37:43 <dgilmore> nirik: but there could be something under a load of 4 that uses lots of ram to say link
18:37:53 <nirik> yeah.
18:37:59 <dgilmore> the last load is dynamically worked out based on avaergae build completion time
18:38:05 <dgilmore> the lask load
18:38:10 <dgilmore> task
18:38:11 <herlo> lol
18:38:14 <misc> couldn't cgroup be used to track the ram used by a set of process ?
18:38:25 <dgilmore> misc: builders are all rhel6
18:38:53 <nirik> so, I guess we all ponder on the problem and try and come up with a good solution...
18:39:02 <skvidal> nirik: I think your suggestion is a good place to start
18:39:07 <skvidal> ask koji upstream for help
18:39:24 <misc> ok so can we track the memory on vm ?
18:39:25 <nirik> it would be nice to have a better solution for arm too...
18:39:33 <misc> I think that's how balooning does, no ?
18:39:36 <dgilmore> nirik: probably the best solution would be to have koji keep track of ram usage during builds and store that. then we could do things nicely
18:39:43 <dgilmore> but thats likely alot of work
18:39:43 <nirik> misc: the problem is that we get a package to build and we don't know how much mem it will need to build.
18:39:45 <skvidal> misc: you mean on the fly allocation of ram?
18:39:52 <misc> ( or maybe polling memory in the vm every minutes, to have some stats )
18:39:53 <herlo> +1 nirik's suggestion
18:40:00 <skvidal> misc: normally the event which tells us it is out of memory is that it OOMs
18:40:01 <nirik> dgilmore: hum... yeah.
18:40:26 <dgilmore> we can somewhat safely make assumptions based on previous builds
18:40:30 <nirik> or perhaps a FTBFS run could also tell us which packages need more mem and we set the list based on that?
18:40:46 <nirik> at least it would be a good seed.
18:41:07 <dgilmore> i suspect that having a task load of more than 4 would cover 95% of cases
18:41:11 <dgilmore> maybe 100%
18:41:12 <nirik> #info need a solution to find builds that need lots of memory.
18:41:13 <ctria> there is definition for package name in koji policies http://fedoraproject.org/wiki/Koji/Policies
18:41:22 <nirik> #idea ask koji developers upstream for a solution
18:41:44 <skvidal> nirik: hmm
18:42:04 <skvidal> nirik: so here's an idea of what i could do on buildvm##
18:42:14 <skvidal> it would take a while, though...
18:42:30 <skvidal> iteratively build every pkg on the 4gb they have available
18:42:37 <skvidal> and see which ones cause it to oom
18:42:53 <nirik> that would take quite a long while...
18:42:56 <dgilmore> channel = method createrepo :: use createrepo has req_channel :: req is_child_task :: parent source */gcc*  :: use heavybuilder source */glibc* :: use heavybuilder source */kernel* :: use heavybuilder
18:42:58 <skvidal> I know :)
18:43:07 <skvidal> dgilmore: -ENOCONTEXT
18:43:09 <dgilmore> thats a snippet of the policy use on arm
18:43:28 <nirik> once we have the cloud setup, we could fire up N builders with 4GB and mass rebuild... :)
18:43:28 <dgilmore> it came aross bas
18:43:44 <skvidal> nirik: so...
18:43:49 <nirik> ceph is apparently another one.
18:43:58 <nirik> webkitgtk is also a pig ;)
18:44:01 <skvidal> nirik: I could convert an existing x86-## box
18:44:05 <skvidal> nirik: to  buildvmhost
18:44:06 <dgilmore> skvidal: http://fpaste.org/QfFj/ thats the full current policy used on arm
18:44:16 <skvidal> nirik: and then i'd have 8+ builders to test with
18:45:00 <nirik> yeah, still will take a while, but less while. ;)
18:45:53 <skvidal> I'm open to other options...
18:45:56 <skvidal> considering our build load
18:46:03 <skvidal> I could probably rob 2 more machines w/o us losing out muchj
18:46:24 <nirik> I'm fine with that, but perhaps we should see if koji dev's have any brilliant ideas we are missing first?
18:46:24 <dgilmore> skvidal: sure. we really only hit the builders really hard for mass rebuilds
18:46:25 <skvidal> our peak afaict has been like 23% use
18:46:36 <skvidal> nirik: you're no fun at all ! :)
18:46:46 <nirik> I know, right? :)
18:46:52 <skvidal> nirik: I can bounce an email to buildsys list
18:46:58 <misc> dgilmore: technically, that's some form of mass rebuild, so maybe this can be done at the same time
18:47:20 <misc> ( or, if someone is gonna rebuild everything, this can produce a list of FBTS and a list of package that need more ram )
18:47:40 <dgilmore> nirik: ill talk with the mikes
18:48:07 <skvidal> dgilmore: even better :)
18:48:12 <nirik> works for me.
18:48:22 <nirik> #action dgilmore to talk to mikes about koji options.
18:48:39 <nirik> misc: we typically only do mass rebuilds when there's a good reason... rpm payload change, gcc change, etc.
18:48:40 <CodeBlock> agh, late, sorry
18:48:41 * CodeBlock here
18:49:08 <nirik> misc: but we hope to use the new cloud setup to mass rebuild to check the FTBFS stuff.
18:49:11 <skvidal> nirik: I think I'd like the opportunity to see if I can do a sensible port of ftbfs over to mockchain, etc
18:49:38 <nirik> yeah, would be good. keeping in mind hopefully that it would be portable to the privatecloud.
18:49:52 <skvidal> it should just be ssh-based
18:50:08 <skvidal> so the only difference is how you get the instances started
18:50:13 <skvidal> not how you spawn off jobs
18:50:26 <nirik> yeah.
18:50:45 <skvidal> question about ftbfs
18:50:55 <skvidal> does that build all rawhide srpms against rawhide?
18:51:20 <nirik> yep.
18:51:34 <nirik> we can ask mdomsch for his scripts.
18:51:55 <dgilmore> skvidal: i believe thats how mdomsch did it
18:51:58 <nirik> I think it was just basically: build everything once, get list of fails, rebuild, cycle until you have no changes.
18:52:19 <skvidal> nirik: nod
18:52:26 <skvidal> nirik: I was checking on which base it built from
18:52:30 <nirik> but there's issues with that too I guess.
18:52:32 <herlo> can someone expand ftbfs for me plz
18:52:32 <skvidal> ie: did he take a single rawhide snapshot
18:52:40 <skvidal> Fails To Build From Source
18:52:42 <nirik> herlo: fails to build from source.
18:52:45 <herlo> tx
18:52:49 <herlo> skvidal wins!
18:52:51 <nirik> ie, you get the src.rpm we have in git and it doesn't work.
18:53:06 <nirik> http://fedoraproject.org/wiki/Fails_to_build_from_source
18:53:25 <herlo> yeah, I have done something very similar with my koji project
18:53:44 <nirik> rawhide can be a challenge, since there's so many moving packages. ;(
18:53:50 * dgilmore needs to run
18:54:08 <skvidal> nirik: which is why I was thinking about taking a snapshot of rawhide into a local mirror
18:54:13 <skvidal> nirik: that all the builders can see
18:54:47 <nirik> you mean for the build repo/binary rpms? or the git repos building from?
18:54:51 <nirik> or both.
18:55:25 <skvidal> build repo/binary
18:55:37 <skvidal> so that what it is building against for the build time doesn't change
18:55:54 <nirik> yeah.
18:56:04 <nirik> although if we can get it down to under a day, it would be the same rawhide. ;)
18:56:18 <skvidal> I cannot fathom how many machines we would need for that
18:56:28 <nirik> large number...
18:56:32 <skvidal> dgilmore: can I convert x86-16,17, 18 over to buildvmhosts?
18:57:20 <nirik> skvidal: I think he had to run...
18:57:27 <skvidal> okay
18:57:34 <nirik> lets continue out of meeting on this...
18:57:36 <skvidal> ok
18:57:39 <skvidal> sorryt for running on
18:57:41 <nirik> anything else for meeting?
18:57:47 <nirik> no worries at all. Good to discuss.
18:58:06 <skvidal> one more item
18:58:26 <skvidal> does anyone else have any interest in ansible and/or tryting to write playbooks?
18:58:33 * herlo does
18:58:38 <herlo> just haven't had time yet
18:58:42 <skvidal> understood.
18:58:47 <herlo> will next week or two for sure
18:58:54 <skvidal> one last thing
18:58:58 <herlo> depending on what gets done with euca, I'd be up for it
18:59:01 <nirik> yeah, I'd like to look at ansible more too.
18:59:09 <skvidal> I didn't hear any horrible screaming about the ssh key thing last week
18:59:15 <skvidal> just some minor grumbling
18:59:27 <skvidal> so.... I'm going to go ahead and make a magic key and put it on lockbox01
18:59:44 <skvidal> and start playing with it on the builder installs I've been working on
18:59:51 <nirik> fair enough.
19:00:03 <nirik> it would only be those instances for now, right? until we expand ansible use?
19:00:09 <skvidal> exactly
19:00:12 <skvidal> nothing else will have it
19:00:23 <nirik> seems sane. Shake out any issues now
19:01:06 <nirik> #info will setup initial ansible ssh keys for testing with.
19:01:19 <skvidal> okay, I'm done with all my openfloor things
19:01:20 <nirik> ok, shall we call it a meeting?
19:01:23 <nirik> :)
19:01:36 <herlo> yup
19:01:40 <herlo> call it
19:01:43 <CodeBlock> I have one quick thing
19:02:00 <nirik> CodeBlock: go for it.
19:02:33 <CodeBlock> we talked about Stickynotes as a pastebin last week, I quickly made a Fedora theme mockup (showed nirik last week, but wanted to get some more voices about it)
19:02:33 <CodeBlock> http://images.srv1.elrod.me/stickynotes-fedora.png
19:02:44 <nirik> CodeBlock: oh, codemaniac was looking to see if he could help with dpsearch. If you catch him on-line, see if he can assist any
19:03:13 <nirik> I think it's a pretty good start... might need more blue. :)
19:03:14 <CodeBlock> nirik: cool, he's been trying to ping me and our schedules have been totally missing each other
19:03:48 <herlo> CodeBlock: oh, cool
19:03:52 * herlo looks
19:04:00 <nirik> #info provide feedback on sticky-notes theme: http://images.srv1.elrod.me/stickynotes-fedora.png
19:04:03 <herlo> oh, very nice!
19:04:30 <herlo> CodeBlock: I like the theme, I bet we could package that up pretty quickly
19:04:48 <herlo> ping me when you have a package and I'll do a review
19:05:02 <herlo> we can get that into the infra repo soon I hope.
19:05:02 <nirik> herlo: it's already up. ;) dcr226 has it up and I am reviewing.
19:05:07 <CodeBlock> herlo: Cool. It needs some coding work, I just wanted to get a working prototype, but I will continue working on it
19:05:09 <herlo> nirik: with the theme?
19:05:19 <herlo> I'm specifically talking about the theme
19:05:23 <nirik> herlo: oh, sorry, no, not the theme...
19:05:27 <nirik> yeah, would be good to get that in too.
19:05:28 <herlo> :) nw
19:05:42 <nirik> ok, calling it a meeting as we are over.
19:05:45 <herlo> nirik: yeah, knew about the stickynotes review
19:05:54 <herlo> CodeBlock: awesome.
19:05:54 <nirik> Lets all continue over on #fedora-admin and #fedora-noc and #fedora-apps.
19:05:58 <herlo> indeed
19:05:58 <nirik> thanks for coming everyone!
19:06:02 <nirik> #endmeeting