19:00:01 <nirik> #startmeeting Infrastructure (2012-02-16)
19:00:01 <zodbot> Meeting started Thu Feb 16 19:00:01 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <nirik> #topic Robot Roll Call
19:00:01 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge
19:00:08 <CodeBlock> here
19:01:30 * nirik waits for folks to arrive
19:01:32 * jac1bat0 here
19:01:54 * LoKoMurdoK here
19:02:23 <ianweller> crooooooooow
19:02:26 * pingou 
19:02:33 <StarBeast> here
19:02:39 * abadger1999 here
19:03:24 * skvidal is her
19:03:25 <skvidal> err
19:03:26 <skvidal> here
19:03:27 <skvidal> even
19:03:39 <nirik> ok, lets go ahead then...
19:03:42 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:03:55 <nirik> Any new folks? Or apprentices that would like to talk about tasks/issues/ideas?
19:04:09 <StarBeast> I am a new here.
19:04:39 <nirik> welcome StarBeast. Can you give a quick intro ?
19:06:07 <StarBeast> Yes. I am living in Switzerland, working at CERN (company that runs collider :) as a system engineer. Using linux for more that 12 years.
19:06:08 <nirik> I'll note here:
19:06:10 <nirik> http://fedoraproject.org/wiki/Infrastructure/GettingStarted
19:06:11 <nirik> and
19:06:20 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice
19:06:34 <nirik> StarBeast: excellent. Welcome.
19:06:51 <StarBeast> Interesting in sysadmin tasks and improving my Python knowlege.
19:07:10 <nirik> sounds good.
19:07:21 <nirik> Anyone else new? or shall we move along...
19:07:22 * x-ip is new too
19:07:31 <StarBeast> Now I am learning Puppet as well. Get a test puppet master in Amazon
19:07:59 <nirik> sounds good. We use puppet quite a bit...
19:08:12 <nirik> x-ip: care to introduce yourself/give a quick bit of background?
19:08:19 <x-ip> sure! :-)
19:08:23 <StarBeast> That is all for now. Waiting for interesting tasks. :) And get some ideas. :)
19:08:45 <smooge> crap sorry
19:09:00 <nirik> welcome smooge
19:09:47 <x-ip> I am Emiliano Dalla Verde Marcozzi (i mean this because i sent my intro to the mailling list as said at:https://fedoraproject.org/wiki/Infrastructure/GettingStarted). I live in Rosario, SAnta Fe, Argentina
19:09:59 <nirik> x-ip: welcome!
19:10:08 <x-ip> i'm a Python Developer, work as python developer in a logistic business ...
19:10:11 <x-ip> nirik, thanks! :D
19:10:29 <x-ip> i'm interested in contribute to Fedora and seems that the infraestructure group is my place :-)
19:10:32 <skvidal> x-ip: any particular code focus you have?
19:10:48 <skvidal> x-ip: webdev? infrastructure tooling?
19:10:53 <nirik> x-ip / StarBeast: Do hang out in #fedora-admin / #fedora-noc / #fedora-apps and ask questions, chime in on things, etc. We can get you pointed at some tasks if we haven't already.
19:10:57 <abadger1999> x-ip is working on some python scripting tasks with me right now.  Hopefully we'll get him working on bigger and bigger tasks soon.
19:11:01 <x-ip> yap skvidal , i like backend development, writing networking servers, shell utils, that kind of stuff
19:11:10 <skvidal> x-ip: hmmm
19:11:29 <skvidal> cool
19:11:30 <abadger1999> skvidal, x-ip: I have him working on the pkgdb-bugzailla sync script right now.. but I think if we have infrastructure tooling tasks, that might be even better.
19:11:48 <skvidal> abadger1999: I can think of a few items that could use some dev time, yah
19:12:00 <x-ip> ^^
19:12:03 <abadger1999> Excellent.
19:12:13 <nirik> sounds good. ;)
19:12:13 <pingou> the ssh key checker? or is that already being taken care of?
19:12:23 <nirik> jac1bat0 was going to work on that. ;)
19:12:27 <abadger1999> x-ip: After you finish the sync changes, look to skvidal for some more tasks ;-)
19:12:35 <pingou> oki :)
19:12:41 <x-ip> abadger1999, ok :-)
19:12:46 <nirik> ok, moving along then...
19:12:52 <skvidal> x-ip: I'm in#fedora-admin all the time
19:12:57 <nirik> #topic F17alpha freeze and tickets
19:13:17 <nirik> So, a reminder to everyone: We are in a pre-release freeze for f17 alpha currently.
19:13:26 <mdomsch> hi
19:13:46 <nirik> Anything in https://fedorahosted.org/fedora-infrastructure/browser/architecture/Environments.png in the pre-release section needs to have 2 +1's from sysadmin-main or releng.
19:13:52 <nirik> welcome mdomsch.
19:14:29 <nirik> .ticket 3136
19:14:31 <zodbot> nirik: #3136 (Fedora17 Alpha - new website) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3136
19:14:32 <nirik> .ticket 3137
19:14:35 <zodbot> nirik: #3137 (Fedora17 Alpha - Verify mirror space) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3137
19:14:36 <nirik> .ticket 3138
19:14:39 <zodbot> nirik: #3138 (Fedora17 Alpha - release day ticket) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3138
19:14:40 <nirik> .ticket 3139
19:14:43 <zodbot> nirik: #3139 (Fedora17 Alpha - verify permissions on content) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3139
19:14:51 <nirik> Those are our release tickets.
19:15:04 <nirik> we can't do several of them yet until alpha is almost ready. :)
19:15:39 <nirik> Any questions on alpha or the freeze?
19:16:02 <nirik> #topic 2 factor auth
19:16:24 <nirik> I was going to look into helping this along, but got sidetracked as usual. ;)
19:16:31 * abadger1999 also
19:16:34 <nirik> will see what we can do over the next week
19:16:43 <smooge> what needs to be moved on?
19:16:50 <abadger1999> So is the next step getting the cgi written and deployed?
19:16:58 <nirik> I think yeah, we need to write the cgi end...
19:17:03 <skvidal> we do
19:17:08 <nirik> and decide for sure on which pam thing we use
19:17:08 <skvidal> you asked me to look at that last week
19:17:10 <skvidal> I have not had time
19:17:18 <skvidal> any of the pam things should work afaict
19:17:23 <nirik> oh, did I? ok. ;)
19:17:35 <skvidal> now - if someone has more cycles to look at it
19:17:37 <skvidal> go for it
19:17:46 <nirik> yeah, just probibly should pick the one thats closest to what we need and most maintained and smallest.
19:17:58 <skvidal> it's really not that difficult a bit of code to write
19:18:16 <skvidal> you take the time + otp + username from the cgi
19:18:18 <nirik> I could probibly hack something up, but I worry about security sensitive code like that.
19:18:28 <skvidal> you pass it over to the yubikey  checker routine
19:18:35 <skvidal> and return 'yes, no or broken'
19:19:14 <nirik> yeah.
19:19:25 <nirik> I can try and poke at it, or anyone else who has time/cares. ;)
19:19:52 <nirik> #topic Staging reorg
19:20:07 <nirik> I think we have a way forward here, but we are in freeze, so we might need to wait until after that...
19:20:20 <abadger1999> So what's the plan for this?
19:20:39 * abadger1999 is unclear since most of what we discussed at fudcon can't happen without more budget.
19:20:40 <nirik> abadger1999: I think we decided to look at app modules with stg in them for stg and without in prod
19:21:17 <nirik> so, there would be a 'include bodhi$env::whatever' and we have a modules/bodhi/ and modules/bodhi-stg for the two different hosts
19:22:10 <abadger1999> Ah -- so going forward with the puppet side of the reorg; holding off on the other changes until later.
19:22:13 <nirik> abadger1999: so, basically this is moving all the *stg* machines over into the production branch, so we can kill the stg branch
19:22:39 <nirik> once thats done we can more easily look at which hosts we could spin up for specific stg testing or which we should keep around like they are now.
19:22:51 <abadger1999> <nod>
19:22:58 <nirik> once we have a private cloud, spinning up containers becomes much easier. ;)
19:23:32 <skvidal> getting them available from outside is... tricky, still
19:23:33 <skvidal> but doable
19:24:13 <nirik> yeah...
19:24:24 <nirik> #topic Applications status / discussion
19:24:37 <nirik> lmacken / threebean / abadger1999 / pingou: any app news?
19:25:10 <abadger1999> new python-fedora went out on monday and we seem to have survived
19:25:27 <abadger1999> If we have any authentication problems though (esp. with tagger) let me know
19:25:47 <pingou> I have been working on gathering easyfix tickets in a simple/single page http://ambre.pingoured.fr/public/easyfix.html
19:25:49 <nirik> I have a bit: I made a pair of packages instances to put packages/tagger on for production. We need to move the config over from staging and get them setup. ;)
19:26:15 <abadger1999> ah yeah -- pingou's news is big.
19:26:23 <nirik> thats nice looking pingou
19:26:40 <pingou> nirik: fedoraproject's source code revised :)
19:26:54 <pingou> so for the design thanks mizmo :)
19:27:10 <nirik> I enabled xml for infrastructure trac. You can add it there now.
19:27:19 <pingou> https://fedoraproject.org/wiki/User:Pingou/easyfix
19:27:33 <pingou> we'll need to move this page at some point, but so far I put it there
19:27:41 <abadger1999> it means that we can stop having an easyfix ticket for app issues in fedora-infrastructure's repository -- instead, we can use the easyfix.html page to show people all the easyfix tickets.
19:27:58 <nirik> yeah
19:28:09 <pingou> I think we should find it a nice place
19:28:14 <pingou> and put some links to it :)
19:29:07 <pingou> only it requires jinja2 which might not be there by default
19:29:38 <nirik> yeah, possibly just on proxies/app servers? or we can figure something out.
19:30:08 <abadger1999> python-jinja2-2.2.1-1.el5.i386
19:30:27 <abadger1999> on bapp01 which is probably where we'd run this and then rsync to the proxies.
19:30:37 <pingou> same version on el6 where I run it, so should work :)
19:30:39 <abadger1999> (at least, I guess that's what we'd do)
19:31:01 <nirik> yeah, sounds sane... we would need it packaged, etc.
19:31:38 <pingou> nirik: I added you as contact for the fedora-infrastructure
19:31:44 <nirik> pingou: thanks
19:32:09 <nirik> ok, any other application news ?
19:32:10 <pingou> F5
19:32:35 <nirik> #topic Upcoming Tasks/Items
19:32:48 <nirik> #info 2012-02-14 to 2012-02-28 - F17 Alpha Freeze
19:32:56 <nirik> #info 2012-02-28 - F17alpha release day
19:33:10 <nirik> nothing else really around in the short term.
19:33:20 <nirik> anything anyone would like to schedule?
19:34:07 <nirik> ok, will toss this one in...
19:34:11 <nirik> #topic Search Engine
19:34:21 <nirik> CodeBlock has been looking at search engines for us...
19:34:36 <nirik> and we think datapark search will possibly work out for us. :)
19:35:05 <nirik> so, he's going to get it packaged up, and then we will setup a instance and see how it works out.
19:35:16 <CodeBlock> indeed, I think I have an RPM ready to be reviewed (maybe possibly)
19:35:33 <nirik> cool.
19:35:33 <pingou> CodeBlock: fedora-review it ;-)
19:35:39 * nirik is happy to review it.
19:36:07 <nirik> I think at worst we have something better than the horrible mediawiki search.
19:36:12 <CodeBlock> nirik: will find out what I have to do to get it reviewable :)
19:36:20 <nirik> at best we have something that indexes all out stuff and does a good job.
19:36:54 <nirik> anyhow, if there's any concerns or issues, please bring them up on list.
19:36:57 <nirik> #topic Open Floor
19:37:07 <nirik> any open floor items/ideas/jokes?
19:37:38 <mdomsch> !
19:37:44 <mdomsch> S3 mirror
19:37:46 <nirik> mdomsch: fire away.
19:37:55 <mdomsch> https://fedoraproject.org/wiki/Infrastructure/Mirroring/Amazon
19:38:40 <nirik> cool. more info there than last I looked.
19:39:27 <mdomsch> question is, can we run this out of FI
19:39:39 <abadger1999> mdomsch: Cool.  Do we want to talk to spevack about this?
19:39:43 <nirik> so, what would we need to do that?
19:40:14 <nirik> a place to run it I guess and a account that is billed?
19:40:31 <mdomsch> I have the billing info now
19:40:54 <mdomsch> separate app servers for it, or run the s3cmd sync job on bapp01?
19:41:07 <nirik> if we have to pay, I anticipate that figuring out the billing will be more complex than all the other issues combined. ;)
19:41:41 <mdomsch> I have the billing part nailed
19:41:46 <nirik> ok, cool.
19:42:04 <nirik> so, I'd think bapp01 is fine as long as it doesn't push it overloaded
19:42:10 <mdomsch> we need to be able to run s3cmd sync on a server somewhere, with access to the netapp, outbound bandwidth to S3, * 7 regions
19:42:15 <nirik> or could it just be in the releng push stuff?
19:42:19 <mdomsch> sure
19:42:25 <mdomsch> releng would work too
19:42:34 <mdomsch> that's why I'm askng - I don't know the right place to run it
19:42:35 <nirik> as long as it's not holding up pushes...
19:42:43 * nirik ponders
19:43:00 <abadger1999> async to the push sounds better.
19:43:15 <smooge> how does s3cmd run over (port wise)
19:43:35 <mdomsch> https
19:44:48 <nirik> this is just updates? or everything or everything + rawhide?
19:45:09 <mdomsch> Everything, update,s rawhide, epel
19:45:41 <mdomsch> ~200
19:45:42 <mdomsch> GB
19:45:50 <smooge> uhm a bit more than that
19:46:32 <nirik> ok. I would have to look, but i would think it would be ok to run on a releng box... or possibly secondary01 I suppose.
19:47:33 <nirik> we could check with dgilmore once he's back on-line.
19:47:52 <nirik> mdomsch: so, I'd say lets figure that for sure, and set it up? (or possibly after the freeze)
19:48:47 <mdomsch> ok
19:49:33 <nirik> at least I see no problems with doing so. ;) Will help out ec2 fedora folks, seems like a win to me.
19:50:28 <nirik> any other open floor stuff?
19:50:36 <StarBeast> Acctually I got some idea, might be a bit silly :). I noticed at #fedora-noc that some nodes sometimes getting a swap out :) But I did not noticed that you are using any analisys tools (I might be wrong). Therefore it is difficult to say how often is happens and why. Nagios is for different tasks.
19:51:36 <nirik> StarBeast: you mean the nagios alerts?
19:51:41 <StarBeast> I am talking about something like rrd-style graphs.
19:51:59 <nirik> the issue is that our noc02 instance has network issues to the rest of our stuff from time to time. ;(
19:52:22 <nirik> the best thing we could fix here would be to monitor a router or gateway and collapse those all into a single alert.
19:53:22 <StarBeast> nirik: I am thinking about some tool that stores history of load avg, swap utilization etc.. in rrd
19:53:32 <nirik> StarBeast: we do have that. ;) Collectd.
19:53:47 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=hosted03&plugin=load&timespan=3600&action=show_selection&ok_button=OK
19:54:01 <StarBeast> Aah... :) I did not noticed it :) Sorry.
19:54:46 <nirik> no problems... some of these things are not very well documented. ;(
19:54:59 <StarBeast> In this case, it is fine.
19:55:32 <nirik> ok, if nothing more will close out the meeting in a minute here.
19:55:43 <StarBeast> We are using Lemon at CERN. Very useful. http://lemon.web.cern.ch/lemon/screenshots.shtml
19:56:01 <nirik> huh. haven't seen that one before.
19:56:23 <nirik> thanks for the info StarBeast
19:56:26 <StarBeast> I guess it is a homemade :)
19:56:37 <StarBeast> But it is an opensource
19:56:43 <nirik> nice.
19:57:12 <nirik> Thanks for coming everyone. ;)
19:57:15 <nirik> #endmeeting