18:00:00 <nirik> #startmeeting Infrastructure (2012-06-14)
18:00:00 <zodbot> Meeting started Thu Jun 14 18:00:00 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic Good morning Fedora
18:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:11 * mdomsch is here
18:00:20 <nirik> who all is around for a exciting, thrilling, wonderous, fedora infrastructure meeting?
18:00:21 * skvidal is
18:00:23 * lmacken 
18:00:27 * rossdylan is here
18:00:28 * ingm4r is
18:00:31 * threebean is here
18:00:48 <sdrfed17> Hi Team. I am Sudhir Menon from India (irc:sdrfed17). with 3yrs of experience in Linux Sys Administration and QA. Would like to contribute to Infrastructure Group.
18:01:02 <nirik> welcome sdrfed17
18:01:28 * ingm4r would like to join the team, too
18:01:33 <sdrfed17> thankyou nirik
18:01:34 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:35 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:01:35 <nirik> would like to ask general questions, they can do so now. Anyone?
18:01:56 <sumitrai> Hi everone I am sumit rai from India, (irc: sumitrai), I have RHCSA, and I would love to be a part of fedora community
18:01:57 <nirik> sdrfed17 / ingm4r: were you more interested in sysadmin tasks? or application development?
18:02:16 <sdrfed17> sysadmin task niik
18:02:41 <ingm4r> Short Line from me: My name is Ingmar (I'm from Germany), I'm working as a Sysadmin since ~5 years
18:02:55 <nirik> excellent. Lots of new folks today. ;)
18:03:02 <ingm4r> so I\m interested in sysadmin tasks, too :)
18:03:02 <sdrfed17> sysadmin task is the thing that i am more interested in, would also like to have my hands on application development as well
18:03:23 <sumitrai> I am interested in sysadmin task too.
18:03:26 <nirik> For the sysadmin side of things, take a look at https://fedoraproject.org/wiki/Infrastructure_Apprentice and if that sounds of interest to you, I can set you up after the meeting (see me in #fedora-admin)
18:03:49 <nirik> for application development, we have a number of apps we work on, and there's a list of easyfix items to look at:
18:03:58 <nirik> http://fedoraproject.org/easyfix/
18:04:58 <nirik> so, we can get you all setup after the meeting. ;)
18:05:06 <nirik> any general questions right now?
18:05:15 <ingm4r> will do nirik
18:05:21 <ingm4r> not yet :)
18:05:40 <sdrfed17> looks good to me the apprentice part
18:05:42 <sdrfed17> nirik
18:06:27 <nirik> great. I can get you setup after the meeting. ;)
18:06:41 <ingm4r> that would be fine.
18:06:41 <nirik> do chime in with questions and comments as they come to you, and again welcome.
18:07:15 <sdrfed17> thank you we will be joining you @ fedora-admin after this meeting
18:07:20 <nirik> #topic Applications status / discussion
18:07:36 <nirik> abadger1999 / threebean / lmacken / pingou / relrod: any application news this week?
18:08:05 <abadger1999> nirik: Sorta sysadminy -- we're about to retire app01.dev :-)
18:08:06 <lmacken> nothing exciting
18:08:17 <nirik> abadger1999: hurray.
18:08:19 <abadger1999> the apps that were being tested on it have moved to pkgdb01.dev and fas01.dev
18:08:46 <nirik> is everyone happy with how staging works these days? is it better than when we had a staging branch?
18:08:46 <abadger1999> in the process, I updated them to use passwordless sudo and made the login/sudo group the commit group for the applications
18:09:05 <abadger1999> which were things we'd talked about migrating our dev boxes to do.
18:09:09 <abadger1999> Seems to be working out fine.
18:09:28 <pingou> worked a bit on HK and our student seems to make some progress as well
18:09:34 <nirik> we did have a upgrade to koji this week. ;)
18:09:46 <pingou> nirik: btw, what about python-bz ? any news ?
18:09:48 <nirik> pingou: cool.
18:09:57 <abadger1999> I think lmacken hit his first stg-was-nonintuitive issue this week (or last week)
18:10:17 <dgilmore> nirik: i have found one bug in koji i need to get fixed
18:10:31 <abadger1999> someone else had added an explicit stg module for something and then we couldn't figure out why committing to master wasn't showing up on stg.
18:10:40 <abadger1999> (in the modules/ directory)
18:10:47 <lmacken> abadger1999: yup, that was confusing at first.
18:10:59 <nirik> pingou: there is a 0.7.0 version. We should retest our stuff with it.
18:11:01 <abadger1999> Not sure if we can do anything about that except remember to check for that.
18:11:09 <nirik> abadger1999: ah yeah. I wonder if there's anything we can do about that... .
18:11:39 <pingou> nirik: release?
18:11:41 <pingou> in testing ?
18:12:06 <nirik> pingou: packages in fedora updates-testing. Looks like it's not been built for epel yet.
18:12:14 <pingou> ok
18:12:21 * pingou notes this on his todo
18:12:22 <nirik> I can do a scratch build if anyone wants
18:13:00 <nirik> ok, cool.
18:13:10 <nirik> dgilmore: what was the bug? in login?
18:14:05 <nirik> #topic Sysadmin status / discussion
18:14:06 <dgilmore> nirik: yeah the web login issue
18:14:15 <dgilmore> its adding a :80 to the url
18:14:24 <nirik> I thought I'd add a section about what sysadminy things we have done over the last week too.
18:14:26 <dgilmore> https:// doesnt work well on port 80
18:14:29 <nirik> dgilmore: ah, bummer. ;(
18:14:43 <nirik> we just finished last night a mass reboot... so everything should be up to date.
18:15:05 <nirik> skvidal revamped out dns this week. :) Please read the readme in the dns repo
18:15:26 <skvidal> if anyone wants to updats our dns SOP
18:15:28 <mdomsch> skvidal: that's huge
18:15:37 <skvidal> to point to the readme in the dns git repo
18:15:38 <mdomsch> thanks for your effort there
18:15:40 <skvidal> please feel free
18:15:52 <skvidal> mdomsch: thanks for that - I hope it will make us all less pained when it comes to proxy time
18:15:54 * nirik can do that.
18:15:59 <skvidal> s/proxy/proxy rotation/
18:16:04 <nirik> yeah, I think it's less error prone for proxys.
18:16:10 <nirik> thanks for working on it skvidal
18:16:13 <skvidal> and it should be less error prone in general
18:16:21 <skvidal> it is VERY hard to get an invalid zone file past it
18:16:36 <skvidal> I hope
18:16:37 <skvidal> :)
18:16:41 <nirik> we also wiped out community01.dev and made a packages01.stg, which I think is mostly working now.
18:16:52 <mdomsch> I've got the S3 mirrors functional in 3 zones now (us-east-1, us-west-1, and us-west-2)
18:16:55 <nirik> yeah, which is something that had happened to us in the past. ;(
18:17:11 <mdomsch> and spent another few nights trying to beat hardlink handling into s3cmd sync
18:17:38 <nirik> mdomsch: any luck with that?
18:17:41 <mdomsch> once that's working, need to parallelize it on 2 dimensions: 1) multiple uploads per upload target
18:17:56 <mdomsch> 2) scan the local file system once, then multiple upload targets in parallel
18:18:06 <mdomsch> as it stands, we're beating the crap out of the netapps on every sync
18:18:18 <mdomsch> as it calculates md5sums on each file before checking in with S3 to see if it has it
18:18:46 <mdomsch> which is the last thing it needs right now - local tree md5sum caching
18:18:53 <nirik> perhaps we could pregenerate that? or get it from the repodata?
18:19:02 <nirik> does it have to be md5?
18:19:20 <mdomsch> unfortunately, yes, md5 only.  S3 returns that as the ETAG
18:19:35 <skvidal> mdomsch: can you generate an md5sum file and go off of datestamp?
18:19:49 <skvidal> mdomsch: b/c datestamp should be a simple stat() hit and not a full file read like md5sum
18:19:54 <mdomsch> though I did just add stashing the md5 in the S3 per-file metadata, so could conceivably add another hash type
18:20:18 <skvidal> mdomsch: then you can assume the md5sum is the same, if the datestamp on the file is older than the last time you ran
18:20:33 <skvidal> (unless someone intentionally set the file mtime/ctime back)
18:21:01 <mdomsch> maybe....
18:21:11 <mdomsch> definitely open to ideas to speed things up
18:21:39 <mdomsch> problem is, mtime/ctime is an easy stat() call locally, but it requires a full HTTP HEAD call for each target remote
18:21:48 <skvidal> mdomsch: you don't need to compare it to remote
18:21:48 <mdomsch> to get it out of the metadata
18:21:54 <skvidal> mdomsch: you just compare it to the last time you ran
18:21:57 <mdomsch> MD5 we get "for free" from the bucket list command
18:22:12 <skvidal> any file with a timestamp > than the last time you ran
18:22:15 <skvidal> you take an md5 of
18:22:25 <skvidal> s/file/local file/
18:22:31 <skvidal> that way you're not hitting EVERY file on the netapp
18:22:38 <skvidal> only those newer than the last execution of your script
18:22:40 <mdomsch> skvidal: ah, yes
18:22:45 <skvidal> and when you're done
18:22:50 <skvidal> you store the md5sum of that file
18:22:56 <skvidal> so - if you need it for any reason
18:22:58 <skvidal> you have it
18:23:05 <mdomsch> yes, that's completely feasible
18:23:05 <skvidal> w/o rereading it from the file itself
18:23:47 <nirik> yeah
18:23:52 <mdomsch> that's exactly in line with what I was thinking
18:24:30 <skvidal> cool
18:24:31 <nirik> cool. Sounds like a number of optimizations possible...
18:24:40 <nirik> ok, moving on?
18:24:46 <mdomsch> so, if there are any new folks
18:24:49 <mdomsch> apprentices etc
18:24:59 <mdomsch> who know python and have time to monkey with it
18:25:06 <mdomsch> I'm very open to the help...
18:25:13 <nirik> excellent.
18:25:26 <nirik> #info assistance with s3 mirroring welcome.
18:25:39 <nirik> any other sysadmin stuff to note from this last week?
18:26:18 <skvidal> the bind managed-keys crap?
18:26:27 <skvidal> if anyone is familiar with named and dnssec
18:26:47 <skvidal> and can figure out why on every startup named belches out that it cannot find some managed-keys in dynamic/<long string>
18:26:55 <skvidal> i would be OVERJOYED to see a solution
18:27:11 <nirik> #info help welcome to track down managed-keys dns warnings
18:27:11 <skvidal> grep for managed-keys in the messages log of any of the nameservers
18:27:13 <skvidal> and you can see
18:27:49 <nirik> yeah, it's an odd one. ;(
18:28:34 <skvidal> an named epylog module
18:28:37 <skvidal> if anyone wants to write one
18:28:47 <skvidal> I'm sure we'd be happy to be a tester of it
18:29:22 <nirik> #info epylog named module welcome to parse named logs.
18:29:33 * nirik should file some of these for apprentice folks. ;)
18:29:53 <nirik> #topic FAD ?
18:30:18 <nirik> So, I sent out an email the other day to judge interest in holding a FAD (Fedora Activity Day).
18:30:27 <nirik> sounds like there is some interest.
18:30:38 <nirik> we need to try and isolate place and time and see who all can make it.
18:30:40 <pingou> I bet there is :)
18:31:08 <nirik> so, what I might do is make a wiki page, and ask people to sign up there and note their place/time prefs.
18:31:24 <nirik> and possibly ballpark costs of flying them to place X or something.
18:31:46 <skvidal> nirik: you know - fudcon in paris - we could colocate a fad w/that
18:31:51 <nirik> anyone have any further thoughts/ideas on this? is security a good topic?
18:31:56 <pingou> skvidal: +1
18:31:56 <skvidal> nirik: I'm sure I could convince eunice that we need to go to paris in the fall.
18:32:07 <nirik> skvidal: ha. yeah!
18:32:16 <pingou> but we should still be able to do one before if we like
18:33:06 <pingou> we should be able to get a room there
18:33:10 <nirik> If folks know of spaces that would be very low cost/free for us to gather at, we could consider them too.
18:33:13 <mdomsch> doubtful I could attend or be of much value for a security-focused FAD
18:33:23 <skvidal> mdomsch: you're always useful
18:33:53 <mdomsch> I think security is too broad though.  I'd like to see a "we will accomplish 1, 2, and with a lot of luck, 3, in 2 days"
18:34:05 <nirik> mdomsch: you're always welcome. ;)
18:34:06 <nirik> yeah...
18:34:10 <mdomsch> I love the ideas on the list so far
18:34:31 <mdomsch> just trim it down to something achievable with a few people who can Get It Done
18:34:53 * pingou wonders about a webapp component
18:34:53 <nirik> yeah, I listed a bunch of possible things...
18:34:58 <pingou> but then it would be 2 groups
18:35:07 <nirik> I think the list is too long to get done all at once there.
18:35:16 <abadger1999> nirik: Smooge had the idea of just getting two-factor auth done.
18:35:30 <abadger1999> That seemed like it was a good focus for a FAD.
18:35:34 <pingou> imho 2 factor should have the priority
18:35:43 <nirik> we might want to focus on things that we could either a) be confident of getting done, b) need to discuss in person more to come up with a plan.
18:35:47 <nirik> abadger1999: yeah.
18:37:06 <nirik> well, I will see about whipping up a web page where we can collect costs and time/place prefs.
18:37:16 <nirik> and we can narrow scope down
18:37:54 <nirik> #action nirik will make a web page to collect possible attendees, flight costs and location / time prefs.
18:38:14 <nirik> #topic Upcoming Tasks/Items
18:38:23 <nirik> #topic Upcoming Tasks/Items
18:38:24 <nirik> #topic 2012-06-18 remove people with pkgdb bugzilla issues.
18:38:24 <nirik> #topic 2012-06-21 to 2012-07-04 Kevin is off on trains and boats.
18:38:24 <nirik> #topic 2012-06-26 Fedora 15 end of life.
18:38:24 <nirik> #topic 2012-06-28 Seth at jury duty.
18:38:24 <nirik> #topic 2012-07-05 nag fi-apprentices
18:38:26 <nirik> #topic 2012-07-12 drop inactive apprentices.
18:38:28 <nirik> #topic 2012-08-07 to 2012-08-21 F18 Alpha Freeze
18:38:30 <nirik> #topic 2012-08-21 F18 Alpha release.
18:38:32 <nirik> ugh.
18:38:34 <nirik> misskey
18:38:36 <nirik> oh well.
18:38:38 <nirik> lots of topics. ;)
18:38:43 <nirik> (those were supposed to be infos)
18:38:49 <striker|rh> holy cow
18:39:01 <ingm4r> DOS...to get back into security :)
18:39:30 <nirik> #topic Upcoming Tasks/Items
18:39:30 <nirik> #info 2012-06-18 remove people with pkgdb bugzilla issues.
18:39:30 <nirik> #info 2012-06-21 to 2012-07-04 Kevin is off on trains and boats.
18:39:30 <nirik> #info 2012-06-26 Fedora 15 end of life.
18:39:30 <nirik> #info 2012-06-28 Seth at jury duty.
18:39:30 <nirik> #info 2012-07-05 nag fi-apprentices
18:39:32 <nirik> #info 2012-07-12 drop inactive apprentices.
18:39:34 <nirik> #info 2012-08-07 to 2012-08-21 F18 Alpha Freeze
18:39:36 <nirik> #info 2012-08-21 F18 Alpha release.
18:39:46 <nirik> anyhow, as noted there, I will be gone the next two meetings. ;)
18:40:06 <nirik> if anyone needs anything from me, please ask me to do it before next thursday.
18:40:18 <skvidal> nirik: I need you to not be gone, kthx
18:40:23 <skvidal> nirik: :)
18:40:30 <nirik> does anyone have any other upcoming tasks or things they would like to note on the schedule?
18:40:49 <nirik> skvidal: working on it. ;) Looking forward to be sitting on the train reading a book looking out the window. ;)
18:41:32 <skvidal> hrmph
18:41:36 <nirik> Oh, our private cloud hardware is supposedly in the datacenter somewhere. We just need it to be located and racked and wired and we can start setting it up.
18:41:44 <skvidal> nirik: and the networking setup
18:42:08 <nirik> yeah.
18:42:21 <nirik> I'm not sure if thats just one switch or two.
18:42:44 <skvidal> and of course whatever it means for ips
18:42:56 <nirik> yeah. we do have an external class C ready for this. ;)
18:43:22 <relrod> sorry, wayyyyy late, but I'm here.
18:43:31 <nirik> oh, also, we are hopefully getting a osuosl02 box... will be good to have 2 machines there so we can HA them or whatever we need.
18:45:16 <nirik> #topic Open Floor
18:45:26 <nirik> any questions, comments, ideas for open floor/
18:46:10 <relrod> well since I missed the app discussion, quick update on fedorahosted automation app stuff
18:46:18 <rossdylan> Fedora badges is coming along pretty well, working on building the nessicary rpm's of the python modules i have been working on
18:46:31 <nirik> relrod: sure...
18:46:35 <nirik> rossdylan: cool.
18:47:20 * nirik noted the ubuntu badges thing thats incompatible with open badges had a 0.2 release the other day.
18:47:46 <relrod> The web side of the fedorahosted automation app is pretty much done, and the CLI I'd say is 75% done. The CLI (at least to the point where I can test it locally) can fully process git requests and Hg requests. I need to get it processing bzr and svn.
18:48:33 <nirik> relrod: how much pain would it be if we had a hosted-agilo01 instance that was just for projects that needed agilo trac plugin/
18:48:33 <relrod> Flask still isn't packaged for el6 yet though. The maintainer is having some issues with the Flask tests not passing on el6
18:49:23 <nirik> ah
18:49:38 <relrod> nirik: Probably not too much pain, you'd just run the CLI on -agilo01 instead of hostedXX
18:49:46 <nirik> ok
18:50:22 <skvidal> item: cgit vs gitweb-caching?
18:50:40 <skvidal> did we come to a conclusion there?
18:50:42 <nirik> #topic cgit?
18:51:05 <nirik> not that I know of. I was going to ask gnome.org folks what they thought of cgit (since they use it there)
18:51:13 <nirik> but I didn't get around to it.
18:51:22 <skvidal> I got an internal email
18:51:25 <skvidal> on the subject
18:51:27 <skvidal> which said
18:51:39 <skvidal> 'cgit is much better'
18:51:42 <skvidal> (more or less)
18:51:51 <nirik> yeah... from looking it seems that way to me.
18:52:04 <nirik> so, I'm fine moving to it.
18:52:17 <nirik> the main downside is broken links.
18:52:20 <abadger1999> +1 for cgit from me.
18:52:24 <nirik> but there's some redirect rules that could help.
18:52:41 <abadger1999> But I've never been as concerned about the broken links as other people.
18:53:06 <nirik> yeah, it doesn't worry me overly. I don't think those links are used much...
18:53:14 <skvidal> abadger1999: I think I am inline with that now
18:53:17 <nirik> if someone hits an old bug with a gitweb link, too bad.
18:53:18 <skvidal> I used to worry about the links
18:53:20 <skvidal> but screw it
18:53:25 <skvidal> it's just how things fall down sometimes
18:54:10 <nirik> I'd be ok adding the redirects to try and make it somewhat nicer, or if we want just try and redirect all those gitweb things to a page that explains we are using cgit and how to search for what they were looking for.
18:54:56 <skvidal> worksforme
18:55:03 <skvidal> oh - I have another sysadmin-y task
18:55:04 <nirik> so, does someone want to lead this? if not, I can add it to my list. ;)
18:55:06 <skvidal> that is a touch herculean
18:55:17 * nirik notes we could test cgit on hosted01/02
18:55:32 <skvidal> nirik: might be easier to test cgit on fedorapeople
18:55:43 <skvidal> nirik: then again maybe those ~ repos will be tricky on fedorapeople
18:56:04 <nirik> there's also pkgs01.stg
18:56:11 <skvidal> nod
18:56:14 <nirik> anyhow, you had another topic?
18:56:30 <skvidal> yah
18:56:32 <skvidal> so
18:56:41 <skvidal> our httpd::websites, etc module in puppet
18:56:42 <skvidal> is
18:56:44 <skvidal> to say the least
18:56:45 <skvidal> complicated
18:56:57 <skvidal> a while back when we moved infra.fp.o to be standalone
18:57:04 <skvidal> I wrote a new httpd::site class
18:57:16 <skvidal> which simplifies how websites can be setup in puppet
18:57:20 <skvidal> it doesn't involve any templates
18:57:28 <skvidal> and makes me less likely to scream
18:57:43 <nirik> yeah, +1 on that
18:57:45 <skvidal> so
18:57:49 <skvidal> we need to move to that more
18:57:51 <nirik> #topic http::websites
18:58:01 <nirik> yeah, fine with me.
18:58:03 <skvidal> we need to convert sites  over and whittle our way off of the other one
18:58:08 <skvidal> just takes people
18:58:17 <nirik> yeah.
18:58:26 <skvidal> another item that is on my todolist but...
18:58:28 <nirik> and getting puppet to do the right thing.
18:58:29 <skvidal> well it's a todolist from hell
18:58:38 <skvidal> iptables templates
18:59:08 <skvidal> my plan is to break iptables templates up into stg/prod templates
18:59:19 <skvidal> this separation is mainly to make sure we keep stg from talking to prod
18:59:40 <nirik> sure... and possibly "untrusted vpn" ?
18:59:41 <skvidal> the idea is for the template to use the heredoc trick
19:00:00 <skvidal> so we have a standard preamble
19:00:14 <skvidal> then if iptables.$iptables_group for that node exists - it gets include
19:00:23 <skvidal> and if iptables.$iptables_datacenter exists - that gets included
19:00:37 <skvidal> and if iptables.$fqdn exits - that gets included
19:00:45 <skvidal> (actually reverse the first two
19:00:49 <skvidal> datacenter, group, fqdn
19:01:01 <nirik> yep. just like the other sane places you already converted to that. ;)
19:01:13 <skvidal> so that we end up being able to add arbitrary rules, per host or per group of hosts (or per datacenter)
19:01:25 <skvidal> w/o having to deal with the defintion problem for iptables
19:01:30 <skvidal> that we deal with in puppet all the time
19:01:41 <skvidal> the other alternative, which I am not advocating but I am throwing out there
19:01:44 <nirik> right. so would we remove the custom rules from nodes then?
19:01:50 <skvidal> yes
19:01:58 <skvidal> we would remove custom rules from node files
19:02:01 <nirik> sounds good to me.
19:02:04 <skvidal> and put them in simple iptables <heredocs>
19:02:07 <skvidal> so the other alternative
19:02:10 <skvidal> that I want to mention
19:02:15 <skvidal> that the dns thing this week made me think about
19:02:35 <skvidal> we could setup iptables in templates - just the dns zone template
19:03:01 <skvidal> in a separate git repo, etc
19:03:11 <skvidal> construct per host and have puppet just run the update-iptables
19:03:15 <skvidal> which sucks down via git, etc
19:03:19 <skvidal> like I said
19:03:22 <skvidal> not advocating
19:03:24 <skvidal> just thinking about it
19:03:30 <nirik> we could, but we don't often change iptables and it doesn't have serial numbers and such... not sure it's worth it.
19:03:51 <skvidal> nod - thr advantage I was thinking of was being able to validate iptables
19:04:19 <skvidal> which is... difficult with the pieces of iptables we'd have to work with in puppet
19:04:33 <nirik> validating iptables in general is difficult. ;(
19:04:44 <skvidal> nirik: true
19:05:00 <misc> especially if you can have a valid iptables config that just block yourself :)
19:05:18 <nirik> I'm happy to simplify and split out what we have now tho for sure...
19:05:27 <nirik> since if you make a mistake now, it affects ALL machines.
19:05:30 <skvidal> the second advantage would be the speed at which we could deploy an iptable
19:05:34 <skvidal> change
19:06:20 <nirik> yeah, currently we don't update often... but there are use cases I suppose.
19:06:27 <skvidal> right
19:06:50 <skvidal> anyway it's something I'm going to be working on so I figured it would be worth mentioning it
19:06:58 <nirik> yeah, sounds good.
19:06:59 <skvidal> if anyone wants to work on it and is familiar with iptables - enjoy
19:07:20 <nirik> #info iptables folks welcome to help with iptables revamp
19:07:37 <nirik> #topic Open Floor (^2)
19:07:44 <skvidal> hah
19:07:48 <nirik> any other items for open floor or other questions, comments? ;)
19:08:19 <ingm4r> just a basic question, if thats ok
19:08:27 <nirik> ingm4r: sure, fire away
19:08:35 <ingm4r> stg ist staging and prd is productive?
19:08:44 <misc> production
19:08:53 <ingm4r> ok
19:09:27 <nirik> ingm4r: yeah...
19:09:42 <nirik> so we try and test things like new package versions and changes in our staging setup...
19:09:53 <nirik> then when they appear fine there, they go to production machines.
19:10:05 <ingm4r> jup, thought so. Just wanted to be sure about the naming.
19:10:16 <nirik> our staging setup is not a complete 1 to 1 mapping, but it has many copies of productions stuff.
19:10:26 <sdrfed17> nirik: so are apprentice guys allowed to work on both staging and production?
19:11:13 <nirik> sdrfed17: sure, the way it works is that apprentices can login to machines and check out a read only copy of our puppet repo... so any changes you make need to be sent throug someone that has commit access.
19:11:25 <nirik> so that way you can see how things are setup and propose patches for review.
19:11:55 <sdrfed17> nirik: ok
19:11:55 <nirik> anyhow, happy to discuss more over in #fedora-admin...
19:12:04 <nirik> we are over time, so lets go ahead and close out...
19:12:09 <nirik> #endmeeting