19:00:00 <nirik> #startmeeting Infrastructure (2013-04-18)
19:00:00 <zodbot> Meeting started Thu Apr 18 19:00:00 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic welcome y'all
19:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:11 * pingou here
19:00:12 * skvidal is here
19:00:13 * threebean here briefly
19:00:17 <jerzyr> here
19:00:19 * puiterwijk 
19:00:21 * cyberworm54 is here
19:00:24 <skvidal> pingou: you beat me to the /me is here!
19:00:27 <skvidal> pingou: noooo!
19:00:36 <pingou> skvidal: I forgot the "is" :(
19:00:37 * skvidal will live in shame
19:00:41 <skvidal> SHAME
19:00:50 <nirik> :)
19:00:52 <jerzyr> present
19:00:53 * pingou learned from the best
19:00:57 * relrod here, but writing a paper
19:01:07 * abadger1999 here
19:01:29 <skvidal> pingou: to be fair I think lmacken was the best when he used to be able to make warren and katzj drop off the channel when he chimed in to rollcall
19:01:42 <pingou> skvidal: I can't compete with that!
19:01:44 <nirik> yeah, that was awesome. ;)
19:01:56 <skvidal> nirik: It took FOREVER to figure out how that was happening
19:01:59 <lmacken> haha
19:02:06 <nirik> good ol dircproxy bug.
19:02:07 <skvidal> lmacken: you remember that, right?
19:02:16 <lmacken> of course :) one of my best hacks
19:02:18 <smooge> here
19:02:20 <skvidal> hah
19:02:27 <skvidal> okay. down to business
19:02:27 <nirik> #topic New folks introductions and Apprentice tasks
19:02:37 <nirik> any new folks or apprentices with questions or comments?
19:02:57 <cyberworm54> >.>
19:03:21 * nirik listens to the sound of snow melting.
19:03:29 <nirik> #topic Applications status / discussion
19:03:40 <nirik> I had a few things here...
19:03:45 * pingou didn't do much this week
19:04:09 <nirik> unless threebean is still here and wants to mention his...
19:04:27 <nirik> pingou: is there still a mail issue on fedocal01.stg? or did we completely fix that now?
19:04:39 <pingou> nirik: I think there is
19:05:19 <smooge> so about summer of code stuff
19:05:33 <pingou> logs are empty but no emails received :(
19:05:42 <pingou> mailq is empty too :/
19:05:56 <nirik> pingou: ok, I can try and help debug that at some point.
19:05:59 * nirik adds to his list.
19:06:16 <pingou> nirik: ah the cron just threw me an error
19:06:42 <pingou> nirik: something db related, let's check this out after the meeting
19:06:52 <nirik> #info threebean created a collectd thing for fedmsg: https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=busgateway01&plugin=fedmsg&timespan=1200&action=show_selection&ok_button=OK
19:07:05 <nirik> #info lots of work on datanommer this week as well upstream
19:07:09 <nirik> pingou: ok.
19:07:13 <nirik> smooge: yeah?
19:08:36 <nirik> oh, I had another big one. ;(
19:08:56 <skvidal> ?
19:09:02 <nirik> #info Bugzilla upgrade is coming up very soon. We need to update stg to new python-bugzilla and test our apps asap.
19:09:20 <nirik> we should also point all stg stuff to parter-bugzilla if it's not already
19:10:39 * nirik listens to more snow melt. ;)
19:10:56 <nirik> hopefully we are already ready for the bz changes, but we should make sure in stg...
19:10:59 <abadger1999> grep -ri bugzilla in puppet or something.
19:11:05 <skvidal> nirik: we're having pollen-valanches here
19:11:05 <smooge> sorry I thought applications meant for application for projects versus the applications stuff
19:11:11 <skvidal> abadger1999: git grep
19:11:26 <puiterwijk> nirik: I have the commit for partner-bugzilla ready
19:11:52 <nirik> puiterwijk: cool. for only stg right?
19:12:04 <puiterwijk> I believe I sent it to you a while back to review, but didn't hear any of it afterwards
19:12:08 <puiterwijk> nirik: yeah, stg only
19:12:23 <nirik> ah, I must have dropped it somewhere, sorry. ;) Please do push it when you get a chance. ;)
19:12:42 <puiterwijk> sure, will do
19:13:23 <nirik> ok, any other applications news?
19:13:33 <nirik> note that we are still frozen until next tuesday...
19:13:48 * nirik needs to make sure to file the release tickets, I didn't get to it yet.
19:13:48 <puiterwijk> nirik: also the app/sys-admin stg stuff, I guess?
19:13:58 <nirik> yeah, was gonna do that in sysadmin
19:14:01 <puiterwijk> ah ok
19:14:06 <nirik> #topic Sysadmin status / discussion
19:14:23 <nirik> so, we have an issue with our stg setup... we want to move apps to use openid there.
19:14:37 <nirik> they reach out to 'username.id.stg.fedoraproject.org'
19:15:09 <nirik> but they cannot reach the outside ip of that due to firewall.
19:15:21 <nirik> we can't use /etc/hosts because we can't do wildcard's there.
19:16:09 <smooge> did you guys determine which firewall?
19:16:20 <nirik> smooge: yes, the rhit controlled one.
19:16:32 <nirik> to fix it we need to have them map every single stg host to a 1:1 nat
19:16:39 <nirik> which I really don't want to do
19:16:40 <puiterwijk> so my idea was to setup an ns01.stg which has its own database for stg.fp.o which the stg hosts use, and uses the normal paths for anything non-stg
19:16:59 <smooge> ugh
19:17:01 <puiterwijk> another idea I had was to add NAT entries to the stg boxes themselve
19:17:11 <skvidal> puiterwijk: split-brain dns aaaaaaaargh
19:17:14 <puiterwijk> smooge: yeah, I know it's ugly, but I'm just trying to come up with ideas...
19:17:17 <smooge> thinks about making id.stg.fedoraproject.org a box that sits at ibiblio
19:17:21 <puiterwijk> skvidal: no, no split-horizon DNS
19:17:38 <puiterwijk> skvidal: just another DNS server entirely
19:17:43 <skvidal> it's 2 different answers dependoing on who is asking
19:17:47 <nirik> I'm not sure about the local NAT... would have to play with that to see if it would really work, because not sure you can do wildcards there at all either.
19:17:53 <puiterwijk> skvidal: I know what split-horizon DNS is
19:18:03 <puiterwijk> nirik: you just redirect ip addresses
19:18:30 <puiterwijk> so it could translate 1.1.1.1 -> 192.168.1.1 (for example) before sending the packet to the NIC
19:19:02 <nirik> yeah, I suppose.
19:19:21 <nirik> ok, I'd like to ponder on this some more, but if someone has a better solution, feel free to suggest it anytime. ;)
19:20:14 <smooge> so why can't id.stg.fedoraproject.org sit at say ibiblio?
19:20:54 <nirik> well, several reasons:
19:21:09 <nirik> a) mixing stg in the same vhost as production stuff.
19:21:20 <nirik> b) would need proxy to talk to it and from it?
19:21:41 <nirik> it might work, but it seems messy to me
19:22:08 <skvidal> nirik: to be fair - what part of our staging environment IS NOT messy?
19:22:25 <nirik> well, yeah.
19:22:26 <skvidal> nirik: if we're going to have to do major forklifting to make this work - is there something else we could change to make this all simpler?
19:22:39 <nirik> smooge: it's a possibility, but I'd like to think about the implications of spreading that out
19:22:39 <smooge> move it all to the cloud?
19:22:44 <skvidal> can we make it all live outside of phx2?
19:23:01 <skvidal> can we stuff it in a cloud-in-a-box off of a spare system?
19:23:18 <smooge> well we don't have much in spare systems
19:23:20 <nirik> we could, but if this dns issue is a lot of work... you are asking for lots MORE work? ;)
19:23:41 <skvidal> nirik: well I was more thinking of longer-range but
19:23:42 <skvidal> yah
19:23:49 <skvidal> I understand where you're coming from
19:23:54 <smooge> I don't mind more lots work if it is lots work once :)
19:23:55 <nirik> yeah, understand...
19:23:57 <skvidal> so I will withdraw that sugestion
19:24:31 <smooge> now there is a GSOC project :).. build an implement a real dev and staging enviroment for us
19:24:31 <nirik> anyhow, we have options, but not clear on the least anoying one. ;)
19:24:41 <nirik> So, any other sysadmin news?
19:25:12 <nirik> #info need to determine solution for username.id.stg.fedoraproject.org in stg, we have several possible, just need to see which is least bad.
19:25:16 <nirik> oh, I have one:
19:25:20 <smooge> I am going to try and get a PPC working
19:25:25 <nirik> #info we have reverse dns for cloud.fedoraproject.org
19:25:29 <smooge> yay!
19:25:34 <pingou> cool :)
19:25:43 <nirik> #info smooge to get new ppc box setup to replace ppc11/ppc12.
19:25:47 <skvidal> nirik: we do?
19:26:02 <nirik> skvidal: we should?
19:26:07 * nirik meant to go back and check it.
19:26:08 <skvidal> dig isn't showing what I'd expect
19:26:16 <skvidal> which is to say - nothing at all
19:26:35 <nirik> hum.
19:26:56 <skvidal> but dig directly to our servers
19:26:57 <skvidal> and it works
19:26:57 <nirik> anyhow, we cna work on that out of band. ;(
19:27:00 <skvidal> yes
19:27:01 <skvidal> sorry
19:27:08 <nirik> #topic Private Cloud status update / discussion
19:27:11 <nirik> any cloud news?
19:27:26 <skvidal> nothing from me
19:28:01 <skvidal> we have 2 boxes ready to be added as compute nodes... just not done yet
19:28:11 <skvidal> smooge: any word on the disk for 03?
19:28:26 <smooge> O3?
19:28:33 <skvidal> fed-cloud03
19:28:40 <nirik> oh yeah, that one with the wacky disk.
19:28:40 <skvidal> the box with the wonky disk?
19:29:11 <smooge> ugh.. I don't know. I got focused on gnome last week and completely forgot where I left that
19:29:12 <smooge> sorry
19:29:24 <nirik> no hurry on it, if you could poke it again that would be good.
19:29:30 <smooge> I will do that before PPC
19:29:41 <nirik> once it's figured out we should reinstall it and 01 and setup 01/03 as a grizzley openstack instance.
19:30:24 <nirik> ok, moving on
19:30:27 <skvidal> smooge: no sweat - thx
19:30:32 <nirik> #topic Upcoming Tasks/Items
19:30:35 <skvidal> nirik: should we use rdo?
19:30:49 <nirik> skvidal: yeah, I think so. ;)
19:30:53 <skvidal> nirik: or should we do it by hand?
19:30:55 <skvidal> ok
19:31:07 <nirik> if it makes sense. I've not looked closely at rdo
19:31:34 <nirik> #info 2013-04-02 to 2013-04-23 ALPHA infrastructure freeze
19:31:34 <nirik> #info 2013-04-22 - announce openid change on hosted.
19:31:34 <nirik> #info 2013-04-23 F19 alpha release
19:31:34 <nirik> #info 2013-04-24 - switch to new sigul servers.
19:31:35 <nirik> #info 2013-04-25 - switch hosted to openid
19:31:36 <nirik> #info 2013-05-01 nag fi-apprentices
19:31:37 <nirik> #info 2013-05-08 drop inactive apprentices
19:31:40 <nirik> #info 2013-05-14 to 2013-05-28 BETA infrastructure freeze
19:31:42 <nirik> #info 2013-05-28 F19 beta release
19:31:44 <nirik> anything folks would like to note or schedule?
19:32:31 <nirik> #topic Open Floor
19:32:35 <nirik> any items for open floor?
19:33:48 <jsmith> nirik: Just a quick status update -- I should have a couple of dedicated servers for you guys in a the next month
19:33:57 <jsmith> nirik: Sorry it's taken so long :-(
19:33:57 <nirik> jsmith: oh, nice!
19:34:05 <pingou> cool :)
19:34:09 <smooge> what kind and where... and no problem
19:34:11 <nirik> no worries, those kinds of things are always welcome whenever they happen.
19:34:44 <jsmith> smooge: Quad-core Xeons, at least 4 gigs of ram (pushing for more), located in our data center in Utah
19:35:00 <jsmith> smooge: I can give you more specifics offline
19:35:04 <smooge> ah cool. thanks
19:35:17 <nirik> cool. I don't think we have anything in utah yet.
19:35:28 <smooge> road trip
19:35:49 <nirik> heh.
19:36:12 <jerzyr> utah jazz
19:36:28 <nirik> ok everyone. thanks for coming. Do continue in #fedora-admin, #fedora-noc and #fedora-apps. ;)
19:36:35 <nirik> #endmeeting