19:00:17 <nirik> #startmeeting Infrastructure (2011-04-14)
19:00:17 <zodbot> Meeting started Thu Apr 14 19:00:17 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:17 <nirik> #meetingname infrastructure
19:00:17 <zodbot> The meeting name has been set to 'infrastructure'
19:00:17 <nirik> #topic Robot Roll Call
19:00:17 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik
19:00:17 <zodbot> Current chairs: codeblock goozbach nirik ricky skvidal smooge
19:00:19 <zodbot> rbergeron: Error: Can't start another meeting, one is in progress.
19:00:21 <rbergeron> ohh
19:00:29 * CodeBlock here
19:00:29 <nirik> humf
19:00:30 <nirik> oops
19:00:32 * rbergeron always does this to you guys
19:00:43 <rbergeron> if anyone pops in, send them to f-m-1
19:00:43 <goozbach> wait, what?
19:00:45 <rbergeron> that's one, not L
19:00:46 <smooge> what
19:00:46 <rbergeron> :)
19:00:58 <goozbach> hah!
19:01:00 <goozbach> ok ok
19:01:02 * goozbach here
19:01:04 <smooge> remember L
19:01:08 <rbergeron> :)
19:01:09 * jpattonwx here, lurking
19:01:49 * jsmith lurks
19:02:40 * sijis is around
19:03:00 * crashmag says hi
19:03:20 <nirik> sorry about that...
19:03:27 <nirik> somehow I didn't realize they were the same time. ;(
19:03:36 <smooge> well we have it first
19:03:47 <smooge> who needs a release anyway
19:04:11 <nirik> ok, lets gets started I guess.
19:04:20 <nirik> #topic New Folks Intros
19:04:28 <nirik> any new folks lurking here this week?
19:04:36 <crashmag> yap :)
19:04:37 <nirik> want to say hi and such?
19:04:45 * jsmith hasn't said hi in a long time :-p
19:05:04 <nirik> crashmag: welcome.
19:05:05 <smooge> I will be able to help people get into infrastructure after this release I believe
19:05:33 <CodeBlock> I can sponsor some people after the semester ends, too
19:05:43 * skvidal is here
19:05:46 <skvidal> sorry I was late
19:05:52 <CodeBlock> skvidal: no. unexcusable.
19:05:53 <jpattonwx> I think I will wait until after semester here too before worrying about sponsorship
19:05:55 <nirik> crashmag: any thoughts on areas you are interested in? do lurk in #fedora-admin and/or #fedora-noc and ask questions and chime in when you can.
19:06:07 <skvidal> CodeBlock: I know - I shall go flagellate myself
19:06:08 <CodeBlock> skvidal: just like my bad grammer. inexcusable*
19:06:45 * nirik points new folks to https://fedoraproject.org/wiki/Infrastructure/GettingStarted lots of good info there.
19:07:07 <crashmag> nirik, well I work for an ITSM company. I realise all of this is based upon trust, and that takes time. So I was hoping that theres a way at working on "lower level" tickets to begin with.
19:08:02 <nirik> crashmag: sure. ;) Do look at our trac instance and https://fedoraproject.org/wiki/Infrastructure_Cleanup_Tasks_2011
19:08:16 <nirik> there are some tasks that don't need much access at all...
19:08:25 <crashmag> started using linux around 02. my geek factor can be measured by my 3 servers at home. was hoping to help out and learn by doing so :)
19:08:56 <nirik> excellent.
19:09:27 <nirik> often lurking in the above channels can get you a feel for things going on, and people are usually happy to explain, etc.
19:09:55 <smooge> oooh itsm I just closed some tickets on that
19:10:08 <nirik> ok, shall we touch on Beta tickets?
19:10:18 <nirik> #topic Fedora 15 Beta tickets.
19:10:58 <nirik> so beta will be released tuesday.
19:11:05 <nirik> we have several tickets to make sure that goes well.
19:11:15 <nirik> .ticket 2704
19:11:16 <zodbot> nirik: #2704 ([Fedora 15 Beta] New Website) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2704
19:11:45 * nirik sees sijis is answering for websites in the other meeting right now. ;)
19:11:46 * CodeBlock looks at sijis
19:12:39 <nirik> we can come back to this one.
19:12:45 <sijis> ok.
19:12:45 <nirik> .ticket 2707
19:12:46 <zodbot> nirik: #2707 ([Fedora 15 Beta] Permissions on mirrors) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2707
19:12:59 <nirik> we need content first, then we can check this.
19:13:16 <nirik> .ticket 2708
19:13:17 <zodbot> nirik: #2708 ([Fedora 15 Beta] Mirror manager repository redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2708
19:13:27 <nirik> will need to check with mdomsch on that one.
19:13:56 <nirik> sijis: any changes in size or anything for the Beta website? any other notable changes?
19:14:41 <sijis> nope. the size will increase a little but not significant
19:14:48 <sijis> about 5-10mb (if that)
19:14:55 <sijis> its because of the countdown banner
19:15:02 <nirik> ok, cool.
19:15:31 <nirik> .ticket 2706
19:15:32 <zodbot> nirik: #2706 ([Fedora 15 Beta] Release Day) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2706
19:15:43 <nirik> .ticket 2710
19:15:45 <zodbot> nirik: #2710 ([Fedora 15 Beta] Infra change freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2710
19:15:45 <smooge> thats yours as boss
19:16:03 <nirik> Thats for release day issues, and a reminder that we are in freeze. ;)
19:16:06 <nirik> smooge: yep. ;)
19:16:12 <CodeBlock> :P
19:16:23 <nirik> I think thats all the beta tickets. Anyone have anything else Beta related?
19:16:26 <smooge> I closed the disk space one. we are way under 1TB at the moment
19:17:21 <nirik> excellent.
19:17:47 <nirik> #topic Upcoming items and outages
19:17:52 <smooge> I will be starting a sync of F13 over to archives soon and a sync of secondary1 over to the netapp also
19:18:12 <nirik> I have a list in the post on the items I know of that are coming up after this freeze.
19:18:27 <nirik> smooge: ok. 13 has a bit more life, but not too much.
19:18:48 <nirik> whats left on secondary1? staging stuff, some video content from old fudcons, etc?
19:18:49 <CodeBlock> smooge: not sure what that entails, but if you need some help and feel like teaching me, by all means, would love to learn :)
19:19:35 <smooge> yeah I sync stuff over just to make it easier for when it is EOL
19:19:44 <nirik> The end of next week we will be having an outage to move db03, nfs01, and pkgs01 (so it will be an outage for building/packagers).
19:20:10 <smooge> and fas01
19:20:43 <nirik> oh, I didn't have that one on my list. That was a rhel6 guest on a xen5 host we wanted to move?
19:21:02 <smooge> correct
19:21:14 <smooge> it will cut a bunch of issues down
19:21:27 <smooge> as it silently reboots a lot and sometimes not so silent
19:21:37 <smooge> it just needs to be built onto virthost03
19:21:38 <nirik> ok. will add to the list. We will figure out exact timing on them and I will send out an outage notice.
19:22:05 <smooge> fas01 contains various certs and such so when it is going down it will degrade other items
19:22:39 <nirik> ok. We will need to determine that. Ideally we can sync it to a new one in advance so the outage window is small.
19:23:03 <nirik> db03 will take the longest, as it will be a dump/reload on new machine.
19:23:24 <CodeBlock> indeed
19:23:32 <smooge> pkgs01 may be the simplest
19:23:47 <nirik> yeah, it should be just shutdown in one place and bring up in another. ;)
19:24:18 <ranjibd> db02 can be done like that too right?
19:24:25 <ranjibd> sorry db03
19:24:43 <nirik> well, no, as we are switching it from rhel5 older db version to rhel6
19:24:57 <nirik> it's now a rhel5 box with a custom postgresql on it.
19:25:07 <nirik> it will go to rhel6 with the normal version.
19:25:43 <nirik> eventually we will move the other db's over to rhel6 too... but not next week
19:26:22 <nirik> after next week we have puppet upgrade and fpca landing, then after final we are going to upgrade nagios/noc01. ;)
19:26:35 <CodeBlock> yep
19:26:38 <nirik> anyone have any other upcoming items they would like to discuss planning on.
19:27:02 * CodeBlock still wants to start playing with moving hosted to EL6, but that is looking like a summer project
19:27:32 <nirik> yeah, that would be great to get done...
19:27:33 <nirik> someday.
19:27:51 <nirik> ok, meeting tickets...
19:27:56 <nirik> #topic Meeting tickets
19:28:00 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority
19:28:42 <nirik> a grab bag of things. ;)
19:29:05 <nirik> #info on cvs01, I am waiting to hear back from the last 2 projects on there. They may move to fedorahosted, or fedorapeople.
19:29:43 <nirik> Anyone see a meeting ticket they would like to comment on?
19:31:11 <CodeBlock> what ever happened with moving blogs off of infra?
19:31:31 <CodeBlock> (i.e. ticket 2591)
19:32:22 <nirik> it's not really moving forward. ;) I think ricky was going to look into it more when he had time...
19:32:38 <nirik> it needs someone talking to wordpress.com or the like... and then talking to the blog users.
19:32:41 <smooge> yeah. it needs a key person to push it forward.
19:32:45 <smooge> sorry slow type
19:32:46 <sijis> yeah, i'm not sure what's left on that.
19:33:06 <smooge> I would say if its not in the ticket.. then everything is left on that :)
19:33:17 * jsmith has some friends at wordpress.com, but they're busy cleaning up from the recent security incident
19:33:19 <sijis> is it contact wp and see if they will host that stuff for us and contacting users and seeing if they need help moving (or are we just moving stuff for them)
19:33:23 <jsmith> I'd be happy to make some introductions
19:33:33 <sijis> oh yeah. i read that this morning
19:33:34 <nirik> jsmith: That would be cool.
19:34:05 <nirik> yeah, contact them and see what they can do for us... then once we know that, contact all our blog users and tell them about it...
19:34:26 <nirik> I think it will depend on what they want to do if we can help move them or if users move their own.
19:34:37 <nirik> and we will need to set a sunset date...
19:34:56 <sijis> i believe they can export all their content with a single click.
19:35:07 <nirik> yeah, should be pretty easy.
19:35:37 <nirik> we also want to explain to them why we are doing this and such... so no one is blindsided or confused.
19:35:56 <sijis> good point. its noted
19:36:27 <nirik> so, if anyone would like to drive that forward, please feel free.
19:36:46 <nirik> #topic Other tickets
19:36:56 <nirik> Any other tickets folks would like to bring up or note?
19:37:32 <ranjibd> nagios upgrade? skvidal was telling last time it didn't work due to some issues
19:37:47 <skvidal> ranjibd: nirik and CodeBlock have the knowledge there
19:37:57 <nirik> yeah, ran into issues with our irc bot, and backed out the upgrade.
19:38:08 <nirik> CodeBlock is planning on doing it again after f15 final.
19:38:17 <nirik> Hopefully it will go smoothly this time. ;)
19:38:39 <CodeBlock> indeed
19:38:50 <ranjibd> puppet upgrade ? can i help any way?
19:38:53 <sijis> is that why we have zodbot and fedbot?
19:39:45 <smooge> possibly.. it was looking at what we could break out
19:39:48 <nirik> sijis: nope. ;) fedbot is one I run locally, mostly for #fedora. It's got some non packaged plugins and such, also it's locked down to not respond to lots of things, which we wouldn't want zodbot to do.
19:40:48 <nirik> ranjibd: it should be pretty easy. Basically just upgrade the master, upgrade a 5 and 6 client and test... it should go smoothly.
19:41:13 <nirik> I don't think splitting zodbot is going to help any.
19:41:22 <ranjibd> nirik: all the clients have same version ?
19:41:43 <skvidal> splitting zodbot won't help the migration - but it will make the bot in general make a lot more since where it should live
19:42:03 <skvidal> having our meetingbot on our nagios server makes no sense to me
19:42:22 <nirik> skvidal: yeah... but where should the meeting part go?
19:42:35 <skvidal> nirik: away
19:42:47 <skvidal> nirik: away from our nagios server
19:42:50 <skvidal> it makes me nervous
19:42:59 <nirik> ranjibd: yeah.
19:43:08 <ricky> For what it's worth, there's no reason that zodbot has to be on the same machine as noc01 to get nagios notifications
19:43:24 <ricky> So moving zodbot to value or something is easy
19:43:27 <nirik> skvidal: ok, we can figure something out.
19:43:37 <nirik> yeah, value might make some sense.
19:43:59 <nirik> we do need to serve the meetbot html stuff.
19:44:24 * nirik makes a note.
19:44:40 <nirik> splitting it won't help the lag on restart unless we split it out by channels...
19:45:02 <nirik> we may be able to prune the channels it's joining...
19:45:43 <ricky> Sorry, I missed the beginning due to internet issues - what does splitting zodbot mean?
19:46:00 <sijis> it would be two bot.. zod and bot :)
19:46:19 <sijis> ok. i need a break. (lame joke)
19:46:26 <nirik> there was talk about splitting it into fasbot, nocbot, meetbot
19:46:33 <smooge> smoogebot
19:46:35 <nirik> sijis: :)
19:46:51 <skvidal> I suggested splitting it logically
19:46:55 <skvidal> into its functional components
19:47:03 <skvidal> and it's authenticated-components
19:47:09 <skvidal> nocbot has more info thant meetboot
19:47:21 <skvidal> fasbot has different access than meetbot or nocbot
19:47:39 <skvidal> conflating all 3 seemed like a recipe for problems
19:47:42 <sijis> isn't that just more stuff to oversee than 1 bot?
19:47:57 <skvidal> also - from a services and meta-services standpoint
19:48:18 <skvidal> it does not make sense to have a user-facing service (zodbot) living on a meta-service server (noc01)
19:48:34 <skvidal> just like it wouldn't make sense to run a webserver for the wiki on our dhcp server
19:48:41 * nirik nods.
19:48:51 <skvidal> that way if noc01 goes down
19:48:58 <skvidal> we don't have to worry about it impacting user-facing services
19:49:00 <nirik> I think moving it is a good idea. Splitting it I am more meh on.
19:49:38 <CodeBlock> I like the idea of splitting it out, but that's me.
19:49:57 <ricky> Same - supybot is pretty simple/stable, and it only really touches data that's public anyway
19:50:06 <ricky> So while I wouldn't mind splitting it out, I don't see it as a huge priority or a huge gain
19:50:20 <skvidal> is everyone on board with moving it OFF of noc01?
19:50:24 <CodeBlock> yes
19:50:25 <nirik> yes.
19:50:27 <smooge> yes
19:50:36 <skvidal> great
19:50:37 <smooge> let me rephrase, YES
19:50:38 <sijis> i do agree that it doens't make sense that its on noc1. it should be in value1/2 as ricky mentioned
19:50:45 <skvidal> value## seems like a winner
19:50:52 <ricky> Yup, only change would be firewall changes to accept supybot-notify + pointing nagios at the new IP
19:51:02 <ricky> Ideally it should not be on a VPN host so that we get notifications if VPN goes down
19:51:04 <nirik> so, how about we move it sooner rather than later... and if someone wants to drive splitting it down the road, they can
19:51:30 <nirik> value already has a webserver for http://meetbot.fedoraproject.org/
19:51:32 <nirik> ?
19:51:59 <ricky> We currently treat value like an app server, so we could proxy that through the proxies like we normally do if we want
19:52:13 <ricky> Or it'd be fine to poke a hole for it too.
19:52:34 <nirik> ok.
19:53:01 <nirik> does someone want to take the task of moving it? I guess I can...
19:53:05 <CodeBlock> heh, less to worry about for the noc01/noc03-tmp switchover then
19:53:11 <CodeBlock> nirik: I can if you don't want
19:53:32 <nirik> CodeBlock: sure, feel free. :) It will require doing it when no meetings are around...
19:53:41 <CodeBlock> when are we looking to do it? And what kind of an outage notif needs sent out?
19:53:51 <smooge> bah.. I think we time it for rbergeron's most important meeting of the week
19:54:02 <ricky> Probably an outage notification to devel-announce + infrastructure
19:54:10 <nirik> well, we could add it to the pile next week, but thats getting lots of stuff.
19:54:16 <ricky> And as a college student, I'm sure you'll be around at 3 AM some days, right?  :-D
19:54:20 <nirik> it shouldn't be frozen, so I suppose we could do it most anytime.
19:54:29 <nirik> oh, that reminds me...
19:54:38 <ricky> (half-kidding - on one hand, changes at 3 AM aren't that great - on the other hand, zodbot's dead simple)
19:55:40 <nirik> What would folks think of the following addition to outage notifications: Once you write up one, have one other person review and approve before sending. It's so easy to mess up timezone stuff, or the like.
19:56:02 <smooge> that would be good
19:56:03 <ricky> Good idea :-)
19:56:09 <CodeBlock> ricky: Yeah 3AM is perfectly fine for me ;D
19:56:10 <skvidal> I have a counterproposal
19:56:11 <smooge> it would make most of my emails though less funny
19:56:13 <ricky> Doesn't have to be formal or anything, just an "OK" in IRC
19:56:20 <skvidal> how about we make all timezones Eastern Standard
19:56:20 <ricky> Right?
19:56:29 <skvidal> so we don't have to mess around with silly things like timezones at all
19:56:30 <skvidal> :)
19:56:32 <nirik> yeah, no red tape, just another pair of eyes.
19:56:33 * skvidal kids
19:56:39 <nirik> skvidal: fine with me. ;)
19:56:44 <CodeBlock> ;D
19:57:02 <ranjibd> +1 for UCT
19:57:06 <nirik> anyone object to the above? will add to sop...
19:57:21 <jsmith> +1 for second set of eyes
19:57:52 <nirik> cool.
19:57:57 <nirik> #topic Open Floor
19:58:01 <nirik> Anything for open floor?
19:58:39 <CodeBlock> 3 weeks or so left in my semester then I'll have a lot more time to be around :)
19:58:39 <ricky> Wow, perfect timing...  this has got to be a first!
19:58:50 * CodeBlock assumes same for ricky
19:59:15 <ricky> Yup, will have a lot more time this year
19:59:36 <nirik> cool. ;)
19:59:59 <nirik> ok, if nothing else, will close out the meeting in a minute here.
20:00:49 <nirik> Thanks for coming everyone!
20:00:52 <nirik> #endmeeting