18:59:58 <nirik> #startmeeting Infrastructure (2012-01-26)
18:59:58 <zodbot> Meeting started Thu Jan 26 18:59:58 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:59:58 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:59:58 <nirik> #meetingname infrastructure
18:59:58 <nirik> #topic Robot Roll Call
18:59:58 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch
18:59:58 <zodbot> The meeting name has been set to 'infrastructure'
18:59:58 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge
19:00:40 * CodeBlock here
19:01:02 * LoKoMurdoK here
19:01:05 * nirik waits for folks to trickle in
19:01:20 <metadave> hello
19:01:45 * lmacken 
19:01:46 * wsterling here
19:02:15 <netSys> hi
19:03:01 <threebean> hi
19:03:22 * jac1bat0 is here
19:03:46 * abadger1999 here
19:03:52 <nirik> #topic New folks introductions and Apprentice tasks.
19:03:58 * pingou here
19:04:12 <nirik> any new folks want to say hi and introduce themselves or apprentices want to talk about tickets?
19:04:21 <metadave> Hello - I'm Dave Parfitt from Buffalo, NY. I write software and fix things. I have been using Linux on and off since the 90's.
19:04:37 <nirik> welcome metadave
19:04:43 <metadave> thanks
19:05:40 <nirik> any other new folks?
19:06:50 <nirik> ok, moving along...
19:06:54 <nirik> #topic 2 factor auth status
19:07:00 <abadger1999> hey metadave!
19:07:11 <metadave> hello abadger1999
19:07:15 <nirik> any news here? /me looks for herlo
19:08:23 * nirik moves on
19:08:29 <nirik> #topic reworking staging
19:08:44 <nirik> We have started in on this... averi has done some work here.
19:09:07 <nirik> we will be working on it more next week/coming weeks
19:09:10 <skvidal> sorry
19:09:15 <nirik> no worries.
19:09:22 <skvidal> was in another window in my brain
19:09:56 <nirik> hopefully sometime in the next few weeks we can nuke the stg branch.
19:10:04 <nirik> I also intend to re-install a few stg machines.
19:10:27 <skvidal> nirik: and if you want to give averi access - he seems to want to wotrk on more of them
19:10:39 <nirik> ok, we can take a look at that...
19:11:12 <nirik> #topic Mass reboot outage later tonight.
19:11:20 <skvidal> yay!
19:11:21 <skvidal> fun!
19:11:26 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3113
19:12:22 <nirik> we have our class A and B reboots tonight.
19:12:26 <nirik> so there will be an outage.
19:12:48 <nirik> http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt is our SOP here.
19:14:27 <nirik> Action for that will likely be in #fedora-noc
19:14:54 <nirik> #topic Applications status / discussion
19:15:09 <nirik> So, this worked somewhat last week, so lets try again. ;)
19:15:35 <nirik> Any application owners have items to discussion about their apps?
19:16:38 <nirik> we have a fas release scheduled for 2012-02-07
19:16:55 <pingou> the audio captcha can already be tested on stg
19:16:58 * abadger1999 checks what his email said
19:17:09 <nirik> pingou: great!
19:17:19 <pingou> thanks to abadger1999 ;-)
19:18:46 <abadger1999> nirik: oops, I announced Feb 2 as cutting the final release.
19:18:59 <nirik> ok.
19:19:05 * nirik fixes
19:19:13 <abadger1999> I think I slipped development stuff a week, but only slipped release a few days
19:19:35 <nirik> fair enough
19:20:22 <nirik> lmacken: any news on bodhi and community?
19:20:34 <nirik> sounds like we had a discussion on list about urls, but did we reach a conclusion.
19:20:42 <lmacken> nirik: I'm working on polishing up a new bodhi release containing the FESCo proventester policy change
19:20:59 <lmacken> it was a 2 line change, but the test suite needs some love to get it back up and running
19:21:04 <nirik> k
19:21:08 <lmacken> as far as community/packages, we need to get it deployed to production
19:21:19 <nirik> yeah
19:21:21 * lmacken has been doing very careful bugfix updates
19:21:37 <lmacken> and that's all I got
19:22:05 <nirik> for deployment, I was thinking 2 servers running the app and 1 db server.
19:22:16 <nirik> then we need to figure out urls.
19:22:20 <lmacken> sounds good. I completely seperate db server?
19:22:24 <lmacken> s/I/A/
19:22:34 <lmacken> right now the only thing that uses a db on community01.dev is tagger
19:22:48 <skvidal> do we have a timeline for that deployment?
19:22:48 <nirik> yeah, so that way this is more self contained... but I guess thats open to discussion.
19:22:56 <skvidal> sorry if I missed it - just trying to catch up a bit
19:23:12 <nirik> do you see any issues with 2 app servers?
19:23:17 * lmacken doesn't
19:23:25 <abadger1999> lmacken: Clue in me on xapian -- is it a db?  Or backed by a db?  Or files?
19:23:27 <lmacken> as soon as those exist, I can deploy it.
19:23:41 <lmacken> abadger1999: xapian is it's own file-based db
19:24:43 <nirik> lmacken: and should we use our existing caching setup? ie, proxies -> haproxy -> varnish -> app1/app2
19:24:45 <skvidal> lmacken: do both app servers have a copy of it - or do they get synced out from a common source?
19:24:48 <abadger1999> lmacken: and community communicates over the network to it?
19:24:52 <lmacken> nirik: that's fine with me
19:25:02 <skvidal> nirik: do we really want to put this all behind the proxies?
19:25:07 <lmacken> skvidal: good question. I haven't thought that part over.
19:25:09 <nirik> might make it even speedier (is that a word)
19:25:28 <lmacken> abadger1999: nope, it's a local db currently
19:25:29 <nirik> skvidal: well, it does help for geographic diversity...
19:25:48 <skvidal> nirik: except that  we lose some of the proxies when things go sideways in phx2
19:25:51 <abadger1999> lmacken: But does it have to be local files?
19:25:58 <lmacken> ok, so we have an indexer that runs every night... we can then have it sync the xapian db to both of our app servers
19:26:02 <abadger1999> or is it tcp/ip capable?
19:26:15 <lmacken> abadger1999: I'm not positive, but I think it's file based only...
19:26:19 <nirik> skvidal: sure, but would direct be any better? I guess wouldn't be subject to proxy failure.
19:26:23 <skvidal> nirik: I guess I was thinking - since this app doesn't tie to any other resource directly (no nfs, for example) maybe this is the time we model it for the future
19:26:41 <nirik> skvidal: yeah, but then it's SPOF on phx2.
19:26:43 <skvidal> nirik: and build it up as an silo
19:26:58 <skvidal> nirik: why not  stick it at ibiblio?
19:27:14 <nirik> skvidal: it needs to index/hit koji/pkgdb/bodhi I think?
19:27:18 <nirik> would be slow from remote...
19:27:34 <skvidal> nirik: oh - I thought the indexing happened locally and was pushed over to the apps
19:27:40 <skvidal> my misunderstanding, then.
19:27:43 <nirik> how about this:
19:28:05 <nirik> I'll start a thread on deployment on the list and we can hash out what questions we all might have and best place to put it, etc.
19:28:19 <pingou> skvidal: the indexing is (for the search) but not the querying of package's info ( lmacken did I got it right ?)
19:28:44 <skvidal> nirik: sounds like a plan - would be handy to know the structure better
19:29:24 <lmacken> pingou: sounds correct :)
19:29:26 <skvidal> lmacken: hate to do this to you - but could you write up a small description of the server intedepdnencies?
19:29:33 <lmacken> skvidal: sure
19:29:40 <skvidal> lmacken: nothing complicated - just want to make sure I know where the bits matter
19:29:46 * nirik nods. that would be great.
19:29:50 <lmacken> skvidal: of just community01.dev stuff?
19:30:06 <skvidal> lmacken: of what we've discussed here, yeah - but ideally both packages and tagger
19:30:13 <skvidal> (which I assume falls under community01.dev)
19:30:22 <lmacken> k, will do
19:30:27 <nirik> also, as a side idea: perhaps we get someone else to deploy it to new machines. ;) Just so we have someone able to do so in case lmacken isn't around and fix any problems with the sop?
19:30:39 <lmacken> that's totally fine with me ;)
19:30:48 <skvidal> nirik: +1
19:30:52 <skvidal> I'm on board with doing that
19:30:53 <lmacken> it /should/ be as simple as just running puppet
19:31:06 <nirik> yeah. ;)
19:31:10 <skvidal> lmacken: if only puppet was as simple as running puppet :)
19:31:13 <nirik> ok, any other applications news?
19:31:21 <lmacken> I did write a sop for it too http://infrastructure.fedoraproject.org/infra/docs/fedorapackages.txt [-]
19:31:27 <nirik> do we have an actual updated list of our apps and who's point on them?
19:32:02 <nirik> http://fedoraproject.org/wiki/Infrastructure/Services I guess. should get updated.
19:32:30 <lmacken> oh right, I forgot about that page.. I'll add packages & tagger
19:32:38 <nirik> lmacken: thanks.
19:32:42 * nirik didn't know of it either.
19:32:47 * lmacken made it a long time ago
19:32:59 <nirik> #topic Upcoming Tasks/Items
19:33:23 <nirik> I have a list in my email agenda... is it worthwhile to spew it to the meeting too?
19:33:35 <nirik> dgilmore will be in phx next week...
19:33:38 <averi> nirik: hey, just read backlog, I confirm what skvidal said, I can work on more hosts if you want me to.
19:33:59 <nirik> averi: great, we can take a look after meeting. ;)
19:34:19 <averi> nirik: I am about to leave, will be back tomorrow after the outage!
19:34:31 <averi> will poke as soon as I login :)
19:34:31 <nirik> If anyone can think of anything for dgilmore to do while out in phx2, we have a ticket:
19:34:33 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3084
19:34:37 <dgilmore> nirik: yeah, need to finalise the list of things to do tehre
19:34:39 <nirik> averi: no worries. Enjoy
19:34:55 <averi> thanks! later
19:34:58 <dgilmore> did the ppc box turn up?
19:35:05 <nirik> smooge: ^
19:35:18 <skvidal> s/turn/burn/
19:35:19 * skvidal hopes
19:35:42 <smooge> dgilmore, box is there
19:35:52 <dgilmore> skvidal: maybe it will burn up
19:35:56 <dgilmore> smooge: gracious
19:36:03 <nirik> also, we want to re-install sign-vault01... however I see it's warnetee ends 2012-04-11
19:36:17 <dgilmore> nirik: hrrm
19:36:19 <skvidal> nirik: womp womp
19:36:29 <nirik> I have now gotten all backups off backup01
19:36:35 <dgilmore> nirik: replacement in 2 days?
19:36:35 <smooge> I think we can get another box or an extension on that warranty
19:36:38 <nirik> who's warnetee ends 2012-06-17
19:36:59 <nirik> smooge: did we plan for a replacement box for that one? I can't recall
19:37:22 <smooge> the reason i didn't get a warrantee extension was that it was a test box that f13 got that made production.
19:37:26 <dgilmore> nirik: im going to do a equalogic firmware update while there also
19:37:33 <skvidal> dgilmore: and junk04?
19:37:41 <nirik> dgilmore: cool. Does that require an outage?
19:37:42 <smooge> I then put in a replacement for first q of next year.. but not sure if that got approved
19:37:54 <dgilmore> nirik: yes, need to take the storage down to do it
19:38:16 <dgilmore> skvidal: is that the one thats broken currently?
19:38:19 <nirik> dgilmore: ok, we should schedule that outage then.
19:38:35 <dgilmore> nirik: yeah, i was thinking next thursday night
19:38:41 <nirik> proposal: repurpose backup01 as a new sign-vault02
19:38:41 <dgilmore> should only take 15 mins
19:38:56 <nirik> then we can extend it's warnetee hopefully.
19:39:07 <nirik> and we have sign-vault01 around if something blows up
19:39:50 <dgilmore> nirik: works for me
19:40:17 <nirik> one question I had was: will backup03's tape drive read backup01 tapes? if so, we should save old tapes for a while in case we need them.
19:40:57 <nirik> Anyone else have anything upcoming they would like to discuss?
19:40:59 <dgilmore> nirik: backup03 has LTO4 drive?
19:41:47 <nirik> LTO-5
19:42:10 <nirik> LTO-5 (Half Height) it says
19:42:10 <dgilmore> nirik: yeah it will be able tor ead LTO4 tapes
19:42:17 <nirik> ok, great.
19:42:43 <nirik> #topic Meeting tagged tickets:
19:42:44 <nirik> https://fedorahosted.org/fedora-infrastructure/report/10
19:42:58 <nirik> .ticket 3094
19:43:02 <zodbot> nirik: #3094 (Assigning a default acl to the pkgs repo) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3094
19:43:05 <nirik> oh yeah, this got addressed.
19:43:12 * nirik will remove meeting keyword.
19:43:18 <nirik> abadger1999: do you need any more info on that one?
19:43:40 <abadger1999> nirik: Nope, We can even EasyFix this one -- it's all in one method.
19:43:46 <abadger1999> I'll update it
19:44:28 <nirik> cool. thanks.
19:44:35 <nirik> #topic Useless Stats
19:44:45 <nirik> I just had some useless stats I thought I would throw out:
19:45:06 <nirik> we are down to just 16 RHEL5 instances now. Which is just over 10% of our total.
19:45:21 <nirik> we are down now to 180 tickets.
19:45:45 <nirik> #topic Open Floor
19:45:51 <nirik> anyone have anything for open floor?
19:45:59 <abadger1999> New infra-related channel  #fedora-apps
19:46:14 <nirik> ah yeah.
19:46:35 <nirik> #info new #fedora-apps channel for application discussion, design, and issues.
19:46:52 <abadger1999> We'll have some fedora-infr-created web app meetings and discussions in there.
19:47:06 <pingou> \ó/
19:47:32 <abadger1999> You can still use #fedora-admin for discussion as well; at least until the traffic for those discussions grows :-)
19:48:14 * nirik sees #fedora-apps as more for design/early discussions or lengthy ones vs admin being for quick issues/bugs/fixes/notifications.
19:48:22 <abadger1999> <nod>
19:48:48 <nirik> anyhow, anyone have anything else? or shall we call it a meeting?
19:49:00 <dgilmore> call it a day
19:49:50 <nirik> ok, thanks for coming everyone!
19:49:53 <nirik> #endmeeting