20:00:46 <mmcgrath> #startmeeting
20:00:46 <zodbot> Meeting started Thu Sep  3 20:00:46 2009 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:57 <mmcgrath> #topic Who's here?
20:01:00 * lmacken 
20:02:06 * abadger1999 shows up
20:02:14 * nirik is hanging around in the back.
20:02:17 * dgilmore 
20:02:39 * jpwdsm is here
20:02:54 <mmcgrath> Ok, well lets get going
20:02:57 * quaid lurks for fun
20:03:32 <mmcgrath> First lets look at meeting tickets.
20:03:34 * dgilmore pust quaid to work
20:03:44 <mmcgrath> And there aren't any so that's good I guess.
20:03:52 <mmcgrath> #topic Infrastructure -- QPID
20:03:59 <mmcgrath> so I've been working on our qpid deployment for community.
20:04:12 <mmcgrath> As it stands I don't think they have any widgets ready for it yet, so it'll sit in staging.
20:04:20 <mmcgrath> I still have a number of questions for someone more experienced with qpid.
20:04:27 <mmcgrath> Specifically around acls, and authentication.
20:04:41 <mmcgrath> I think the orbited interaction will all be completely anonymous at the qpid layer.
20:04:50 <mmcgrath> but I want to know what that means and how to lock it down.
20:05:02 <mmcgrath> I don't think we want people inserting their own routes or anything.
20:05:06 <lmacken> the clients will still have to auth, but most likely through a guest account
20:05:20 <mmcgrath> lmacken: the 'clients' the desktop apps?
20:05:26 <lmacken> or the web browser
20:05:45 <mmcgrath> well, here's the rub right now, and it might be fixable but i'm not sure.
20:06:00 <mmcgrath> I haven't figured out how to get qpid to do anonymous and auth based at the same time.
20:06:14 <mmcgrath> When I do auth=yes in the config, no anonymous connections are allowed.
20:06:25 <mmcgrath> when I do auth=no, all user / password combos are accepted.
20:06:33 <mmcgrath> which means the acl system isn't useful.
20:06:48 <lmacken> Hmm.. well, I assume J5's AMQP.js can auth
20:06:54 <lmacken> and I think guest/guest is enabled by default in qpid
20:07:38 <mmcgrath> It'd probably be worth it for you guys to get a working widget together and a working client up and going in staging so I can test
20:07:53 <lmacken> yep, will do.
20:08:11 <lmacken> I'll just make a basic widget that just spits out all messages that it receives
20:08:16 <lmacken> and we can roll from there
20:08:17 <mmcgrath> K, so I'll sit tight until that time.
20:08:27 <Oxf13> oh Im' here, sorry (:
20:08:29 <lmacken> J5 was planning on releasing his AMQP.js stuff today, I think.
20:08:34 <mmcgrath> worksforme
20:08:48 <mmcgrath> most of the stuff I've been testing had that reply_to flag stuff and got replied to.
20:08:57 <mmcgrath> I'm not sure if that's what you guys have in mind for most stuff, or some stuff or what.
20:09:14 <lmacken> (for an example of how Moksha/Orbited is currently working with live widgets, check the source code for moksha.csh.rit.edu)
20:09:18 <mmcgrath> lmacken: another thing I'm a little confused on is how messages work while I'm away.
20:09:22 <mmcgrath> or will they not?
20:09:41 <mmcgrath> lmacken: like, will messages get queued if a client isn't connected?
20:09:50 <mmcgrath> will some messages get queued depending on the message?
20:09:51 <Oxf13> mmcgrath: I think that depends on the queue
20:09:57 <mmcgrath> there's still a lot of question marks like that in my head.
20:10:04 <Oxf13> whether or not it's durable with guaranteed delivery or not
20:10:04 <mmcgrath> Oxf13: will there be lots of different queues?
20:10:05 <lmacken> mmcgrath: hmm, we'll have to figure those logistics out
20:10:10 <Oxf13> mmcgrath: there can be
20:10:16 <mmcgrath> like will you have a queue and I have a queue?  or will there just be a cvs queue?
20:10:20 <lmacken> usually we're just doing multicast to anyone that is subscribed, so if you're not subscribed, you don't get the msg
20:10:24 <Oxf13> every client is essentially a queue
20:10:36 <mmcgrath> Oxf13: but what if the client is not connected?
20:10:38 <J5> lmacken: I released it 30 minutes ago
20:10:39 <smooge> sorry back
20:10:46 <mmcgrath> that's where the durable queue comes in?
20:10:48 <mmcgrath> it'll just store it?
20:11:01 <Oxf13> mmcgrath: I think so yea.  It depends on how the queue is created when the client connects
20:11:04 <J5> lmacken: It can't auth yet, but that is easy to add
20:11:05 <mmcgrath> I guess that's where I'm still a little confused, it sounds like we'll have an all the above type setup.
20:11:23 <Oxf13> for some of the stuff I want to use the bus for, I think we'll have to have durable queues
20:11:28 <J5> we don't need durable auth
20:11:29 <Oxf13> for things like notifications of cvs changes and builds
20:11:39 <J5> durable queues that is
20:11:40 <Oxf13> and for triggering autoqa events
20:11:45 <mmcgrath> Oxf13: but is the stuff you have in mind also goign to work with community and their external brokers?  or all internal brokers?
20:11:49 <Oxf13> particularly if we tie package tagging into pass/fail of QA events
20:12:00 <Oxf13> mmcgrath: I think it would be all internal, but I'm really not sure.
20:12:11 <Oxf13> I don't know what the community folks have going, I've not come up to speed yet :/
20:12:17 <mmcgrath> Oxf13: honestly I don't even know that the difference matters the way we have it setup.
20:12:26 <mmcgrath> well the way I have it setup now.
20:12:35 <mmcgrath> and I still don't quite get the difference between an exchange and a queue
20:12:46 <Oxf13> I don't know all the terms yet either
20:12:51 <mmcgrath> anyway, I still have some reading to do.  but we do have a functional working setup right now.
20:12:52 <Oxf13> I need to read up and maybe take the RH training course
20:12:55 <mmcgrath> but not one that's ready for production.
20:12:59 <J5> exchange is just the broker with bindings to queues
20:13:13 <mmcgrath> J5: but an exchange can have multiple queues right?
20:13:21 <J5> so you send messages to exchange and it sorts it into a queue
20:13:32 <lmacken> yeah, you send messages to exchanges, not queues.
20:13:37 <J5> like a mail sorting facility
20:13:45 <Oxf13> ah right
20:13:53 <Oxf13> the sender is essentially done once it gets into the hands of the exchange
20:14:00 <mmcgrath> so the queue itself is really just a storage bin?
20:14:03 <mmcgrath> it's literally just a queue?
20:14:04 <Oxf13> then the exchange worries about ensuring all the queues get the message correctly
20:14:05 <J5> yep
20:14:21 <J5> messages can be split into multiple queues
20:14:29 <mmcgrath> Ok, so my next question on that then (and I won't jack the whole meeting for this)
20:14:40 <mmcgrath> if jesse and I are waiting for our builds to finish.
20:14:53 <mmcgrath> and koji sends a message that jesse's build is finished, and my build is finished.
20:15:00 <mmcgrath> how does it get setup so that jesse gets his message and I get mine?
20:15:05 <lmacken> J5: is your deployment proposal ready for prime time?  meaning, can we schedule a meeting with the AMQP/QMF guys yet?
20:15:09 <Oxf13> mmcgrath: here is how I envision it
20:15:23 <Oxf13> mmcgrath: koji would message to the exchange that "foo build completed"
20:15:26 <mmcgrath> do I have to set up some sort of filter to only display my builds and ignore jesses? or does the broker some how route it to me via how I'm logged in?
20:15:28 <J5> mmcgrath: depends, but you can rout on pieces of the message
20:15:45 <Oxf13> the exchange would know what queues such notifications go to, one being our future notification server
20:15:49 <J5> you route to different queues with bindings
20:16:10 <J5> lmacken: sure
20:16:12 <lmacken> yeah, depends if we're doing topic-based routing... so if you were listening to fedora.builds.# vs. fedora.builds.kernel
20:16:21 <Oxf13> the notificatoin server woudl have a queue on the exchange.  The exchagne would deliver a copy of the "build finished" message to the notification server, the server would then generate whatever it wants to generate, like emails, an rss update, etc.. basedon the message
20:16:39 <mmcgrath> lmacken: topics, that's the part I was confused about.  not queues.  Whats the difference between an exchange and a topic?
20:16:46 <abadger1999> mmcgrath, Oxf13: the information  you're talking about would be useful outside... whether we go directly through qpid to get it there, I don't know.
20:16:47 <Oxf13> mmcgrath: I don't think you and I would have personal queues on the exchange.
20:16:57 <mmcgrath> Oxf13: <nod>
20:16:59 <abadger1999> having it available externally will make testing easier ...
20:17:06 <lmacken> mmcgrath: 'topic' is a type of exchange
20:17:13 <abadger1999> especially since we don't have a good solution for testing AGPL stuff inside infra.
20:17:16 <lmacken> along with fanout, etc
20:17:18 <J5> mmcgrath: topic is like the subject field in an e-mail, except you can user wild cards to do a match
20:17:19 <Oxf13> mmcgrath: My thought would be that the notification server would have a queue on the exchange, and would take care of the logic of delivering the email to you and me
20:17:46 <mmcgrath> Ok, well.  Besides the obvious fact that I have some more docs to read, I think it'd be helpful to get at least something up in staging.
20:17:47 <Oxf13> mmcgrath: the notification server would have the logic to see that "smolt" build finished, and here are the people that care about smolt builds, so I'm going to email each one of them.
20:17:51 <lmacken> Oxf13: yeah, the 'community hub' would be consuming the message, and could handle the RSS/Email/etc logic
20:17:54 <mmcgrath> Oxf13: yeah
20:18:09 <J5> mmcgrath: there are different types of match rules, topic being one of them
20:18:18 <mmcgrath> J5: got'cha.
20:18:19 <Oxf13> mmcgrath: the autoqa server would have it's own queue for such messages on the exchange
20:18:25 <mmcgrath> yeah
20:18:46 <Oxf13> and then the autoqa server would consume the message, see it's a finished build for smolt, and then kick off any automated testing setup for smolt specifically, packages in general
20:18:53 <mmcgrath> I think my confusion on this is, after the few server client tests I've written, I realize hwo many different ways the brokers can be setup and how messages go around.
20:19:01 <mmcgrath> I think it'll be helpful to just get some use cases in and going.
20:19:07 <Oxf13> then once autoqa has done its thing, it'll announce the results back on the bus, and...
20:19:16 <Oxf13> mmcgrath: yeah
20:19:19 <Oxf13> the scope could be huge
20:19:33 <lmacken> yeah, I think after once we get a simple usecase running, we should sit down with the AMQP guys and make sure we're on the right track
20:19:37 <Oxf13> it's so flexible that it makes it hard to zoom in on a specific thing
20:19:47 <mmcgrath> J5: lmacken: another question for you, can I make the externally available qpid server any arbitrary hostname?  Or does it have to be at the same address that the URL in the browser is using?
20:19:58 * mmcgrath seems to remember some browser security 'feature' that prevented them being different
20:20:01 <mmcgrath> but I might have made that up
20:20:10 <lmacken> Orbited requires that it be under the same domain iirc
20:20:19 <mmcgrath> lmacken: same domain or same hostname?
20:20:23 <lmacken> I don't think it can do cross-domain request, at least not until HTML5
20:20:34 <lmacken> mmcgrath: domain I believe
20:20:34 <J5> mmcgrath: domain
20:20:46 <lmacken> by HTML5, I mean, newer browsers.
20:20:51 <J5> lmacken: you lose cookies for cross domain
20:20:54 <mmcgrath> K, that's good to know.
20:21:14 <mmcgrath> Ok, so that's really all I've got on this for the moment.
20:21:26 <mmcgrath> J5: lmacken: do you have any ETA you might have some basic foo up in staging?
20:21:45 <J5> I'll have basic subscription in JS done tomorrow
20:22:01 <lmacken> we'll try and get something up next week
20:22:07 <J5> and then setup is a sinch if all we are running is tests.  So Monday or tuesday
20:22:37 <lmacken> (http://www.rabbitmq.com/faq.html has some good explanations for messaging scenarios and such)
20:22:40 <abadger1999> And remember to put it all in rpms for now :-(
20:22:43 <mmcgrath> k, sounds good.  I'll make sure to have my schedule mostly available during that time so any hiccups that do come up can be dealt with.
20:23:03 <mmcgrath> lmacken: thanks
20:23:03 <J5> cool
20:23:18 <mmcgrath> Ok, anyone have any other questions / comments on this?
20:24:32 <mmcgrath> alllrighty
20:24:35 <mmcgrath> #topic smolt
20:24:35 <abadger1999> Are we still following the multiple brokers route?
20:24:47 <mmcgrath> abadger1999: yeah, we'll have a single (or HA) internal broker.
20:24:54 <mmcgrath> and then each proxy server will have it's own sort of 'proxy broker'
20:24:59 <abadger1999> <nod>
20:25:27 <mmcgrath> So just a couple of notes on the smoltdb, I've made some interesting changes to what queries are being run.
20:25:39 <mmcgrath> we still seem to be having daily issues with render-stat but it's in much better shape then it was.
20:26:01 <smooge> cool.
20:26:08 <mmcgrath> #topic Infrastructure -- pgpool
20:26:19 <mmcgrath> So db2 has been running out of open connections.
20:26:29 <smooge> wheee
20:26:30 <mmcgrath> mostly because of poor pooling in our applications.
20:26:44 <mmcgrath> pgpool is a sort of fake postgres server that does this for the applications.
20:26:51 <mmcgrath> I'm working on deploying it in staging right now.
20:27:14 <mmcgrath> I think it'll be useful to mirrormanger, transifex and possibly koji not that we've had any issues with db3
20:27:42 <mmcgrath> Anyone have any questions or concerns on that?
20:28:02 <mmcgrath> I'm debating whether to get this in to production asap or wait until next week.
20:28:17 <mbonnet> mmcgrath: koji uses persistent connections, pooling will be of no benefit.  Also, db3 has been a rock lately, lets not mess with it. :)
20:28:17 <mmcgrath> Only because I think I'm the only one that knows how to use it, and I'm going to be gone from tomorrow until monday.
20:28:28 <mmcgrath> mbonnet: I'm inclined to agree.
20:28:42 <mmcgrath> mbonnet: but even with persistent connections, if I do a load test against koji.fedoraproject.org, won't I eventually use up all the connections?
20:29:21 <mbonnet> mmcgrath: no, I think we've got MaxServers in httpd set lower than the max db connections (or we should)
20:29:27 <mmcgrath> ahh, got'cha.
20:29:43 <mmcgrath> even if we don't, no sense in fixing what isn't broken.  db2 is where our current pain lies so we'll just focus on that.
20:30:11 <mmcgrath> The thing that was odd for me to wrap my head around with this is, unlike alchemy that does pooling per client (like app1 and app2 each have their own pools).  pgpool does it completely external to that.
20:30:16 <mmcgrath> anyway... moving on.
20:30:23 <mmcgrath> #topic Infrastructure -- IPv6
20:30:31 <mmcgrath> So, domsch just sort of threw this all together and got it working.
20:30:36 <mmcgrath> It's worked fairly well.
20:30:46 <mmcgrath> We had a couple of issues that were external to us
20:30:59 <mmcgrath> but daumas as you've seen has gotten some of that all fixed
20:31:20 <mmcgrath> There's been some odder issues like what Oxf13 saw yesterday and the day before.
20:31:31 <mmcgrath> I'm a little worried that other people are running into similar issues.
20:31:42 <mmcgrath> so keep an eye out.
20:31:44 <Oxf13> for me, it was a broken local router
20:31:48 <Oxf13> advertising ipv6 when it shouldn't
20:32:04 <mmcgrath> but it was also very non-obvious to troubleshoot, so some people may not even realize it.
20:32:09 <mmcgrath> ahwell, another First for Feodra ;)
20:32:28 <mmcgrath> #topic Infrasturcture -- rkhunter
20:32:39 <mmcgrath> So I think I've got rkhunter just about configured where it's tolerable.
20:32:52 <mmcgrath> I'm going to make an SOP for it explaining what the emails mean, and what types of false positives are likely.
20:33:09 <mmcgrath> Any questions there?
20:33:16 <lmacken> can rkhunter detect the thing from that time
20:33:19 <lmacken> ?
20:33:25 <lmacken> from The Incident
20:33:26 <mmcgrath> actually I think it would have
20:33:33 <lmacken> I had a patch written a while back, but I'm not sure what happened
20:33:50 <mmcgrath> The way I have it configured presently it would have notified us
20:33:55 <lmacken> excellent
20:34:14 <mmcgrath> which reminds me I need to add another config because it HATES the fact that /usr/bin/.ssh.hmac exists on Fedora machines :)
20:34:55 <mmcgrath> anywho, moving on.
20:35:12 <mmcgrath> #topic Infrastructure -- RHEL5.4
20:35:17 <smooge> joy
20:35:18 <mmcgrath> So we've got a RHEL5.4 upgrade to do.
20:35:30 <mmcgrath> honestly every one of these upgrades to date has been fairly painless.
20:35:34 <mmcgrath> I'm hoping this will be as well.
20:35:45 <mmcgrath> The basic way we've done this in the past -- start with staging (obviously)
20:35:52 <mmcgrath> Once that goes upgrade those servers in production
20:35:59 <mmcgrath> so that'd include the proxy servers, app servers, and db servers.
20:36:05 <mmcgrath> which is a big chunk of the important stuff.
20:36:23 <mmcgrath> Next, pick a physical host in PHX to upgrade and make sure virtualization is still working :)
20:36:30 <mmcgrath> then the rest of those
20:36:33 <mmcgrath> then everything else.
20:36:35 <mmcgrath> kind of one by one.
20:36:39 <smooge> one issue we had with the 5.1 -> 5.2 update was with our xen servers. You had to upgrade the dom0 before you could do the domU's
20:36:41 <mmcgrath> trying to align reboots as we can.
20:36:58 <smooge> it was listed as being needed for later ones too... but not sure if that was the case
20:37:22 <mmcgrath> smooge: that sounds familiar, the newer dom0's worked with the older domu's but not the other way around.
20:38:01 <smooge> so in that case, you need to do xen9 first and then the staging
20:38:05 <mmcgrath> Anywho, that's not urgent urgent, but I'd like to have it done before the beta freeze.
20:38:25 <smooge> I can start on it next week.
20:38:26 <mmcgrath> smooge: do the release notes dictate the same thing for this round or will it not be an issue.
20:38:29 <mmcgrath> ?
20:39:08 <smooge> i am reading through them. its mentioned as a best practice.. but I think they spend more time on KVM!!!!! than Xen this tim
20:39:40 <mmcgrath> heh
20:39:41 <lmacken> do we have any kvm guests yet for testing?
20:39:42 <mmcgrath> quite likely
20:39:49 <mmcgrath> lmacken: app7 is a kvm guest
20:40:01 <lmacken> ah, and app7 is already serving production apps... nice
20:40:05 <mmcgrath> https://admin.fedoraproject.org/haproxy/proxy1/
20:40:08 <mmcgrath> yeah
20:40:21 <mmcgrath> that's worth mentioning as well, if we have any issues with app7, just turn it off, you won't hurt anything :)
20:41:03 <mmcgrath> that does kind of contradict what I just said about esting in staging but we did do some special testing with app7 ahead of time.
20:41:15 <mmcgrath> G_work's been working on getting it and xen6 ready for a while, he's been using the 5.4 beta.
20:41:24 <mmcgrath> Anyone have any questions on this?
20:41:37 <smooge> none from me
20:41:49 <mmcgrath> Alllrighty
20:42:13 <mmcgrath> I think with that I'm done.  Nothing new on the PHX move, some meetings with our main contact have been moved to next week.
20:42:16 <mmcgrath> so
20:42:19 <mmcgrath> #topic Infrastructure -- Open Floor
20:42:24 <mmcgrath> anyone have anything they'd like to discuss?
20:42:29 <abadger1999> Note: orbited and same origin: http://en.wikipedia.org/wiki/Same_origin_policy  Cross site is per hostname (unless there's a browser bug).  domain name == hostname
20:43:20 <abadger1999> Ah, I took off draft status on the licensing page: https://fedoraproject.org/wiki/Infrastructure_Licensing
20:43:26 <smooge> move to phx ( did I miss that?)
20:43:31 <mmcgrath> abadger1999: that's what I remember hearing about.
20:43:31 <abadger1999> Should we list that anywhere/announce it?
20:43:33 <smooge> oh wait you covered it..
20:43:37 <mmcgrath> smooge: just said it a bit ago.
20:43:40 <mmcgrath> not really much else going on.
20:44:12 <mmcgrath> abadger1999: good question, totally up to you.
20:44:21 <mmcgrath> I'd be worried about starting a flame fest myself
20:44:26 <abadger1999> heh :-)
20:44:28 <mmcgrath> but it is an important step for us.
20:45:24 <abadger1999> k  I'll figure something out.
20:45:26 <mmcgrath> smooge: we should probably figure out what we want to go where based on size and A requirements.
20:46:07 <smooge> I am working on an ascii picture today
20:46:13 <mmcgrath> sweet
20:46:15 <mmcgrath> smooge: what for?
20:46:17 <abadger1999> The only thing to note as a change is that I made a policy on AGPL apps until we get tired of it/come up with better: staging has to be like production (rpms + hotfixes listed in trac) and no AGPL apps on publictest.
20:46:29 <smooge> rack space.. or where you talking about soemthing else?
20:46:36 <mmcgrath> abadger1999: <nod>
20:46:42 <mmcgrath> smooge: yeah rackspace
20:46:42 <lmacken> smooge: speaking of ascii pictures: http://ditaa.sf.net
20:46:53 <mmcgrath> smooge: sorry, thought you were working on network diagrams or just plain art :)
20:47:24 <smooge> oooooooh pretty
20:47:39 <lmacken> yeah, I started playing around with that tool last night.. pretty neat
20:47:42 <smooge> i wonder how that will render my old maryilyn monroe ascii calender
20:48:37 <mmcgrath> hah
20:48:51 <mmcgrath> abadger1999: might be good to send another reminder to the list as to our new rules.
20:49:01 <mmcgrath> other then that we've got 12 minutes left, if no one has anything we can close it early.
20:49:06 <abadger1999> mmcgrath: Will do.
20:49:27 <mmcgrath> Ok, well if no one has anything else, I'll close in 30
20:49:50 <smooge> say goodnight gracie
20:49:58 <mmcgrath> good night gracie!
20:50:01 <mmcgrath> Ok, that's that
20:50:05 <mmcgrath> #endmeeting