22:07:47 <smooge> #startmeeting
22:07:47 <zodbot> Meeting started Sat Dec  5 22:07:47 2009 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:07:47 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
22:08:10 <smooge> Mike McGrath talking
22:08:23 <smooge> Metrics: The difference between problems and measurements
22:09:02 <smooge> Brief introduction, New Repo is Slow, Lunch? Nope. Nagios
22:09:35 <smooge> When you have an issue what you do to get rid of symptoms and what are the real problems
22:10:10 <smooge> if your eye is red, its a symptom. if you have a car in your eye. thats a problem
22:10:34 <smooge> examples of measurements are given: load, temperature, hits/second, page swaps per second 24 lbs of ribs are all measurements
22:11:10 <smooge> problems: OOM Killer, bad offsite mirroring, and Bernie Madoff are all problems
22:11:52 <smooge> Example: koji is slow, why because it shells out to createrepo and was slow
22:12:12 <smooge> first tool was sar and all the lines looked fine.
22:12:23 <smooge> second tool was iperf and the network looked line
22:12:46 <smooge> after this it was get a profile tool with python
22:13:48 <smooge> this required source changes to get the profiler called which meant he wrapped various 'def's
22:14:09 <smooge> the tool he used was hotshot though there are faster ones..
22:14:40 <smooge> this gave out dumps which gprof2dump made pictures of speed area.
22:16:26 <smooge> mike then showed the picture output which is a pretty picture that shows that 87.5% of its time doing a posixpath exists 44192 times
22:17:57 <smooge> root cause turns into NFS is too slow and we go over how could make it better
22:19:20 <smooge> looking at the code it turned out the call was redundant and could be removed.
22:19:42 <smooge> the call being a os.path.exists(foobar)
22:20:00 <smooge> nagios and the pain that it is
22:20:40 <smooge> mirror manager is known to be our most redundant thing.
22:20:53 <smooge> one day we got a nagios saying it was down.
22:21:25 <smooge> checked to see if it was down (which it was).. then go to sar for load/disk/cpu/memory/swapping/ and then iperf for network
22:22:00 <smooge> next steps were looked at (load? apache-status showed a lot of mirror-list requests)
22:22:19 <smooge> why and where were these waiting mirror-lists coming from?
22:26:10 * ricky is on the edge of his seat :-)
22:26:48 <smooge> looked at what the changes were made and found a commit to mirror-manager happened
22:27:34 <smooge> Mike got Matt D to help him go through and found a line that did readlines() versus ohter commands. This caused a huge increase amount of memory or place
22:28:00 <smooge> Options: revert code, add more servers, more proxy caching, use something other than readlines???
22:28:32 <smooge> what will they do? tune in tomorrow.. same McGrath channel same McGrath time
22:28:41 * ricky falls off :-)
22:29:07 <smooge> actually the last option was taken.. it was use f.readline versus f.readlines.. using xreadlines might do in the future
22:30:01 * ricky remembers this issue - xreadlines is deprecated now
22:30:11 <smooge> ah
22:30:22 <smooge> but the code is python 2.4 where its not :/
22:30:27 <smooge> anyway... back to the lecture
22:30:31 <smooge> broad stroke
22:30:40 <smooge> [or maybe it is and I missed it.]
22:30:42 <ricky> I think it was deprecated in 2.4+
22:31:19 <smooge> Broad Strokes: Clearly identify the problem, Clearly identify what the solution should be, Baseline, Baseline, Baseline!
22:33:26 <smooge> Questions:
22:33:43 <smooge> What tools do you use to profile code?
22:34:11 <smooge> Hotshot was what was used.. and how to do wrapper functions.
22:36:02 <smooge> function maps then can be used to find your performance issues
22:38:02 <smooge> we need to do more profiling and will put ricky on doing it after the move
22:38:37 <smooge> ... or at least someone :)
22:40:19 <ricky> After the move = winter break, so happy to help then :-D
22:41:02 <smooge> question #2 ... (real question... ) where is ricky
22:41:35 * ricky is at CMU procrastinating writing an essay
22:42:12 <smooge> they are not impressed with your excuses.. you could have procrastinated here.
22:42:38 <ricky> Heh
22:42:42 <smooge> they really hope you will do well, and make them proud.
22:42:48 <smooge> anyway lecture over.
22:42:49 <ricky> Thanks :-)
22:42:54 <smooge> #endmeeting