17:35:17 #startmeeting 17:35:17 Meeting started Fri Apr 9 17:35:17 2010 UTC. The chair is mchua. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:35:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:35:33 #meetingname Feature profile interview - dmalcom on Python features in F13 17:35:34 The meeting name has been set to 'feature_profile_interview_-_dmalcom_on_python_features_in_f13' 17:35:51 dmalcolm: Tell us about yourself a bit. Who are you? Where are you from? How did you get started working on Fedora, on Free Software, with Python, on Python? 17:36:07 (And then we'll dive into each feature from there - we'll play things by ear a bit.) 17:37:53 I'll be cleaning up this transcript afterwards (on wiki, so we have slightly nicer formatting) and sending out the link for a +1 and any edits before we trumpet things out. 17:38:04 Hi, I'm David Malcolm. I got interested in Linux maybe 10 years ago, working on various things in the GNOME community 17:39:12 In my day job I work at Red Hat, and am in the fortunate position of being paid to work on Free Software (yay!) 17:39:51 \o/ 17:40:03 I learned Python a few years ago, and it quickly became my favourite programming language 17:41:09 Red Hat is now paying me to work on making Python better, and I get to split my time between the "upstream" python.org project, within the Fedora project, and within RH's products 17:41:39 What is it that you like about Python? 17:41:57 (The audience I'm aiming this writeup for, btw, is Python programmers - "how can we make Fedora more awesome for you.") 17:42:44 It fits my mental model of programming very well: things that should be simple are simple, but it can scale up to handle the complicated things as well - without introducing needless complexity as it does it 17:43:20 so I can write a simple script to Get Stuff Done, but it can potentially develop into something larger 17:43:38 * mchua nods. Let's talk about some of the features in F13 that might be of interest to Python programmers. 17:44:04 I've got 3 on my list here - parallel installation of python3, debugging tools, and systemtap probes. 17:44:14 Any that we're missing, or any particular one you'd like to start with first? 17:44:31 that order works for me 17:44:56 All righty. 17:44:59 Python 2 and Python 3 17:45:04 #topic Parallel installation of Python 2 and Python 3 17:45:15 So - what is this feature? Why is it cool? 17:45:28 * mchua goes to copypaste the descript, and we can tweak/annotate 17:45:32 Python 3 fixes a lot of long-standing issues (if I was unkind I would call them "warts") in the language 17:45:37 A packaged version of python 3.* will be provided within Fedora 13 as an optional component, parallel-installable with the Python 2 stack. 17:45:40 The critical system components that use Python 2 (yum, anaconda) shall continue to use Python 2. 17:45:44 17:45:46 * mchua nods 17:46:02 but the cost of doing that is that a _lot_ of things change from Python 2 to Python 3 ; in some ways you can think of them as different languages 17:46:29 yes: we use Python 2 extensively within Fedora 17:46:49 much of Fedora's web infrastructure is written in Python 17:47:39 and the system tools like the updater ("yum"), the installer ("anaconda"), a slew of graphical config tools ("system-config-*") 17:47:57 Hello, lmacken! Just in time for the Python interviewness. dmalcolm is explaining the parallel-installable python {2, 3} feature. 17:48:21 lmacken: We're doing that, then debugging tools, then systemtap tools, in that order. We'll be done within a half-hour (I hope). 17:48:27 mchua: cool, I'm going to be lurking -- got back to back meetings starting in a few 17:49:06 * mchua nods 17:49:29 lmacken: before you go, can you give us a quick sentence or two on who you are and what you do and how you're related to Python + Fedora / these features? 17:50:01 when we talk about installing Python, there are three things: the core "runtime", the "standard library", and a host of other 3rd-party modules on top 17:50:31 dmalcolm: /me nods. 17:50:49 the standard library is often described as "batteries included" as it does a lot, but even so there's a need for the 3rd-party modules 17:50:58 So this Python 2 vs Python 3 switch is something that a lot of Python developers (web devs especially?) will be facing soon? 17:51:10 * dmalcolm isn't sure how many we have for Python 2 in Fedora, but there are a lot 17:52:00 there are hundreds if not thousands of modules, some of which need other modules, and they're all at some point on a spectrum of support for python 3 17:52:38 some new modules were written directly for Python 3; others are pre-existing modules that already support it, some have only just begun porting 17:52:39 mchua: Hi, I'm Luke, and I solve hard problems with Python. I've written, deployed, and currently maintain a variety of web-apps in Fedora's Infrastructure -- all written in Python using the TurboGears framework. I also help maintain close to 100 python modules for Fedora & EPEL. 17:53:02 mchua: as for these specific features -- dmalcolm has been driving them, and I haven't been involved 17:53:10 dmalcom: /me nods. and as time goes by, both the python 2 and python 3 universes will evolve - but they're separate universes, because of the differences between the language versions? 17:53:35 and some modules are basically "dead" in that the code is written and it works, and has worked for years, but may need prodding to get it to work with Python 3. 17:53:51 lmacken: User perspective also important. :) I might come back and poke you about this when you're out of meetings (or quickly in the office next week) if you don't mind, once we've got dmalcolm's dev perspective on those features. 17:54:03 (that's probably an overharsh way of saying that) 17:54:12 mchua: yeah, sounds good. you going to be around for this next week? http://live.gnome.org/Hackfests/Python2010 17:54:15 dmalcolm: Ah, we can edit. ;) 17:54:19 dmalcolm: you too ^^ 17:54:53 lmacken: YES. One of the things I'd like to do in thinking about this feature profile as part of a "make Fedora a better place for Python hackers campaign" is thinking about what to do while hanging out there. 17:55:06 lmacken: I'm attending the PyGTK hackfest next week, yes 17:55:10 mchua: Remy is bringing his OVC/Storyteller team out next thursday for the hackfest, and BarCamp on saturday 17:55:20 mchua: yep, that sounds good to me 17:55:53 So I'd like to start with this interview with dmalcolm - then we'll clean it up, and then the three of us can talk gameplan on the fedora python list. 17:55:57 17:56:01 if that sounds good. 17:56:08 dmalcolm: (Sorry to keep interrupting your flow.) 17:56:22 * dmalcolm will just keep typing :) 17:57:03 Python provides a tool called "2to3" which can automatically convert much Python 2 code to Python 3, provided the code follows some rules 17:57:49 Unfortunately it's often not clear which modules are ported yet, and if they need conversion 17:58:16 For Fedora 13, we provide two Python stacks, the Python 2 one, and the Python 3 one. 17:58:46 For the Python 3 one, we've tried to provide RPM packages of Python code that's known to work with Python 3 17:58:58 See https://fedoraproject.org/wiki/Features/Python3F13#Porting_status 17:59:46 One approach we could have followed was to simply run "2to3" on everything, but doing that you have no guarantee that the end-result actually does what it's meant to 18:00:11 So these packages in F13 have been tested to work with Python 3? 18:00:11 So if you see a "python3-foo" RPM in Fedora 13, you know that it should _actually_work_ 18:00:14 yes 18:00:32 we're not just "throwing them over the wall" 18:01:22 we've gone through various modules, picked the ones that are known to work (possibly requiring steps to make them do so e.g. "2to3"), and tested them 18:01:59 At the same time, we had to make some cleanups to RPM to support multiple Python stacks 18:02:12 and I added some tests to the "rpmlint" tool for this 18:02:15 and we're doing this in part because we need the python3 stuff ourselves. 18:03:13 (basically: compiled .py files get cached as .pyc files, but the format's different between different minor-releases of Python, so we fixed some things to ensure that these are valid) 18:03:36 My hope is that for Fedora 14 we can start cutting over some of our tools to Python 3 18:04:02 I helped port RPM's Python bindings so that they can work with Python 3 (this is in rpm-4.8.0 IIRC) 18:04:30 One other thing I did was write a tool to help people port their C extension modules 18:04:57 One nice thing about Python is that it makes it very easy to write wrapper code that bridges between Python and C 18:05:07 and there's a lot of this code around 18:05:18 Unfortunately it needs some changes between Python 2 and Python 3 18:05:31 I ran into this porting RPM's python bindings 18:05:57 Half the work requires thought, but the other half is fairly mindless, once you get the hang of it 18:06:18 So I wrote a tool to help with the mindless parts, which I called 2to3c, in homage to the 2to3 tool 18:06:36 https://fedorahosted.org/2to3c/ 18:07:08 * mchua looks at the time - (I'm happy to stay as long as you want to talk about this, but will defer to your schedule - good to go 'til 2:30 or 3?) 18:07:16 John Palmieri used this to help him port the DBus python bindings 18:07:29 * dmalcolm has a very open schedule 18:07:52 dmalcolm: Nice. I see the download/usage instructions - looks like it's a pretty new project that's looking for testers/feedback/help. 18:08:11 See http://bugs.freedesktop.org/show_bug.cgi?id=26420 18:08:21 Yes, it's rather "bleeding edge" right now 18:08:32 (Help would be most welcome!) 18:10:42 So hopefully we now have an excellent Python 3 platform in Fedora 13: I believe we have a well-tuned build of python3, and a good selection of add-on modules available via rpm 18:10:57 and this should be useful to people looking to port their code 18:11:05 or to learn the language 18:11:30 arguably Python 3 is easier to learn than Python 2; a lot of unnecessary complexity was removed 18:11:36 So, looking at this feature - I see instructions on how to test it at https://fedoraproject.org/wiki/Features/Python3F13#How_To_Test. 18:11:50 Are those still the best set of instructions for people going "cool, how do I start?" 18:12:42 good question 18:13:38 should that be in How To Test, or in User Experience? 18:14:22 dmalcolm: Both, if they're the same thing. ;) 18:17:00 (I think that section could be improved) 18:17:18 both from the POV of testing, and from marketing 18:19:01 perhaps we can mark those as needing attention, and move on? 18:19:09 Sure. 18:19:16 #action "how to test + user experience" needs some love 18:19:34 The other questions from this section were: 18:19:37 Show me an example. 18:19:37 Give me a few technical details. 18:19:37 How did this feature come into being? 18:19:49 I think we've covered most of those somewhere above, but just in case - last comments on parallel-installable? 18:20:01 It's something that people have wanted for a while 18:20:20 There have been a few proposals on the matter on the mailing list 18:21:05 Ensuring that it was independent of the Python 2 stack was the most important detail, so that we can be sure we don't break it 18:21:29 "Don't cross the streams!" 18:23:55 How is this looking? 18:27:48 dmalcolm: You made a ghostbusters reference. We're all good. :) 18:28:01 Moving on to systemtap probes! 18:28:04 #topic systemtap probes 18:28:11 the other two topics should be quicker, I think. 18:28:21 So. 18:28:25 What is it? 18:28:26 Why do I care? 18:28:29 * mchua looks up summary 18:28:39 https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2 18:28:54 I've written a "top"-like tool that shows you all python function calls per-second across the whole system 18:29:04 "Systemtap allows event tracing of programs when they have static probes inserted. This allows for tracing specifics of an application on a higher level that is meaningful to the application user so they don't have to know the exact source code details for tracing what is happening. Language runtimes can benefit from this by exposing events that make sense to users of those languages/runtimes." 18:29:14 and another one that shows you them hierarchically 18:29:19 I'm... not much of a Python programmer, but I've done a bit of Python dev, and this confuses me - I'm not sure how this applies yet. 18:30:17 so, Systemtap is a tracing/probing/monitoring tool 18:30:58 the idea is (metaphor alert) that you can stick probes under the hood of the engine and see what's going o 18:31:01 the idea is (metaphor alert) that you can stick probes under the hood of the engine and see what's going on 18:31:20 in the past, most of the places where you could probe where in the kernel 18:31:25 in the past, most of the places where you could probe were in the kernel 18:31:31 (where's my grammar) 18:31:47 * mchua laughs 18:32:29 for Fedora 13, I've added places to the Python 2 and Python 3 runtimes that you can monitor 18:32:40 specifically: Python function calls 18:33:12 so you can write scripts that e.g. watch for calls to a particular module, or watch for calls of a particular Python function 18:33:25 across the whole computer, or just in a given process 18:33:46 Nice. 18:34:00 and as examples, I provide precanned scripts 18:34:19 * mchua found https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#How_To_Test with code examples - perhaps at the hackfest next week I can ask you (or mjw, might be a better option) to do a screencast of the quickest demo of this possible. 18:34:20 (a) shows you the function call and return hierarchy for all Python that's running 18:34:29 (b) "top" for Python 18:34:53 so these ought to be useful as is, and people can write their own using systemtap's mini-language 18:35:31 #link https://fedoraproject.org/wiki/Features/SystemtapStaticProbes#.22top.22_for_Python_function_calls 18:35:34 super-cool. 18:35:47 #action provide screencast of the python systemtap static probes 18:35:55 dmalcolm: What sort of Python programmers might care most immediately about this? 18:36:00 (is that the magic syntax?) 18:36:05 dmalcolm: Yep. 18:36:23 are there particular types of projects that this would solve an immediate pain point in, etc? 18:36:24 I showed this to stickster running on a program that he wrote and his eyes lit up 18:36:38 Why's that? 18:36:56 what did it let him understand (or understand more quickly) that wasn't available before? 18:36:58 if nothing else, it's a great teaching tool: you can see what your code is doing, directly 18:37:21 that in itself is lovely. 18:37:25 so it's something tha teven novices can use. 18:37:32 um, "that even" 18:37:37 * mchua not sure wehre typing skillz went this morning 18:37:40 one other use case: a busy python-based website could use this for profiling 18:37:44 s/wehre/where 18:37:49 see what parts of the Python are getting used a lot 18:37:55 Yeah, python web frameworks were my first thought. 18:37:59 or where the time is going 18:38:35 So we've got the video for an example - I can also ask on the python list if anyone else wants to chip in an example. 18:39:02 Any technical details about these probes you'd like to point out? 18:39:12 and also - possibly related - How did this feature come into being? 18:39:38 I should mention that this relates to work done by Sun/Apple on DTrace, which is an analogue to SystemTap 18:39:47 How did you start working (I presume with Mark Wielaard, who's the overall systemtap feature owner) on it? 18:39:51 * mchua nods - that's good to note 18:40:12 there have been some patches to add this support to Python floating about on the upstream bug tracker for a while - but for DTrace 18:40:43 Mark (and others) have added some partial DTrace compatibility to Systemtap 18:41:11 so that it looks like (during the Python build) that we're running DTrace, but actually it's all shimming into Systemtap 18:41:36 so there was some work from the systemtap folks on ensuring that this stuff could work 18:41:55 and I did some profiling, to ensure that there wasn't a performance cost to all of this 18:42:11 (actually, in early versions of the work there was, and we fixed it) 18:42:45 hope that makes sense 18:43:28 * mchua nods 18:43:32 Makes sense to me. 18:43:58 I'm still trying to figure out how a Normal Python Programmer would get started with this coolness, but I think a screencast will take care of it. 18:44:05 Anything else you can think of re: probes? Or last feature? 18:44:25 (I think a pair of screencasts is the way to go) 18:44:36 (showing rather than telling) 18:44:44 * mchua nods 18:44:54 Ok, debugging! 18:44:58 #topic Easier python debugging 18:44:58 yay 18:45:04 #link https://fedoraproject.org/wiki/Features/EasierPythonDebugging 18:45:20 "The gdb debugger has been extended so that it can report detailed information on the internals of the Python 2 and Python 3 runtimes. Backtraces involving Python will now by default show mixed C and Python-level information on what such processes are doing, without requiring expertise in the use of gdb. 18:45:25 We have also extended gdb with new commands, aimed at Python developers, allowing them to see what a process is doing at the Python level. 18:45:29 We believe this ability is unique to Fedora, and will be valuable for Python developers seeking additional visibility into their CPython processes." 18:46:08 One of the great things about Python is how easy it is to wrap external libraries (e.g. written in C) 18:46:41 the downside of this is that if one of these libraries has a bug, then that bug takes out the whole of the Python process, without giving you a nice Exception/traceback 18:47:06 #link https://fedoraproject.org/wiki/Features/EasierPythonDebugging#Detailed_Description 18:47:09 has a nice example of this. 18:47:10 er, "nice" 18:47:16 Since we added the ABRT tool, I see a lot of Python crashes 18:47:34 #link http://fedoraproject.org/wiki/Features/ABRT 18:47:35 ...which typically aren't crashes in Python itself, they're crashes in the libraries 18:47:43 #action mchua see if we can get a better "what's ABRT?" link 18:47:48 * mchua nods. 18:47:56 I've spent a lot of time debugging these things, and I wanted to make my life easier 18:48:21 For example, in Fedora 12 (I believe), we shipped GTK-2.18, which contained 18:48:39 Alex Larsson's bug rewrite of how GTK writes stuff to the screen 18:48:51 greatly reducing on-screen flicker 18:49:03 but the downside is that a few applications broke 18:49:22 An example turned out to be the "istanbul" screencast-recording tool 18:49:35 Figuring that out was "fun" 18:50:11 Python has long had a set of macros for gdb that let you connect to a running (or dying) python process and debug what's going on 18:50:16 but they're fiddly to use 18:50:26 and they assume the process is only "lightly broken" 18:50:44 For example, they add a "pyo" command, for printing python objects 18:51:03 In theory, it's equivalent to "print" in Python on that object 18:51:24 but if the object is internally corrupt, if you run it, you'll merely get another crash :( 18:52:02 The other big problem is that the macros really assume you're proficient with gdb 18:52:16 and know your way around the insides of Python 18:52:21 #link http://www.gnu.org/software/gdb/ 18:52:47 So I started looking for a better way of doing this 18:53:24 In Fedora 12 (I believe), Fedora gained a shiny new version of gdb 18:54:15 ( http://sourceware.org/gdb/wiki/ProjectArcher ) 18:54:56 Various people worked on improving C++ debugging, but one of the by-products of that was that gdb 7 now has the ability to be extended using Python 18:55:51 A bunch of Red Hatters added this; it's now possible to write Python code that hooks into the debugger, to pretty-print data types 18:56:02 what I did was to use this 18:56:14 and I wrote Python code that knows about the insides of ...Python itseld 18:56:16 and I wrote Python code that knows about the insides of ...Python itself 18:56:46 so you now have Python code running inside the gdb process, which knows how to scrape data out of another dying process 18:57:06 the practical upshot is that it's now possible to attach to an already-running Python process with gdb 18:57:15 and type: 18:57:16 py-list 18:57:35 will show you the python source code that's currently running 18:57:37 py-bt 18:57:46 ...will show you a Python-level backtrace 18:57:50 py-up 18:57:56 ...will take you up the call stack 18:58:00 py-down 18:58:07 ...will take you down the call stack 18:58:24 and when you print data, it will tell you what the data is, in a meaningful way 18:59:02 That might be an interesting screencast to do as well - I'll tinker around and see if I can figure it out, actually... this shouldn't be too hard. 18:59:04 so rather than being told the hexadecimal address of where the object is stored in RAM, gdb should tell you that e.g. you have a [1, 2, 3] 18:59:05 I think. 18:59:09 list 18:59:10 #action mchua try makign this creencast 18:59:19 #action mchua learn to spell 18:59:40 the caveat is that it works well on i686, but less well on x86_64 18:59:56 it ought to work on Python 3, but I think there are some bugs there 19:00:03 most of my testing has been on Python 2 19:00:25 I've set it up so that if you install python-debuginfo, it should all Just Work 19:00:55 dmalcolm: would you recommend trying this on python 2 then? are either in the "maybe, still testing" phase? 19:01:14 where should bug reports go? ;) anything for people to watch for? 19:01:23 Plus, now if ABRT detects a crash of a python process, the report should automatically the file/line information at the Python level and the values of all of the Python vars, rather than just hexadecimal noise 19:02:14 I think I still have some testing to do on Python 3 for this, so I'd recommend trying it out on python 2, with i686 19:02:35 please file bug reports against "python" and "python3" as appropriate 19:02:54 (this stuff lives in the -debuginfo subpackages of those src.rpms) 19:03:55 Nice. 19:03:57 Ok, will do. 19:04:11 If you see a Python traceback inside gdb, then that's likely a bug in my code 19:04:27 (please file a bug if you do see this) 19:04:47 The code tries to be robust in the face of arbitrary breakage of the process being debugged 19:05:02 (we are trying to debug crashes, after all!) 19:05:17 * mchua grins 19:05:36 I also recently got this code into upstream, into python's SVN repository 19:05:45 Now, this is something that was made for Fedora - this is the first place it's come out? 19:05:51 * mchua nods 19:05:51 Yes 19:06:03 oh, how many upstreams were involved here? there's fedora, gdb, python... 19:06:15 * mchua trying to hit the 4 F's for this (friends == upstreams) 19:06:47 I believe Debian and Ubuntu have a version of my patch, though I believe their version of gdb doesn't have all of the patches needed to fully support all the extension commands (though the prettyprinting should work for them) 19:07:09 s/a version/an earlier version/ 19:07:36 So this work is going out to other places as well. 19:07:52 Yup 19:08:05 and it's likely to be in Python 2.7 when that comes out 19:08:12 * mchua crosses fingers 19:08:13 though it works fine with 2.6 19:08:26 (it's in SVN for it) 19:08:32 so it's a nice example of Fedora being a place where innovation happens in free software, then goes out to benefit the rest of the system. 19:08:45 testing + feedback on this one is the most helpful thing people can do? 19:08:47 thanks 19:08:51 yes 19:08:54 please test 19:08:57 or are there ways you'd like to extend it right now, that folks could pick up on? 19:09:39 as I said, I've tried to make it robust, but there are plenty of surprising ways in which a complicated program + libraries can fail 19:09:50 so if you see Python tracebacks inside gdb, please do file bugs 19:10:03 * mchua almost done - then I ask you "what do you do when you're not hacking?" and if you have any other thoughts, and then we're finished. 19:10:06 * mchua nods. 19:10:10 also: suggestions for ways of making Python easier to debug would be good 19:10:15 noted! 19:10:28 for Fedora 14 I want to take this further, e.g. maybe adding python-level breakpoints to gdb 19:10:50 (in general, I'd say, hearing about how we can make life better for python devs == helpful - because we're python folks in fedora too.) 19:10:59 Hm. So if folks are interested in that, how should they get started? 19:11:00 (for f14) 19:11:08 when I'm not hacking: hanging out with my wife and cat, pottering about in our garden 19:11:51 Sounds like the good life. 19:12:05 when it's not raining! 19:12:41 One nice thing about this feature is that although it's quite "low-level", the code is written in Python 19:13:21 So a Python developer with an idea for making this better, may well be able to do so directly 19:13:33 * mchua nods 19:13:39 as I said, I'm keen on hearing ideas for improvement 19:13:47 #action mchua make sure there's a "getting started - so you want to help? do X!" link in there somewhere 19:14:13 I'll see what links I'm missing once I clean up the transcript and will ping you + luke + toshio + python list from there... I'm likely to be hacking on this next week in the back corner of the python sprint. 19:14:28 I have a very keen, not-at-all-vested interest in making Python easier to debug! 19:14:38 * mchua is not actually a useful python hacker for that sprint, but can possibly jump in and be a docs ninja at some point - mostly wants to be exposed to python hackers again more, it's been a while 19:15:13 excellent 19:15:58 mchua: so shall we call it done for now? 19:16:30 I hope to blog about some of this, too 19:16:49 dmalcolm: Yes - thanks for spending so much time with this. 19:16:56 I'll clean it up before the weekend ends. 19:17:00 thank you! I guess we drastically overran 19:17:03 * mchua hoping for tonight, but we'll see. 19:17:04 sorry 19:17:04 :) 19:17:13 dmalcolm: No, I was worried about your schedule more - thanks for taking so much time! 19:17:17 This is great stuff. 19:17:20 likewise 19:17:21 thanks 19:17:40 * mchua will email log 19:17:41 * mchua waves 19:17:46 #endmeeting