20:09:24 <jonmasters> #startmeeting Fedora ARM Tech Talk - Debugging ARM vexpress kernels with gdb
20:09:24 <zodbot> Meeting started Fri Feb 15 20:09:24 2013 UTC.  The chair is jonmasters. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:09:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:09:44 <jonmasters> #chair bconoboy clark|w dmarlin j_dulaney pwhalen
20:09:44 <zodbot> Current chairs: bconoboy clark|w dmarlin j_dulaney jonmasters pwhalen
20:09:50 <j_dulaney> jonmasters:  A few minutes
20:09:53 <j_dulaney> But, go ahead
20:09:58 <j_dulaney> I'll follow along
20:10:50 <jonmasters> ok, thank you everyone (and those reading the archives) for joining the second Fedora ARM technical talk
20:11:00 <jonmasters> to follow along, you will need the following:
20:11:18 <jonmasters> 1). A copy of the vexpress (Versatile Express) release of F18, which you can download from the wiki:
20:11:24 <jonmasters> http://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
20:11:41 <jonmasters> 2). A copy of the "debuginfo" package that matches the kernel from the F18 release:
20:11:47 <jonmasters> http://armpkgs.fedoraproject.org/packages/kernel/3.6.10/8.fc18/armv7hl/kernel-debuginfo-3.6.10-8.fc18.armv7hl.rpm
20:12:16 <jonmasters> 3). A suitable gdb to use to debug the kernel. I generally build my own cross compilers using crosstool-ng, however, you can get one from Linaro:
20:12:24 <jonmasters> https://launchpad.net/linaro-toolchain-binaries/trunk/2013.01/+download/gcc-linaro-arm-linux-gnueabihf-4.7-2013.01-20130125_linux.tar.bz2
20:12:39 <jonmasters> (Al Stone will include gdb in the future RPM releases available from Linaro)
20:13:06 <jonmasters> Make sure you have followed the instructions to get a vexpress system setup
20:13:37 <jonmasters> vexpress, or Versatile Express, is a development platform sold by ARM as part of their Keil tooling
20:13:57 <jonmasters> it is a physical system that includes an ATX-like motherboard with slots for two daughterboards
20:14:18 <jonmasters> on the motherboard, there are many devices, such as audio interfaces, MMC controllers, flash memory, and so on
20:14:34 <jonmasters> there is room provided for two additional daughterboards:
20:14:44 <jonmasters> 1). A CoreTile
20:14:47 <jonmasters> 2). A LogicTile
20:15:08 <jonmasters> The CoreTile provides whatever latest and greatest CPU ARM are working on in a removable form factor
20:15:39 <jonmasters> The LogicTile contains an FPGA that allows various SoC (System-on-Chip) specific hardware to be tested before it is fabbed in a silicon foundary
20:16:49 <jonmasters> ARM CPU designers and developers use the Versatile Express platform to test new designs, combining the CoreTile (e.g. a Cortex-A9 containing several Cortex-A9MP cores) with a LogicTile containing their magic SoC hardware bits they are making, for example Calxeda would use a process such as this to design their highbank chip
20:17:19 <jonmasters> The latest and greatest stuff is A15 (Eagle), for which various TC (Test Chips) were initially made available as CoreTiles for Versatile
20:17:39 <jonmasters> Now, not everyone has a Versatile Express hardware platform, but it has another use
20:18:07 <jonmasters> because Versatile is the reference platform made by ARM, it is natural that various emulation software would target (pretend to look like) a Versatile Express
20:18:15 <jonmasters> this is the case for qemu
20:18:28 <jonmasters> qemu provides a "vexpress" model of the physical Versatile Express hardware platform
20:18:59 <jonmasters> it does not implement all of the Versatile hardware features, but it does include the main ones, such as the flash, audio, LCD controller, and so on
20:19:21 <jonmasters> it is this model (emulation of physical hardware platform) that we target in Fedora with the vexpress images
20:19:35 <jonmasters> Let me clear up another confusion too
20:19:58 <jonmasters> People often get confused between qemu used in virtualization like on x86, and qemu used to emulate physical hardware
20:20:37 <jonmasters> When you are doing "real" KVM-style virtualization, like on A15 or x86_64 systems, you are really only using bits of qemu to provide IO for the virtualized system
20:21:00 <jonmasters> When we are using qemu here, however, we are talking about providing a full emulation of a physical hardware platform
20:21:15 <jonmasters> It is unfortunate that there are two very disjoint uses of qemu because this confuses many people
20:22:19 <jonmasters> It also means, for example, that when we get to virtualization later on in Fedora when we are doing the A15 (kernel-lpae) we may use qemu to provide some IO that looks like versatile just because the virt folks in the community are lazily going with vexpress as the environment
20:22:43 <jonmasters> but that does not have any direct bearing on our support for "vexpress" as a hardware target
20:22:46 <jonmasters> does that make sense?
20:23:12 <jonmasters> j_dulaney: ?
20:23:30 <pwhalen> it does
20:23:33 <jonmasters> ok
20:23:41 <j_dulaney> It does
20:23:43 <j_dulaney> Sorry
20:23:44 <jonmasters> so with that cleared up, let's talk about the kernel
20:24:03 <jonmasters> so, there are actually 3 different "vexpress" systems
20:24:20 * j_dulaney is burning some serious cycles
20:24:27 <jonmasters> that is the original one, which is very legacy, and had IO devices located at different memory addresses, etc.
20:24:35 <jonmasters> that is known as "versatile" within the kernel
20:24:41 <jonmasters> (we don't support that)
20:24:57 <jonmasters> then, there is the vexpress we are targeting, which provides a Cortex-A9 based environment
20:25:12 <jonmasters> then, there is a newer A15 (Eagle) model that we are not yet supporting
20:25:28 <jonmasters> so if you ever look at the code in the kernel you will see various different possibilities :)
20:25:39 <jonmasters> When I say vexpress, I mean the A9 one we are using today
20:25:55 <jonmasters> now, first I will suggest we all boot a vexpress system
20:26:15 <jonmasters> but without using bconoboy 's boot script, because I need to make some changes to it
20:26:34 <jonmasters> so, let's boot up a vexpress system in non-graphic mode. Here is what we will do:
20:26:56 <jonmasters> extract the tarball containing the vexpress F18 release
20:26:59 <jonmasters> you should see this:
20:27:08 <jonmasters> [jcm@independence Fedora-18-vexpress-xfce-armhfp]$ ls
20:27:08 <jonmasters> boot  Fedora-18-vexpress-xfce-armhfp.img
20:27:23 <jonmasters> in the boot directory, you will have some of the following files (but not all yet):
20:27:33 <jonmasters> [jcm@independence boot]$ ls
20:27:33 <jonmasters> boot-vexpress                        vmlinux-3.6.10-8.fc18.armv7hl
20:27:33 <jonmasters> initramfs-3.6.10-8.fc18.armv7hl.img  vmlinuz-3.6.10-8.fc18.armv7hl
20:27:43 <jonmasters> take a look at that boot-vexpress file
20:27:49 <jonmasters> in there, you will see the following snippet:
20:28:08 <jonmasters> if [ $GUI = 1 ]; then
20:28:09 <jonmasters> qemu-system-arm -machine vexpress-a9 -m 1024 -net nic -net user \
20:28:09 <jonmasters> -append "rw root=/dev/mmcblk0p3 rootwait physmap.enabled=0" \
20:28:09 <jonmasters> -kernel "$KERN" \
20:28:09 <jonmasters> -initrd "$RAMFS" \
20:28:09 <jonmasters> -sd "$IMAGE" $FDTARG
20:28:11 <jonmasters> else
20:28:13 <jonmasters> qemu-system-arm -machine vexpress-a9 -m 1024 -nographic -net nic -net user \
20:28:15 <jonmasters> -append "console=ttyAMA0,115200n8 rw root=/dev/mmcblk0p3 rootwait physmap.enabled=0" \
20:28:17 <jonmasters> -kernel "$KERN" \
20:28:19 <jonmasters> -initrd "$RAMFS" \
20:28:21 <jonmasters> -sd "$IMAGE" $FDTARG
20:28:25 <jonmasters> fi
20:28:27 <jonmasters> we are going to run qemu-system-arm manually
20:28:29 <jonmasters> make sure you have qemu-system-arm installed
20:28:46 <jonmasters> rostedt: as an aside, thanks for joining us, logs will be available for you to catch up
20:28:55 <rostedt> thanks
20:29:07 <jonmasters> clark|w: perhaps you want to send a private copy to Steven now
20:29:26 <jonmasters> ok, so with the qemu system model installed, we're going to run qemu manually
20:29:30 <jonmasters> like this:
20:29:42 <jonmasters> qemu-system-arm -machine vexpress-a9 -m 1024 -nographic -net nic -net user -append "console=ttyAMA0,115200n8 rw root=/dev/mmcblk0p3 rootwait physmap.enabled=0" -kernel vmlinuz-3.6.10-8.fc18.armv7hl -initrd initramfs-3.6.10-8.fc18.armv7hl.img -sd ../Fedora-18-vexpress-xfce-armhfp.img
20:30:07 <jonmasters> You see there that I have simply copy-pasted the invocation from the boot script, substituting in the correct kernel, initrd, and disk image
20:30:23 <jonmasters> If you were to hit enter, that would boot now
20:30:31 <jonmasters> You can do this if you like, but shut it down afterwar
20:30:34 <jonmasters> * afterward
20:31:05 <jonmasters> the qemu model supports a few things directly, such as loading a kernel and initramfs
20:31:20 <jonmasters> it also supports loading a dtb if it is told to do so (we have not here)
20:31:30 <jonmasters> qemu also contains an internal gdb "stub"
20:31:59 <jonmasters> A gdb stub provides a mechanism for gdb (the GNU Debugger) to remotely control a "target" (in this case a system emulation model of a vexpress)
20:32:12 <jonmasters> gdb requires a kernel with full symbolic information
20:32:47 <jonmasters> that means an unstripped kernel, or a much bigger kernel vmlinux (not not vmlinuz) that still contains all of the link-time information about which source files various code came from
20:33:18 <jonmasters> fortunately, we keep these around in Fedora. You can get one by finding the kernel package in Koji corresponding to an installed kernel. I'll save you the bother here. You want this:
20:33:28 <jonmasters> http://armpkgs.fedoraproject.org/packages/kernel/3.6.10/8.fc18/armv7hl/kernel-debuginfo-3.6.10-8.fc18.armv7hl.rpm
20:33:56 <jonmasters> that "kernel-debuginfo" package contains all of the debugging information in DWARF format that is usually stripped from our "production" Fedora binaries
20:34:10 <jonmasters> I usually take a kernel-debuginfo package and manually extract it, as follows:
20:34:31 <jonmasters> [jcm@independence vexpress]$ rpm2cpio kernel-debuginfo-3.6.10-8.fc18.armv7hl.rpm | cpio -idv
20:34:57 <jonmasters> Doing this will result in a lot of files being extracted under a "usr" directory within the current working directory
20:35:01 <jonmasters> e.g.:
20:35:10 <jonmasters> [jcm@independence vexpress]$ ls -l usr/
20:35:10 <jonmasters> total 4
20:35:10 <jonmasters> drwxrwxr-x. 3 jcm jcm 4096 Feb 15 14:59 lib
20:35:30 <jonmasters> on a normal system, if you installed this directly it would mean stuff would go in /usr/lib
20:35:52 <jonmasters> but we are not installing directly because we are wanting to play with armv7hl kernel binaries on an x86_64 host system
20:36:07 <jonmasters> you will find some fun bits in
20:36:21 <jonmasters> [jcm@independence vexpress]$ ls -l usr/lib/debug/lib/modules/3.6.10-8.fc18.armv7hl/
20:36:21 <jonmasters> total 77876
20:36:21 <jonmasters> drwxr-xr-x.  5 jcm jcm     4096 Feb 15 14:59 extra
20:36:21 <jonmasters> drwxr-xr-x. 11 jcm jcm     4096 Feb 15 14:59 kernel
20:36:21 <jonmasters> -rwxr-xr-x.  1 jcm jcm 79734393 Feb 15 14:59 vmlinux
20:36:54 <jonmasters> because the kernel Kbuild system is wired up that /lib/modules/kernel_version/ contains links to various parts of the kernel build, Fedora also keeps the vmlinux file from a kernel compile in there
20:37:19 <jonmasters> this means that when the debuginfo scripts (within the redhat-rpm-config) run, this all gets stashed in that location
20:37:29 <jonmasters> anyway, that huge vmlinux file is what you want
20:37:53 <jonmasters> I copy it over to where I already have the "boot" subdirectory from my vexpress image. This means, my vexpress "boot" directory looks like:
20:38:06 <jonmasters> [jcm@independence boot]$ ls -l
20:38:06 <jonmasters> total 93440
20:38:06 <jonmasters> -rwxr-xr-x. 1 jcm jcm     1970 Feb  1 21:35 boot-vexpress
20:38:06 <jonmasters> -rw-r--r--. 1 jcm jcm 12542803 Feb  1 21:35 initramfs-3.6.10-8.fc18.armv7hl.img
20:38:06 <jonmasters> -rwxr-xr-x. 1 jcm jcm 79734393 Feb 15 15:01 vmlinux-3.6.10-8.fc18.armv7hl
20:38:07 <jonmasters> -rw-r--r--. 1 jcm jcm  3392832 Feb  1 21:35 vmlinuz-3.6.10-8.fc18.armv7hl
20:38:15 <jonmasters> Notice that I renamed it to vmlinux-kernel_version
20:38:36 <jonmasters> so now I have something I can use with gdb, it's time to get the corresponding source code
20:38:57 <jonmasters> you could take the SRPM from Koji when you get the debuginfo and prep that
20:39:02 <jonmasters> but I am using Fedora GIT
20:39:05 <jonmasters> so:
20:39:35 <jonmasters> in my /data/work/Fedora_GIT directory I have previously done a "fedpkg clone kernel"
20:39:47 <jonmasters> followed by a "fedpkg switch-branch f18"
20:40:06 <jonmasters> then, I have made a working git branch with "git checkout -b f18-lesson"
20:40:35 <jonmasters> and I have reset the contents of that branch to the version of the source closest to the RPM I am playing with:
20:40:50 <jonmasters> $ git reset --hard 0689821d3cb11e672f32a9909a0396d3c0da4314
20:41:17 <jonmasters> if you were on a more recent kernel and were debugging the latest and greatest, you of course would just have it handy
20:41:34 <jonmasters> now, with a copy of the Fedora kernel git, you can prep a source tree:
20:41:50 <jonmasters> $ fedpkg --dist f18 prep --arch armv7hl
20:42:02 <jonmasters> You want to do that way for two reasons:
20:42:11 <jonmasters> 1). You're on a non-f18 branch so it doesn't know the "dist" tag to use
20:42:31 <jonmasters> 2). You want to tell it to prep for "armv7hl" for completeness, or it will prep for x86_64, which is technically pedantically incorrect
20:42:43 <jonmasters> after doing this, you will see a copy of the kernel source in:
20:43:07 <jonmasters> [jcm@independence kernel]$ ls -ld kernel-3.6.fc18/linux-3.6.10-6.fc18.armv7hl
20:43:07 <jonmasters> drwxr-xr-x. 24 jcm jcm 4096 Feb 15 15:07 kernel-3.6.fc18/linux-3.6.10-6.fc18.armv7hl
20:43:33 <jonmasters> change into that directory
20:43:44 <jonmasters> so mine looks like:
20:43:56 <jonmasters> [jcm@independence linux-3.6.10-6.fc18.armv7hl]$ ls
20:43:56 <jonmasters> arch                      ipc
20:43:56 <jonmasters> block                     Kbuild
20:43:57 <jonmasters> config-arm-generic        Kconfig
20:43:57 <jonmasters> config-arm-highbank       kernel
20:43:57 <jonmasters> config-arm-imx            kernel-3.6.10-armv5tel-kirkwood.config
20:43:59 <jonmasters> config-arm-kirkwood       kernel-3.6.10-armv7l.config
20:44:01 <jonmasters> config-arm-omap           kernel-3.6.10-armv7l-highbank.config
20:44:03 <jonmasters> config-arm-tegra          kernel-3.6.10-armv7l-imx.config
20:44:05 <jonmasters> config-arm-versatile      kernel-3.6.10-armv7l-omap.config
20:44:07 <jonmasters> config-debug              kernel-3.6.10-armv7l-tegra.config
20:44:09 <jonmasters> config-generic            lib
20:44:11 <jonmasters> config-i686-PAE           MAINTAINERS
20:44:13 <jonmasters> config-local              Makefile
20:44:15 <jonmasters> config-nodebug            merge.pl
20:44:17 <jonmasters> config-powerpc32-generic  mm
20:44:19 <jonmasters> config-powerpc32-smp      net
20:44:21 <jonmasters> config-powerpc64          README
20:44:23 <jonmasters> config-powerpc64p7        REPORTING-BUGS
20:44:27 <jonmasters> config-powerpc-generic    samples
20:44:29 <jonmasters> configs                   scripts
20:44:31 <jonmasters> config-s390x              security
20:44:33 <jonmasters> config-sparc64-generic    sound
20:44:35 <jonmasters> config-x86-32-generic     temp-armv5tel-kirkwood
20:44:37 <jonmasters> config-x86_64-generic     temp-armv7l-highbank
20:44:39 <jonmasters> config-x86-generic        temp-armv7l-imx
20:44:41 <jonmasters> COPYING                   temp-armv7l-omap
20:44:43 <jonmasters> CREDITS                   temp-armv7l-tegra
20:44:45 <jonmasters> crypto                    temp-armv7l-versatile
20:44:47 <jonmasters> Documentation             temp-x86-32
20:44:49 <jonmasters> drivers                   temp-x86-64
20:44:51 <jonmasters> firmware                  tools
20:44:53 <jonmasters> fs                        usr
20:44:57 <jonmasters> include                   virt
20:44:59 <jonmasters> init
20:45:01 <jonmasters> --- end flood ---
20:45:03 <jonmasters> wcohen: as an aside, welcome, this is logged so you can catch up later, or someone will send you a backlog privately
20:45:06 <jonmasters> ok, continuing...
20:45:11 <jonmasters> so with a kernel tree prepped, and a vexpress system image setup, and a debuginfo kernel extracted, all we need now is a debugger!
20:45:38 <jonmasters> There is not (yet) a generic cross-gdb in Fedora for ARM that can run on x86_64 and target arm
20:45:51 <jonmasters> however, there are great cross compilers originally packaged by dhowells
20:45:59 <jonmasters> and there will be a gdb
20:46:09 <jonmasters> in the interim, please use the Linaro gdb from this location:
20:46:17 <jonmasters> https://launchpad.net/linaro-toolchain-binaries/trunk/2013.01/+download/gcc-linaro-arm-linux-gnueabihf-4.7-2013.01-20130125_linux.tar.bz2
20:46:28 <jonmasters> I keep all of my toolchains in /data/toolchains
20:46:37 <jonmasters> including many I have built myself with crosstool, and so on
20:46:39 <jonmasters> so I have:
20:46:57 <jonmasters> [jcm@independence linux-3.6.10-6.fc18.armv7hl]$ ls /data/toolchains/linaro/gcc-linaro-arm-linux-gnueabihf-4.7-2013.01-20130125_linux/
20:47:06 <jonmasters> arm-linux-gnueabihf  bin  lib  libexec  share
20:47:22 <jonmasters> now, going back to the kernel source directory we extracted before
20:47:35 <jonmasters> if you're in the right directory you will see MAINTAINERS and Documentation and so on
20:47:53 <jonmasters> from there, you want to fire up gdb to debug the kernel vmlinux as follows:
20:48:07 <jonmasters> /data/toolchains/linaro/gcc-linaro-arm-linux-gnueabihf-4.7-2013.01-20130125_linux/bin/arm-linux-gnueabihf-gdb ~/vexpress/Fedora-18-vexpress-xfce-armhfp/boot/vmlinux-3.6.10-8.fc18.armv7hl
20:48:34 <jonmasters> so I am inside the kernel source directory, and I am specifically calling the Linaro gdb and passing it the path to my local copy of the vmlinux kernel
20:48:49 <jonmasters> you will see output ending in:
20:48:59 <jonmasters> Reading symbols from /home/jcm/vexpress/Fedora-18-vexpress-xfce-armhfp/boot/vmlinux-3.6.10-8.fc18.armv7hl...done.
20:48:59 <jonmasters> (gdb)
20:49:24 <jonmasters> now, on another terminal, you want to start up gdb as before, but adding two additional parameters
20:49:58 <jonmasters> oops, I mean not gdb but qemu on another terminal
20:50:03 <jonmasters> so that should have been:
20:50:09 <jonmasters> now, on another terminal, you want to start up qemu as before, but adding two additional parameters
20:50:33 <jonmasters> one parameter will tell the qemu model to start up its internal gdb stub and listen for gdb to connect to it on local port 1234
20:50:46 <jonmasters> the other parameter will tell qemu not to start executing until it is told to via gdb
20:50:57 <jonmasters> those are the -s and -S parameters in this line:
20:51:10 <jonmasters> qemu-system-arm -machine vexpress-a9 -m 1024 -nographic -net nic -net user -append "console=ttyAMA0,115200n8 rw root=/dev/mmcblk0p3 rootwait physmap.enabled=0" -kernel vmlinuz-3.6.10-8.fc18.armv7hl -initrd initramfs-3.6.10-8.fc18.armv7hl.img -sd ../Fedora-18-vexpress-xfce-armhfp.img -s -S
20:51:20 <jonmasters> if I run that now:
20:51:32 <jonmasters> [jcm@independence boot]$ qemu-system-arm -machine vexpress-a9 -m 1024 -nographic -net nic -net user -append "console=ttyAMA0,115200n8 rw root=/dev/mmcblk0p3 rootwait physmap.enabled=0" -kernel vmlinuz-3.6.10-8.fc18.armv7hl -initrd initramfs-3.6.10-8.fc18.armv7hl.img -sd ../Fedora-18-vexpress-xfce-armhfp.img -s -S
20:51:32 <jonmasters> pulseaudio: set_sink_input_volume() failed
20:51:32 <jonmasters> pulseaudio: Reason: Invalid argument
20:51:32 <jonmasters> pulseaudio: set_sink_input_mute() failed
20:51:33 <jonmasters> pulseaudio: Reason: Invalid argument
20:51:39 <jonmasters> <and it stops here>
20:51:55 <jonmasters> ok, now in the gdb window, I connect to that qemu using a special gdb command:
20:52:09 <jonmasters> (gdb) target remote localhost:1234
20:52:10 <jonmasters> Remote debugging using localhost:1234
20:52:10 <jonmasters> 0x60000000 in ?? ()
20:52:17 <jonmasters> (gdb)
20:52:34 <jonmasters> now I have connected gdb to the qemu gdb stub and gdb is controlling qemu
20:52:54 <jonmasters> the qemu is not running until we tell gdb to start it
20:53:05 <jonmasters> but first, let's set a breakpoint to stop execution later
20:53:13 <jonmasters> the classical example is "start_kernel"
20:53:36 <jonmasters> start_kernel is always the first generic high-level C code executed when a Linux architecture gets out of early init
20:53:39 <jonmasters> so let's do that:
20:53:50 <jonmasters> (gdb) break start_kernel
20:53:50 <jonmasters> Breakpoint 1 at 0xc060a50c: file init/main.c, line 467.
20:53:57 <jonmasters> and now, let's start qemu:
20:54:04 <jonmasters> (gdb) c
20:54:04 <jonmasters> Continuing.
20:54:05 <jonmasters> Breakpoint 1, start_kernel () at init/main.c:467
20:54:05 <jonmasters> warning: Source file is more recent than executable.
20:54:05 <jonmasters> 467	{
20:54:37 <jonmasters> no output occurred because this is before the kernel outputs anything, but qemu did run the early kernel code
20:54:51 <jonmasters> now, we can use regular gdb commands like "n" (next) to step over the kernel setup:
20:55:02 <jonmasters> (gdb) n
20:55:02 <jonmasters> 476		smp_setup_processor_id();
20:55:02 <jonmasters> (gdb)
20:55:02 <jonmasters> 482		boot_init_stack_canary();
20:55:03 <jonmasters> (gdb)
20:55:03 <jonmasters> 484		cgroup_init_early();
20:55:05 <jonmasters> (gdb)
20:55:07 <jonmasters> 486		local_irq_disable();
20:55:09 <jonmasters> (gdb)
20:55:11 <jonmasters> 487		early_boot_irqs_disabled = true;
20:55:13 <jonmasters> (gdb)
20:55:15 <jonmasters> 493		tick_init();
20:55:17 <jonmasters> (gdb)
20:55:19 <jonmasters> 494		boot_cpu_init();
20:55:21 <jonmasters> if I keep going, soon I will see output in my qemu windo
20:55:23 <jonmasters> * window
20:56:09 <jonmasters> once you get beyond rest_init gdb will seem like it is not stopping and the kernel keeps going
20:56:36 <jonmasters> this is because at that point the kernel has begun running the scheduler and is in an entirely different code path (initramfs init)
20:56:49 <jonmasters> but we can still ctrl-c in gdb and stop the kernel
20:56:57 <jonmasters> ^C
20:56:58 <jonmasters> Program received signal SIGINT, Interrupt.
20:56:58 <jonmasters> 0xc00ea2c8 in __mod_zone_page_state (zone=0xc06a2940 <contig_page_data+832>,
20:56:58 <jonmasters> item=NR_ACTIVE_ANON, delta=delta@entry=-1) at mm/vmstat.c:223
20:56:58 <jonmasters> 223		if (unlikely(x > t || x < -t)) {
20:56:58 <jonmasters> (gdb)
20:57:08 <jonmasters> in my case, it was well into boot when I did that
20:57:24 <jonmasters> I can type "c" to continue booting as normal
20:57:36 <jonmasters> Fedora release 18 (Spherical Cow)
20:57:36 <jonmasters> Kernel 3.6.10-8.fc18.armv7hl on an armv7l (ttyAMA0)
20:57:37 <jonmasters> localhost login:
20:57:43 <jonmasters> now, how is this useful?
20:57:49 <jonmasters> well, it is useful in several ways
20:58:13 <jonmasters> 1. You can now use gdb to stop the entire qemu model at any time to see how the kernel is working
20:58:24 <jonmasters> 2. You can step through kernel bootup to understand how Linux works
20:58:38 <jonmasters> 3. You can confirm what the kernel is doing at a given moment
20:59:02 <jonmasters> When I told Paul recently that I knew the updated kernels for 3.7 were booting, this is because I ran them in this fashion
20:59:25 <jonmasters> I can see that they boot, they just are unable (with a dtb) to see the serial interface to tell us
20:59:46 <jonmasters> I will be using gdb and some kernel hackery experience to debug what is wrong with that, using this approach
21:00:05 <jonmasters> You can also use gdb with openocd in place of qemu's gdb stub
21:00:21 <jonmasters> if you have a hardware debugger like a Flyswatter from tincantools attached to e.g. a PandaBoard
21:00:36 <jonmasters> then you can run openocd after attaching it, and it will listen on e.g. localhost:1234
21:00:54 <jonmasters> and then you can connect from gdb in the same way to debug physical hardware just like you debug a qemu model
21:01:03 <jonmasters> I can show that another time
21:01:15 <jonmasters> but for now, I think I have covered the introduction I wanted to cover here
21:01:24 <jonmasters> are there questions, comments, suggestions? Was this helpful?
21:01:53 <pwhalen> excellent Jon, thanks for doing this, it was very helpful
21:02:25 * j_dulaney thought so
21:02:33 <jonmasters> well, good. If there are questions I will answer them here or in #fedora-arm
21:02:36 * j_dulaney will poke at it some more
21:02:42 <j_dulaney> Roger
21:02:44 <jonmasters> but if there are no specific questions at this moment, I will end this
21:02:50 <j_dulaney> ack
21:02:52 <jonmasters> going once...
21:03:01 <jonmasters> going twice...
21:03:07 <jonmasters> #endmeeting