18:03:58 <jwb> #startmeeting 18:03:58 <zodbot> Meeting started Fri May 31 18:03:58 2013 UTC. The chair is jwb. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:03:58 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:04:05 <jwb> #meetingname fedora-kernel 18:04:05 <zodbot> The meeting name has been set to 'fedora-kernel' 18:04:12 <jwb> #meetingtopic Fedora Kernel 18:04:17 <jwb> #topic init 18:04:32 <jwb> we'll give this a shot this week, even though it's been pretty quiet 18:04:34 <jwb> who's around? 18:04:39 * jforbes is here 18:04:45 <davej> holla 18:05:21 * jforbes notes pretty quiet is relative 18:05:34 <davej> heh 18:05:56 <davej> jforbes: after we've done a runthrough of the releases, maybe you can give some back-story about this weeks fun 18:06:07 <jforbes> Sure thing 18:06:56 <davej> ok, so let's do that first. first up: F17. jforbes ? 18:07:02 * nirik is lurking. 18:07:20 <jwb> #topic F17 18:07:26 <jforbes> Sure, F17 rebase to 3.9.4 is still sitting in updates testing... It really needs karma 18:07:53 <jwb> #info F17 at 3.9.4 in updates-testing. Needs karma, as usual 18:08:05 <jforbes> As in, it has none, and has been in testing for a week now 18:08:25 <jwb> #link https://admin.fedoraproject.org/updates/FEDORA-2013-9123/kernel-3.9.4-100.fc17 18:08:29 * nirik can test later today 18:08:49 <jforbes> Otherwise, fairly quiet on F17, but that's to be expected 18:09:09 <davej> given we're not doing more 3.8's, we might have to push that ourselves in a week or so 18:09:29 <davej> no, wait, what's current in 17 ? did 3.9.2 make it ? 18:09:40 <jforbes> No, 3.8.13 is current 18:09:45 <davej> ouch 18:10:01 <davej> yeah, we're going to have to force the issue there, what with new CVEs coming up all the time 18:10:02 <jforbes> Right, this is the first rebase 18:10:10 <nirik> odd. 18:10:11 <jwb> we did that with 3.8.13 18:10:28 <jwb> i had to give it karma to get it into a state that we could push to stable 18:10:43 * nirik has kernel-3.9.3-100.fc17.x86_64 on his f17 test box and no offer for .4... 18:11:15 <jwb> there was an edit 18:11:37 <jwb> looks like it was changed from 3.9.3 to 3.9.4 while a push was ongoing 18:11:38 <jforbes> there was since 3.9.4 came out before .3 made it to testing 18:11:48 <jforbes> but it shows 3.9.4 is in testing 18:12:04 <jforbes> Since the 25th 18:12:09 <jwb> bodhi is lying 18:12:13 <jwb> koji latest-pkg f17-updates-testing kernel 18:12:14 <jwb> Build Tag Built by 18:12:14 <jwb> ---------------------------------------- -------------------- ---------------- 18:12:17 <jwb> kernel-3.8.13-100.fc17 f17-updates jforbes 18:12:26 <nirik> yeah, something failed. 18:12:30 <jwb> edit while in the middle of a push probably screwed it up 18:12:36 <nirik> possibly the change happened during a push. yep 18:12:39 <jwb> of course, i thought bodhi wouldn't let you do that 18:12:42 <jwb> but hey! 18:13:09 <davej> so what needs to happen there ? manual re-push ? 18:13:17 <jforbes> trying to fix it now 18:13:44 <jforbes> okay, unpushed and repushed 18:13:55 <jforbes> it should go with the next sync? 18:14:15 <davej> maybe it'll get karma now ;) 18:14:29 * nirik was just about to fire a 17/18 push. ;) 18:14:37 <jforbes> davej: I wouldn't count on it too much, 3.8.13 only got 1 18:14:48 <davej> on to 18 ? 18:14:53 <jforbes> sure 18:14:54 <jwb> #topic F18 18:15:18 <davej> so one bug that seems to be affecting a bunch of people since the rebase that's kinda concerning is that we've got a bunch of machines that won't boot due to some EFI changes 18:15:38 <jwb> i have a scratch build running right now with a fix for that 18:15:44 <jwb> koji latest-pkg f17-updates-testing kernel 18:15:44 <jwb> Build Tag Built by 18:15:44 <jwb> ---------------------------------------- -------------------- ---------------- 18:15:45 <davej> oh, awesome 18:15:47 <jwb> kernel-3.8.13-100.fc17 f17-updates jforbes 18:15:48 <jwb> http://koji.fedoraproject.org/koji/taskinfo?taskID=5445919 18:15:51 <jwb> oops 18:15:58 <davej> I really hope that works out, so we don't have to ship f19 like that 18:16:02 <jwb> it's basically a revert of 2 patches 18:16:11 <davej> that's pretty much my only real concern in 3.9.x right now 18:16:16 <jwb> once i test it out here locally, i'll ask people to try it out 18:16:34 <davej> great 18:16:46 <jwb> #info efi bug causing some machines not to boot in 3.9. 18:16:57 <jwb> #action jwb to try scratch kernel and ask for feedback in the bug 18:17:18 <jwb> that bug exists on all 3.9 kernels, so impacts f19-f17 18:18:00 <davej> am I misremembering, or was that mostly HP's ? 18:18:08 <jwb> HPs, and toshiba 18:18:12 * nirik has had some weird lockups of his main virthost on 3.9.x, but not much info gathered yet. Just goes dead, nothing in logs. ;( 18:18:12 <davej> ok 18:19:13 <davej> anything else for 19 ? 18:19:19 <jwb> nirik, there's a macvlan/mactap issue that breaks host machines 18:19:24 <jwb> davej, that was 18 18:19:34 <davej> derp, yeah, that's what I meant ;-P 18:19:42 <jwb> though really... kind of one and the same 18:19:43 <brunowolff> Is f19 going to be fixed for rawhide kernels? 18:19:52 <jwb> brunowolff, ask the systemd people 18:19:58 <jwb> #topic F19 18:20:13 <jwb> so f19 is still chugging along on the 3.9 train. will stay there 18:20:17 <jwb> at least for GA 18:20:24 <jwb> same concerns as f18 18:20:29 <nirik> jwb: oh yeah? bug number handy? 18:20:33 <jwb> nirik, sec 18:21:08 <jwb> 954181 18:21:25 <nirik> thanks 18:21:45 <jwb> i have nothing else for f19 18:22:06 <jwb> #topic rawhide 18:22:25 <jwb> rawhide is at 3.10-rc3 at the moment. fairly stable, despite davej finding weird bugs everywhere 18:22:31 <jwb> #info at 3.10-rc3 18:22:43 <jwb> we also included the kernel-install patches from haraldh 18:22:45 <brunowolff> I thing PAE updates are still not making the right kernel the default. 18:23:09 <jwb> brunowolff, then please continue to update the bug 18:23:21 <jwb> haraldh and kay need to fix kernel-install 18:23:43 <jwb> and if people want rawhide kernels to work on f19, then you need to lobby them to do the same fixes on f19 18:23:53 <brunowolff> I have been waiting for a new kernel since the systemd fix. But I'll try uninstalling and reinstalling one if there isn't an update soon. 18:24:02 <jwb> there isn't really anythign we can do in the kernel spec itself to fix this, aside from reverting the change to use kernel-install 18:24:08 <nirik> rawhide is updating correctly for me now. 18:24:16 <nirik> (with the latest systemd) 18:24:17 <brunowolff> I have done so,e lobbying in the bug. 18:24:26 <jwb> nirik, i'm guessing you aren't using PAE or a flavor kernel? 18:24:33 <nirik> yeah, 64bit 18:24:45 * nirik doesn't think he has any 32bit stuff anywhere anymore. 18:24:55 <jwb> nirik, maybe just try installing kernel-debug ? 18:25:11 <jwb> that would tell us if kernel-install has an issue with flavor kernels, or just PAE 18:25:16 <nirik> I can sure. 18:25:16 <brunowolff> As a work around I have been manually updating the grub.conf files. But I might just move the machines to rawhide to avoid this as rawhide is working fine right now. 18:25:28 <jwb> brunowolff, wait, what? 18:25:53 <jwb> i thought you just said rawhide kernel-install wasn't working for the PAE kernel? 18:26:05 <brunowolff> I do on my one rawhide machine. 18:26:18 <jwb> you do what? 18:26:29 <nirik> which kernel, on what machine doesn't update? ;) 18:26:30 <brunowolff> But I also saw it on my f19 machines running rawhide kernels, before things broke worse. 18:26:39 <jforbes> I have a rawhide guest booting issue on 32bit, but I haven't had time to chase it down 18:26:40 <jwb> oh. that's to be expected 18:27:01 <jwb> because the systemd people didn't build fixes for f19 18:27:36 <jforbes> Yeah, kinda sad that it broke rawhide nodebug for earlier releases 18:27:41 <nirik> kernel-debug updates grub ok here in rawhide. 18:28:12 * nirik removes kernel-debug before he accidentally boots in it. ;) 18:28:12 <brunowolff> The problem is with PAE kernels. 18:28:24 <jwb> brunowolff, PAE kernels in rawhide, or PAE kernels on f19? 18:28:33 <jwb> because if it's the latter, it's to be expected at this point 18:28:41 <brunowolff> The --package parameter is always kernel now, when it should be kernel-PAE for PAE kernels. 18:29:19 <jwb> WHICH RELEASE 18:29:59 <brunowolff> The PAE stuff broke when the switch was made to kernel-install, because when kernel-install calls new-kernel-pkg it uses the --package kernel option. 18:30:35 <brunowolff> When it was called from the rpm scripts it was called with --package kernel-PAE when appropriate. 18:31:14 <nirik> brunowolff: there might be another variable in there. /etc/sysconfig/kernel 18:31:14 <jwb> moving on 18:31:21 <brunowolff> Now things are so broken on f19, the problem isn't visible because nothing is getting added to grub and grub2 conf files. 18:32:22 <nirik> brunowolff: you need to be more clear. ;) a) kernel-PAE in f19 doesn't get added to f19? b) kernel-PAE in rawhide doesn't get added to rawhide? c) kernel-PAE in rawhide doesn't get added to f19? 18:32:48 <jwb> f19 is known to be broken. MOVE ON 18:33:00 <nirik> fair. 18:33:18 <jwb> #topic kernel testing overview 18:33:24 <brunowolff> There were different behaviors at different times, because a second bug occured which masked the first. Thislatter bug has now been fixed in rawhide. 18:33:52 <jforbes> Okay, this week has been the big push to get kernel testing up and running 18:34:12 <brunowolff> I haven't retested in rawhide because there hasn't been a new kernel since the latter bug was fixed. But the fix shouldn't have any affect on the problem I was seeing. 18:34:44 <jforbes> Generally speaking things are moving, the regression test box has all VMs installed and configured (F17/F18/F19/Rawhide all 32bit and 64bit) 18:35:37 <jforbes> We have had some serious delays due to the box not coming back from reboot and an issue with sda last night making it remount / read only 18:36:38 <jforbes> During both of those, the delays were made much worse due to not having full serial access and the java management/kvm being suboptimal, but the infra guys have been working hard to get everything in place 18:36:58 <nirik> hopefully we can finish getting serial sorted out soon. 18:37:09 <jforbes> At this point, the box is back up, and before any further config is done with the harness, we need to get serial working correctly. 18:37:35 <jforbes> As of today, we have access to the serial box, so we just have to make sure the host is talking to it correctly 18:37:59 <jforbes> The total list of outstanding issues for regression/stress testing: 18:38:09 <jforbes> 1) Serial access to host as described above 18:40:07 <jforbes> 2) A bit more polish in how we handle the scheduling of tests. The decision was made to never interrupt a test run even if a new kernel is available. Now the harness is calling a scheduler for every kernel build, and that scheduler will boot the correct VMs, but if they are in mid test I need to figure out how to best make sure they start as soon as they are finished 18:41:05 <jforbes> 3) The rawhide instances are supposed to run trinity once they finish the regression test until a new kernel is available for testing. At the moment, systemd doesn't like trinity. I need to make them work together any better 18:42:20 <jforbes> I am confident 1 can be fixed today, and the other issues shouldn't take too much time after. I have a couple of ideas on how to best handle 2, I just need to see which one is better 18:43:14 <jforbes> And thanks to nirik and smooge for dedicating so much of their time to getting things working this week 18:43:30 <nirik> no worries. Sorry it's been such a pain. ;( 18:44:38 <davej> in parallel with jforbes setting that up, jwb has been working on getting some bits of ltp into our test suite, and I've been working on some IO testing scripts to try as many different storage setups as we can think up 18:45:15 <davej> it's a bit crappy right now, but seems to work 18:47:08 <davej> think that's about it. open floor ? 18:47:15 <jwb> #topic open floor 18:48:21 <brunowolff> I retested Bug 964023 and PAE kernels are still not being set to default in rawhide (this is the discussion from earlier in this meeting). 18:48:51 <nirik> brunowolff: fun. file a new systemd bug I'd say. 18:49:04 <brunowolff> The bug is already assigned to systemd. 18:49:36 <brunowolff> That is not the bug about kernels not getting put in the conf files at all. 18:49:52 <nirik> you have the number handy? 18:50:05 <brunowolff> 964023 18:50:21 <nirik> thanks 18:52:46 <davej> ok, let's close it out. I think we're done here. 18:52:56 <jwb> #endmeeting