20:00:01 #startmeeting FESCO (2010-02-23) 20:00:01 #meetingname fesco 20:00:01 Meeting started Tue Feb 23 20:00:01 2010 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:01 #chair dgilmore notting nirik skvidal Kevin_Kofler ajax pjones cwickert mjg59 20:00:01 #topic init process 20:00:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:06 The meeting name has been set to 'fesco' 20:00:08 Current chairs: Kevin_Kofler ajax cwickert dgilmore mjg59 nirik notting pjones skvidal 20:00:11 * pjones yawns 20:00:14 * skvidal is here 20:00:34 i've been h(ighl)it! 20:00:37 Present. 20:00:37 Here 20:01:03 * notting is hee 20:01:05 * notting is here 20:01:12 * dgilmore is here 20:01:35 I guess we are only missing cwickert. :) 20:01:47 #topic #339 Stale Feature Pages 20:01:51 .fesco 339 20:01:52 nirik: #339 (Stale Feature Pages) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/339 20:02:03 There are 2 of these... shall we do them one at a time? 20:02:13 yeah 20:02:16 first: https://fedoraproject.org/wiki/Features/NouveauDisplayPort 20:02:46 This is at 80%. 20:03:11 i thought bskeggs said that what was there is functional 20:03:18 i can mark it 100% if it makes you feel better. 20:03:24 whats laking and can it land soon 20:03:34 ajax: whats left to do? 20:03:37 it works about as well as intel displayport, last i tried it. 20:03:39 comment from the fesco ticket: 20:03:42 dgilmore: just bugs really. 20:03:44 "For NouveauDisplayPort?, the current status is pretty much as listed on the feature page. Where it's been tested it works with the limitation that the VBIOS has to have initialised it first. I'm not sure if you wish to advertise it as a feature or not with that limitation still in the current codebase." 20:03:55 I think it's a case of "this explains what is really a suite of features, and each of them which has been implemented is basically done, but not all are implemented" 20:04:03 ajax: ok, so as is its in good shape 20:04:16 yeah, it's usable. 20:04:38 So let's keep it. 20:04:40 The release note doesn't really seem to explain it 20:04:41 pjones: yeah, so perhaps we could have those that are not implemented move to a f14 feature and put the ones that are done for f13 at 100%? 20:05:01 maybe we sould ask for more documentation to put into the release notes 20:05:01 mjg59: true. i can smack ben around to get it updated. 20:05:12 I think it could do with a certain amount of cleanup to explain what the limitations are, but otherwise it looks fine 20:05:13 nirik: yeah, if my understanding is right, at least. ajax: am I getting this right? 20:05:27 specifically making sure the VBIOS is initialised 20:05:50 pjones: well. i wrote a DP support page ages ago for F11 that was for all the drivers 20:05:51 ajax: mjg59: if it requires vbios, what happens on S/R? 20:05:55 this page is clearly derived from that. 20:06:02 nirik: then again, the vbios bit is orthogonal to that 20:06:08 What exactly does the VBIOS thing mean for a user? 20:06:18 If your machine doesn't do magic, this won't work 20:06:25 VBIOS here means "you have to have booted with the DP monitor plugged in" 20:06:25 Most machines do magic 20:06:33 Kevin_Kofler: it means we're entirely at the whim of the bios vendor 20:07:04 So basically we need to 1. tell the user to boot with the monitor plugged in and 2. hope their BIOS works, which will usually but not always be the case, right? 20:07:13 I guess that's release note material. 20:07:33 yeah. basically: no hotplug, and the occasional vendor may be crappy. 20:07:41 Kevin_Kofler: right. which is better than before, where it was "even if you boot with the monitor plugged, X won't work 20:07:51 s/X/KMS/ 20:08:03 * nirik is +1 to keeping it, but would like it updated and move to 100% for the things that are landed. 20:08:03 (since nouveau is a KMS-only driver now) 20:08:11 which is to say: this is exactly like my TiVo and HDMI. 20:08:25 keep +1 from me too 20:08:30 +1 20:08:38 +1 20:08:41 +1 20:08:55 (but get it updated) 20:08:59 +1 to keep 20:09:07 but it needs updtaing 20:09:11 +1 to that 20:09:24 #agreed Feature is kept, but needs updating to reflect things that have landed and been implemented. 20:09:37 +1 to keep 20:09:42 second: https://fedoraproject.org/wiki/Features/BetterWebcamSupportF13 20:09:57 This is at 90% 20:09:59 well Hans updated the webcam drivers 3 days ago ... 20:10:20 * skvidal is inclined to say the same thing as the last one 20:10:25 keep it but get it updated 20:10:30 not sure what's missing on this. but then, also not sure what's _improved_ since F12 on this. 20:10:47 ajax: More hardware support. 20:11:02 "Rebase gspca usb webcam driver + sub drivers to latest upstream, this adds support for the following webcam bridge chipsets: benq, cpia1, sn9c2028; and support for new devices and many bugfixes in other gspca-subdrivers " 20:11:02 yeah, if 100% means all cameras work, I don't think thats going to happen. ;) 20:11:10 Kevin_Kofler: the page doesn't indicate which ones are newly supported from F12, that i can see. 20:11:16 sounds like an improvement to me ;) 20:11:25 (unless the new drivers are completly broken) 20:12:01 is hans around? 20:12:04 keep +1 20:12:11 skvidal: i agree 20:12:19 skvidal: not afaict 20:12:21 hansg is on PTO this week, so I couldn't find him to talk to him earlier. 20:12:27 it would be nice to know what exactlyt was added improved on 20:12:43 +1 to "keep, please update the page" 20:13:11 yeah, +1 to that. Updating to show whats changed/added/etc would be very nice. 20:14:15 +1 20:15:09 * nirik waits for more votes. 20:15:17 +1 20:15:22 +1 20:15:26 +1 20:15:27 i guess i did not actually do that 20:16:01 also +1 20:16:18 #agreed Feature is kept, but needs updating to reflect things that have landed and been implemented. 20:16:21 Updating is a good idea inseed. 20:16:26 *indeed 20:16:43 ajax / pjones: would you guys be willing to ping feature owners to update? 20:17:19 I think it'll be next monday before I see him on irc, but sure. 20:17:33 ok. I can update the ticket too, but just in case. ;) 20:17:37 ok, moving on... 20:17:40 #topic #340 Determine and approve F11 end of life date 20:17:42 nirik: i was planning to update the dp one myself and get ben to review it, since he's on my team and i wrote over half that text in the first place 20:17:46 .fesco 340 20:17:47 nirik: #340 (Determine and approve F11 end of life date) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/340 20:17:53 ajax: that would be lovely. 20:18:14 * dgilmore is +1 to the proposed date 20:18:39 * Kevin_Kofler votes +1 too, any date is really fine with me. 20:18:49 +1, sounds fine. 20:18:57 +1, seems ok. would be +2 if it was tomorrow. 20:19:14 last update push would be about 2 weeks before that 20:19:14 +1 to the june 11th 20:19:17 indeed. +1 here as well as long as there is no shout from rel-eng about it... 20:19:30 Sure, +1 20:19:37 nirik: +2 if releng screams, I like watching the suffering 20:19:38 :) 20:19:39 * cwickert is sorry to be late 20:19:57 jwb: so why do we say the eol is after the last push by so long? 20:20:22 Yeah, that doesn't make sense. 20:20:35 mostly i say that because people don't listen and bug me about submitting updates closer to EOL anyway 20:20:41 The last push should be within a day of the EOL or the EOL date makes no sense. 20:20:47 +1 20:20:53 well, i think the idea is that we don't dump a bunch of stuff directly to stable on the EOL date with no recourse if it's broken 20:21:00 notting, exactly. 20:21:18 Still, a strong -1 to having 2 weeks between last push and nominal EOL. 20:21:18 also, updates like "new upstream release" are useless in a release destined for EOL 20:21:26 the last month there should only be security updates 20:21:52 or serious bugfixes, IMHO 20:21:53 -1 to dgilmore's proposal as well, at least bugfixes need to be allowed. 20:22:14 dgilmore, we shut off CVS branching how far before the EOL date? 20:22:16 There's no point in pushing bugfixes to a release that's going to stop getting security support shortly afterwards 20:22:23 jwb: 1 month 20:22:25 when F-13 is released we no longer allow new packages for F-11 we should also limit updates to F-11 to security and critical(packages doesnt work) fixe 20:22:35 Anything that encourages people to continue running unsupported releases is irresponsible 20:22:44 dgilmore: I'd say "bugfixes" more in general. 20:22:44 mjg59, agreed 20:22:54 Kevin_Kofler: no critical bugfixes 20:22:55 And that includes things like the latest point release of KDE. :-) 20:23:03 Kevin_Kofler: if you need a bug fixed update already 20:23:06 well, if we say we are supporting a release, we should support it... 20:23:08 about the only case i can see for bugfixes is if the upgrade path is broken 20:23:17 nirik: Right. 20:23:22 And supporting means fixing bugs. 20:23:27 Even if they're not critical. 20:23:35 Kevin_Kofler: this is an area where you are really wrong and unhelpful to the users experience of fedora 20:23:39 nirik: What do we mean by supporting a release? If there's data loss, hardware damage or security issues, we'd push the update 20:23:43 It also means the last update push should be on EOL day, not 2 weeks before. 20:24:01 look, i'm not going to do a push on the day of EOL 20:24:07 Well, they're tautologically not critical. 20:24:07 if you want it a week before, fine 20:24:18 but the day of EOL is just a waste of god damned time 20:24:34 EOL on day X means EOL on day X, not X-7 or X-14. 20:24:44 What does an update on EOL day *mean*? 20:24:47 That means no more updates starting from X+1. 20:24:54 vote please so i can get on with my day 20:24:54 Anyone running that OS after that day no lnoger has security updates 20:24:59 mjg59: well, sure... but if it's something that is a serious bugfix I could see pushing it... if only so someone could still upgrade, etc. 20:25:04 mjg59: it means the last day that we will push security fixes 20:25:07 well, at very earliest we should have the last push the day /after/ EOL, since we don't give specific times on anything. 20:25:12 anyhow, we are drifting into a new topic here... 20:25:17 mjg59: What dgilmore said. :-) 20:25:29 I think the proposed date was approved. 20:25:33 so if there is a critical security fix we should push it but really we need to encourage updating 20:25:33 * nirik goes to count votes. 20:25:34 nirik: yes, it was. 20:25:35 It was 20:25:42 If we don't push security updates already a week before EOL, we're lieing to our users about the EOL date. 20:25:53 #agreed F11 end of life date will be 2010-06-11 20:25:57 well if you care about security you should move to FX ... X > 11 anyway 20:26:13 so last day security updates don't really matter 20:26:14 but well 20:26:27 thanks nirik 20:27:12 ok, shall we move on? 20:27:16 #topic #342 Look into dnssec-conf continual failure 20:27:20 .fesco 342 20:27:21 nirik: #342 (Look into dnssec-conf continual failure) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/342 20:28:02 They don't seem to be around. 20:28:06 yet another case of "feature introduced in updates is screwed up". 20:28:08 man, if only we didn't push major updates to stable releases. 20:28:25 basically dnssec-conf has had issues, we want to try and help maintainers fix things... 20:28:31 the right thing to do here is to back dnssec out of all the default configs. 20:28:44 (and fix the config updating crap...) 20:28:47 DNSSec was an approved F12 feature. 20:29:03 yeah, it seems more 'bugfix gone horribly wrong' 20:29:04 The problem is that apparently hosts we communicate with requested some implementation changes and those broke things. 20:29:09 Kevin_Kofler: yes, but the stuff that's hitting people updating wasn't. 20:29:39 Or folks who use Fedora on their DNS servers, not sure which it was. 20:29:47 pjones: it seems like dnssec is a half assed thing that doesnt really work as advertised and is super fragile 20:29:56 so, how do we help here? is there any fesco people who would be willing to help the mainainers with the packaging issues? 20:29:58 But in any case, somebody nagged Fedora's DNS maintainers to change stuff which caused the breakage. 20:30:02 dgilmore: I think that may be more true of the bind implementation than dnssec itself. 20:30:07 * dgilmore responded to the devs with my configs for rawhide breakage 20:30:22 pjones: possibly its all binds fault 20:30:24 I think blaming dnssec-conf is besides the point here 20:30:26 (bind is not great. shocking.) 20:31:50 since things are broken for me im happy to work with the devs to try make it right 20:31:53 https://admin.fedoraproject.org/updates/F12/FEDORA-2010-1890?_csrf_token=2ab25137c542ae325729d40d662f0fb4c913217c shows an update being pushed without any positive karma 20:31:57 the biggest issue here is touching config files in scripts like that... and secondly not enough time in testing. 20:32:05 so right now the case is that people doing "yum update bind" on f12 get a bind.conf that won't start bind because it's got a dnssec config block that points at a non-existing cert file 20:32:23 pjones: that was fixed I think. New issues are different. 20:32:24 er, key file, not cert file. 20:32:33 So, yeah, what nirik said 20:32:38 mjg59: yes, it was pushed to stable directly. 20:32:48 nirik: huh. zaitcev blogged about it happening to him yesterday afternoon. 20:32:49 The update wasn't thought through well enough, and there wasn't an opportunity for testing 20:33:08 mjg59: the maintainer even wanted a special quicker push to stable directly. 20:33:20 of course he did. 20:33:21 Looks like an abuse of direct stable pushing. 20:33:22 pjones: odd. 20:34:06 right, so lets move on from pointing fingers and go to: how do we keep this from happening further, and how can we help the maintainers fix things? (sounds like dgilmore is willinng to work with them on this?) 20:34:06 Kevin_Kofler: but aren't they all? 20:34:07 https://lists.isc.org/pipermail/bind-users/2010-February/078726.html is the issue 20:34:19 https://lists.isc.org/pipermail/bind-users/attachments/20100205/33ed56d5/attachment.eml to be more relevant 20:34:29 pjones: Uh no, "one-character typo fix to fix a broken dependency" is safe to push directly to stable. 20:34:32 This is not. 20:34:51 Where to draw the line is hard. 20:35:00 So, we had a problem with the fact that limited-lifespan keys were included in a package without any means of automatically updating them 20:35:21 Kevin_Kofler: I don't agree. 20:35:35 mjg59: Yeah, so they tried to rush a fix for that and that broke things. 20:36:22 but things are still broken in some ways... see dgilmore's and zaitcev's setup? 20:36:31 So, was the update the least-impact way of fixing the immediate problem? 20:36:33 pjones: Say a package accidentally "Requires: gilbc" and this breaks updates for everyone who has an older version of that package, how is it not a good idea to push something directly to stable which just fixes "gilbc" to "glibc"? :-) 20:36:34 there are even tracker bugs for all the bugs. ;) 20:37:14 (of course packages are unlikely to Require glibc, this was just an example. ;-) ) 20:37:41 Kevin_Kofler: why do you think the maintainer isn't just as likely to typo something in the fix as he was in the original? 20:37:47 (I ask this because I've done it...) 20:38:28 Because he diffs what he's about to commit and he only changed that Requires. At worst it's still broken and doesn't make things worse. 20:38:35 so - I think having a rule which is 'no direct to stable' updates makes sense 20:38:42 skvidal: I don't. 20:38:47 See discussion above. 20:38:51 I think realizing that we may need to circumvent that rule 20:38:52 on occasion 20:38:54 is just reality 20:38:59 Kevin_Kofler: people are really bad at following processes like the one you just assumed everybody follows perfectly. 20:39:01 *really bad* 20:39:03 if the circumstances are so desperate 20:39:10 in other words 20:39:13 the rule makes sense 20:39:17 unless we're in a dire emergency 20:39:22 bodhi says that this update spent 4 days between submission and being pushed 20:39:22 and in that case all bets are off anyway 20:39:25 pjones: I always diff what I commit. 20:39:28 skvidal: im in complete agreement 20:39:30 Why did nobody ntice the breakage in that time? 20:39:43 mjg59: it was submitted on a friday I think... no pushes sat/sun/pushed out monday. 20:39:46 Kevin_Kofler: I'm glad you're great at it, but I don't think your perfection generalizes very well. 20:39:48 So I don't want Bodhi to stop me from pushing stuff to stable just because some people can't be bothered to diff their changes. :-/ 20:39:49 mjg59: b/c it wasn't pushed anywhere in the interim, so no one could notice 20:39:51 mjg59: it was never in updates-testing. 20:39:57 Ok. 20:40:16 (That said, I'm by no means perfect.) 20:40:16 mjg59: because no one tested it 20:40:21 Ah, yes, looking at the days makes more sense 20:40:41 (I just try hard to make sure I don't screw up what I commit, because I surely would otherwise. ;-) ) 20:40:50 * nirik notes the maintainer also requested a push to stable in epel, but the epel policy of 2 weeks in testing was observed instead. 20:40:59 Ok. So things had been broken since December 20:41:10 I think it's pretty clear that waiting a few more days wasn't going to result in the world ending 20:41:24 mjg59: indeed 20:41:25 right. 20:41:30 Kevin_Kofler: i'm not sure why the entire system should be tailored to suit an individual developer's needs. (see also the request today that all users of QT check that the buildroot is stable before ever trying to push anything) 20:41:39 So, what was the rationale for the immediate push? 20:41:41 so, how can we communicate to maintainers better or enforce a policy that avoids issues like this? 20:42:04 just forbid direct pushed to stable? 20:42:27 are we trying to solve that now, or solve 'get working dnssec out'? 20:42:32 disable them in bodhi and require manual pushes by rel-end in a *real* emergancy 20:42:42 notting: good question. both? but one might be easier than the other... ;) 20:42:44 (except for security) 20:42:47 cwickert: -1 20:42:50 notting: Are any of us sufficiently well-versed in dnssec to fix this ourselves? 20:42:55 mjg59: i certainly am not 20:42:56 We should fix this problem. 20:43:00 Kevin_Kofler, i was just thinking loud 20:43:15 Kevin_Kofler: The problem is that an update went to stable without any testing. 20:43:19 How do we solve that problem? 20:43:26 notting: BTW, for Qt updates, KDE SIG will try to use a safer process next time (but that's a bit OT here). 20:43:35 Kevin_Kofler: in what cases does not being able to push to stable cause you issues? 20:44:11 proposal: we should not allow push direct to stable, in the event of severe emergency, we can call emergency sessions of fesco to override. or simply churn testing REALLY fast 20:44:22 nirik: Important one-line bugfixes take twice as long to go out. 20:44:34 Often the issue can be very annoying and the fix trivial. 20:44:36 2nd proposal: dnssec needs to be fixed or reverted at this point it's impacting others 20:44:40 That's when I hit "push to stable". 20:44:42 If they're important one-line bugfixes, how did the buggy version get into stable in the first place? 20:44:43 Kevin_Kofler: but would not that bug have been found in testing if the update in which it happens had to go to testing? 20:44:47 skvidal: +1 20:44:50 nirik: No. 20:45:08 Usually the update spent time in testing, then a day after it goes stable people notice issues. 20:45:15 A lot more people run stable than testing. 20:45:34 Kevin_Kofler: The fix for "things aren't tested" isn't "test them by pushing them to stable"! 20:45:49 * notting backspaces, just types "what mjg59 said" 20:46:03 skvidal: I don't think there is a clear revert path here... 20:46:31 nirik: okay s/or reverted// 20:46:50 the probelm is it needs to learn how to play nice with lots of different bind configs 20:46:55 I would probibly be a weak +1 to no stable pushes directly... but I would say we should have a clear override setup for it in place... 20:47:28 skvidal: +1 20:47:31 The problem we have at this point would appear to be that there are diverse configurations and that the updates have potentially altered them further 20:47:32 nirik: security only and only with it having been tested an oked by someone other than the developer 20:47:37 on proposal 1 that is 20:47:38 so we could address Kevin_Kofler's concern for fixes. 20:47:54 What would be the override procedure? 20:48:06 OK by 1 rel-eng member would be fine with me (I could just bug rdieter ;-) ). 20:48:16 how much direct to stable do we have now thats not security? 20:48:28 nirik: if we have ANY then that's probably not good 20:48:30 nirik: lots 20:48:41 skvidal: well, there's at least dnssec-conf I suppose. ;) 20:48:41 nirik: alot of people push straight to stable always 20:48:52 At this point, it might be best to just forget about dnssec-conf in 11 and 12 20:49:02 Everyone that's ging to be broken by it probably already has been 20:49:13 There's no magic way to speed up fixes for that 20:49:16 mjg59: we at least need to issue an update that cleanly disables it 20:49:21 And no clear way to revert to the original state 20:49:23 mjg59: on the premise that anything that requires a 50 line %post to fix is going to cause more issues? 20:49:39 notting: Right. We (as in Fedora) fucked up, admins are going to have to cope. 20:49:40 Bodhi initially didn't allow direct stable pushes. 20:49:43 notting: :) 20:49:43 It didn't work out at all. 20:49:49 Many people complained. 20:49:52 also, how hard is it to change bodhi to not allow stable ? 20:49:53 The focus should be on ensuring that this doesn't happen again 20:49:54 Then lmacken implemented them. 20:50:07 Please don't go backwards. 20:50:12 Kevin_Kofler: that doesn't disprove anything about 'many people' being wrong 20:50:14 Forced bureaucracy is really unhelpgul. 20:50:19 *unhelpful 20:50:37 notting: Except those many people brought out several practical issues which resulted from this. 20:50:37 Blocking direct pushes to stable would help, but doesn't solve the fundamental problem that things are getting pushed without being adequately tested 20:50:40 Kevin_Kofler: how is requiring testing backwards? 20:50:50 or at least requiring the opportunity for testing? 20:50:55 pjones: It's going back to something which was tried and didn't work. 20:51:01 nirik, I just checked, shouldn't be too hard to only allow releng or something like that 20:51:06 Kevin_Kofler: we need to make it work 20:51:07 right, so perhaps we should get folks working on/counterproposing a new updates setup... 20:51:09 your anecdote doesn't in any way imply that it didn't work. 20:51:21 Kevin's said that even the current procedure of updates-testing doesn't result in enoguh testing 20:51:25 (plenty of people complain about taxes, but look where we'd be without them...) 20:51:51 without socialised health care ;) 20:51:54 Maybe there just aren't enough people using updates-testing. 20:51:55 Bugs were taking ages to get fixed because they'd have to wait for at least 2 pushes. 20:51:57 Which is a pretty stunning indication that we shouldn't be pushing *anything* except security fixes, because we have no confidence that they're tested 20:51:59 http://fedoraproject.org/wiki/Desktop/Whiteboards/UpdateExperience 20:52:24 mjg59: It just means updates-testing is of limited use and most stuff should just go directly to stable. :-) 20:52:34 Kevin_Kofler: In the rest of the world, it takes longer than a week for fixes to appear on users' systems 20:52:43 It turns out that there's a reason fr that 20:52:48 Kevin_Kofler: lets stop doing releases and just make everyone use rawhide 20:52:51 But we want to be better than the rest of the world. ;-) 20:53:04 can we have a vote on the proposals? 20:53:07 gholms: there certainly aren't, but that can't possible effect packages that never show up there at all. 20:53:09 dgilmore: The idea is that maintainers don't push risky changes to a release and especially not directly to stable. 20:53:13 skvidal: i've lost track of them in the insanity 20:53:14 But not all changes are risky. 20:53:18 since it seems like everyone is arguing only with kevin 20:53:22 proposal: we should not allow push direct to stable, in the event of severe emergency, we can call emergency sessions of fesco to override. or simply churn testing REALLY fast 20:53:24 * nirik personally thinks the epel process has been working nicely... 20:53:30 Kevin_Kofler: but people have proven to not observe that 20:53:31 skvidal: +1 on both. 20:53:32 We have one single change here which is bad. 20:53:38 We should get that change fixed. 20:53:41 nirik: I think time-based is probably a hang up - but.... 20:53:43 Instead of "fixing" a policy which is not broken. 20:53:47 Kevin_Kofler: that's why he had a second proposal. 20:53:48 One incident can always happen. 20:53:52 skvidal: +1 20:53:53 It's just that single incident. 20:53:56 And the policy is /obviously/ broken. Everybody seems to know it. 20:54:12 pjones: This is complete nonsense. 20:54:27 2nd proposal: dnssec needs to be fixed and documented how to unscrew it manually at this point it's impacting others 20:54:27 Existing policy is broken, but I feel a little awkward about voting without a solid idea what our replacement policy is 20:54:27 I'll assume by "this", you're referring to your own comments. 20:54:30 Hardly anybody outside of FESCo sees stable pushes as "obviously broken". 20:54:38 skvidal: -1 20:54:38 Kevin_Kofler: one update that hoses all users of a package outweighs 200 bugfixes pushed 5 days earlier 20:54:45 Kevin_Kofler: Then they can get themselves elected next time 20:55:00 there have been other cases as well... recall dbus ? 20:55:04 I'd (grudgingly) approve if there was an override procedure which can actually work, like approval by 1 rel-eng member. 20:55:15 skvidal: +1 to your second proposal. although that seems more common sense than anything else :/ 20:55:22 mjg59: force every update to go through testing. pushed to stable only via enough karma. security updates cen be rushed though by ensuring quick testing 20:55:23 So you're using as evidence against a change that those who don't have to deal with the repercussions don't notice that something is a problem? 20:55:25 It'd prevent total stupidity and we can always find one rel-eng member to approve our urgent pushes. 20:55:25 really? 20:55:25 notting: :) 20:55:32 But emergency FESCo sessions?! Come on… 20:55:38 FESCo really has other things to do! 20:55:53 Kevin_Kofler: like argue with you all day? 20:55:57 How about a fesco ticket, folks vote in ticket? 20:56:02 nirik: works for me 20:56:07 Kevin_Kofler: implying rel-eng as a gatekeeper of what is stable or tested enough to go out seems to be putting it at the wrong level 20:56:23 seriously, if you don't think 10 minutes to discuss whether or not an update is really so critical that it doesn't need a testing push isn't within fesco's perview, you could always resign. 20:56:25 dgilmore: Would we have any other means? Push direct to stable after two weeks? 20:56:33 +1 to dnssec needing fixing, but -1 to the "ban stable pushes" nonsense. 20:56:39 mjg59: i would honestly say no 20:56:50 notting: Then put QA or whoever in charge. 20:56:54 +1 here to both with that change. (and if dgilmore would be willing to work with the maintainers that would be great) 20:57:03 Or heck, even 1 FESCo member. :-) 20:57:29 But so that you can actually ping 1 person on IRC for an urgent push without having to summon a whole committee! 20:57:58 * nirik suspects often you could find 5 fesco folks on irc on pretty short notice. 20:58:01 Calling a meeting takes at least 2 days. 20:58:10 Kevin_Kofler: lets use a ticket. 20:58:17 So, I'm fine with the idea of using a ticket for this 20:58:17 Kevin_Kofler: of course you'd say 1 FESCo member, becuase from your statements, it sounds like you would intentionally sabotage the process and approve everything 20:58:21 That also takes at least 1-2 days to get 5 votes. 20:58:31 a ticket is fine by me. 20:58:36 notting: I'd not approve everything! 20:58:39 Just what makes sense. 20:58:48 But I think it's going to take a few cycles before people settle down on what "urgent update" actually means 20:58:48 Kevin_Kofler: look, if we have to, we can use a phone tree just like the PTA does. 20:58:52 we will also need to change some docs for this I bet... new packages could then no longer go direct to stable. 20:58:56 If somebody queues "update from fooapp-1.0 to fooapp-2.0" directly to stable I'll veto it! 20:59:05 Because "one line bugfix" doesn't fit my definition unless it's deleting people's files 20:59:37 nirik: no comment. 20:59:38 anyhow, votes? 20:59:44 mjg59: If there's no risk, then I don't see what's the problem with letting it go directly to stable. 20:59:59 Kevin_Kofler: "no risk" is *not* the same as "urgent push" 21:00:10 Kevin_Kofler: how on earth can you still believe that there's ever no risk? 21:00:17 * notting is +1 to disabling pushes directly to stable. 'fesco ticket' is fine for now, but would like to have something better. 21:00:20 Right, but both are criteria in favor of going directly to stable. 21:00:21 actually, one sec 21:00:35 jwb: the implication here is that people would like testing pushes more frequently. is that even feasible? 21:00:40 pjones: Because there's no evidence to the contrary? 21:00:44 I think we are at +3 / -1 for the first proposal. 21:00:53 Broadly speaking, I'm +1 21:00:57 We have one single update which was screwed up. 21:01:05 notting, i do them daily. for f13-testing more than once. beyond that, no not much 21:01:07 This doesn't mean that ALL updates are risky! 21:01:12 notting: not really more than once a day 21:01:13 But, ideally, I'd like to ensure that we have a policy that we agree on before changing things 21:01:25 jwb: daily i can handle. i just didn't want us to be stuck where we could only push them weekly 21:01:37 All we need to do is telling people to judge risks more carefully. 21:01:44 A blanket ban is always the wrong answer. 21:01:58 then should we write this proposal up, get a firestorm^H^H^H^H^Hfeedback on mailing lists and vote on it next week? 21:01:59 This is a social problem, a technical/bureaucratic solution is bound to suck. 21:02:09 It's also not going to solve the problem. 21:02:19 The update can go through 1 testing push and still be broken. 21:02:21 Kevin_Kofler: but it's clear that social answer isn't going to work... as we continue to see this happening... 21:02:25 (and be pushed to stable anyway) 21:02:29 you're claiming that the problem is poor judgment about the risk. this is incorrect. the problem is poor judgment about the need. 21:02:35 nirik: This is about ONE SINGLE UPDATE. 21:02:42 Kevin_Kofler: would you like some more examples? 21:02:44 It's not a problem which deserves a solution affecting everyone. 21:02:55 dbus, dnssec, thunderbird 21:03:01 Kevin_Kofler: We hear you. We disagree. 21:03:02 those are 3 off the top of my head 21:03:11 jwb: I can find more from #fedora logs... 21:03:12 jwb: The other 2 were security updates. 21:03:36 They'd still be allowed under this policy (and banning direct stable pushes of security updates is completely asinine, it leaves machines open for exploitation). 21:03:57 actually 21:03:57 There have been other non security updates that have caused breakage/serious issues. 21:04:11 i'm all for having direct stable security pushes require QA signoff. we'd need to scare up some resources 21:04:15 defining further qa mechanisms for security updates might be appropriate, but it's separate from this particular discussion. 21:04:19 but there's no reason those shouldn't be tested 21:04:40 Then why can't we allow all direct stable pushes as long as QA signs off? 21:04:43 critical path now requires karma on f11/f12 ? or is that only f13? 21:05:05 Proposal: We write a draft policy on stable updates and vote on it next week 21:05:18 Kevin_Kofler: because security updates have necessity prima facae. 21:05:19 nirik, f13 only 21:05:26 jwb: ok, thanks for the info. 21:05:33 That gives us an opportunity to discuss various consequences with affected teams 21:05:34 mjg59: +1... who is writing it though? :) 21:05:39 I'm happy to start 21:05:43 pjones: So do updates for "this package is completely broken", which some direct stable pushes were about. 21:05:49 mjg59: actually, then -1 to skvidal's first proposal, and +1 to mjg59. best to have a full policy to vote on 21:06:01 skvidal: How do you feel about that? 21:06:05 I'm fine with withdrawing 21:06:18 Kevin_Kofler: those packages can't do harm by not being updated - which means them going to testing is perfectly appropriate. 21:06:20 mjg59: excellent. Did you see the Whiteboard proposal for updates link? might look at that as well for more ideas/context. 21:06:27 nirik: Yup 21:06:39 mjg59: I'll be glad to read what you start with and help 21:06:47 * nirik too would be happy to help 21:06:55 Collecting feedback is definitely a good idea, I hope we'll get some strong negative feedback for this proposal. ;-) 21:07:05 I have a feeling that we're getting inundated with stupid bureaucracy. 21:07:15 * pjones is +1 on mjg59's plan as well. 21:07:16 Kevin_Kofler: I have a feeling I'm getting to the limit of my patience with you 21:07:23 Kevin_Kofler: tread carefully. 21:07:41 First the critical path crap, now this. 21:07:46 ok, so on this topic we are going to try and write up a draft. and dgilmore is going to work with the dnssec-conf maintainers? 21:07:55 da 21:07:55 nirik: yes 21:07:56 Sounds good 21:07:59 nirik: that sounds like a plan 21:08:03 Maintainers need to be more careful about what they do, just banning things will NOT solve the problem. 21:08:20 It'll just make life harder for those of us who DO care about providing a stable and bug-free experience to our users. 21:08:28 #agreed dgilmore will work with dnssec-conf maintainers. mjg59 is going to write up a updates proposal for further consideration. 21:08:43 #topic #341 provenpackager request - chkr 21:08:45 Kevin_Kofler: we all care about that. stop throwing insults around about the rest of our conviction, okay? 21:08:46 .fesco 341 21:08:48 nirik: #341 (provenpackager request - chkr) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/341 21:09:05 there was feedback on the sponsors list. All positive I think. 21:09:18 yes, all feedback was positive. +1 from me. 21:09:19 +1 on this; he does a lot of work. 21:09:32 +1 here. mono needs more folks to care for it. 21:09:52 +1, I don't have a personal opinion, but all the feedback was positive, so why not? 21:10:05 +1 21:10:20 I dunno - if he can't even get ralf mad at him, what kind of guy would we be approving? :) 21:10:22 * skvidal kids 21:10:23 +1 21:10:26 +1 21:10:34 #agreed chkr is approved as provenpackager. Congrats. 21:10:43 +1 21:10:55 #topic Fedora Engineering Services Tickets/Discussion 21:11:05 So, I talked about this some at the end of last meeting. 21:11:09 skvidal: I hope I can get Ralf to shout at YOU, I know he hates bureaucracy. ;-) 21:11:11 mmcgrath: you around? 21:11:23 nirik: yo 21:11:25 I filed some random tickets: https://fedorahosted.org/fedora-engineering-services/report/6 21:11:45 * mmcgrath has seen some and done some. 21:11:50 err done one 21:12:11 * jds2001 is involved in pinging the sigs 21:12:17 gotten some response, actually :) 21:12:18 So, should we have fesco members file things? should we consider and approve things in this meeting to be tickets? Should we advertise this to get more people doing tickets? 21:12:37 what level of tickets are we looking at? 21:12:43 * nirik waits for fesco folks to look at the tickets he filed and see if they think they are usefull or the like. 21:13:06 do we want to file ones for 'fix broken deps'? 21:13:11 notting: so far, 3 tickets, 1 closed 2 in progress. 21:13:14 notting: the idea is that these are things fesco would like to know/get done, but fesco members are busy. So a pool of people willing to put in a few hours a week would work on them. 21:13:16 notting: that's certainly reasonable 21:13:19 and similar lower-level fixes? 21:13:39 notting: I would think so, for stable releases... 21:14:14 but I would suggest it be: find out why deps are broken, make sure tickets are filed and maintainer knows about it, if no response or fixes from maintainer, step in and fix yourself. 21:14:32 i think we do need to advertise this more - we currently have two contributors, mmcgrath and myself :D 21:14:43 nirik: ok. we'd probably have to advertise that 'may require provenpackager' 21:14:45 I think this is a great way for us to get things done that we just never get around to/have time for. 21:14:52 notting: agreed. 21:14:53 i meant to add some bugs to that list. bad me. 21:15:09 so, for the meeting record... 21:15:22 notting: certain people wont be suited for certain tasks, for sure. 21:15:55 nirik: ok. existing tickets look fine, let's file more and see what sort of pickup we get 21:15:57 fes ticket 1 Investigate Fedora versions in VPS providers - mmcgrath did this. The major providers do offer Fedora 12. 21:16:06 any ideas for how to promote this more? 21:16:07 Amazon doesn't. 21:16:11 people should only take tasks they are capable of 21:16:22 gholms: yeah, noted. being worked on. 21:16:40 fes ticket 2: SIGs roundup and pinging - jds2001 is working on this. 21:17:07 fes ticket 3: Fedora as Android Development platform - mmcgrath was going to look at this. 21:17:18 Linode has only F11. :-( 21:17:43 Kevin_Kofler: yeah, at least it's a supported release currently. 21:17:44 You can upload any arbitrary VM image yourself, but they don't have F12 as a premade offering. 21:18:07 we could send them a nicely worded email asking them to provide a f12 instance. 21:18:36 and/or pass this info off to the cloud-sig to do so. 21:18:45 Should I bring that up at the cloud-sig meeting? 21:19:09 i thought we were working on creating ec2 images as part of the f13 release process 21:19:25 so while it sucks that there is not f12 images 21:19:29 We are; it's still a work in progress. 21:19:32 gholms: please do. Perhaps a "contact VPS provider" SOP could come out of it? 21:19:38 going forward we will upload official images 21:20:07 I'm a bit concerned about the sheer number of VPSs out there. 21:20:18 Who do we contact, whose image formats do we supply, etc. 21:20:27 * nirik notes these are the kind of proactive things I would like to see happen more rather than just us reacting to exising things. 21:20:57 gholms: sure, it's a open question... but I think we could open a dialog with them... and/or see what the cloud sig can do. 21:21:28 How do we initiate such a dialog? 21:21:36 so, do we add more tickets and advertise? who adds tickets? anyone in fesco? anyone? 21:22:01 (Sorry for straying off-topic; I'll wait for open floor) 21:23:03 gholms: a nicely worded email to start with I would say? "Dear VPS provider, we noticed you provide Fedora 10 and 9, those releases are end of life. Would you consider offering F12? Would you like assistance in making an image for your needs? we have a f12 image in format X,Y,Z available. Please contact cloud-sig@ if you have questions" 21:23:24 (i'm adding some ideas to the fes trac. typing is slow.) 21:23:26 * nirik wonders if the rest of fesco wandered off to lunch. 21:23:34 nirik: i think for now, anyone in fesco should add tickets. advertisement, hm. mail to devel-announce? mail to fedora-announce? ? 21:23:42 * gholms understands the question better now 21:24:37 mmcgrath: thoughts on advertising? 21:24:48 nirik: a tad late for lunch, maybe dinner? :D 21:25:07 nirik: once we've got enough stuff to do, I'll send an official announcement out and start handing out tasks. 21:25:59 I think we may not want to advertise to fedora-announce at least at first, as we don't want some new contributor to try a ticket that they can't handle... 21:26:06 yeah 21:26:31 ok, so, please add sensible tickets at your leasure. 21:26:37 anything else on this for now? 21:27:14 ok, moving on... 21:27:23 #topic #314 Wordpress bundles libraries 21:27:41 This was not on the agenda because I didn't think there was any new info... but there was today. 21:27:47 .fesco 314 21:27:50 nirik: #314 (Wordpress bundles libraries) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/314 21:28:08 upstream closed their ticket WONTFIX 21:28:10 http://core.trac.wordpress.org/ticket/12124 21:28:39 Upstream's viewpoint here is understandable 21:28:50 yeah, it's easier for them... 21:28:55 And for most users 21:29:19 well, it's kindof understandable, and kindof not. 21:29:22 Upstream sucks. 21:29:29 they should really be making it so you can do it /either/ way. 21:29:38 They should at least use unmodified libraries so it's easy to plug in system versions. 21:29:38 so, would anyone like to try and convince upstream ? or should we give up on that? 21:29:43 But they just don't care. 21:29:49 I don't think "convincing" is going to do anything 21:30:05 A minimally invasive patch that allowed either approach would be the best way to convince them 21:30:07 I'm wondering if this needs discussion on a mailing list instead of just submitting a ticket and hoping whoever answers agrees with us. 21:30:17 well, the fedora maintainer didn't really try and convince them... they said "hey, would you not bundle these" 21:30:40 It's a case where we're asking them to do work that currently doesn't improve their lives 21:30:47 mjg59: The problem is that fixing their mess is very invasive as they modified all the stuff they imported. 21:30:51 mjg59: yes, doing the work would probably be a good step. 21:31:04 Kevin_Kofler: I'm drawing a line between the modified versions and the basically unmodified ones 21:31:05 Which is why we have this exception request in the first place, otherwise just plugging in the system versions would be easy. 21:31:51 so, where do we go from here? 21:32:19 FPC recommended not granting an exemption for the basically unmodified libraries 21:32:22 * nirik notes we lost cwickert to net split 21:32:33 So I guess we ask the maintainer to do the necessary work in unbundling them 21:32:43 * dgilmore is -1 to granting an exception 21:33:18 I think we can grant an exception for the modified ones (though we'd obviously prefer not to) 21:33:28 nirik, sorry, i was also busy, customer called with an emergancy 21:33:34 cwickert: no problem. 21:33:39 pjones: indeed 21:34:06 yeah, I am -1 to the simply bundled ones getting an exception. I think we were waiting to hear from FPC again about the modified ones. 21:34:31 has anybody come up with a technical solution so far? 21:35:03 the technical solution would seem to be "don't bundle the unmodified ones, and make sure the code loading them checks the system path when it doesn't find something" 21:35:25 pjones, right, but that's a pretty massive change 21:35:28 -1 to allow bundling the unmodified ones. 21:35:31 which is unlikely to be a particularly difficult job, though it is a lot of typing. 21:35:46 For the modified ones, somebody needs to look into how modified they are. 21:36:01 If it's not feasible to fix them, I guess we have no other choice than allowing them. 21:36:16 I think we need to split this discussion in two and discuss the character of the modifications separately. 21:36:25 ok, should we just defer again until hearing from FPC on the modified ones? or ? 21:36:31 pjones: agreed. 21:37:38 ok, lets defer again and talk about it all after the next FPC meeting (whenever that is) 21:38:10 any objections? 21:38:22 nope 21:38:23 nope 21:38:28 no 21:38:38 #topic Open Floor 21:38:47 Anyone have anything for open floor? 21:39:10 Not unless you have any more cloud-related stuff to add. 21:40:02 not off hand. 21:40:52 * nirik will close the meeting in a minute if nothing else comes up 21:41:39 Thanks for coming everyone! 21:41:44 #endmeeting