15:00:24 #startmeeting FESCO (2018-05-04) 15:00:24 Meeting started Fri May 4 15:00:24 2018 UTC. 15:00:24 This meeting is logged and archived in a public location. 15:00:24 The chair is bowlofeggs. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:24 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:24 The meeting name has been set to 'fesco_(2018-05-04)' 15:00:25 #meetingname fesco 15:00:25 #chair maxamillion dgilmore nirik jsmith sgallagh bowlofeggs tyll jwb zbyszek 15:00:25 #topic init process 15:00:25 The meeting name has been set to 'fesco' 15:00:25 Current chairs: bowlofeggs dgilmore jsmith jwb maxamillion nirik sgallagh tyll zbyszek 15:00:30 hola 15:00:31 .hello2 15:00:31 maxamillion: maxamillion 'Adam Miller' 15:00:33 .hello2 15:00:34 sgallagh: sgallagh 'Stephen Gallagher' 15:00:54 morning 15:01:00 cool, that's quorum 15:01:04 let's get this moving 15:01:15 #topic #1877 large number of packages FTBFS in F28 15:01:15 .fesco 1877 15:01:15 https://pagure.io/fesco/issue/1877 15:01:19 bowlofeggs: Issue #1877: large number of packages FTBFS in F28 - fesco - Pagure - https://pagure.io/fesco/issue/1877 15:01:44 i'm not entirely sure why this is a fesco ticket 15:01:55 or what action is expected of fesco - seems like a releng issue? 15:02:49 In a lot of ways it is, I'm also not really clear as to what we're supposed to advise on or make a decision about 15:03:07 there was some major process breakdowns on the releng side during the mass rebuild 15:03:07 well, I think the orig comment had some info there... 15:03:19 suggesting we call out the issue or try and help it. 15:03:46 yeah that's a good point 15:04:04 I'm also not sure the status currently, I think tyll_ and zbyszek were rebuilding everything that was on the failed list again to get a better list 15:04:08 * mboddu is here 15:04:18 between communications not being sent, and a outage in the middle of it 15:05:09 mboddu: what is the status on the f28 ftbfs's, from your perspective? 15:05:25 sorry for being late 15:05:32 * nirik notes there is a f29 one on the horizon. 15:06:06 bowlofeggs: I agree with dgilmore, there were some snags in implementing it, but atm tyll_ is rebuilding them 15:06:24 I think at this point all that can be done is work to make sure the f29 rebuild goes smoothly 15:06:33 .hello till 15:06:34 tyll: till 'Till Maas' 15:06:56 yep, it's too late to do anything meaningful for F28 15:07:23 are there f28 packages that are broken and/or can't install due to this? 15:07:49 * zbyszek is not aware of any 15:07:51 not that I know of 15:07:53 should fesco just acknowledge the problem and trust releng to sort it out? 15:08:05 bowlofeggs: +1 15:08:14 it might be nice to do some test runs in the weeks leading up to f29 mass rebuild and make sure it all actually works. 15:08:17 my next steps are to make sure that there are proper bugs for all FTBFS bugs and then start a cleanup process of FTBFS packages, i.e. announcements to devel list and retirement after 6 weeks or so 15:08:37 well, I think we should discuss the retirment process 15:09:00 that the ftbfs bugs were not filed until way after the mass rebuild was a big issue 15:09:01 zbyszek: i'm not sure, but i *think* there might be a policy about retiring FTBFS packages already? 15:09:20 bowlofeggs: I don't think there is 15:09:25 if there isn't, there probably should be 15:09:27 there is a process to do this before branching but we did not follow it 15:09:29 part of that is on me because I did not follow up strongly enough to make sure it was done 15:09:32 Well, there must have been, since we had a script that did it 15:09:36 Didn't we? 15:09:56 https://fedoraproject.org/wiki/Fails_to_build_from_source 15:10:02 sgallagh: I re-used the script that I used to report long-time orphaned pkgs for this 15:10:14 sgallagh: to create a report 15:10:15 we have scripts, some of which could use work to make them better, butthe processes are all manual 15:10:20 there is multiple steps 15:10:26 that page does say that FTBFS packages get autoremoved 15:10:30 * sgallagh nods 15:10:31 https://docs.pagure.org/releng/sop_deprecate_ftbfs_packages.html 15:10:41 dgilmore: steps that have to happen in-order? 15:10:47 maxamillion: yes 15:10:51 zbyszek: does that page satisfy you that we have a documented policy about FTBFS? 15:11:15 bowlofeggs: yes, but that policy is significantly different from "retire after 6 weeks". 15:11:19 maxamillion: the triggers for many of the steps are the person running them remebering to do so 15:11:30 yeah the policy says N+2, which seems way too long to me 15:11:30 I wonder what we could use to automate in-order tasks across different systems.... 15:11:38 :) 15:11:47 zbyszek: 6 weeks was the timeout for orphaned pkgs 15:11:51 maxamillion: func! 15:11:53 We (including me) voted to retire packages after 6 weeks, but I now think that was a mistake. 15:12:03 maxamillion: puppet? 15:12:06 * bowlofeggs runs 15:12:39 I fear that there are *many* ftbfs packages with significant deps. 15:12:52 yeah i'm sure there are 15:13:05 should we defer this ticket for a few weeks to see what releng can accomplish? 15:13:07 zbyszek: usually the warnings about retirement get people to actually do something about them :-) 15:13:20 we should also definitely stop retiring before beta. 15:13:37 Often a rash of non-responsive maintainer bugs ensues 15:13:41 it sounds like we don't really know how many FTBFSs there are or how many deps they have 15:13:43 nirik: Or at least before final freeze 15:14:12 we could make a new ticket to re-think the FTBFS policy 15:14:30 i personally think N+2 is too long, but 6 weeks is too short :) 15:14:41 bowlofeggs: they do have that new ansible knock-off called bolt 15:14:48 6 weeks is only in rawhide 15:14:53 maxamillion: ah interesting 15:15:18 bowlofeggs: it is 6 weeks reminders for packages that failed for at the N-2 rebuild, so these are two timeframes 15:15:21 I think the timeout should be lower if the package is uninstallable 15:15:32 tyll: oh i see. yeah that is reasonable then 15:16:11 proposal: let's defer this ticket to give releng some time to sort it out and revisit in 2 weeks 15:16:23 bowlofeggs: +1 15:16:29 and hope for some more data in the ticket... like numbers from the f28 one 15:16:51 bowlofeggs: +1 15:16:59 bowlofeggs: +1 15:17:03 bowlofeggs: actually, https://docs.pagure.org/releng/sop_deprecate_ftbfs_packages.html said there would be one week notification for pkgs that failed for N-2 rebuilds 15:17:05 That's fine with me as long as we have a plan made public on devel-announce at least a week before the mass rebuild 15:17:52 haha ok so nirik and sgallagh both wanted adjustments to the proposal 15:17:58 lemme rephrase 15:18:05 bowlofeggs: +1 (also I can continue to work on getting the bugs, packages and deps ready) 15:18:05 FWIW, I think the timeout should be just 6 weeks when a package fails to rebuild in a mass rebuild, *and* becomes uninstallable because of missing deps. 15:18:51 proposal: we defer this issue for 2 weeks. we want data on the ticket (numbers from f28 rebuild), and we want a public plan in devel-announce at least a week before the mass rebuild 15:18:55 sgallagh: public in what way? one of the steps is to announce the mass rebuild ahead of starting, that was one step that was not done in f28 15:19:01 zbyszek: there used to be a seperate cleanup for pkgs with broken deps 15:19:05 bowlofeggs: -1 15:19:32 dgilmore: you don't like the public plan part? 15:19:35 I think the ask for numbers and a public plan are not useful without sufficient direction 15:19:45 bowlofeggs: no I think the ask is poorly formed 15:19:53 dgilmore: I think tyll is working on this 15:19:55 dgilmore: I just want there to be a clear understanding with our packagers what will happen if their packages don't rebuild successfully, 15:20:09 They should know this before the mass rebuild occurs 15:20:14 bowlofeggs: I also want to know what the public plan means outside of what is supposed to happen already 15:20:42 sgallagh: that is part of the communication that is supposed to happen already 15:20:44 I'd also like to propose that we consider successful build from source of any package included on a blocking install media to be a blocker to the release. 15:20:52 So we don't end up with abandoned critical software 15:20:53 we've been on this for about 20 minutes now - do we want to continue - or do we want to just got with the prior deferal proposal that had enough votes to pass? 15:21:08 sgallagh: the person who ran the mass rebuild had never done it befer, did not follow the SOP and missed pieces 15:21:09 err s/blocking install media/default install from blocking media/ 15:21:17 I should have provided better supervision 15:21:33 What about we go with the proposal, but open a ticket to discuss the policy in parallel? 15:21:34 dgilmore: You can't be everywhere. I'm not pointing fingers 15:21:38 sgallagh: the mass rebuild honestly does not protect us for that 15:21:41 zbyszek: yes i think that is a good idea 15:21:51 also the cleanup of long-time FTBFS pkgs is not really part of the mass rebuild process (or was not in the past) 15:21:54 dgilmore: It provides us an opportunity to identify some cases though 15:21:55 i'd like this ticket to stay focused ont eh f28 FTBFS not the policy 15:22:06 sgallagh: it really does not do a reasonable job of making sure software is not abandoned 15:22:09 let's do policy in its own ticket if we want to reassess it 15:22:10 If we get to Freeze and a critical FTBFS hasn't been fixed, we can find a new owner 15:22:37 sgallagh: that is very heavy, but sure 15:23:03 well, the owner could also just be trying to fix it and not have a fix yet... like the ppp thing... 15:23:04 proposal: let's defer this ticket for 2 weeks to give releng/tyll/zbyszek time to sort things out some more. let's also file a new ticket to revisit FTBFS policy 15:23:15 bowlofeggs: +1 15:23:17 nirik: indeed 15:23:24 how about whoever wants to change the policy files the ticket with their proposal? 15:23:24 nirik: Amended: if it hasn't been fixed or efforts to do so communicated in the BZ 15:23:25 +1 15:23:32 bowlofeggs: +1 15:23:42 bowlofeggs: +1 15:23:58 nirik: afaiu the policy is not the problem but the lack of time to actually follow it ;-) 15:24:18 tyll: the lack of well-known policy is also a problem, imo 15:24:23 +1 to defer. I prefer if we want to change the policy we have a proposal... a 'lets discuss the policy' ticket will be not helpfull IMHO 15:24:36 nirik: I'll draft a proposal 15:24:40 but if everyone wants to do that, great. 15:24:50 zbyszek: +1 15:25:01 zbyszek++ 15:25:02 sgallagh: Karma for zbyszek changed to 2 (for the f28 release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:25:17 counting, i have +6 15:25:22 maxamillion: ? 15:25:27 zbyszek: the current policy is at https://docs.pagure.org/releng/sop_deprecate_ftbfs_packages.html 15:25:34 bowlofeggs: +1 15:25:54 cool 15:25:54 sorry, completely blanked ... thought I'd voted 15:26:05 zbyszek++ 15:26:05 mboddu: Karma for zbyszek changed to 3 (for the f28 release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:26:16 #agreed let's defer this ticket for 2 weeks to give releng/tyll/zbyszek time to sort things out some more. let's also file a new ticket to revisit FTBFS policy 15:26:25 #action zbyszek will proposal a new policy in a new ticket 15:26:39 s/proposal/propose/ 15:26:51 haha 15:26:52 #undo 15:26:52 Removing item from minutes: ACTION by bowlofeggs at 15:26:25 : zbyszek will proposal a new policy in a new ticket 15:27:02 #action zbyszek will propose a new policy in a new ticket 15:27:11 mboddu: this next ticket is relevant for you too 15:27:14 there are currently 539 pkgs that did not rebuild since the F27 rebuild AFAICS 15:27:19 #topic #1878 Please change "Everything" directory to something less 15:27:19 inaccurately comprehensive 15:27:19 .fesco 1878 15:27:19 https://pagure.io/fesco/issue/1878 15:27:22 bowlofeggs: Yup and I am still around :) 15:27:22 bowlofeggs: Issue #1878: Please change "Everything" directory to something less inaccurately comprehensive - fesco - Pagure - https://pagure.io/fesco/issue/1878 15:27:38 mattdm: if you are around, we are discussing the "Everything" ticket 15:28:00 ok so i said a lot on this ticket, but i think there's something i did not make clear about my position 15:28:07 This is a expensive change, I also think that mattdm's proposed names are not the most ideal either 15:28:24 my position is not that fedora should not rename it, it's that FESCo should not mandate that Fedora rename it 15:28:30 making the change does make some sense 15:28:41 because i don't see a favorable ratio of benefit to cost 15:28:56 so if releng and infra want to put the effort into doing it, i do not oppose that 15:28:57 whats the frontrunner for the renamed thing/ 15:28:58 ? 15:29:06 but i do oppose FESCo declaring that they must do this 15:29:24 nirik: i don't think there is a favored name yet 15:29:41 nirik: mattdm is heavily pushing his option and rejecting all others, but I do not see a strong contender from anyone 15:29:50 * tyll is wondering if there are really people irritated by the name or if it is a theoretical problem 15:29:52 I was thinking we might have a middle ground... 15:30:10 mattdm is the person pushing the hardest for his proposal 15:30:26 ask everyone affected to move this to be a configurable thing... and as a low pri do that... with a deadline out like f30 or something. 15:30:26 * zbyszek likes his own proposal of adding a README instead 15:30:32 tyll: i work on infra on a project that uses that name and i barely notice it myself - i have a hard time thinking that many non-infra/releng people see the name very often 15:30:48 https://pagure.io/fesco/issue/1878#comment-508452 15:31:16 even then I do not see mattdm as overly pushing for it, more than he feels its the best choice 15:31:16 bowlofeggs: It has been difficult when drafting blogs and other comms around modularity 15:31:22 This is a massive change, I would actually avoid it if its not necessary, but ... 15:31:25 ot another possiblity: we have talked about redesigning our mirror content entirely... take this into consideration when doing that. someday. 15:31:30 The best we have is "the traditional Fedora repos" 15:31:40 When trying to differentiate 15:31:51 https://pagure.io/fesco/issue/1878#comment-508287 has all of my proposals 15:31:59 The natural way would be to use the variant name, but "Everything" would be confusing in that context 15:32:15 sgallagh: well that's just blog writing - that doesn't have to tie in to what every script/application in the release pipeline names things, imo 15:32:15 I feel it would be a lot of work, and I am not keen on it, but I would do it if the FPL feels its important for us to do. 15:32:52 sgallagh: i agree it is hard to think of good wording when *talking* about it, and it would be nice to settle on nice branding there, but does that require us to force infra/releng to do a bunch of work for something people rarely see? 15:32:59 s/rarely/rarely, if ever/ 15:33:06 I don't really see much benefit to the change and it's been expressed many times that it's non-trivial work to accomplish 15:33:07 nirik: maybe we reccomend that it be defered for a mirror redesign? 15:33:12 nirik: +1 15:33:14 i was a fedora user for *years* and never saw the word "Everything" 15:33:17 Situations where the external and internal branding differ tend to get confusing for the people doing the talking 15:34:01 bowlofeggs: Everything goes back to F7 from memory 15:34:04 e.g. FreeIPA is called Red Hat Identity Manager when shipped as RHEL, but all documentation uses the FreeIPA nomenclature because it's too hard to change from upstream 15:34:06 sgallagh: i'm pretty close to the "internal branding" and i barely notice it 15:34:26 i mean, bodhi uses the label "Everything" too, and even i see that word rarely 15:35:17 but as i said, i'm not opposed to others changing it of their own will, but i am opposed to fesco mandating a change because i just don't see enough benefit to justify that demand 15:35:28 * sgallagh nods 15:35:34 bowlofeggs: That is the reason why I am against the change, as it is very intrusive for some simple name change (like we never had bad nomenclature) 15:35:40 i'm not saying there is no benefit, just a disproportionate ratio 15:35:42 I'm not sure where I fall 15:35:57 bowlofeggs: +1 15:36:05 But I wanted to provide the necessary context 15:36:11 there are so many more important things for us (Fedora i mean, not fesco) to do than this 15:36:25 bowlofeggs: +1 15:36:27 if we make people work on this, they won't be working on things that affect users much more heavily 15:36:39 * nirik nods 15:37:12 if we do a mirror redesign it could do more things that are of benifit... make content more syncable/better modules for mirrors, etc 15:37:45 nirik: i don't think it's just the mirror that is hard to change though, right? 15:38:05 like, pungi and bodhi and probably tons of scripts and who knows what else 15:38:21 pungi does not need changes, the configs for it do 15:38:25 ah 15:38:46 mirrormanager will need changes 15:39:11 proposal: FESCo is not opposed to the name changing, but also does not choose to enforce it. the decision is left up to the people who would do the work 15:39:45 I dont maintain a mirror, but does mirror maintainers run any special scripts? 15:39:45 yes, there's lots of changes. I am just saying if we are making them, we could make them with more changes that get us more advantage 15:39:56 If they do, then they might have to update their scripts as well 15:40:07 bowlofeggs: +1 15:40:18 mboddu: for everything not really 15:40:18 bowlofeggs: Let's be honest, that proposal is equivalent to FESCo saying "don't do it". Which is a reasonable proposal, but let's be clear about it. 15:40:35 sgallagh: I'd prefer that too 15:41:01 nirik: Okay, thats good to know, since it might require for them to change their scripts which we cannot control 15:41:26 sgallagh: well i don't think it's quite that - that would be "FESCo opposes this change." 15:41:34 perhaps we could defer again and see if we can get mattdm and adamw here (since they were the other two who had strong opinions) 15:41:35 Counter proposal: FESCo does not feel that the benefits outweigh the implementation cost and recommends that nomenclature be changed in documentation instead. 15:41:36 mboddu: some mirrors write custome mirroring scripts, some use generic approaches 15:41:51 sgallagh: +1 15:41:51 sgallagh: i do not oppose the change happening, i just dont' think it's right for us to force it. that's not the same as "dont' do it" 15:42:28 sgallagh: perhaps of instead at the end you use first 15:42:38 leave the door open for the change 15:42:42 but not mandate it 15:42:44 bowlofeggs: Without us "forcing" it, we all know that the only other way it would happen is by our primary donor forcing it. 15:42:45 I'm good with either sgallagh or bowlofeggs' proposal 15:42:53 dgilmore: I'd be fine with that 15:42:53 bowlofeggs: I think it's our role to decide some things. 15:43:12 proposal: FESCo does not feel that the benefits outweigh the implementation cost and recommends that nomenclature be changed in documentation first. 15:43:20 dgilmore: +1 15:43:24 dgilmore: +1 15:43:28 +1 obviously 15:43:40 sgallagh: if our primary donor forces it, that's fine - they pay the people who would be doing the changes :) 15:44:05 +1 15:44:15 i'm fine wtih sgallagh/dgilmore's proposal too, but i prefer communicating that we also don't oppose it 15:44:16 well, the changes can be made in each area by those groups... 15:44:18 so +1 15:44:45 yeah, I am +1 to either 15:45:04 dgilmore: +1 15:45:09 i think that's 6 15:45:14 dgilmore: +1 15:45:18 cool, 7 15:45:30 Not sure mine counts, but +1 :) 15:45:32 bowlofeggs: this is clearly a change with effects in many different components, so it's something that should be decided publicly, not something to be undertaken by a single person. And FESCo is the way to do the public discussion. 15:45:33 #agreed FESCo does not feel that the benefits outweigh the implementation cost and recommends that nomenclature be changed in documentation first. (+7, 0, -0) 15:46:12 bowlofeggs: that's why prefer sgallagh/dgilmore's stronger version 15:46:28 zbyszek: i can see that point, but i also think it could be reasonable to have the implementors agree together to do it and only have FESCo step in if we think it's important and they can't agree 15:46:36 anyways, we have an agreement anyway 15:46:39 so let's move on 15:46:52 last issue 15:46:54 #topic #1886 Draft Fedora 30 schedule 15:46:54 .fesco 1886 15:46:54 https://pagure.io/fesco/issue/1886 15:46:55 bowlofeggs: Issue #1886: Draft Fedora 30 schedule - fesco - Pagure - https://pagure.io/fesco/issue/1886 15:47:59 +1 here... we can always adjust later 15:48:04 whoah crazy - i thoughti was reading the f29 schedule and it starts with "fedora 29 release" and i was like "uh what?" 15:48:06 haha 15:48:19 bowlofeggs: We're trying everything backwards this release! 15:48:36 what nirik wrote 15:48:39 +1 15:48:41 +1 15:48:42 +1 15:48:43 Yeah, +1 15:49:02 we have an in-ticket +1 from jwb 15:49:15 +1 15:49:21 maxamillion: ? 15:49:26 +1 15:49:36 sorry, multi tasking badly ... I'm also on the phone 15:49:39 cool, 8 15:50:05 #agreed FESCo approves the very futuristic F30 schedule (+8, 0, -0) 15:50:15 #topic Next week's chair 15:50:21 it should be Fedora XXX :) 15:50:21 volunteers? 15:50:30 nirik: haha like superbowls 15:50:33 I should ask: will we have quorum? 15:50:52 i will be here 15:50:53 I'll be around, but with RH Summit next week, it might not be true for everyone 15:50:54 I kinda like the idea of delaying f30 for a year 15:51:08 I'll chair if we have it 15:51:10 I will be here next week 15:51:13 * nirik will be here. 15:51:17 I'll be here 15:51:36 cool 15:51:53 #action dgilmore will chair next week, assuming we reach quorum (please e-mail the list if you can't make it ahead of time) 15:52:03 Uh 15:52:06 bowlofeggs: sgallagh said he would, but I can also 15:52:09 the ()'s was aimed at all of us, not dgilmore 15:52:13 oh sorry 15:52:15 misread 15:52:17 #undo 15:52:17 Removing item from minutes: ACTION by bowlofeggs at 15:51:53 : dgilmore will chair next week, assuming we reach quorum (please e-mail the list if you can't make it ahead of time) 15:52:20 s'all good 15:52:25 bowlofeggs: it is fine 15:52:30 #action sgallagh will chair next week, assuming we reach quorum 15:52:55 #action FESCo members, please e-mail the list if you can't be here next week so we don't hold a meeting just to find out we dont' have quorum 15:52:57 I have an item for Open Floor, when we get there 15:53:02 #topic Open Floor 15:53:04 sgallagh: go! 15:53:19 With F28 released, it's probably time to start talking about elections 15:53:27 * dgilmore has a thing to stay also 15:53:34 sgallagh: that is what I had 15:53:40 heh 15:53:40 yep 15:53:45 sgallagh: elections process started Wednesday 15:53:51 it is underway 15:53:57 i'm sad because i didn't get a full year in my term due to f28 being on time and f26 being really late :) 15:54:26 bowlofeggs, run again 15:54:34 Southern_Gentlem: yeah i plan to :) 15:54:40 https://pagure.io/Fedora-Council/tickets/issue/201 15:55:01 * dgilmore notes that his seat is up and he will not be running for FESCo again this time around 15:55:38 dgilmore: :( you will be missed 15:55:42 election results are due June 7th 15:55:44 yeah. ;( 15:55:59 You will be missed, but you also definitely served your time :) 15:56:10 * bowlofeggs proposes that since dgilmore won't be on fesco anymore that fesco mandate that fedora only support x86_64 as retribution 15:56:25 .fire bowlofeggs 15:56:25 adamw fires bowlofeggs 15:56:25 bowlofeggs: I will fight you 15:56:28 haha 15:56:38 figuratively 15:56:52 bowlofeggs: Maybe not a wise thing to say when you're running for re-election :-D 15:56:58 adamw has fired so many brave fedowarriors 15:57:08 sgallagh: ahaha true 15:57:37 any other thoughts on this, or other things for open floor? 15:57:54 * sgallagh has nothing 15:58:19 closing in 45 s 15:59:04 #endmeeting