15:00:11 #startmeeting FESCO (2020-03-30) 15:00:11 Meeting started Mon Mar 30 15:00:11 2020 UTC. 15:00:11 This meeting is logged and archived in a public location. 15:00:11 The chair is zbyszek. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:11 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:11 The meeting name has been set to 'fesco_(2020-03-30)' 15:00:11 #meetingname fesco 15:00:11 #chair nirik, ignatenkobrain, decathorpe, zbyszek, bookwar, sgallagh, contyk, mhroncok, dcantrell 15:00:11 The meeting name has been set to 'fesco' 15:00:11 Current chairs: bookwar contyk dcantrell decathorpe ignatenkobrain mhroncok nirik sgallagh zbyszek 15:00:22 o/ 15:00:24 o/ 15:00:31 morning 15:00:33 sgallagh: should I do the chairing? 15:00:41 * mhroncok needs to leave in 45 minutes 15:00:45 I don't think he's coming, so yes. 15:00:59 .hello2 15:01:02 dcantrell: dcantrell 'David Cantrell' 15:01:10 .hello psabata 15:01:10 contyk: psabata 'Petr Šabata' 15:01:16 .hello2 15:01:17 sgallagh said that he's not feeling well, so I'm taking over as chair 15:01:18 bcotton: bcotton 'Ben Cotton' 15:01:19 .hello2 15:01:22 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 15:02:05 We have 5, counting decathorpe and mhroncok, right? 15:02:27 yhp 15:02:30 *yup 15:02:45 #topic #2358 Drop the restriction on tagging for fedora-release 15:02:45 .fesco 2358 15:02:46 zbyszek: Issue #2358: Drop the restriction on tagging for fedora-release - fesco - Pagure.io - https://pagure.io/fesco/issue/2358 15:02:50 zbyszek: 6? 15:02:58 yeah, I count six 15:03:04 Oh, nirik, right. 15:03:14 * nirik waves 15:03:25 I was so thrown off by the new greeting in Fedora-land, that I miscounted: o/ 15:04:11 .hello2 15:04:12 bookwar: bookwar 'Aleksandra Fedorova' 15:04:16 So... I don't know, I'm obviously biased, by I really don't see the need to the special procedure here. 15:04:39 I commented in the ticket a moment ago. 15:04:53 I'm in favor of relaxing the process here but I'd like to see some sanity tests added. 15:04:58 Unless we have those already. 15:05:07 as I said in the ticket, I'd be ok dropping fedora-release/fedora-repos from that rule... but also I really think people should coordinate with releng anyhow before making changes... and I don't see how this is too much of a bottleneck. 15:05:32 nirik: so now it happens that any provenpackager can ge tin the changes 15:05:37 I not opposed to removing fedora-release from the set 15:05:38 and than the build is not tagged 15:05:49 right. 15:05:58 I don't see that as "coordinating" 15:06:30 do we have a history of provenpackagers pushing in changes, builds failed to tag and releng saying: oh no, this change should not be here? 15:06:35 in the interest of not pushing updates to it all the time, releng may well have serveral things they want to push out at once in an update. If someone comes along and pushes their pet bug it messes that up 15:07:06 to me the restriction seems to be in place due to leftover process which may or may not be valid anymore 15:07:13 should we get additional input from releng? 15:07:22 Also, you can destroy Fedora from many packages, not just this one 15:07:29 well, it was put in place because someone pushed changes that had to be reverted 15:07:42 as mhroncok says, that can happen in many other packages 15:07:54 anyhow, as I said I am ok with relaxing those, so I will stop arguring. 15:08:02 nirik: do we put this protection in place for all reverted changes? 15:08:09 ok, vote? 15:08:10 if releng's process is/was relying on data in that package for the purpose of different composes, maybe this warrants a restriction 15:08:21 but maybe they've changed their processes now 15:08:30 no, but like I said, I agree with you, so I am not sure how I can agree more. 15:08:39 :) 15:08:43 nirik: agree more! 15:09:46 dcantrell: I don't think so. That package is mostly a release of systemd presets (by number of lines), and that's where the most common changes are. 15:10:11 but it's also important for other reasons too... but then we have a lot of important packages. 15:10:20 ok. I recall at one point fedora-release containing repo configuration data, which may or may not be used at compose time 15:10:28 but honestly, I'm fine relaxing this now in the name of progress 15:10:37 proposal: drop fedora-release and fedora-repos from the secure-boot channel 15:10:40 if it causes problems, we can revisit the process/policy 15:10:47 dcantrell: thats fedora-repos. 15:10:49 +1 15:10:53 +1 15:10:55 +1 15:11:07 nirik: ah right, stuff moves around all these years 15:11:14 +1 15:11:14 +1 15:11:19 +1 15:11:20 * dcantrell goes back to combing grey beard 15:11:20 yeah. 15:11:31 +1 my own proposal too 15:11:41 #agreed APPROVED (+7, 0, 0) 15:11:54 Thanks ;) 15:11:57 #topic #2360 RPM 4.16 and sqlite db changes for Fedora 33 15:11:58 .fesco 2360 15:11:59 zbyszek: Issue #2360: RPM 4.16 and sqlite db changes for Fedora 33 - fesco - Pagure.io - https://pagure.io/fesco/issue/2360 15:12:36 do we need more votes here? 15:12:46 I'm +1, but urge close consultation with releng/infra due to the datacenter move... 15:12:50 dcantrell and bookwar IIRC 15:13:15 I am +1 on 2360 15:13:17 Voted +1 to both in the ticket. 15:13:28 the more sqlite the better 15:13:34 they'd like to ship 4.16 ASAP, so that's why this is on agenda, otheriwse it would just be approved by the end of the 7 days window 15:13:44 decathorpe: we're at +5/+4 or so, but only 4 days have passed, so we'd need to wait another few days according to procedure. By voting here, we can speed this up. 15:14:14 ok, thanks. for the logs, I voted +1/+1 in the ticket as well. 15:14:22 this is an example of "we actually wait for the approval" kind of change, so it's good to try to reward them by being fast :D 15:16:07 I didn't vote in the ticket, but I am +1 now 15:16:30 bookwar: vote? 15:16:38 * mhroncok was +1 to both in the ticket 15:16:41 they may need to not switch to sqlite as soon as they wanted, but definitely we should do it when we can. ;) 15:16:51 nirik: I still plan to open that releng ticket for bootstrap mock in koji 15:17:06 mhroncok: sure. 15:18:26 OK, I'll continue without bookwar's vote, since we're at +7 anyway 15:18:47 #agree APPROVED (both parts) (+7, 0, 0) 15:18:58 #topic DNF prompts for GPG key import for "repo_gpgcheck=1"-repositories despite "rpm --import"-ing the keys first 15:19:01 .bug 1768206 15:19:03 zbyszek: 1768206 – DNF prompts for GPG key import for "repo_gpgcheck=1"-repositories despite "rpm --import"-ing the keys first - https://bugzilla.redhat.com/1768206 15:19:23 zbyszek: are you proposing it as a fesco blocker? 15:19:37 Right now, I want to gather feedback. 15:19:48 what sort of feedback? 15:19:56 zbyszek: I looked briefly after you added it to the agenda, but I was not able to read it all yet 15:20:11 I mean, it's a bug... it should get fixed... ? 15:20:18 zbyszek: do you have a "tl;dr why this is a fesco concern"? 15:21:15 tl;dr: a change in cisco repo exposed a design choice in dnf that causes a new hard-to-answer question to be asked, and also breaks dnf-makecache.service from completing. 15:21:46 The issue slips between the cracks: it's not a bug in dnf, not a bug in the repo definitions, and not a bug in dnf-makecache. 15:21:47 the cisco repo didn't change... 15:22:02 it's just that we are now including it in images... 15:22:13 it's a dnf bug IMHO 15:22:26 yeah, what I'm reading here in BZ says this is a dnf bug 15:22:38 or at least a bad user experience 15:22:56 yeah it's pretty annoying, and it breaks the makecache service :/ 15:23:28 So my first question is: do we consider this a problem at all? 15:24:01 I would think dnf would read the keys imported into the rpmdb if it doesn't have the key already itself... if it still cannot find the key it should fail if it's running non interactively. 15:24:05 * mhroncok was not able to grasp it all yet, sorry 15:24:38 yes, it's definitely a problem. It breaks CI stuff that tries to run 'dnf check-update' and dnf--makecache and confuses users. 15:25:48 I think it'd be best to change behaviour of dnf here. 15:25:52 I suppose if it cannot be solved in dnf before release, we could look at backing out the add of the repo to images/composes... but the workstation working group wanted that. 15:27:49 nirik: if it comes to either/or, I think it better to postpone the add to F33 15:27:55 it's 15:28:37 yeah, I'd agree. we may want to just FYI the workstation working group on the issues... 15:28:45 Hmm, that'd be unfortunate. So let's wait to see if an easy fix in dnf is possible. 15:29:13 * mhroncok leaves in 15 15:29:20 Right, let's move on. 15:29:31 #topic Fedora 33 System-Wide Change Proposal: ELN Buildroot and Compose V2 15:29:49 We don't have a ticket for this one yet, but we discussed it last week already. 15:30:15 bookwar: are you still here, you haven't said anything in a while? 15:30:29 i am here, sorry there was an overlaping call 15:30:45 thanks for bringing it 15:31:02 I think there has been still usefull feedback on the list... perhaps wait another week for any further tweaks before voting? 15:31:25 we are not voting here, are we? 15:31:25 nirik: yeah, the discussion is ongoing, I think it'd be too early to vote. 15:31:31 ack 15:31:58 I'll just say that I'm generally in favor. I think it's a cool idea and we should try it out... 15:32:02 i think we essentially go back to the same request as we had for additional buildroot: we ask for some infrastructure, and we make everything optional 15:32:23 it is just branded differently now 15:32:47 yeah 15:32:58 zbyszek: do you feel like there questions still unanswered? 15:33:42 bookwar: for one, it's unclear what will happen if people don't like the eln conditionals in some package 15:34:06 they don't apply them, we build package from fedora master branch as is 15:34:11 there was the question of if .eln should update or be older than .fcXX (which may have been since answered, I haven't read devel list yet today) 15:34:22 currently it seems that if you don't want conditionals, you will get a pull request with them 15:35:03 Your analogy with code upstream/downstream is actually nice: we have a mechanism to override upstream decisions if they don't like our patches or have some other bug that can't be fixed. If we relied on taking upstream code unchanged, we wouldn't be able to fix things. 15:35:04 mhroncok: you will have a suggestion, which you can reject, if you reject - we go with the master 15:35:18 bookwar: and if it FTBFS? 15:36:02 zbyszek: i am not against branching as a tool, but this is the next layer of the overrides which happen after ELN 15:36:37 mhroncok: I guess ideally a upstreamable patch? 15:36:47 mhroncok: If it builds in Fedora and not in ELN there should be a reason for it 15:36:57 we need to see this reasons 15:37:11 that's one of the goals of the project - to know such reasons 15:37:28 But ELN has a different package set, it's obvious that things will FTBFS and FTI, even without any actual bugs. 15:37:34 bookwar: what if the reason is: this optional build dependency is not in ELN? 15:37:46 zbyszek: we adjust the package set then 15:38:09 bookwar: so eventually, we build even more packages and aliante even more maintainers? 15:38:24 we said it there, we inherit rawhide in the buildroot, we will work gradually on that content set, not cut everything from teh start 15:38:27 bookwar: have you seen the dependency graphs produced by the minization efforts? Fedora package set is close to a strong graph. 15:38:33 bookwar: I just don't want people to complain because somebody "force" pushed changes into master they do not want 15:39:01 several redhatters as well as nonredhatters have suggested that this approach does not work for them 15:39:03 decathorpe: we are not going to force anything 15:39:24 mhroncok: then these redhatters will work via branch, but that branch is not going to be ELN 15:39:27 but what's the conflict resolution going to look like then? 15:39:40 bookwar: I am afraid I cannot trust that statement after the recent internal e-mail exchange we had 15:39:55 (about not forcing that is) 15:40:49 mhroncok: in that conversation i said we are not going to setup branching 15:40:59 and we are not going to setup it in ELN 15:41:10 in internal RHEL infrastructure we use branching for years 15:41:16 and we will keep using it 15:41:24 I have the feeling that there's a big difference in the estimation of the extent of patches required, and also in the effect of patches not merged. 15:41:35 you also implied that as a RH sponsored packager, I can be told to use codnitionals 15:41:39 *conditionals 15:42:07 and lot of RH sponsored packagers said on the mialing list they don't want conditionals and you keep repeating that they won't be forced, I simply don't trust this 15:42:08 mhroncok: that's not what i said 15:42:14 My guess is that a) a large number of packages will require *some* conditionals, and b) even a few packages not building deep in the stack will cause a cascade of failures. 15:42:40 mhroncok: if you have personal issue with my statement - let's resolve it separately 15:43:01 no, I dont have any personal issues with your statement. I have technicall issues with this change proposal 15:43:15 * nirik wishes we had a better story for conditionals in general, but it's a hard balacing problem. ;( 15:43:16 as well as others 15:43:24 and you are rejecting that feedback 15:44:28 I am very sorry, I could argue about this more, I really need to go, you know where to reach me 15:44:48 perhaps you should try and have a higher BW conversation outside the meeting? 15:44:54 mhroncok: then let me repeat: there is no enforcement possible, I don;t have such power, i don't plan to get it, i don't ask for it. We want to try the conditional approach. Some people will collaborate and use the benefits it brings, some people won't and they will live as they are. 15:45:43 Branches are possible in RHEL internal structure or maybe in CentOS stream, we don't want to create them now in Fedora 15:46:36 bookwar: OK, let's discuss a specific example: systemd. The package is fairly complicated, and I expect it might be quite different in RHEL (e.g. different cryptlibs, whatnot). 15:46:45 mhroncok: and in opur private argument was that you as redhatter actually have a direct benefit of the conditional. So i don't see reason for you to go against it. But if it is your decision to go against it, i can not override it. 15:47:05 And let's say it doesn't build and doesn't install in ELN, because of that. 15:47:30 Once systemd-libs FTI, a large fraction of the distro is broken. 15:48:03 zbyszek: we need to see a reason why it fails. We can not generically. 15:48:49 yes, if ELN compose will fail for 2 months with us not being able to get it working, then there is no point building it 15:48:53 then we sit together and decide what's next 15:49:20 * decathorpe whispers: using eln branch 15:49:43 OK, so let's say that the resolution of the tehcnical discussion is that a heavy conditional is needed, because different BRs and some reverts of upstream patches are necessary. 15:49:56 And the Fedora maintainer says no. 15:50:49 It's a question of a contingency plan for such cases. You cannot just say "we'll discuss when the issue comes". 15:51:05 another thing we could do is have "-eln.spec" files in the master branch for unusual cases that need more work to resolve. that way rawhide isn't disrupted and we avoid a branch 15:51:23 eln build system can use the -eln.spec file if it exists, otherwise the regular one 15:51:38 zbyszek: the contingency plan is to use master of Fedora Rawhide, and adjust ELN not the source of the package 15:51:51 for me this would all be easier to discuss specifics with something actually in progress that we could look at 15:52:08 bookwar: but we're discussing the case where the master branch FTBFS or FTI in eln 15:52:26 what mechanism will you use to adjust ELN? 15:52:47 zbyszek: if Fedora Rawhide works and eln doesn't, this means that the reason of the failure is the divergence between eln and fedora 15:53:14 since divergence is compose config, component set and build flags - we can adjust that on eln side 15:53:58 well this assumes that no conditionals are present in master? 15:54:25 because even with only one %if eln there will be divergence between rawhide and eln 15:54:27 bookwar: it's usual to require tweaks to spec files between different fedora releases. And that's OK, because the whole purpose of releases is to make things different. And the same applies to ELN. Making ELN exactly like rawhide would be pointless. 15:55:24 zbyszek: if eln becomes pointless we discard it 15:55:51 decathorpe: so the case where you accpet one, but won't accept more? and won't remove that one? that seems... pretty unfriendly 15:56:07 bookwar: Dunno, I would like to like the proposal, but there are some obvious failure points in the proposal, and they are neither acknowleged nor is a suitable resolution proposed. 15:56:39 nirik: no, I mean: eln will undoubtedly diverge from rawhide as soon as there is a conditional in one spec file 15:57:01 zbyszek: well, it's a spectrum right? on one end maintainers that have 1 spec in rawhide that they merge to all other branches and has tons of conditionals... and the other end you have completely different specs on each branch that you have to manually change each branch/no merging. 15:57:12 decathorpe: sure 15:57:32 nirik: exactly 15:57:45 zbyszek: eln is a _possibility_ for maintainers of downstream and upstream to work together, it is an option. If they don't want to work together on the component in fedora then the downstream goes away and works on their fork. 15:58:12 we already have a major stakeholder for this change which is kernel 15:58:42 even if we have everything else as Fedora Rawhide does, building ELN kernel alone brings benefits 15:58:48 nirik: that's the point that I and I think others are trying to make: that for we *know* that there are packagers who very much prefer branches to conditionals 15:59:02 so we want to start with it, and see how it goes with other components 15:59:20 sure, but I am really hoping most packages will not need any conditionals for this. 15:59:30 zbyszek: "preferring the branch" is not a decision of fedora packager 16:00:21 bookwar: it is. Why not? I have some packages this way, some other, and mix and match both approches when and how I see fit. 16:00:22 zbyszek: when fedora packagers "prefer eln branches", it means they prefer downstream to work separately, and it is up to downstream to decide how this work is going to happen then 16:00:23 sure it is... at least currently 16:00:42 bookwar: no, we were talking abotu fedora branches here... 16:00:43 nirik: sure, but we'd still like to see using branches as a contingency mechanism instead of "we tried, but packager X is blocking us, so bye bye ELN" 16:01:13 bookwar: I disagree. the package maintainers should have the ability to create a branch if they see that the best way to coordinate their work. forcing them to work in a specific way goes against that 16:01:32 the problem with eln branches (at least from my view) is that it would be vastly more work. You would have to try and keep that in sync with rawhide... 16:01:42 to that point, on the RH side I think we need stronger rules around RH packagers working in Fedora than we do now 16:02:12 nirik: "git merge master"? 16:02:19 dcantrell: the request for branch comes from perception that eln branch will be maintained by someone else, not a fedora owner of the master branch 16:02:21 nirik: I completely agree, but that's my opinion. I think for some projects a branch might work better for them and they can sort out how merges and syncs work. 16:02:32 dcantrell: and we don't have that "someone else" in fedora 16:02:33 decathorpe: but you have a eln branch because the maintainer didn't want to take conditionals... 16:02:57 so you have to fix up merge conflicts at least... 16:03:00 merge conflicts indicate that manual work would be necessary anyway 16:03:12 bookwar: sure, I get that. I would like to see this proposal eliminate that perception. the ELN responsibility is a key part here, and that's from the RH side I think 16:03:15 whether you're on a separate branch or not 16:03:52 decathorpe: you can design conditionals in a way so that they don't mess with the version bump 16:04:00 but I think they want the manual work to be at the collection level instead of at the package level, since they don't have a pile of packagers signing up for this work.... 16:04:11 decathorpe: designing sync between branches in this way is harder 16:05:04 bookwar: for what it's worth, I'm not against this change, but without a contingency mechanism in place I think it's doomed 16:05:20 so perhaps the change should say "we really want to avoid eln branches, but if we get to a place where they make sense, we would bring it back up to fesco/devel and decide if we wanted to have them, or do something else" 16:05:46 nirik: +1 16:05:56 nirik: yes! that's all I would want here. 16:06:24 nirik: if we get to a place where ELN effort fails, then we need to reconsider this effort and do something else 16:06:58 bookwar: and now is exactly the time to ask about this. 16:07:16 I just wonder why do you consider needing a separate git branch for select packages a failure 16:07:42 well, it would dramatically increase the work needed... 16:07:52 decathorpe: i explained it, i don't want to fork fedora, i want to make fedora flexible 16:08:29 Changing the subject a bit: I asked on the mailing list, but didn't get any reply. What are the initial changes from rawhide that are planned? 16:08:31 well, is fedora 31 a fork of rawhide? maybe, but that's really stretching the definitions here 16:08:47 E.g. compile flags for the avx2 stuff on amd64? 16:09:19 zbyszek: avx, kernel, everything else is just existing conditionals 16:11:24 Compilation options are set through macros defined by redhat-rpm-config... I'm talking about changes to that package for example. 16:11:51 (Not how they'll be made, but what changes are planned...) 16:11:55 zbyszek: you do realize that this is already approved change? 16:12:08 the avx2 part? 16:12:18 bookwar: no? 16:12:57 zbyszek: you can look at the kernel spec... it already has a bunch of '%if fedora X, %else Y things... 16:13:03 zbyszek: this change https://fedoraproject.org/wiki/Changes/Additional_buildroot_to_test_x86-64_micro-architecture_update was accepted by FESCo 16:13:31 but it was replaced by this current one, right? 16:13:59 extended, yes 16:14:42 but we have discussed how we would make that change, and that part is not different now 16:15:08 it is the same configuration, and same impact 16:15:13 we aren't doing that one still tho right? there's not 2 new buildroots? 16:15:29 nirik: we want to do it in the ELN buildroot 16:15:42 right, ok, thats what I thought... but this was confusing me. 16:16:17 bookwar: so, the avx2 flags, any other changes are planned for eln? 16:17:21 nirik: I'm not worried about the kernel too much, because it's self-contained and special in many regards. 16:17:42 I'm trying to figure out what changes that impact building and running of other packages are planned. 16:18:08 zbyszek: kernel and avx2 are starting points. Whatever other changes may come - they should come to ELN SIG, which is supposed to scope them, and if it causes the build issues like you describe, which will lead to a complete compose failure for weeks, then ELN SIG won't accept them 16:18:33 OK, makes sense. 16:19:12 that sounds reasonable to me as well. 16:19:59 as i said with addition of existing conditionals 16:20:20 OK, I don't have anything else on this subject for now. 16:20:23 Anyone else? 16:21:47 Let's move on. 16:21:48 #topic Next week's chair 16:21:53 volunteers? 16:22:23 just one volunteer? 16:23:09 sorry, friday and monday I'm busy with university :( 16:23:22 I can. 16:23:39 #action contyk will chair next meeting 16:23:43 thanks 16:23:50 #topic Open Floor 16:24:46 ...crickets... 16:25:02 OK, let's wrap this up. Thanks all. 16:25:03 #endmeeting