15:00:00 #startmeeting FESCO (2019-05-17) 15:00:00 Meeting started Fri May 17 15:00:00 2019 UTC. 15:00:00 This meeting is logged and archived in a public location. 15:00:00 The chair is contyk. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:00 The meeting name has been set to 'fesco_(2019-05-17)' 15:00:02 #meetingname fesco 15:00:02 The meeting name has been set to 'fesco' 15:00:04 #chair nirik, bowlofeggs, jforbes, zbyszek, bookwar, sgallagh, contyk, mhroncok, otaylor 15:00:04 Current chairs: bookwar bowlofeggs contyk jforbes mhroncok nirik otaylor sgallagh zbyszek 15:00:06 #topic init process 15:00:14 .hello psabata 15:00:15 contyk: psabata 'Petr Šabata' 15:00:15 morning 15:00:19 .hello2 15:00:20 jforbes: jforbes 'Justin M. Forbes' 15:00:47 .hello2 15:00:48 sgallagh: sgallagh 'Stephen Gallagher' 15:01:53 hi 15:02:05 yay, quorum 15:02:10 .hello2 15:02:10 bookwar: bookwar 'Aleksandra Fedorova' 15:02:42 .hello2 15:02:43 .hello2 15:02:43 bcotton: bcotton 'Ben Cotton' 15:02:46 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 15:02:47 hi all, sorry for missing last time, was on the road 15:03:05 no worries 15:03:20 oh, I'm in a tram, I'll have to drop off for a few minutes in half an hour 15:03:26 well, let's get this started then 15:03:29 .hello2 15:03:30 bowlofeggs: bowlofeggs 'Randy Barlow' 15:03:34 I only put two topics on the agenda anyway 15:03:45 #topic #2128 Policy needed: Modules built against EOL Fedora releases 15:03:47 .fesco 2128 15:03:48 contyk: Issue #2128: Policy needed: Modules built against EOL Fedora releases - fesco - Pagure.io - https://pagure.io/fesco/issue/2128 15:03:49 https://pagure.io/fesco/issue/2128 15:04:16 So, my proposal there was heavily-flawed, as mhroncok pointed out 15:04:39 so perhaps s/rawhide/oldest supported stable release/ ? 15:04:50 nirik: I don't love that as an answer either 15:05:08 * contyk is reading the comments 15:05:15 why not just say it has to be built against any supported release? 15:05:18 including epel? 15:05:23 Because it means that we're shipping content on fN that doesn't have the security hardening of other packages and modules 15:05:23 I guess that for fedora this isn't so bad, because support is only 13 months... 15:05:26 I'm fine with "any supported release" 15:05:38 sgallagh: well, we are still shipping that release... 15:05:55 Right, we are basically saying it has to be rebuilt once per year 15:06:05 sgallagh: i think it's ok because if we thought it was a real security issue we should fix it in the release anyway, right? 15:06:08 I'm not saying I won't settle for it. 15:06:18 I'm making sure we understand what we're settling *on* 15:06:22 bowlofeggs, mhroncok: but doesn't have the same problem that sgallagh proposal had 15:06:25 * nirik is ok with 'any supported' or 'oldest supported' 15:06:38 > Packages built against older glibc etc. will work with new glibc etc., but not the other way around. 15:06:41 ? 15:06:49 there are two cases where people use this limited modular buildrequires setup 15:07:02 one is that java case which builds on the oldest one and doesn't rebuild until they have to 15:07:05 zbyszek: we keep the "make it work" part on the mainatiner 15:07:14 the other was the rust module which only builds on rawhide and ships to older releases 15:07:30 because they build statically and don't have the build deps in older platforms 15:07:36 contyk: Does rust not link against glibc even? 15:07:50 it does link 15:08:07 zbyszek: yeah i agree with mhroncok - our policies should be about what is safe/allowed. if it doesn't work the maintainer can solve it without policy 15:08:09 even if it does, is that an issue? 15:08:19 bowlofeggs: +1 15:08:20 they do it the other way around 15:08:46 contyk: Can you add antecedents to "it" and "that" in your sentence? 15:08:52 I don't know what you were replying to 15:08:57 OK, in that case I'm fine with "any supported release" 15:09:22 they build with the latest hardening flags, so that security aspect is solved 15:09:41 contyk: I was concerned about it not actually running against FN-2 15:09:42 if they build in rawhide and then ship in f29; unless the packages then don't work because of glibc incompatibility 15:09:45 But that's not my problem 15:09:50 in which case I would say it's up to the maintainer to solve it 15:09:54 Agreed 15:09:57 or stop doing that 15:10:17 OK, let's assume that we go with "any supported release"... how do we enforce that? 15:10:25 my module (rpick) just gets built against all releases anyway (rather than build once and ship many), and i'd think maybe most people would just do that? 15:10:34 When does that enforcement occur? 15:10:43 bowlofeggs: yes, most would and should 15:11:05 If I build a module on F28 for FAnything the day before F28 goes EOL, what do we do about shipping that module on F29 and F30? 15:11:07 but in some cases they might not want to do that and I'd like to leave that to them (and deal with breakage if it occurs) 15:11:10 Do we nuke it from the repos? 15:11:16 What if it's a default stream? 15:11:16 sgallagh: when FN is released, (somebody|something) opens bugs for module that were only built on FN-2 15:11:41 sgallagh: is it possible to know which release each module we ship was built against? 15:11:47 it is 15:11:49 if the module is not rebuilt on at least FN-1, it gets "orphaned" 15:11:52 maybe s/possible/easy/ ☺ 15:12:14 if it is not a default stream, we nook it from the repos 15:12:16 I think mhroncok's proposal is sane 15:12:26 if it is a default stream, I have no idea 15:12:32 how will the user know if they are using a module that got nuked? 15:12:38 mhroncok: I think we probably can't nuke it, especially if it shipped in the frozwn repo 15:12:40 *frozen 15:12:45 will they just stop getting updates for it and not know it is no longer supported? 15:13:00 just like with packages, yeah 15:13:01 But we *can* make sure that they can't ship updates for it without fixing the BRs 15:13:02 I'd like to follow the 6 weeks orphaned period, but I have no idea how to do that in modular world 15:13:11 sgallagh: nuke from rawhide at least 15:13:14 that seems kinda bad to me 15:13:17 mhroncok: That's fair 15:14:03 so in theory, we still have F28 built packages in the entire F30 life time, but that is a compormise 15:14:08 acknowledging that this happens with rpms because maintainers disappear, it does seem "more bad" that a module i'm using could become unsupported mid release because it was built against a retired fedora, and i won't be informed 15:14:28 mhroncok: I think it's a reasonable compromise. 15:15:14 bowlofeggs: Well, it's not any different than a packager disappearing and no one noticing that no updates show up 15:15:18 bowlofeggs: well effectively, this is the same - no updates in either scenario 15:15:37 would this only happen *because* the maintainer disappeared? 15:15:39 We never really notice until someone starts non-responsive policy or it becomes FTBFS on a mass-rebuild 15:16:00 sgallagh: and even FTBFS packages stay forever 15:16:01 it is a bit worse though because there's mroe to it than the main package 15:16:05 We can also paper over this a bit during the mass-rebuild policy. 15:16:11 because there can be CVEs in the dependencies 15:16:25 if an RPM maintainer disappears, at least the openssl on my box is still maintainer 15:16:33 We have code to detect the BRs at that point, so we can file the mass-rebuild bugs for them then 15:16:43 not if the openssl maintainer disappears 15:16:49 if a module maintainer disappears and that module has openssl, the attack surface is bigger 15:16:57 the effect of a missing maintainer is more significant here 15:17:08 in theory 15:17:16 in practice, it's still the same person 15:17:24 not really 15:17:28 either maintaining 100 java packages in one module, or in Fedora 15:17:49 not always the same person 15:17:56 yeah i'd say usually not the same person 15:18:05 I could make a module that bundles openssl and anyone using my module would override the "system" version 15:18:20 if I then stop maintaining it, they will be stuck with my version, not knowing 15:18:21 contyk: you should not be able to do that IMHO 15:18:33 We should *probably* also have a policy disallowing module streams of crypto libs 15:18:34 contyk: but this is the decision a person makes when choosing to use the module 15:18:35 if we allow this, we are doomed anyway 15:18:42 bookwar: yes 15:18:47 Just as a general safety precaution 15:18:51 it's not just crypto though - that was just an example 15:18:56 we can not guard people from those 15:18:59 I think we're diverging from the main topic here. Let's not go into Modularity just yet. 15:19:00 mhroncok: well, that's one aspect of modules -- ship the deps that your app needs 15:19:01 CVEs can happen in any package 15:19:15 when they choose to use such a module, they need to subscibe to the news regarding it 15:19:17 zbyszek: isn't modularity the topic here? 15:19:21 mhroncok: if my app requires some special openssl, it would be reasonable to put in there 15:19:27 with crypto it's highly questionable 15:19:30 but in principle, yes 15:20:01 bowlofeggs: yeah, but it's ageneral problem slash feature of modularity, it applies the same to every modular package. 15:20:09 yes 15:20:55 but i think allowing modules to stay around that were built n-m releases ago exacerbates this problem' 15:20:56 proposal: we agree on the general rule 15:21:11 and we figure out the "enforcement" later 15:21:23 yes, +1 15:21:24 perhaps at eol of a release we run reports for any modules still using that release? 15:21:31 the scale of this is still so small that we can just talk to the maintainers 15:21:38 mhroncok: That's fine. 15:21:44 like if f30, at date of EOL has a module that was built when f28 was branched and that module has an openssl vuln, that seems worse than what would happen with ursine packages 15:21:47 could you restate the general rule? 15:22:03 I brought it up because it may be harder to implement than we immediately realize. 15:22:05 Maybe we could just open bugs against all modules built against the old release with "please rebuild"? 15:22:08 bowlofeggs: we shouldn't allow this in default modules, but for custom ones - this is a choice people make. We just really need to make sure we communicate it - you are using a weird module it is your headache when it is not maintained anymore 15:22:09 ... 15:23:04 wasn't this what freshmaker was supposed to do? 15:23:07 Proposal: Policy for modules in Fedora is that all modules being shipped in the stable repositories must have been built against a currently-stable platform" 15:23:12 bookwar: that sounds like someting appropriate for copr, but not official fedora repos, imo 15:23:16 nirik: one of many things, the scope has changed quite a bit 15:23:22 -" 15:23:33 sgallagh: does that handle the rust case tho? 15:23:34 sgallagh: +1 15:23:36 sgallagh: does "currently stable" include rawhide? 15:23:42 depending on if you include rawhide there. 15:23:43 Any module that is capable of building against one platform but running against all of them must be always be built against a supported platform. Modules built against EOL platform MUST be rebuilt on a newer one. 15:23:47 * sgallagh sighs 15:23:49 OK, revising 15:23:49 sgallagh: but that may be hard to deal with in practice 15:23:56 mhroncok: +1 15:24:05 Proposal: Policy for modules in Fedora is that all modules being shipped in the stable repositories must have been built against a currently-active release of Fedora or Rawhide" 15:24:12 i like the policy as a rule, but i think it may be hard to implement 15:24:19 sgallagh: +1 15:24:32 this also deals with the EPEL case 15:24:35 so that's good 15:24:40 mhroncok: +1 sgallagh +1 (I think they say the same thing, pick whichever better wording) 15:24:50 sgallagh: +1 too, what nirik said 15:24:51 mine doesn't say fedora or epel 15:24:53 what happens if the rebuild against supported platform fails? 15:25:05 sgallagh's explicitly mentions Fedora, so it rules out EPEL IMHO 15:25:09 Yeah, mine also doesn't mention epel 15:25:10 I'm OK with both 15:25:11 * sgallagh sighs 15:25:15 .ello2 15:25:15 One more try... 15:25:18 I don't think we need to worry about epel right now... since it's not possible 15:25:24 I'd rather exclude EPEL from this 15:25:25 Proposal: Policy for modules in Fedora is that all modules being shipped in the stable repositories must have been built against a currently-active release of Fedora, EPEL or Rawhide 15:25:33 grr, one sec 15:25:36 building on EPEL and then shipping in Fedora doesn't sound right 15:25:51 probably one point to this, who is going to rebuild modules when some release goes EOL? 15:25:51 Proposal: Policy for modules in Fedora and EPEL is that all modules being shipped in the stable repositories must have been built against a currently-active release of Fedora, EPEL or Rawhide 15:25:55 and change all modulemds? 15:26:06 probably worth deciding on this prior restricting it 15:26:06 yeah i'm concerned abotu the rebuild too 15:26:11 ignatenkobrain: We already discussed that and deferred that question 15:26:16 both who will do it, and what will happen when it fails 15:26:21 The policy can stand before we automate it 15:26:45 because if it fails, we don't have a way to inform users that they are now using soemthing that can't be supported 15:26:45 I think it's fine; there are basically three cases 15:26:49 I'm +1 for whatever (with EPEL, without EPEL) 15:26:49 and that can happen mid-release 15:26:52 so that can be surprising 15:27:00 1. you build for all platforms and you're always fine; this is handled by releng and mass rebuilds 15:27:08 2. you build only in rawhide and ship everywhere, you're fine 15:27:24 3. you build only in the oldest release and ship everywhere, you need to update your modulemd every release and rebuild 15:27:27 bowlofeggs: As I said before, that is no different than a package maintainer getting busy and not releasing the latest updates from upstream. 15:27:41 If the package still *works*, great. 15:27:54 sgallagh: i disagree - as i said before, the impact can be worse due to inclusion of deps they don't maintain 15:28:17 i.e., it is not the same level of impact as what happens with ursine builds 15:28:44 with ursine, one rpm is unmaintained. with this, an RPM and n of its deps are unmaintained 15:28:56 bowlofeggs: The set of people doing this right now is small enough that I'm willing to assume we don't need to legislate around it yet 15:28:59 bowlofeggs: including deps they don't maintain inside the module gets us back to the static linking problem 15:29:09 jforbes: yeah 15:29:46 And as such, it should not be allowed without an exception similar to static linking 15:30:05 if we at least had a way to inform users that they are using a module that was built against a now-unsupported fedora release, that would help my concern a bit (but not entirely) 15:30:11 jforbes: We no longer require exceptions for that. Just virtual Provides 15:31:22 let's vote? 15:31:30 * nirik thinks we aren't going to solve enforcement today...needs more thought 15:31:36 sgallagh's Proposal: Policy for modules in Fedora and EPEL is that all modules being shipped in the stable repositories must have been built against a currently-active release of Fedora, EPEL or Rawhide 15:31:47 +1 15:31:49 +1 15:31:51 +1 15:31:57 +1 15:31:58 +1 15:32:07 +1 (but i'm worried that we can't actually enforce it) 15:32:12 even though technically this might mean shipping rawhide modules in EPEL 15:32:19 we might fine tune that once someone does that 15:32:26 +1 15:32:29 contyk: If it works, do we care? 15:32:36 maybe not 15:32:48 If it doesn't work, probably the maintainer will care 15:33:29 zbyszek: ? 15:33:32 contyk: i imagine that's what the rust sig will want to do 15:33:45 mhroncok: He said he'd have an outage about now 15:33:50 oh 15:33:52 okay 15:34:08 #agree Policy for modules in Fedora and EPEL is that all modules being shipped in the stable repositories must have been built against a currently-active release of Fedora, EPEL or Rawhide (+8, 0, -0) 15:34:28 I was expecting to spend the whole two hours on this 15:34:38 we're super effective today 15:34:42 we can spend the remaining time on the enforcment part :D 15:34:44 #topic #2127 Election Interview Questions — FESCo (Jun 2019) 15:34:44 fesco++ 15:34:46 .fesco 2127 15:34:48 https://pagure.io/fesco/issue/2127 15:34:48 contyk: Issue #2127: Election Interview Questions — FESCo (Jun 2019) - fesco - Pagure.io - https://pagure.io/fesco/issue/2127 15:35:21 accepted by policy? 15:35:48 seems so 15:35:48 i'm fine with reusing the same questions we've been using the last few elections 15:35:55 I only saw +3 when making the agenda 15:35:55 I guess those are ok... they are a bit stale, but meh 15:36:17 nirik: that means I can copy&paste my answers, too 15:36:18 it's not like somebody has time to change them anyway, right? 15:36:52 contyk: sure! 15:37:02 ok, accepted by policy 15:37:19 #agree The same set of questions will be used again (+5, 0, -0) 15:37:26 mhroncok: i have my answers prepared for the next round ) 15:37:38 #topic Next week's chair 15:38:10 I won't be around next week 15:38:18 probably 15:38:25 contyk: please count me +1 to the vote on mhroncok's proposal 15:38:53 * contyk makes a mental note 15:38:59 I will be around for most of the meeting, depends on how long it goes. 15:39:18 anyone going to the opensuse conference? 15:39:23 I guess I can do it, if no one else is chiming in. 15:39:24 * mhroncok doesn't know yet. might not be around 15:39:35 I'm not sure if I'll be around yet. I have some travelling in the evening. 15:39:53 alright 15:40:00 #action sgallagh will chair next meeting 15:40:02 sgallagh++ 15:40:03 contyk: Karma for sgallagh changed to 6 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:40:08 #topic Open Floor 15:41:18 contyk: any update on default modular packages in the buildroot? 15:43:34 mhroncok: not much new but I had a meeting with the MBS folks yesterday to brainstorm the changes needed on their side to support this 15:43:50 there is one open technical question still but I think we can proceed 15:44:05 What's the remaining technical question? 15:44:29 MBS using inheritance to create the buildroots means you can't have those defaults that depend on conflicting streams 15:45:10 long term we should make MBS somehow inject modular metadata into its buildroots and use dnf to resolve it but that would be a major change 15:45:17 and it's not needed right now 15:45:19 contyk: should default streams not depend on anything that conflicts with anything else, and so on transitively 15:45:26 ? 15:45:41 zbyszek: technically they can and dnf can deal with it 15:45:54 zbyszek: e.g. A:1 and B:1 being the defaults, A:1 depending on B:2 15:46:01 zbyszek: Our current policy in Fedora is that default streams may not do that. 15:46:16 zbyszek: with dnf if you install something from A:1, you enable B:2 and make the B:1 default unavailable and vice versa 15:46:16 (Though it's not enforced by any technical means) 15:46:49 since we have this policy that sgallagh mentions, we don't have to deal with the MBS issue right away 15:47:16 but dnf can handle it 15:48:12 closing in a minute unless there's anything else? 15:48:22 well, if the question is non modular build root... 15:48:45 the non-modular buildroot is not affected by this, even without the policy 15:48:51 we need still to make the new buildroot compose and update koji/createrepoc 15:49:27 or wait for epel8 to test things. ;) 15:49:33 so I think sgallagh made the changes to createrepo 15:49:45 regarding koji I'd need to check but that was also planned out 15:49:47 Yes, the 0.14 release should have the necessary changes 15:49:48 yes, both createrepo and koji patches are done and merged. 15:49:55 nice 15:49:57 Koji had the patches merged, not sure if there has been a release 15:50:01 but we still need to make the modular buildroot compose... 15:50:05 then I just need to file the specific requirements for the composes :) 15:50:11 there has not been, but we can backport. 15:50:12 then we need that MBS change 15:50:34 and then we can roll it all out 15:50:59 oh right, I guess it has to use the new buildroot too... ok 15:51:26 yeah 15:51:28 contyk: has there been any discussion of mbs changes needed for epel8? or this would support that too? 15:51:39 nirik: this covers EPEL8 as well 15:52:33 as far as I know, the EPEL plan is still the same -- decompose the RHEL8+CRB content into a purely ursine tag + one tag per module, all linking to external repos 15:52:51 ok. in epel8 the default/buildroot modules will be a bunch of external repos... in fedora it will be a buildroot repo for that? 15:52:52 then we'll need MBS records for the RHEL8 modules, pointing to those tags 15:53:05 a compose of RHEL8 defaults that gets linked with the ursine buildroot, just like in Fedora... 15:53:07 and that's it 15:53:41 nirik: in EPEL it would be the ursine tag (also an external repo) + a bunch of externalr repos for each module 15:53:57 nirik: in Fedora an ursine tag (not an external repo) + one external repo with the default modules, all in one 15:54:01 almost the same 15:54:06 right. 15:54:26 for epel8 we split the rhel8 appstream, for fedora we combine the modules into an appstream. ;) 15:54:34 contyk: thanks for the update 15:54:48 alright 15:54:52 anything else? 15:55:29 okay 15:55:36 #endmeeting