15:00:17 #startmeeting FESCO (2018-10-01) 15:00:17 Meeting started Mon Oct 1 15:00:17 2018 UTC. 15:00:17 This meeting is logged and archived in a public location. 15:00:17 The chair is zbyszek. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:17 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:17 The meeting name has been set to 'fesco_(2018-10-01)' 15:00:18 #meetingname fesco 15:00:18 The meeting name has been set to 'fesco' 15:00:18 #chair nirik, maxamillion, jsmith, jforbes, zbyszek, tyll, sgallagh, contyk, bowlofeggs 15:00:18 Current chairs: bowlofeggs contyk jforbes jsmith maxamillion nirik sgallagh tyll zbyszek 15:00:20 #topic init process 15:00:23 .hello2 15:00:23 o/ 15:00:23 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 15:00:26 .hello2 15:00:26 .hello psabata 15:00:26 jforbes: jforbes 'Justin M. Forbes' 15:00:26 morning 15:00:29 .hello2 15:00:29 contyk: psabata 'Petr Šabata' 15:00:32 evening 15:00:32 sgallagh: sgallagh 'Stephen Gallagher' 15:00:48 .hello2 15:00:49 maxamillion: maxamillion 'Adam Miller' 15:00:49 (I’m also at AnsibleFest, so I may not be fully here) 15:00:56 +1 15:01:02 I'm at AnsibleFest as well 15:01:45 I'm counting 6, so we have quorum 15:01:46 .hello2 15:01:47 bowlofeggs: bowlofeggs 'Randy Barlow' 15:01:58 Let's start 15:02:07 #topic #1974 Problematic blocker for F29: dnf 'offline' module tracking 15:02:10 .fesco 1974 15:02:12 zbyszek: Issue #1974: Problematic blocker for F29: dnf 'offline' module tracking - fesco - Pagure.io - https://pagure.io/fesco/issue/1974 15:02:13 https://pagure.io/fesco/issue/1974 15:02:15 yay 15:02:31 so I was hoping to comment on this one before this meeting but didn't get to it 15:02:42 me too ;) 15:02:57 * sgallagh threw together a straw man on the ticket a little while ago. 15:03:03 But I'm with sgallagh on this one 15:03:34 The "soft behaviour" approach I don't understand at all 15:03:36 * nirik just got up, so goes to read recent comments. 15:04:16 zbyszek: I’ll try to explain, give me a moment 15:05:23 In brief, the major issue here is that DNF historically only sees the "target" state of the system. 15:05:41 It takes all of the repodata it has available at the current moment and makes its decisions based on that: 15:05:44 * contyk is in a call so a little distracted :| 15:05:52 Asked to update? Is there a higher NVR in the set? Then update. 15:06:19 But the problem is that because modules filter some things out of the solver, the *current* state is important, not just the destination state. 15:06:39 Right. And modules are all about not always installing the latest version, so they cannot work with the always-install-latest beahviour. 15:06:41 Because if some repodata is unavailable (the u-t example being the obvious one), then the filtering logic may get confused 15:07:25 Moreover, since the set of subpackages offered by a particular module stream may change over time, if you only have the latest modulemd, it might lose track of those as well 15:08:09 sgallagh: so if i enabled u-t and install a module, that module's metadata (md?) file stays on my system somewhere? 15:08:11 (For example, I carry libuv in the nodejs module because it's not new enough in F28 non-modular. However, if it got updated there, I might drop it in a module update) 15:08:26 which would mean dnf does have the info it would need to know that i got that nvr from a module? 15:08:28 That *specific* case will end up being safe, but the decision will be made without the filtering logic 15:08:42 bowlofeggs: It does not do this *today* 15:09:00 ah, but your proposal is that it could/should keep that md file? 15:09:17 That is the recommended way to resolve this issue: that DNF needs to maintain a lookup table matching installed RPMs to the modulemd associated with them 15:09:27 s/the/my/ 15:09:41 Sorry s/the recommended/my recommended/ 15:09:42 bowlofeggs: yes 15:09:48 and it could assume that if there is no modulemd then the rpm came from a "traditional" repo? 15:09:58 I think we should keep modulemd files for every installed modular RPM and the latest seen modulemd file for every enabled stream 15:10:17 contyk: I concur 15:10:20 but there is a problem dmach raised -- you might get a defective "latest" modulemd from updates-testing 15:10:21 that does sound pretty sensible to me, though i am far from an expert on modularity ☺ 15:10:28 sounds like dnf folks aren't too convinced? 15:10:29 contyk: How do you mean? 15:10:58 sgallagh: you dnf --enablerepo=updates-testing-modular module install foo 15:11:00 nirik: Anyway, FESCo doesn't need to dictate the implementation 15:11:13 We need to decide on what criteria we're willing to block F29 release. 15:11:22 sgallagh: and dnf records the modulemd for that one; you then decide it's broken but dnf remembers the modulemd because it was the last one it saw (assuming foo is enabled) 15:11:46 so you need to drop it somehow, reverting back to the latest from the now enabled repos (stable updates) 15:12:05 I think a clean or somesuch command would help there, even though it'd be a workaround and not very intuitive 15:12:07 contyk: one assumes `dnf module disable foo:stream` would suffice 15:12:17 on the other hand it's a corner case and this will probably only happen when you deal with test updates 15:12:19 so, if we say we require this for final... is there any way it would be implemented before next tuesday? 15:12:20 contyk: I don't think this is much different from the case where you install a package from u-t and then disable u-t and upgrades don't happen for a while 15:12:33 sgallagh: you don't always want to disable the module, especially if it's a low level one 15:12:45 zbyszek: that's true 15:13:00 zbyszek: He didn't say it clearly, but he meant "consider also the case where we accidentally shipped bad/invalid metadata in u-t" 15:13:12 Saving bad data needs a way to recover from it 15:13:24 I think that may be overkill for blocking F29 though 15:13:33 I'd prefer to assume we won't ship bad data 15:13:49 that's fair 15:14:03 OK, so if that happens either a) release updated version with fixed metadata or b) require the user to disable the module ? 15:14:20 sgallagh: heh, as the bodhi maintainer i'm sad to say shipping bad data has happened on my watch 15:14:35 (i accidentally broke all dnf/yum repos by accidentally turning on tagger) 15:14:44 bowlofeggs: I assume the best of you. Prove me right :) 15:14:45 it happens :) 15:14:47 haha 15:14:54 i will say it's not frequent 15:15:18 bowlofeggs: Well, it would be difficult in this case to ship invalid data without a bug in MBS 15:15:26 Since we're basically just copying it into place 15:15:35 how much should we assume that people using u-t know how to recover from weird situations? 15:15:53 i would think we can expect a bit more from those users than the average user (not necessarily a lot more) 15:15:54 bowlofeggs: "Smarter than the average bear", I'd say 15:15:57 yeah 15:16:37 sgallagh: I don't know that I would go that far, people frequently copy paste commands to enable update testing to get a specific fix listed in a bug or similar 15:17:04 We don't want to make updates-testing harder to use for people who are willing to do so, but I don't think that is a huge risk here either 15:17:16 sgallagh: can you describe (or repeat, if you already did), how we could handle the case of bad metadata in your recommended approach? 15:17:18 jforbes: Like I said, I think the bad-data case is *our* problem, not so much the user's 15:17:29 And shipping a new update with fixed modulemd would hopefully fix most cases. 15:17:46 And the workaround of disabling the module and forcibly downgrading/removing the RPM is not terribly onerous 15:18:05 we do have distro-sync... 15:18:42 sgallagh: +1 15:18:49 do we have module-sync? 15:18:52 sgallagh: does distro-sync work on modules? 15:18:58 bowlofeggs: Yes 15:19:04 jforbes: that's kinda what I meant with the "clean or somesuch" :) we don't 15:19:19 ok, to avoid confusion :) 15:19:21 bowlofeggs: Though, it's probably worth our making a statement that it must continue to do so] 15:19:25 distro-sync works on modular packages 15:19:31 as part of our blocking criteria 15:19:33 cool 15:19:35 it doesn't do anything the currently non-existent modular database 15:19:47 ok so we need to make a decision on what our criteria are 15:19:48 right, what contyk said 15:20:18 * nirik is worried how late this is. :( 15:20:30 I think "distro-sync must put all module RPMs at the currently-known versions for any enabled streams" should be one 15:20:40 contyk: i'm not sure i follow that non-existant mdb comment - do you specifically mean it doesn't work if u-t is enabled, a module is installed, and then a distro-sync is run? 15:20:57 and that we would need something like sgallagh's proposal for that to work? 15:21:10 sgallagh: yeah that sounds good to me 15:21:39 wait a sec 15:22:40 While I do agree that we need these protections in place, I am also concerned about making large dnf changes this late in the game. How invasive is the changeset for non modularity users? (I know, we don't know the answer yet, but it should be considered when we do) 15:22:57 sgallagh: that doesn't sound good. If I do 'install module A with u-t enabled', then 'disable u-t', then install B, I don't expect rpm to downgrade A 15:23:00 To make that slightly more clear: A distro-sync must set all the RPMs, modular and non-modular, at the latest version for all module streams in accessible repositories" 15:23:00 well, I dont know that there is a changeset. ;) 15:23:20 zbyszek: Where do you get "install B downgrades A" from? 15:23:28 That wasn't anywhere in the conversation that I saw 15:23:43 from what you said "must put all module RPMs at the currently-known versions" 15:23:57 zbyszek: I said *distro-sync* must do this 15:24:00 Not install or upgrade 15:24:12 Ah, sorry, I misread. 15:24:19 No worries. 15:24:32 yeah this is concerningly late 15:24:57 bowlofeggs: dnf currently doesn't store modulemds in its database or anywhere, so if you lose access to the repodata and don't have it cached, dnf forgets what RPMs on your system are modular 15:25:06 but as jforbes pointed out, there are copy-pasta users out there who this might negatively affect if we release with what we have now too :/ 15:25:14 bowlofeggs: the hope was to make dnf remember modulemds for enabled and installed content to avoid that 15:25:31 bowlofeggs: the u-t case is just a simple reproducer/example of where you generally lose the repodata 15:25:37 yeah cool 15:25:59 the proposal does sound sound to me, but i also do worry about big changes this late in the cycle 15:26:11 is it a big change? 15:26:13 Another example would be a bad mirror that missed the modulemd, etc. 15:26:26 I worry how long we will have to delay... this code isn't written yet right? 15:26:32 I don't think it's necessarily a big change 15:26:36 but what do I know :) 15:27:00 Our team is also lending resources to DNF to try to do this for them. 15:27:15 you could just keep the modulemds on the disk and concatenate that with whatever comes from repodata 15:27:16 I think if we make this a hard requirement for F29 GA, we face a big delay, even in the best-case scenario 15:27:31 i guess the question might be "which is worse? shipping the current dnf, or holding the release and possibly risking a buggier dnf" 15:28:11 or if you are an optimist: which is better? ☺ 15:28:14 it's hard to guess without the code existing yet or the upstream dnf folks not yet wanting to write it. ;( 15:28:22 adamw: Since I notice you're commenting on the bug, you may want to be involved in the conversation as well 15:28:38 oh, you're talking about it now? 15:28:41 Yup 15:28:50 :) 15:29:00 i honestly don't have a strong opinion about what the outcome ought to be, from a qa perspective we just need it to be clear what is expected to be 'done' here by f29 final 15:29:21 * sgallagh nods 15:29:41 My proposal would be to *not* delay F29 for this 15:30:13 I think this mostly affects modularity users, so just say "modularity is still in preview", and ship F29. 15:30:20 would we be willing to allow a dnf update to stable to address this issue post release? 15:30:26 zbyszek: In F29, all users are modularity users 15:30:30 I wouldn't want to say it's in preview 15:30:42 The repos are enabled for everyone and some packages (e.g. Maven) are only available now as modules 15:30:50 yeah module repos are enabled on all systems now 15:31:05 and some packages are becoming module only (like a big part of the java stack according to devel list) 15:31:12 from my point of view this is a major issue 15:31:17 well, I suppose we have next meeting before freeze... we could try and get code and convince upstream and decide next week? or ? 15:31:31 as sgallagh said before, we probably don't need extra recovery mechanisms to clean up corrupted modulemd database 15:31:36 I wouldn't block for that 15:31:39 ack 15:31:52 but keeping the very minimum on your system, I'd really like to see that before GA 15:31:54 I think we want to block on situations that could result in getting incompatible updates 15:31:54 ack 15:32:00 i'd +1 a week of code proposing and further discussion 15:32:06 (meaning updates that don't remain on the selected stream) 15:32:19 stratis is another module only one I think... and thats a f29 change... 15:32:24 * sgallagh nods 15:32:25 the argument that all users are module users now is compelling me to think this is important for f29 too 15:33:44 proposal: let's give modularity+dnf folks a week to propose code and/or ideas and revisit next meeting 15:33:49 Honestly, I think I could settle for "distro-sync must work as expected for available repos" 15:34:06 As long as we have that, we can get out of any otherwise unexpected situations 15:34:11 sgallagh: and 'available' means 'enabled' ? or 'ever been enabled' ? 15:34:16 bowlofeggs: +1 15:34:32 I would really like to see some sort of estimates as to a) how long the dnf changes will take to actually implement, and b) how risky/invasive those changes are before making a final call. We don't have those. Hopefully we could get them in the next week. 15:34:40 sgallagh: yeah that's a good point, but i'd think we'd need to define "as expected" a bit too 15:34:56 nirik: The repo must be enabled and available. But the module RPMs must not change stream from "ever been enabled" 15:35:23 They *can* downgrade if the repo only has older versions for that stream, but must not change streams 15:35:38 bowlofeggs: +1 15:36:17 Well, I'm hoping that by constraining the problem to the above, DNF can get us a fix even if it's not the complete one we hope for with the modulemd local DB 15:36:53 Can we as FESCo agree that the distro-sync example I cited above is at least the minimum we need to be comfortable shipping? 15:37:12 As I said, as long as we have that, users have an escape hatch from other issues. 15:37:36 yeah, that seems reasonable... 15:37:37 sgallagh: can you write up a proposal that expands the "as expected"? i'm in favor of the general idea, but would like something that defines the expectation a bit 15:37:40 sgallagh: I can agree with that. 15:37:43 like that stuff about not changing streams 15:38:01 Yeah, I'll add something to the ticket 15:38:04 * sgallagh starts typing 15:38:29 and what happens if the u-t update *is* a new stream? 15:39:04 i guess that would be like installing an rpm from an enhancement u-t update that doesn't exist yet in the stable repos? 15:39:12 i.e., a distro-sync would do nothign? 15:39:31 sorry, s/enhancement/newpackage' 15:39:51 or does distro-sync remove packages it doesn't see in teh distro? 15:39:57 i guess maybe it does? /me can't remember 15:40:22 Do we have agreement that sgallagh posts in the ticket, and we just ack bowlofeggs' proposal to wait a week for discussion? 15:40:33 ah, distro-sync does not remove packages it doesn't find in the distro 15:40:46 i have one RPM i made for myself on my local system and distro-sync does not propose removing it 15:41:09 bowlofeggs: An update of a new stream is sort of like a new package. It should not impact the update process 15:41:28 sgallagh: yeah that's how i see it too now that i think about it some ☺ 15:42:02 bowlofeggs, others: I just submitted my proposal to the ticket 15:42:16 thank you 15:43:06 https://pagure.io/fesco/issue/1974#comment-534269 for any looking for the link 15:43:24 Thanks 15:43:31 hmm 15:43:42 so that would mean if there was a new default module it wouldn't enable on distro-sync? 15:43:44 so "dnf update" could change streams? 15:43:57 sgallagh: +1 15:44:18 contyk: I wasn't setting a constraint on that in this proposal 15:44:35 nirik: I expressly avoided talking about what the non-modular RPMs should do in that proposal 15:44:49 But yes, a non-Modular RPM *may* become a modular RPM of a default stream 15:44:56 If you want me to add that to this proposal, I can 15:45:14 I doubt we can list all cases. ;) 15:45:23 Right, but I think that one's probably worth it, actually 15:45:33 Since it's going to become common as packages start switching over 15:45:39 true 15:46:08 +1 to the amendment 15:46:58 OK, we have been on this for 45 minutes, and we have a concrete proposal 15:47:10 Amended 15:47:14 See the same link above 15:47:43 proposal: table for a week, discuss proposal in the ticket, revisit next week? 15:48:00 zbyszek: Seems reasonable, but given time constraints please discuss... soon :) 15:48:09 +1 15:48:19 sgallagh: +1 15:48:33 I don't know, this seems like something totally different than what the bug is about 15:48:39 I'm for the proposal I think, but it would be nice to get more input and think about it a bit instead of a few minutes monday morning. ;) 15:48:42 and I think it already works, too (good) 15:48:43 +1 to table, but yes it *has* to be discussed next week. 15:49:05 i could +1 rediscussing/checking-in next week 15:49:12 but i also like the proposal 15:49:48 +1 to another week of "offline" discussions and proposals 15:50:00 bowlofeggs: I like the proposal, I would also like to see some feedback from the dnf team on the proposal 15:50:14 jforbes: Reasonable 15:50:39 jforbes, nirik: sgallagh's proposal in the ticket is *not* being voted here 15:50:40 sure 15:50:52 contyk: I think we're probably 99% of the way there, but I still see at least one case that wouldn't work, which is the original proposal I made this morning. 15:51:03 zbyszek: I know, I voted on your motion to table 15:51:14 Install an RPM from a module that we no longer have any repo reference ofr 15:51:15 *for 15:51:35 This has to cover at least ensuring that this doesn't get "upgraded" back to non-modular 15:52:06 Anyway, I'll stop talking now. 15:52:45 ok, for clarity: sgallagh, bowlofeggs, contyk, can you please vote on *my* proposal explicitly 15:53:03 zbyszek: +1 15:53:03 zbyszek: and your proposal is which one? 15:53:18 zbyszek> proposal: table for a week, discuss proposal in the ticket, revisit next week 15:53:30 zbyszek: +1 15:53:53 zbyszek: +1 15:54:31 #agreed We table this for a week to give modularity+dnf folks time to discuss and respond to the proposed requirement. We will revisit next meeting. (+6, 0, 0) 15:54:34 #info proposal for requirements for DNF behaviour at GA: https://pagure.io/fesco/issue/1974#comment-534269 15:54:37 #info given time constraints, we ask for the discussion to start now 15:54:53 The next one should be easier. 15:54:58 #topic #1927 F29 Self Contained Change: Origin 3.10 15:54:58 .fesco 1927 15:54:58 https://pagure.io/fesco/issue/1927 15:55:00 zbyszek: Issue #1927: F29 Self Contained Change: OpenShift Origin 3.10 - fesco - Pagure.io - https://pagure.io/fesco/issue/1927 15:55:12 several of us voted in ticket for this one 15:55:19 * zbyszek is counting 15:55:37 * jcajka is around if there are any questions 15:55:37 +1 here 15:55:43 we're at +4 in the ticket: jforbes, jsmith, zbyszek, bowlofeggs 15:55:58 contyk ? 15:56:01 +1 15:56:05 zbyszek: You missed me. We're already +5 15:56:09 Well, +6 with contyk 15:56:15 +7 with nirik ☺ 15:56:37 Right. 15:56:57 #agreed The update of the Change from 3.10 to 3.11 is approved (+7, 0, 0) 15:57:21 #topic Next week's chair 15:57:36 I can do it 15:57:44 Sorry for the late announcement today, I returned from ASG2018 late yesterday 15:57:58 #action jforbes will chair next meeting 15:58:03 #topic Open Floor 15:58:22 thanks jforbes 15:58:33 https://pagure.io/fesco/issue/1970 15:58:37 * bowlofeggs looks around at how wide and open this floor is 15:59:12 smooge: Do you have a specific question? 15:59:26 it is more of an ongoing issue 15:59:42 s/ongoing/developing/ 16:00:05 This needs more manpower to be resolved. 16:00:12 tibbs had a package for EPEL only which got orphaned by someone else 16:00:49 Actually the best I can figure out is that the act of unretiring that package removed my access completely. 16:01:10 And then the package was orphaned a few days after it was unretired. 16:01:20 haha that's weird 16:01:39 I believe it was intentional to start the six week clock between orphaning and retirement. 16:02:23 But I didn't receive any notice that any of this had happened, and I didn't notice the state of the package between the unretirement and the orphaning. 16:02:31 So I don't know exactly what step removed my access. 16:02:44 tibbs mentioned that ticket number so I wanted to bring it up in open floor as a heads up that FESCO may be seeing some upset packagers 16:02:53 https://pagure.io/fesco/issue/1970#comment-532232 16:03:04 tibbs: I think the issue with python2-matplotlib is not directly related to teh main part of that ticket 16:03:09 that seems like a non ideal process... 16:03:12 Well 1970 is the ticket and that's my comment requesting coordination. 16:03:29 f29 blocker review meeting is starting up in #fedora-blocker-review now 16:03:34 sorry for the slow typing.. I am trying to make sure I had accurate references 16:04:14 I didn't see the ticket at all until I was mentioned in it and basically just replied to the mention. 16:05:10 tibbs: fwiw, I think what you are asking for in the last comment seems reasonable. 16:05:30 I think this is a corner case with EPEL-only packages that we didn't consider. 16:05:49 Maybe we can continue the discussion in the ticket? 16:07:30 I wonder how many cases like this we have 16:07:38 EPEL only packages? 16:08:31 zbyszek, do you mean EPEL only packages or do you mean that would be alternative corner cases? 16:08:56 note that you can override the bug owner for fedora or epel, ie, set that to orphan... but yeah 16:09:51 not epel-only, but rather "we want to retire the Fedora package, but keep the epel one" 16:10:43 most of the epel python2 stub packages 16:11:06 Then it sounds like we need to adjust the policy to take this case into account. 16:11:09 well in this case was this ever in Fedora? 16:11:55 things like python34 python36 etc would also show up like that.. (and the php, perl etc ) 16:12:18 also there's the other case... something we want to retire in epel, but keep in fedora... 16:12:53 and that case too :) [as nirik writes what I was working on] 16:12:53 see also: https://pagure.io/fedora-infrastructure/issue/6241 16:14:05 .. thought he had written about this a long time ago elsewhere .. 16:15:17 has nothing more to add on this at the moment 16:15:33 ideally we should document all these cases here... ;( 16:16:36 * zbyszek has nothing more to add either 16:17:00 Let's continue the discussion in the ticket. 16:17:17 Concrete proposal are welcome. 16:17:27 Anything else for open floor? 16:18:07 So see next week 16:18:08 #endmeeting