17:01:49 #startmeeting FESCO (2021-04-27) 17:01:49 #meetingname fesco 17:01:49 #chair nirik, ignatenkobrain, decathorpe, zbyszek, sgallagh, mhroncok, dcantrell, cverna, Conan_Kudo, Pharaoh_Atem, Son_Goku, King_InuYasha, Sir_Gallantmon, Eighth_Doctor 17:01:49 #topic init process 17:01:49 Meeting started Tue Apr 27 17:01:49 2021 UTC. 17:01:49 This meeting is logged and archived in a public location. 17:01:49 The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:49 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:01:49 The meeting name has been set to 'fesco_(2021-04-27)' 17:01:49 The meeting name has been set to 'fesco' 17:01:49 Current chairs: Conan_Kudo Eighth_Doctor King_InuYasha Pharaoh_Atem Sir_Gallantmon Son_Goku cverna dcantrell decathorpe ignatenkobrain mhroncok nirik sgallagh zbyszek 17:01:57 .hello2 17:01:58 .hello2 17:01:58 ignatenkobrain: ignatenkobrain 'Igor Raits' 17:02:01 .hello2 17:02:01 sgallagh: sgallagh 'Stephen Gallagher' 17:02:03 .hello churchyard 17:02:04 dcantrell: dcantrell 'David Cantrell' 17:02:07 mhroncok: churchyard 'Miro Hrončok' 17:02:31 morning 17:02:32 sgallagh: I've updated the meeting page 17:02:55 Thank you 17:02:58 .hello2 17:02:59 decathorpe1: Sorry, but you don't exist 17:03:07 .hello2 17:03:08 zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' 17:03:31 * decathorpe1 questions existence 17:03:53 sgallagh: also tagged https://pagure.io/fesco/issue/2598 with meeting 17:04:19 Ah, I missed that one. 17:04:21 Thanks 17:04:41 OK, we have quorum, so let's get started 17:05:20 #topic #2598 F35 Change: Package information on ELF objects 17:05:20 .fesco 2598 17:05:21 sgallagh: Issue #2598: F35 Change: Package information on ELF objects - fesco - Pagure.io - https://pagure.io/fesco/issue/2598 17:05:54 .hello ngompa 17:05:55 Eighth_Doctor: ngompa 'Neal Gompa' 17:06:02 FWIW, after the lengthy discussion of fedora-devel, I'm not comfortable with making this "on by default". 17:06:19 I don't think this is a good feature at all :/ 17:06:20 I'm inclined to agree. (I caught up on it this morning, fortunately) 17:06:22 I have not followed 100% of the devel discussion - was the problem of package content churn addressed? 17:06:22 zbyszek: how could this be opt-in? 17:06:23 I think it's a wonderful feature, but has some non-negligible risks. 17:06:33 agreed, I don't really like this proposal 17:06:39 Oh, wait, sorry, wrong ticket. 17:06:43 hello o/ 17:06:48 I'm not opposed to it in Fedora, but I do not like the idea of on by default 17:06:57 Sorry, I didn't even look at the subject. 17:07:06 I don't know what would make sense for it to be partially on 17:07:07 * nirik is with mhroncok... how can this be opt in? 17:07:07 Just so we're clear, I started with New Business 17:07:16 like how would you even make this reasonably work? 17:07:21 So this is the ELF annotation issue 17:07:34 sgallagh: Hmm, the agenda said debuginfod as the only item... 17:07:54 that is my fault 17:07:56 zbyszek: Yes, but mhroncok alerted me to one other item 17:08:01 I added this after the agenda was sent 17:08:04 Oh, OK. Apologies. 17:08:16 Nope, I should have been louder about the agenda change 17:08:35 Anyway, let's cover this one first, then move to debuginfod 17:08:39 For *this* subject, we're gathering information to be able to reply to the question about CoW. 17:09:03 I hope to provide a bigger reply on the mailing list within a few days. 17:09:26 opt-in may also be pointless on this ELF one anyway. I don't see a huge advantage here and it feels like it's shifting a lot of the RPM db in to ELF metadata 17:09:28 so, table to next week? 17:09:42 nirik: +1 17:09:46 nirik: +1 17:09:49 nirik: +1 17:10:06 +1 17:10:14 * mhroncok is now sorry for bringing it to the agenda :D 17:10:20 .fire mhroncok 17:10:20 adamw fires mhroncok 17:10:23 :) 17:11:10 mhroncok: my fault, I should look at topic. 17:11:31 I'm okay with tabling it. I had some thoughts, but I'll reply on the list 17:12:01 #info We will revisit this topic next week after zbyszek provides feedback on the devel list 17:12:09 #topic #2597 F35 Change: Debuginfod By Default 17:12:09 .fesco 2597 17:12:10 sgallagh: Issue #2597: F35 Change: Debuginfod By Default - fesco - Pagure.io - https://pagure.io/fesco/issue/2597 17:12:23 I think it's a wonderful feature, but has some non-negligible risks. 17:12:25 * sgallagh ducks the incoming messages 17:13:14 yeah, from a feature perspective, this seems nice, but the privacy and security angle should be considered 17:13:15 agreed. 17:13:39 Could we tie this into Software (and the KDE equivalent)? 17:13:39 But I think doing it is worth it over the risks... is fche here today ? 17:13:51 Similar to how we get one-time approval for 3rd-party software? 17:14:10 one-time approval would be nice, but hard 17:14:11 I was going to bring this up with the council, but the next meeting was not until this week 17:14:20 for a council policy weigh-in sort of thing 17:14:38 very hard, as lots of different clients. ;( 17:14:58 nirik: what about the novel concent of a config file? ;) 17:15:32 decathorpe1: if the default is off, people won't discover this 17:15:40 and if the default is on, we are where we are now 17:15:52 well, my understanding is that this just provides an interface for clients... you would need to change the interface to get it to have some sort of opt in thing... 17:15:52 I think I could make a fairly strong argument that if a user calls `gdb `, their natural expectation is to be able to debug that stuff 17:16:09 indeed 17:16:12 And that our current approach (telling them to run a DNF command) does not meet that expectation 17:16:18 indeed 17:16:39 but I'd also argue, that downloading stuff (to execute) from the interwebz is also nto expected 17:16:59 mhroncok: I think we could easily patch gdb and the 5 other most common tools to give a hint (if $DEBUGINFO_URL not set, say "consider installing elfutils-debuginfod-client") 17:17:09 I'd counter that with "nothing is downloaded unless it has an approved signature" 17:17:37 s/downloaded/executed/ 17:17:38 sgallagh: with dnf yes, with this new stuff there is no signature 17:17:48 hi nirik 17:17:57 okay, *that* sends up alarm bells for me 17:18:04 didn't know we were on the agenda 17:18:15 fche: Sorry, I sent the agenda very late this week 17:18:27 zbyszek: how many people would say 'n' ? 17:18:28 :) 17:18:51 sgallagh, yeah, re. no signature, well what sort of signature should we invent? 17:19:09 nirik: I don't know. But I think some would. 17:19:20 sans IMA or such build-time signatures on individual files, what else is there? 17:19:29 I doubt it. It sounds like a dialog people would learn to just say y to... ;( 17:19:50 I am trying to debug thing... oh yeah, I have to say yes to this silly prompt 17:19:56 fche: IMO, we want a signature that is generated at package build time on the offline build system, and can be used by the client to verify that the file it got from the debuginfo server matches. 17:19:57 zbyszek, (btw it's not 5 tools but more like 20) 17:20:04 fche: OK, I guess I was thinking that this would download the RPMs that provided that debuginfo 17:20:15 But it's just downloading the individual files? 17:20:29 sgallagh: it does not use rpms directly (but it reads them from koji on the server end) 17:20:30 sgallagh, correct. 17:20:41 it consumes signed koji rpms and extracts things on the fly on request 17:21:04 but no signatures are passed through? 17:21:19 I mean you could assemble the signed rpm on the client and verify it, but that would make it a lot slower I suspect. 17:21:22 But there's no client-side validation that they're getting trusted content? 17:21:24 there are no per-file signatures to pass through. 17:21:29 so there's no way to verify that you're actually getting what you asked for? 17:21:54 The risk of a MitM attack there is fairly high, I'd think. 17:22:07 couldn't we do signatures on the files? 17:22:18 well, it's a https connection to fedora-infra... 17:22:25 when importing into a debuginfod thing, just gpg sign them or something? 17:22:31 Eighth_Doctor, would defer to the IMA proposal for that as a thing then? 17:22:33 Eighth_Doctor: we rejected the ima change... which might have been able to do this. ;) 17:22:45 y'all are making this too complicated 17:22:52 we don't need IMA to solve this problem 17:23:08 fche: did you consider something that'd be like the IMA per-file signatures, but that'd be distributed separately, so that debuginfod can forward per-file sigs to the client? 17:23:14 I like what Eighth_Doctor says 17:23:53 out-of-band privileged distribution of per-file signatures (which don't exist yet at all) ? 17:23:56 either do something like detached signatures as part of debuginfod protocol, or attached signatures into the files themselves and gdb grows a way to verify them 17:24:09 IMA is overkill for this purpose 17:24:15 * nirik notes again this is not just gdb 17:24:36 whatever, not the point 17:24:41 if you're asking debuginfod to Sign files as it serves them, that still requires the client to trust the keys; the server would have to have private keys (which become a juicy target too) 17:24:53 fche: you can't avoid that 17:24:54 not sure that helps 17:24:57 you can't avoid being a target 17:25:06 anyone who says otherwise is a liar 17:25:07 nirik: if we had signatures, the client that downloads the files would also do signature verification, so the number of programs that make use of this doesn't matter. 17:25:27 zbyszek, again the issue there is having a trust chain. 17:25:36 the server that serves the files does not need to be the server that signs and uploads the files to the server 17:25:39 if the sigs come from the build system, on a per-file basis, with well distributed keys, okay that might work maybe 17:25:56 fche: they are proposing a completely seperate chain. 17:26:01 if the sigs come from private keys held by the debuginfod server, then you have a single attack point, the same thing as we're worried about with the status quo 17:26:17 fche: what mhroncok and nirik said. 17:26:38 the sigs come from private keys held by the srever that also signs the RPm packages, or another sevrer similar to that one 17:26:53 that server sends the signed files to the debuginfod server 17:26:58 sigul already knows how to do signatures of random things 17:27:02 so we sign on demand? 17:27:10 10,00000 files? 17:27:27 that's tricky 17:27:31 what about shipping something like SHA256 hashes of debuginfo files with the normal non-debuginfo packages? that shouldn't amount to too much data but would allow the debuginfod client to verify that the known-good-and-verified checksum matches the downloaded file 17:27:36 3.87 million binaries with buildids 17:27:58 decathorpe1: that might be very big 17:28:10 decathorpe1, https://sourceware.org/bugzilla/show_bug.cgi?id=27758 << something like that was proposed 17:28:25 not as protection against a penetrated server but as a way to detect en-route or later corruption 17:28:55 note that that ticket is an entirely different thing 17:29:21 yeah, just shipping-hashes reminded me. 17:29:38 decathorpe1, shipping hashes with the basic binary rpms is ... IMA ? 17:29:39 when we generate the debuginfo files during rpmbuild, we could hash them, and store that to the buildid directory 17:30:04 we would only hash what is debuginfo'ed 17:30:12 IMa is hashing everything 17:30:16 no? 17:30:23 debuginfod also serves source code files, ftr 17:30:38 right, that could easily get ugly 17:31:08 if you need to store hashes of 12256768 file in the rpm that is otherwise just 4 KiB :D 17:31:16 12256768 files 17:31:25 never mind then 17:31:36 so just to be clear - the only thread model of interest here - is a penetrated server? 17:31:51 apparently, yes 17:32:06 seems to be what this discussion is latched on to. ;) 17:32:34 fche: also another type of man-in-middle-attack where the client is tricked into taking to a different server, e.g. through a compromized TLS cert. 17:32:40 is there a Security team :) who can assess risks of that, or give advice how to detect/prevent that at the server side? 17:32:47 Red Hat has one 17:32:48 zbyszek, TLS compromises I think are beyond our scope 17:32:57 (note that I am not necessarily against this, I just think that we should explore all other options and ask legal+council for approval) 17:32:57 that implicates the entire download chain system 17:33:18 there's a whole bigger problem if we have TLS compromise 17:33:21 Yeah, I think I'm less concerned about this so long as the downloader is properly validating the server cert 17:33:26 we can add a TLSA record for it.. that would help perhaps? 17:33:34 probably 17:33:34 sgallagh, yeah it does, via libcurl defaults 17:33:35 mhroncok: +1, I think council should have an opinion here. 17:33:37 Beyond that, it's a multi-layered attack, which reduces the effectiveness 17:34:00 I will bring this up with the council on Thursday 17:34:27 I think it would be nice to talk with the RH security folks. 17:34:30 dcantrell, you promised that last time :-) :-) 17:34:37 fche: meetings are not weekly 17:34:41 aha! 17:34:43 only teasing 17:34:46 :) 17:35:04 zbyszek: I think that's a reasonable idea. the next question is _which_ security team should be consulted 17:35:04 but yeah please do hook us up with whatever other teams need to opine on this, we'd love to talk and explore what else is practical 17:35:04 let's wait for what council says 17:35:05 I'm happy to hear what the council thinks, but this is a technical thing so I would think it would be in our wheelhouse. ;) 17:35:38 mhroncok: I think the council doesn't have enough information right now. We should figure out the technical security details first. 17:36:07 perhaps we could see where everyone is at? a) +1 change as listed, b) +1 change, but no default, c) -1 change until security issues are more dealt with 17:36:14 zbyszek: ok, but we need to do this async, just brianstorming here every week is not moving us any further 17:36:21 mhroncok: +1 17:36:22 mhroncok: ack 17:36:40 nirik: b 17:36:44 ok, an actionable question on the issue# would be great 17:36:53 After this discussion, I'm not concerned about the security angle any longer. 17:37:02 I am a) personally. Happy for security improvements, but don't want to hold it up and other distros are already using it. 17:37:35 I would like the Council to chime in on the potential for information leakage, but I suspect that we can solve that by the information policy rather than a tech solution. 17:37:54 I'm not decided, either a or b 17:38:09 sgallagh: information leakage is not very big here. In fact it's probably lower than what the mirrors already get from update requests. 17:38:19 so far I'm inclined o say "sounds nice, but please opt-in only for the people who understand the implications" 17:38:36 decathorpe1: that's 'b' 17:38:45 b it is, then. ;) 17:38:55 zbyszek: Well, it's a stronger indication of what software is actually being RUN on those systems (vs. just installed) 17:39:04 but yeah, I'm not too worried about it either. 17:40:39 OTOH, it's only with a server under our control, so the information is rather tightly held... 17:41:00 what if the server is compromised! 17:41:03 (just kidding) 17:41:55 So... I think I was the person who was questioning the security aspect most strongly. If everyone else thinks I'm being overly paranoid, I can accept that. 17:42:05 how big a vector is that really in the end? how hard is it to feed a debugger a tampered with debuginfo and get... anything ? 17:42:09 would y'all like me to phrase a backup option in the Change, for a graceful -1 in the sense of "if not approved for Default, this feature is still worth mentioning as a Release Notes, and this is how to install it?" 17:42:34 fche: I like that idea 17:42:38 fche: I don't think it will be needed 17:42:44 (not that I'd prefer -1 but :) 17:43:16 Honestly, I think I've talked myself to voting +1 on the proposal as it stands 17:43:21 somebod please count the abcs 17:43:40 I'm too tired for that :D 17:44:12 fche: do you know if anybody is doing fuzzing of the debuginfo data users? 17:44:30 nirik and sgallagh are the only ones who would pick a? 17:44:49 libdwarf? 17:45:00 zbyszek, not aware. but to be honest, even Valid crafted dwarf could do some damage, with correct debugger type tools. 17:45:15 yeah, so far: me: a sgallagh: a, decathorpe1: b, mhroncok: a or b no other votes? 17:45:19 I am leaning more towards a) too 17:45:27 another possible compromise position is a) for non-root users only ? 17:45:32 I voted a. 17:45:34 I voted b 17:45:36 sorry. 17:45:42 zbyszek: make up your mind! :D 17:45:50 * fche hands zbyszek a fiver 17:46:39 ok, so 3 for a, 2 for b... 17:46:53 I vote b 17:47:07 ok, 3 and 3 then... 17:47:38 considering we need 5 for approval, my hesitating vote would not help 17:47:46 any of you b folks would be swayed by further changes/discussion? 17:47:50 who's the third 'a' vote? 17:48:00 me :) 17:48:00 cverna 17:48:02 cverna 17:48:03 zbyszek: cverna... I guess he was just leaning... 17:48:16 yeah count me in a) 17:48:25 Sorry, I missed that. 17:49:01 so I guess we just punt and retry next week and see if we can get more votes/change minds? 17:49:09 if there was a consent prompt, I'd be aAAA :D 17:49:28 nirik: I am afraid it won't help 17:49:29 what more info can I bring to y'all for next time? not sure 17:49:35 nirik: but we can try once more 17:49:39 nirik: we could accept 'b' provisionally, and maybe "upgrade" to 'a' later on. 17:50:00 we could, but is that not already in place? 17:50:25 mhroncok: you need to do configuration by hand. 17:51:06 fche: do you want explciit fesco approval for easy opt-in before we decide about making it the default? 17:51:09 fche: a consent prompt isn't going to be possible right? 17:51:33 nirik, not really easy.... if only one could prompt a sysadmin during a rpm -i or something :) 17:52:09 mhroncok, manual opt-in is already possible (Via the env var.). we could interpret a 'b' vote as a request for a non-default subrpm that activates that env var in profile.d 17:52:24 ('a' would be doing that profile.d change in some default subrpm) 17:53:29 fche: and I think we can approve that here if you want us to 17:53:36 Oh, I see one thing now. /etc/profile.d is a bad way to do this, because it's only effective for things which go through a shell. 17:53:56 So for example, systemd-coredump cannot make use of this. 17:54:10 zbyszek, systemd has env files that could set it up for them, but yeah 17:54:24 fche: yuck! 17:54:32 well yeah but baby steps 17:54:38 fche: Can it be made to read a config file? 17:54:43 * cverna needs to drop, thanks everyone o/ 17:55:01 zbyszek, yeah probably 17:55:29 +1 to baby steps 17:55:47 -1 to figure out the implementation on a meeting at 8 PM 17:56:02 mhroncok: it's good that we still have 5 minutes until 8. 17:56:09 :) 17:56:10 4 17:56:15 zbyszek: :D 17:56:56 I think we're close here. 17:57:01 sgallagh: do we have a "volunteer" to talk with RH security people? 17:57:05 fche: when constent prompt would be very very hard, would a warning be easier? 17:57:23 mhroncok, think we could do something like that 17:57:31 isatty -> notify on stdout kind of thing 17:57:34 zbyszek: I'm honestly unsure it is necessary. 17:58:21 fche: e.g. -- Ok user, I am now going to destroy your computer, you have 3 seconds to terminate me. If you wish to disable this warning, edit this config that way. If you wish me to never do this, edit it in this another way. Sleeping 1...2...3...booom 17:58:59 that way, we still have this on by default, so poople know about it and thay can use it 17:59:07 and if they are paranoid, they knw how to disbale it 17:59:08 mhroncok: Sounds good, as long as we can play some 8-bit music at the same time. 17:59:16 chiprom 17:59:17 (the sleeping but is non neccessary, a message would do) 17:59:29 s/but/bit/ 17:59:43 beep beep beep Are your debuginfo are belong to us 18:00:41 OK, I think we've gone past the "useful" portion of this discussion. 18:01:01 the message thing was actually a serious proposal 18:01:06 that would make me vote a) 18:01:11 As I understand it, we don't have enough votes to pass this yet, but we seem to agree in general that we *want* to get to that point. 18:01:16 but yes, please end this meeting :D 18:01:25 So let's enumerate the specific things we need to do to get to "yes" and then call it a day 18:01:32 +1 18:01:35 sgallagh: +1 18:01:58 1) Someone needs to talk the Security Response Team for a threat analysis. 18:02:07 2) We need to ask Council about the security implications 18:02:17 2) (and privacy) 18:02:26 3) We need to investigate the feasibility of a consent dialog. 18:02:36 Oops, 2) was supposed to say privacy indeed. 18:02:47 Council isn't the right place for security decisions 18:02:51 right 18:02:59 Am I missing anything? 18:03:12 I don't think so 18:05:49 OK, I'll volunteer to reach out to the security folks. dcantrell can you talk to Council? 18:05:53 And who wants 3)? 18:05:59 yes, I'll talk to council 18:06:10 #action sgallagh to talk to Security team about a threat analysis 18:06:11 as much as I'd liek to explore 3, I won't be able to 18:06:24 #action dcantrell to speak to Council about privacy concerns 18:06:35 fche: Can you provide us with some options by next week? 18:07:35 re. consent dialog and such? sure 18:07:50 thank you 18:08:04 #action fche to explore consent dialogs/warnings 18:08:26 #topic Next Week's Chair 18:08:45 Who wants it? 18:10:10 ... 18:10:17 sgallagh: I'll take it if nobody else will. 18:10:31 You've been stuck with it too frequently of late 18:10:36 I'll do it again next week 18:10:51 #action sgallagh to chair next week's meeting 18:10:56 #topic Open Floor 18:10:56 sgallagh++ 18:11:43 sgallagh: thanks 18:12:00 Anything for Open Floor? 18:12:35 happy Fedora 34 release day 18:13:51 Indeed 18:13:55 OK, I'm calling it. 18:13:57 #endmeeting