18:01:09 #startmeeting FESCO (2013-05-29) 18:01:09 Meeting started Wed May 29 18:01:09 2013 UTC. The chair is t8m. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:09 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:01:15 #meetingname fesco 18:01:15 The meeting name has been set to 'fesco' 18:01:22 #chair abadger1999 jwb mitr mmaslano notting nirik pjones t8m sgallagh 18:01:23 Current chairs: abadger1999 jwb mitr mmaslano nirik notting pjones sgallagh t8m 18:01:31 Nobody here but us chickens 18:01:33 morning everyone. 18:01:34 Hello 18:01:35 #topic init process 18:01:37 Greetings 18:01:43 Hello everyone 18:01:45 hello 18:02:29 * notting is here 18:02:34 hi 18:03:43 should we wait a short while for jwb? 18:04:00 hi, sorry 18:04:33 ok let's start 18:04:36 #topic #1113 Using PIE by default on AMD64 18:04:44 .fesco 1113 18:04:45 t8m: #1113 (Using PIE by default on AMD64) – FESCo - https://fedorahosted.org/fesco/ticket/1113 18:05:19 so, I think we are all agreed that we don't want to do this for f19 right? so, it would be f20... and it would need a mass rebuild. So, I don't think we need to be very hasty here... we can take time and gather info, etc. 18:05:26 In general, I continue to favor just leaving this decision up to the package maintainers, except for the previously-agreed categories. 18:05:47 So Jakub seems firmly opposed to have PIE on by default. 18:05:54 t8m: I tend to defer to him in these situations. 18:05:58 nirik, I agree that we do not have to be hasty 18:06:51 I still haven't seen a compelling set of applications that would be noticeably harmed by the ~3-4% penalty if the default were filpped. 18:07:17 basically, all of them. 18:07:24 OTOH the side effect of invalidating prelink does seem to be noticeable (... on LibreOffice, where we could debate whether it shouldn't be _hardened_build anyway) 18:07:25 mitr: it's not actually clear if it is 3-4% tho... 18:07:56 jwb: For applications that spend most of their time waiting for users' input and have a 100ms latency budget, the 3-4% don't make a difference, do they? 18:07:59 jwb, basically none of them as 99% of code is already in shared libraries which are PIC 18:08:42 nirik: fair point, but then again we don't know what to measure even if the methodology could be improved 18:09:13 well, to start with, some actual fedora packages with our compiler flags, etc... 18:09:23 right. so since people don't care because the apps are idle most of the time, we should build with -O0 so we can debug things more easily 18:09:34 your logic doesn't really make sense 18:09:49 on the other hand I agree with Jakub that most important for security are other things than address space randomization 18:10:30 jwb: I can't actually see an obvious rebuttal to the -O0 argument 18:10:31 I don't think the argument about idling is really valid, what's more important is that most of the code is already PIC 18:10:54 mitr, that means you don't know what it does 18:10:56 t8m: Well, yeah, "don't use C" where this discussion becomes moot 18:11:05 jwb: I would actually *agree* with your -O0 argument, if only because we can barely ever debug anything with our debuginfo. 18:11:18 ugh. ok, i'm done. 18:11:33 jwb: Yeah, if it tripled the binary size or something, that would be worrisome. But, in principle, if the performace doesn't matter, then it doesn't matter. 18:11:33 i don't agree PIE should be default everywhere and i don't think -O0 is a good idea. 18:11:45 this is just silly 18:11:53 jwb: Believe me, I agree with you. 18:12:15 as we wouldn't forbid to opt out of the hardened build I don't think having the default flipped would be a problem for anyone though 18:12:20 t8m: As for "most of the code being PIC", you only need one non-randomized place to override. (It does get easier if you have more of them, but not linearly) 18:12:38 The irony is that the only examples I can think of that would really benefit from a 3% performance increase are the network servers that we already have on the mandatory list (apache, MariaDB, etc.) 18:12:38 * nirik is torn between just listening to jakub and trying to gather more info, but it's muddy what all that info should be. 18:12:44 given the number of customers (admittedly, not fedora) that intentionally disable selinux for real/imagined performance impact of the same order of magnitude as discussed here, I don't think we can wave away the performance impact 18:12:54 mitr, not from the security POV but from the code efficiency 18:13:13 sgallagh: Yes, precisely. 18:13:44 notting: by the same token, we continue to ship with selinux enabled by default despite the (real or imagined) performance impact. 18:13:51 abadger1999, +1 18:14:43 are we seriously ignoring the entire purpose of SELinux with this argumentation? 18:14:49 abadger1999: the point is that the performance difference matters. and i think we can agree that selinux is a far far better security benefit than PIE, so we eat the cost there. i don't think PIE is a big enough benefit to counter that. 18:14:55 18:15:03 yes 18:15:12 notting, I can agree with that 18:15:14 also, PIE is not trivially disablable. 18:15:54 So really we want to know -- what is the actual performance degradation with our compilation flags +/- pie. 18:16:04 so do we want to vote on this proposal or does somebody feel that more data can persuade him/her to change his mind? 18:16:11 abadger1999, no 18:16:14 abadger1999, performance degradation of what? 18:16:17 abadger1999: Do you have a threshold in mind that would make you decide one way or the other? 18:16:25 we want to know that and we want to know exactly how beneficial the security is 18:16:28 abadger1999: I would like to know that yeah, but it will of course vary. 18:17:05 I don't think you can measure any real performance degradation on regular Fedora apps as most of the performance sensitive code is already PIC 18:17:06 jwb: sure -- my criteria wasn't exclusive. 18:17:15 t8m: I'm still kind of hoping for resolution of the prelink side effect 18:17:16 f. e. the crypto libraries 18:17:21 jwb: the pie security benefit is merely a decreased chance of pre-compiled exploits working 18:18:17 mitr: I think they're mutually exclusive but I could be wrong. 18:18:26 notting: both precompiled and individually-generated; ("precompiled" == sending the same binary to many users is where prelink makes a difference) 18:18:37 abadger1999: they are right now, but it's just software :) 18:18:38 notting: When bressers explained it to me, it sounded like more than that. 18:18:39 notting: and I'm understanding correctly, right, that the big performance gap is at startup, not as the program runs? 18:19:14 mitr: ok, 'canned' exploits, i guess? 18:19:16 pjones: For PIE it is an ongoing cost 18:19:20 pjones: Some of the numbers halfie got were that "PIE" builds are actually faster 18:19:24 notting: ie: it would also take longer for an attacker to generate an exploit that relied on finding out addresses. 18:19:34 mitr: right, and we still don't really know why. 18:19:38 pjones: yes 18:19:41 okay, I think I've got this mostly swapped in again now. 18:19:43 * nirik nods. 18:19:53 mitr, pjones, it is probably within the error of the measurement 18:20:03 mitr, I would ignore those numbers and assume that PIE performance <= non-PIE. 18:20:40 t8m: does seem like it's merely hysteresis or something, yeah. 18:20:51 halfie: Well, it does kind of make the other numbers questionable... However at this point I'm really not looking for a specific % theshold in the decision 18:21:30 the shorthand of performance would be, "all code performs as PIC code, not just linked in library code" 18:21:47 don't think there's a performance cost other than that? 18:22:14 mitr, if you run the benchmarks, you will find my numbers accurate enough. Still I would like to go ahead with the assumption that PIE performance <= no-PIE performance. 18:22:41 notting: Depends on the particulars of the proposal (whether -z now is added as well) 18:23:18 Let's summarize: 1. PIE is about 3-4% slower than non-pie code in binary 2. this change does not affect code in shared libraries 3. PIE disables prelink slowing startup of libreoffice (for smaller apps the startup time difference is negligible) 4. PIE security benefits are not so important 5. we would still allow opt out of PIE if we change the default 18:23:29 "I'd reiterate that the advantages of address space randomization on x86-64 are grossly exaggerated" <=== I don't agree with this. On 32-bit Fedora, stack address has 11 bits of entropy which is trivial to brute-force. You can't use such plain brute-force attacks against 64-bit ASLR. 18:23:47 t8m, 1. only for some applications (not generally!) 18:24:08 1 is only for 1 application in an old paper no? 18:24:28 halfie, that's because many applications have the computationally demanding code in shared libaries 18:24:51 nirik: It was on the SPECcpu benchmark set IIRC, which is a reasonably large pile of code (IIRC including a version of gcc :)) 18:24:55 t8m: right, shared libs are already pic, so they've already got this hit 18:25:39 mitr, except it is a special code in the sense that it is in binaries and not in libraries as most of the code is :) 18:25:42 and not our compiler flags. :) 18:26:10 (http://www.spec.org/cpu2006/Docs/ "Benachmark Documentation") 18:26:26 just from my workstation: du -s /usr/bin/ 18:26:26 410028 /usr/bin/ 18:26:37 du -s /usr/lib64 18:26:42 1803100 /usr/lib64 18:26:48 t8m, right, and this difference is only visible for 100% CPU bound apps 18:26:50 I disagree with point 4. "PIE security benefits are not so important" 18:26:54 can you explain why is this? 18:26:57 On 32-bit Fedora, stack address has 11 bits of entropy which is trivial to brute-force. You can't use such plain brute-force attacks against 64-bit ASLR 18:27:04 No-one has been able to mount *pure* brute-force attacks against 64-bit ASLR. (assuming there are no address leakage problems). 18:27:14 t8m: so your point is it'll effect roughly 1/5 of the code we execute? 18:27:16 halfie, there are various techniques how to overcome the randomization 18:27:23 t8m: includes a bunch of junk in subdirs. 18:27:29 pjones, yep 18:27:49 t8m, citation required. can you show me a real working exploit which bypasses / overcomes 64-bit ASLR with a remote attack? 18:27:51 t8m: /usr/lib64/*.so.* is ~680MB 18:29:09 halfie, it depends on the type of bug, type of the server - for example on forking servers it is very much possible because after the fork the address space is always the same so you can try multiple times 18:30:32 t8m, and yes, ASLR can't solve the problem itself. we need something like grsecurity's feature to detect such crashes caused by brute-force attempts. 18:30:56 ^^ it is almost trivial to implement such crash and alert systems. 18:31:30 no 18:31:39 I think we can vote just now on the proposal given that we can always vote again if this does not pass and we get more compelling reasons. 18:32:00 t8m: Your argument sounds like "it's not 100% so let's not bother trying". PIE already covers a bunch of forking servers. It would help ensure attacking Linux is more work than it's worth. 18:32:00 "We need to stop using C" BTW note that most non-C languages are not impacted (at least Python, Perl and Ruby already have the core of the language in a shared library) 18:32:42 mitr, good point. 18:32:50 mitr: Very little is impacted, I'd bet 80% of the distribution code is in shared libraries. 18:32:53 bress, note that I'm for changing the default, playing devil's advocate 18:32:54 Maybe even more. 18:33:14 t8m: Understood. I missed some scrollback, I thought your position was unlike you ;) 18:33:40 On that note, all important servers are already covered. 18:33:51 This change is really to catch the desktopy stuff. 18:33:54 bress: The numbers we have are a roughly equal split between *bin and *lib (but obviously unweighted by frequency of usage) 18:33:58 bress: Yes, we know. 18:34:00 t8m, http://www.vnsecurity.net/2012/02/exploiting-sudo-format-string-vunerability/ .. these guys had very hard time with local exploitation. Imagine if they were faced against 64-bit ASLR remotely. 18:34:44 mitr: Fair enough then. I did make that up :) 18:35:02 would it be possible to run the specYYYY benchmark with fedora default flags then again with PIE and get some kind of approx performance change? 18:35:16 mitr, given that shared libraries are expected to be shared, they are probably used more than the binaries :D 18:35:39 nirik: I don't think anyone has spec access. It's quite expensive IIRC. 18:35:45 ah, bummer. 18:35:54 nirik, and if that gives 2.9% or 4.8% instead of 3.6% does it change anything? 18:35:59 * nirik didn't know that was the case. do we have some other useful thing. 18:36:18 nirik: halfie did some benchmarks. 18:36:22 t8m: it might. if it was 1.0 or 10% would it? 18:36:26 They were unexciting. 18:36:34 bress: using different flags than what we use, yes. 18:36:37 nirik, lot of my benchmarks overlap with SPEC. 18:36:42 t8m: if you were making a point that it's somewhat irrelevant... I tend to agree... real world code that we're running in Fedora is more important. 18:37:04 nirik, if you are interested you can checkout https://github.com/kholia/unSPEC 18:37:11 nirik: Besides, somethign like spec is heavily CPU intensive, if something needs CPU performance, turn off PIE (things like video encoders could benefit here). 18:37:18 the spec2006 thing is from 2012 with not our flags, etc. 18:37:21 abadger1999, first I don't expect it to be 1 or 10% and second yes, real world code is more important 18:37:39 bress, video encoders are already PIC in shared libraries 18:37:48 In which case PIE is free. 18:38:17 t8m: I suppose, I can't think of anything that doesn't use encoders in libraries. 18:38:53 t8m, exactly, and same applies to audio audio encoding apps. there is no difference in peformance. I expected PIE builds to take a heavy beating but it didn't happen (do to all code being in the libs) 18:39:45 So out of curiosity, is anyone actually against this proposal? 18:40:00 It would also be a big PR event for Fedora. 18:40:17 bress: I don't like the anti-prelink side effect (though I'm not sure whether that makes me a -0.5 or +0.5 ... ) 18:40:22 there is one class of applications (chess engines) which are "real-world" apps and where there is a slowdown of <3.0%. I am very OK with putting such apps in a whitelist. 18:40:22 I wouldn't say I am against it so much as reluctant to approve it over jakubs objections. I think he's smart and usually knows what he's talking about. 18:40:44 (real world encoders don't use the codecs in fedora anyway. ahem) 18:40:45 nirik: I agree with you 18:40:58 mitr: Yeah, the loss of prelink could be a downside, but I also expect the things that benefit from prelink, we want to protect with PIE. 18:41:01 would it be productive for you to talk with him directly and see if you can convince each other? or is that likely to be a bad idea? :) 18:41:04 bress: with the current information and since the proposal allows maintainers to opt out, I'd be for the proposal. 18:41:26 #proposal Make the _hardened_build default to 1 and allow it to be switched off for performance sensitive applications. 18:41:34 Is jakub against PIE, or for prelink? I've not talked with him. 18:41:43 bress, against PIE 18:41:44 * notting is -1. would prefer consensus with tools maintainers. 18:41:47 bress: For desktop applications that users have to manually start, PIE + prelink (== addresses change every 14 days) is perfectly appropriate I think 18:42:00 bress: https://fedorahosted.org/fesco/ticket/1113#comment:10 18:42:01 -1 18:42:04 I'm still not convinced that the benefits outweight the costs here. I'm -1 18:42:07 +1 18:42:09 +1 18:42:17 +1 18:42:24 halfie, +1 18:42:46 kseifried: heh, sorry -- only fesco members get a counted vote for an actual proposal. 18:42:48 wait, fesco counts votes from everyone now? 18:42:49 * nirik notes this is a fesco vote. ;) 18:42:50 kseifried, halfie, unfortunately your votes do not count :) 18:42:53 notting, no 18:42:57 -1 18:43:02 Given that we already have an easy opt-in, I think forcing it is unnecessarily polarizing. 18:43:22 so thats -4 +3 ? 18:43:30 good decision, for now 18:43:36 sgallagh, : one note: security options that are opt in don't get traction/acceptance/work. if people cared about security we wouldn't need PIE to begin with :P 18:43:47 kseifried: Believe me, I understand that. 18:43:47 kseifried, yep 18:43:48 * mitr will go with "0", still not sure how to weigh it 18:44:04 That said, I opted in to PIE on SSSD, so it *does* sometimes gain traction :) 18:44:08 kseifried: halfie Has already done quite a lot of work to ensure the important packages are PIE 18:44:12 also has anyone reported problems with the apps that currently have PIE enabled? 18:44:26 * pjones is also -1, preferring that the people making the tools didn't recommend against using them this way 18:44:26 #info proposal is rejected (+3 -4 0:1) 18:44:31 #undo 18:44:31 Removing item from minutes: 18:44:34 * nirik is also 0 18:44:43 #info proposal is rejected (+3 -5 0:2) 18:44:53 How do we have 10 votes? 18:44:53 #undo 18:44:53 Removing item from minutes: 18:45:03 yeah, I might have miscounted. 18:45:03 #info proposal is rejected (+2 -5 0:2) 18:45:36 kseifried: I think that postgresql (tom lane) had some problems. but I could be misremembering the flags he was trying to add. 18:45:41 i do encourage halfie, kseifried, bress at all to work with the compiler & toolchain folks on a new proposal 18:45:48 notting, +1 18:45:53 notting, thanks, will do 18:45:53 notting++ 18:45:56 This comment from Jakub makes it sound like we need to fix some PIE bugs. I trust jakub, we need to understand what's happening here. 18:46:05 yeah, agreed. 18:46:15 if those issues can be worked out it would be lovely. 18:46:17 bress, I am afraid it won't be possible due to ABI 18:46:25 um, 'et. al.', not 'at all'. not sure how i managed to self-damnautocorrect 18:46:38 let's go on 18:46:51 #topic #1117 Generalize policy about privilege escalation and Administrator user accounts 18:46:52 t8m: Suggesting something is impossible is unwise ;) 18:47:01 Thanks for taking up the issue guys, I appreciate it. 18:47:05 bress, impossible without breaking ABI 18:47:45 .fesco 1117 18:47:46 t8m: #1117 (Generalize policy about privilege escalation and Administrator user accounts) – FESCo - https://fedorahosted.org/fesco/ticket/1117 18:48:10 unless i missed something, no proposal yet. table until there is one? 18:48:23 please 18:48:27 yep -- did someone offer to write aproposal? 18:48:29 notting, yep, I should have probably cleared the meeting flag 18:49:05 #info no proposal yet 18:49:24 We could also close the ticket if no one is willing to take charge of this. 18:50:16 perhaps but I'd leave it open for a while 18:51:05 We don't have anything else on agenda. 18:51:16 #topic Next week's chair 18:51:19 okay -- it's just -- we're all here. If none of us is volunteering, it's not magically going to get written :-) 18:51:46 #undo 18:51:46 Removing item from minutes: 18:52:13 #info Anybody is encouraged to create a concrete proposal for generalizing the policy 18:52:14 abadger1999: Perhaps that should be flagged as a discussion topic at Flock? 18:52:19 We don't need it sooner than that. 18:52:33 I'm interested in helping with this policy FWIW. 18:52:46 (feel free to tell me to shut up and go away if I'm out of line) 18:52:51 yep, it does not have to be a FESCo member 18:52:55 bress: You are quite welcome to :-) 18:53:11 bress: That would be fantastic. 18:53:14 please do 18:54:13 Anytime you guys have security related thigns like this, feel free to send them my way. 18:54:13 #topic Next week's chair 18:54:48 Anybody volunteers? 18:55:16 I can do that 18:56:12 I may or may not be around next week. It's going to be a busy week at $DAYJOB 18:56:38 * mmaslano too 18:57:55 I'm sorry I'm having internet connection outage just now... 18:58:37 #action mitr to chair FESCo next week 18:59:02 t8m: would you like someone else to finish out the meeting? or you back on line ok now? 19:00:14 #topic Open Floor 19:00:14 Still some packets get through :) 19:00:52 I wanted to note https://bugzilla.redhat.com/show_bug.cgi?id=967385 to fesco folks. It wasn't referred to us (yet), but if anyone has strong opinions or would like it on the agenda down the road... 19:01:23 do you have a tldr summary? 19:02:25 livecd-tools used to remove root password (no password, anyone can su). this was fixed and the default is now 'locked'. 19:02:44 however, our live media removes the password/lock in kickstart, so we have the same behavior as in the past. 19:02:53 Right, as I recall that was the outcome of a CVE 19:02:54 it's debatable if this is a security problem or not. 19:03:17 If there is a "liveuser" with no password and full sudo root access, there's no security difference, is there? 19:03:19 it does seem to break kdesu tho to have a locked root account. 19:03:51 mitr: another step/single point people would need to go thru... so it's a slight difference, but dunno if it's worth it. 19:03:56 mitr: sudo instead of su breaks kde stuff 19:04:08 one note on this: when you do the "install to disk" option it won;t let you finish until you deal with the root password in the GUI Installer, so that's good 19:04:11 mitr: Well, we could theoretically randomly-generate liveuser with an arbitrary suffix, which would prevent automated remote attacks, I guess 19:04:22 sgallagh: Might just as well ask for a password then 19:04:30 and another note: locking the root account is a pretty standard configuration and should be supported without breaking things 19:04:31 kseifried: that's an important point 19:04:32 yeah, this only affects the live media running live. not installs. 19:04:45 mitr, : yeah, I checked cause I was worried it might do something bad =) 19:04:45 mitr: Well, the user doesn't actually need to know the username on a live media, they're automatically logged in 19:05:17 sgallagh: That's true for the user but not for the attacker we are presumably worried about? 19:05:28 What attack vector is being considered here? 19:05:51 * nirik didn't mean for this to be a full topic, but ok. ;) 19:05:52 one note: any local access (e.g. crappy php app/any applictions/etc now means root access 19:05:57 which was probably not intended. 19:06:21 * mmaslano was told to chair rest of the meeting. t8m's internet is down 19:06:21 (I suppose i'm just confused; for a live image that has no remote access and manual setup required for installation, I can't see that much reason to be concerned) 19:06:21 get shell access on live media somehow, su -> root. vs get shell access on live media somehow -> su liveuser -> sudo -i root 19:06:28 giving the user account access to root is one thing for a live system but giving any application/running code root access was probably not the intention 19:06:46 right. 19:07:43 mitr: Well, I was forgetting that sshd is disabled by default on the live media. 19:07:46 so, in any case doing any change for f19 seems crazy to me. ;) 19:07:51 but something to look at for 20+ 19:08:04 I think I agree with comment #19 19:08:08 So I was on the wrong track 19:08:09 you guys should document this and warn people, as ANY running code can access root. web browser plugins, you name it. 19:08:37 kseifried: they could also using sudo too no? 19:08:42 I'm sorry I lost connection for a while 19:08:54 kseifried: I can't see an obvious alternative for the live OS. "To start trying out Fedora, pick a secure password you probably won't need again?" 19:09:07 t8m: https://bugzilla.redhat.com/show_bug.cgi?id=967385 19:09:16 t8m: http://www.fpaste.org/15302/54548136/ 19:09:17 mitr: sudo: access all, no passwd for the liveuser account 19:09:30 sgallagh, mmaslano, thanks 19:09:40 kseifried: I'm afraid I don't understand. 19:09:42 mitr: or else make sure you document that any running code has root access cause I bet people weren't expecting that. 19:09:47 mitr: the root password is blank. 19:09:52 so any code can simply "su -" 19:09:55 kseifried: So is liveusers' password. 19:10:32 so currently anycode could also 'sudo -i' 19:12:15 right. but what if the user runs a web application that isn't srun as liveuser? that web app now == root. 19:12:34 That was true before as well (httpd -> liveuser -> root) 19:12:41 like I said, if this is intentional you need to document this 19:12:56 Documenting this is worth looking into, yes. 19:12:57 kseifried: right, or we need to figure out a better way to setup the live media 19:13:14 yup. 19:13:23 or both 19:13:33 kseifried: Sandboxed X session? 19:13:37 t8m: Something worth considering - restrict su/sudo/login/etc. usage from system service accounts 19:13:52 or lock it down using pam to consoles/etc 19:13:57 interactive, etc. 19:14:12 but there are definitely better ways to handle it rather than simply having a blank root password 19:14:16 kseifried, that would be too limiting 19:14:29 t8m, : what specifically would be to limiting? 19:14:29 but mitr's proposal might be doable 19:15:03 * nirik thinks this is a good discussion to have, but perhaps on list? or outside meeting? I don't think we need to solve this here now. 19:15:03 kseifried, you couldn't for example run ssh user@somewhere su - 19:15:14 nirik, +1 19:15:27 we definitely won't solve this here 19:15:32 t8m, : sorry, not tracking you 19:15:36 nirik: right 19:16:48 Anything else for Open Floor? 19:17:30 I'd like to go on record with a Congratulations! to everyone that helped in getting Fedora 19 Beta out the door without slippage. 19:17:35 Fantastic achievement folks1 19:17:49 sgallagh, +1 19:18:21 yes indeed. :) 19:19:52 huzzah! :-) 19:20:28 I will close the meeting in a minute if we do not have anything else to discuss. 19:21:26 #endmeeting