15:05:57 #startmeeting Server Working Group Weekly Meeting (2014-04-15) 15:05:57 Meeting started Tue Apr 15 15:05:57 2014 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:05:57 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:06:10 #chair sgallagh mizmo nirik davidstrauss Evolution adamw simo tuanta mitr 15:06:10 Current chairs: Evolution adamw davidstrauss mitr mizmo nirik sgallagh simo tuanta 15:06:17 .fas tuanta 15:06:17 tuanta: tuanta 'Truong Anh Tuan' 15:06:19 #topic roll call 15:06:27 .fas adamwill 15:06:28 adamw: adamwilltest 'Adam Williamson (test)' - adamwill 'Adam Williamson' 15:06:33 .fas duffy 15:06:34 mizmo: mikeduffy 'xiaofeng.wu' - duffy 'Máirín Duffy' - cduffy 'Charles Duffy' - dduffy 'David Duffy' - pduffy 'Pat Duffy' - kdpkd 'kev duffy' - dufflebag 'Martin Duffy' - thequbit 'Tim Duffy' 15:06:38 .hellomynameis kevin 15:06:39 nirik: kevin 'Kevin Fenzi' 15:06:41 what the 15:06:43 heh, that test account's still there, apparently 15:06:56 mizmo: looks like it's a fuzzy search :) 15:06:57 Hello 15:07:02 mizmo: .fas is a wildcard search. ;) Thats why we added the hellomynameis. ;) 15:07:14 hellomynameis is too long! lol i was trying to be slick 15:07:41 i was just copying tuanta 15:07:50 i'm not scheduled to wake up for another two hours or so 15:08:08 anyone else? 15:08:37 and can the Duffy table *please* either order some drinks or free up the space for other patrons ;) 15:08:53 * nirik wonders if we should just make it 'hello' or perhaps 'hi' :) 15:09:22 #info present: adamw, mizmo, tuanta, mitr, nirik, simo 15:09:56 #info Agenda Item: Drafting a test plan 15:10:12 * mizmo pours some pints of guinness for the duffy clan 15:10:15 #info Agenda Item: Should we have a default caching DNS server? 15:10:26 oh, yeah, of course it'd be a clan 15:10:34 and there's me reading scottish-set fiction right now 15:11:22 does anyone have any other agenda items? 15:11:27 mizmo: what you haven't read how guinness puts harmful stuff in their beers? :-P 15:11:56 adamw: should we talk about this journald remote protocol thing ? 15:12:05 is this something we should care for in F21 ? 15:12:08 simo, don't tell me i dont want to know! 15:12:12 +1 on installing Unbound by default. 15:12:23 simo: what's the reference for this? 15:12:24 davidstrauss: it is more nuanced thn that 15:12:29 unbound is the caching dns server? 15:12:34 also, we're not on that agenda item yet 15:12:36 davidstrauss: one of our deliverables is the domain controller role 15:12:38 * adamw waves order papers, frowns 15:12:41 which comes with bind 15:13:03 Oh, I mistook the #info line with moving to that item. 15:13:04 simo: is the journald remote protocol thing a devel@ reference? i'm like 500 mails behind. 15:13:09 adamw: the reference is a thread on fedora-devel: F21 Self Contained Change: Remote Journal Logging 15:13:26 ah, okay. let's add it, if we have nothing to say, no problems. 15:13:31 adamw: Also, I am present. 15:13:34 ok 15:13:39 #info Agenda item: Remote journal logging 15:13:44 #info also present: davidstrauss 15:13:50 OK then 15:14:05 #topic Drafting a test plan 15:14:32 So with my QA hat on, I figured that now the products are reasonably defined, it'd be a good time to try and come up with test plans for them 15:15:02 i'm not aiming to do the job in the meeting :), just a heads-up and discussion starter thing 15:15:31 so, likely we need 3 plans? a base server role framework one for all roles and then role specific addons for each role? 15:15:40 the idea being initially to define what we ought to test (in an ideal world) to validate a Server product release 15:16:18 nirik: however exactly you want to frame that semantically, yeah, we at least need to test the 'framework' bits, and also the functionality of any supported roles 15:16:37 (a corollary of that is probably that supported role propositions better come with testing resources...) 15:16:46 er, proposals. 15:17:09 we may be able to grab some upstream related tests for some things... ie, does freeipa have a set of tests or docs on such? I'm sure there is some postgresql test docs. 15:17:13 #info adamw has volunteered himself to draft a broad initial test plan for the Server product, an attempt to define the required scope of test coverage to validate a Server release 15:17:25 good thinking 15:17:52 #info nirik suggests checking for already-existing upstream test plans (e.g. freeipa, postgresql) 15:18:02 adamw: what is the scope of Fedora QA ? 15:18:08 I am always a bit fuzzy on hat 15:18:18 simo: excellent question ;) 15:18:19 as much as we can do, but no more? ;) 15:18:20 I would assume that you do not test the product works flawlessly 15:18:30 but that it can be properly installed and the roles also installed ? 15:18:30 simo: sure we do. all bugs are products of your overactive imagination. 15:19:03 actually, nirik's answer is fairly close to the truth of it 15:19:25 we still have very little upstream testing of freeipa :( 15:19:31 but we are working on it 15:19:43 the approach we're taking here is to define (quite broadly, we don't need a lot of detail) these 'ideal' test plans, and then do the fun bit of figuring out how much we can practically undertake to achieve 15:20:00 adamw: ok can w also define priority ? 15:20:01 we do have some freeipa test cases already: https://fedoraproject.org/wiki/Category:FreeIPA_Test_Cases 15:20:03 I'd say it would be worth tossing out a bunch of things, then prioritizing them... 15:20:13 for example I think it is a proprity that the role framework works 15:20:14 simo: i'd probably make that a second-stage thing, but it's something to bear in mind 15:20:19 ie you can get to install the domain controlle role 15:20:23 yeah, anything that's 'fundamental' gets fairly high priority 15:20:30 however if then the role has bugs, that is not a big concenrs 15:20:34 so initial deployment and later configuration of roles 15:20:34 *concern 15:20:42 as long as the install finishes and it "starts" 15:20:46 i would like to have at least *some* functional testing of our primary role 15:20:54 analogous to the way we currently do some basic tests of the desktop 15:20:57 adamw: sure 15:21:16 it's kind of a laugh test - you don't really want to put out a product that just fails extremely obviously out of the box, even if it's 'just a regular bug' that 'can be fixed with an update' 15:21:19 adamw: do you have any testing infrastructure to deal with multiuple machines interacting ? 15:21:48 adamw: because a simple test that will go a long way is this 15:21:59 simo: we don't have something permanent / shareable / formal really, afaik (red hat does, fedora doesn't). we tend to knock up ad hoc VM farms if we need to test something like that. 15:22:00 1. install fedora server 15:22:07 but it can be done. 15:22:08 2. install fedora workstation on another machine 15:22:14 or we can just formalize the VM farm process. 15:22:17 install domain role on server 15:22:22 join workstation to server 15:22:25 create user on server 15:22:31 login as that user on workstation 15:22:37 there in 6 simple steps :) 15:22:40 simo: beaker exists and runs $somewhere in fedora-land; I don't know how much it's been used 15:22:42 right 15:22:48 and as a bonus you are testind TWO products :) 15:23:23 mitr: i thought we only ever had a test instance of beaker 15:23:30 adamw: I have no idea 15:23:31 but imbw :P 15:23:38 i'll check with tflink, he'd probably know 15:23:56 yeah, there's a test type instance. it's not doing anything that I know of. 15:24:12 #info simo suggests simple deployment of domain controller role on server machine and domain join / user creation on a Workstation machine as a basic smoke test for domain controller role 15:25:32 so, yeah, i'll probably just work on the basis of drafting something up and sending it to both server@ and test@ for review, but if anyone wants to help or contribute thoughts, please do, the ones so far have been helpful for sure 15:25:55 adamw: as for database role, we can do something similar, just on a single machine 15:25:57 simo: one thing that would be really useful if you have the time (or a minion who does :>) would be to go through the freeipa test cases at https://fedoraproject.org/wiki/Category:FreeIPA_Test_Cases and check for bitrot 15:26:07 you'd have to identify an application to "test" the db though 15:26:08 +1 simo 15:26:10 some of them date back a few years 15:26:23 I would rather do real world application testing than synthetic upstream tests thouhg 15:26:26 that feels like something it shouldn't be hard to find stress tests for 15:26:32 * davidstrauss has to duck out in a few minutes to head to Pantheon and set up for the systemd hackfest. 15:26:35 true 15:26:35 those are good but already done by upstream/rh (I hope) 15:26:37 yeah, for db... install, confirm you can connect with test user, confirm you can update/commit some test data and query it. 15:26:54 simo: well, a *lot* of the issues we find in validation are interaction issues 15:26:56 a real scenario IMO adds more value to the whole ecosystem and may uncover bugs that escape synthetic testing 15:27:07 nirik, you meant we will do that from command line? 15:27:16 simo: not really "this code doesn't do what it was meant to do" bugs but "this specific version of this bit doesn't work with this specific version of this other bit at this time" stuff 15:27:28 so you can't rely on upstream/downstream testing so much for that 15:27:51 tuanta: sure, we could have a test script or something... 15:27:53 nirik, we can write a test script for that 15:27:57 yeah 15:28:00 yes, it makes sense 15:28:01 anyhow, OK if we jump to the next item so davidstrauss can get in anything he has to say before he leaves? 15:28:08 ack 15:28:23 adamw: maybe we can discuss again once you have a draft 15:28:23 yup 15:28:25 thanks for the input so far 15:28:29 more ideas may spring once we have something more defined to look at 15:28:35 i know how that is :) 15:28:57 #topic Should we have a default caching DNS server? 15:29:04 davidstrauss: anything you want to put in at the top? 15:29:30 Just that we've found this immensely valuable on our Fedora boxes at Pantheon. 15:29:42 * jsmith always adds one to his Fedora servers as well 15:29:46 We were overloading Rackspace's before implementing one, and picking was hard. 15:30:01 #info davidstrauss and jsmith have found caching DNS very valuable in real-world deployments 15:30:02 * nirik is a big fan of unbound... 15:30:04 We started with pdns-recursor, but it *still* lacks DNSSec validation in 2014. 15:30:23 * adamw was thinking there was something built into NM, but that's client-side caching, isn't it. 15:30:24 We're transitioning to Unbound, maybe 70% complete. 15:30:40 jsmith: what do you use? 15:31:04 adamw: that's the same thing AFAIK, support for running/managing a system-wide caching daemon 15:31:06 I haven't found a single benefit for PDNS-Recursor over Unbound, and I wouldn't wish using BIND as a bundled caching DNS server on anyone. 15:31:17 adamw: Used to use bind, now using unbound 15:31:26 we are currently using unbound on our servers as well 15:31:49 looks like a case of 'student body, unbound' 15:32:02 The biggest integration challenge is having it automatically configure upstream servers from DHCP or an initial list populated into resolv.conf 15:32:19 the dnssec-trigger stuff does this fine on clients... 15:32:27 We should not ship a Fedora server that goes right to root for all lookups 15:32:34 +a million 15:32:38 +1 15:32:44 +1 15:33:06 so what we need is something like a simpler NM plugin like dnssec-trigger... 15:33:07 There is a NetworkManager integration for Unbound, but it looked pretty unmaintained. 15:33:07 shall we take a vote just to put a stamp on it? 15:33:30 yes, I think it's worth, adamw 15:33:40 davidstrauss: Wouldn't the biggest challenge be coexisting with a DNS server role? (Obviously not today's concern, but we do need to have some plan) 15:34:04 ok sorry 15:34:09 I got distracted a few minutes 15:34:24 I wanted to bring this up because it is *not* straightforward for server 15:34:25 mitr: Not really, if it's just for local use. We have our DNS servers listen on everything but ::1 and 127.0.0.1 and Unbound listen on those loopbacks. 15:34:26 for example 15:34:33 how do we deal with is on a domain controller ? 15:34:46 the role allows for installing bind as the DNS server 15:34:53 so just to be clear, we are just talking about doing caching for the system running Server itself here 15:34:57 of course we can decide to run both unbound and bind on the same machine 15:34:57 not a 'caching DNS server role' 15:34:59 simo: unbound on localhost, dns server on other ips? 15:35:00 It is not possible, by the way, to have the resolv.conf system talk to a non-standard port. 15:35:04 davidstrauss: ignoring people shouting about one more daemon, that's clearly a clean solution 15:35:09 we just need a modification so that bind does not listen on 127.0.0.1 15:35:18 but sounds like a waste and potentially confusing 15:35:28 * nirik is also not a bind fan. ;) 15:35:37 Proposal: the Server product should have DNS caching functionality enabled by default 15:35:46 (patches to the wording welcome if it's silly) 15:35:49 davidstrauss: can't you add a port to the ip address of the nameserver option ? 15:35:55 simo: Nope. 15:35:58 interesting 15:36:02 AFAIK 15:36:03 adamw: +1 in general; re: timing, F22 together with the rest of the OS? 15:36:09 I'm +1 to the idea, but we need some NM integration. perhaps we could ask NM folks if they would be willing to make something for us? 15:36:15 simo: We looked into that. It would have been easier. 15:36:16 adamw: well I would like to get there 15:36:25 like I said, I thought NM actually *had* some DNS caching functionality. but imbw on that. 15:36:26 but dhcp passed resolvers is also a problem 15:36:34 should we have these things are blockers ? 15:37:04 adamw: it has integration witrh dnsmasq 15:37:08 adamw: it can use dnsmasq I think... not sure about unbound. 15:37:10 simo: I'd say that whatever works for all the complex setups Workstation needs to handle is very likely to work for Server well 15:37:15 ah, yeah, that's what I was thinking of 15:37:16 adamw: and there is serious work going on to do an unbound plugin 15:37:23 but I think they are targeting F22 ? 15:37:28 simo: yes 15:37:31 so, I wouldn't make this a blocker, but a nice to have. ;) 15:37:53 mitr: problem is for F21 we have only dnssec-trigger, and I do not think that is good enough for now 15:37:59 not sure, it *may* work 15:37:59 #info there is an effort underway to write a plugin for NetworkManager to integrate with unbound, which appears to be the WG's caching nameserver of choice 15:38:07 #undo 15:38:07 Removing item from minutes: INFO by adamw at 15:37:59 : there is an effort underway to write a plugin for NetworkManager to integrate with unbound, which appears to be the WG's caching nameserver of choice 15:38:18 #info there is an effort underway to write a plugin for NetworkManager to integrate with unbound, which appears to be the WG's caching nameserver of choice, but it is on track for F22 not F21 15:38:22 simo: I'd really prefer using the NM-led solution, not Server doing something special just to get in F21 15:38:33 mitr: indeed 15:38:38 seems like we have a consensus on that 15:38:53 so should we make this default conditional to people getting a plugin for NM first ? 15:38:57 mitr: Consider this a nudge to NM ;-) 15:39:01 adamw: we also have another problem 15:39:05 now that I think of it 15:39:14 server will often be used as a base for virtualization 15:39:31 simo: How does that affect this? 15:39:37 in that case dnsmasq is used by default by virt-manager/libvritd right ? 15:39:40 davidstrauss: I'm generally following pspacek and psimerda's work, they don't need more nudging :) 15:39:47 Proposal: We would like the Server product to have DNS caching functionality by default, but we believe this can be done at a project-wide level as of Fedora 22 and do not intend to NIH it for the Server product only Fedora 21 15:39:51 is that going to caue conflict again if unbound is present ? 15:40:03 simo: yeah, we have amusing interaction issues with libvirt's dnsmasq quite a bit. 15:40:17 simo: Can libvirt switch to Unbound as well? 15:40:17 i don't know specifically about this case, but it sure seems like something to watch out for. 15:40:19 simo: AFAICS it listens "only" on the virt interface 15:40:23 adamw: I'll take an action item to ask Dan Berrange 15:40:25 adamw: +1 15:40:51 This is not just an optimization to me. With Unbound, it's a major enhancement to security. 15:40:55 #action simo to work with virtualization and networking folks to make sure any caching DNS plans consider libvirt's use of dnsmasq 15:40:56 One more remark before I go. 15:40:59 Also, eventually, containers - we'll want to forward DNS from the containers to the host in most cases 15:41:21 Some recursors break Unbound DNSSec validation. Google's, for example, validate but don't pass on the right info. 15:41:26 adamw: +1 15:41:29 mitr: right 15:41:48 davidstrauss: I think they have fixed that now 15:41:55 the one I recall is that we had some fun with libvirt's dnsmasq getting nested - when we started installing Boxes by default, networking stopped working in VMs because of libvirt's dnsmasq running in the guest 15:41:59 Dyn's also breaks Unbound 15:42:00 davidstrauss: fedora-devel has been talking a lot about ways unbound can workaround broken recursors 15:42:13 well, unbound+dnsssec-trigger 15:42:14 davidstrauss: Dyn ? 15:42:35 any more votes on the proposal? is anyone opposed? 15:42:54 +1 with all the caveats 15:43:00 +1 from me too 15:43:46 * adamw +1s his own proposal 15:44:26 #agreed We would like the Server product to have DNS caching functionality by default, but we believe this can be done at a project-wide level as of Fedora 22 and do not intend to NIH it for the Server product only Fedora 21 15:45:01 do we need any further action items at this point? do we have someone on whatever the team is that's working on this NM/unbound approach? 15:45:30 adamw: I tend to keep myself up to date on this stuff 15:45:38 so I can try to be liason when needed 15:45:38 cool. 15:45:54 #info simo will try to act as liaison to the folks working on NM/unbound integration 15:46:42 okey dokey, a few minutes for the last topic... 15:46:56 #topic Remote journal logging 15:47:19 simo: did you have something specific here? (I've not caught up to the thread yet) 15:47:28 I have to say I brought this up because the current plans seem to make it very hard to have secure remote logging 15:47:32 I'm also behind I'm afraid 15:47:38 they are planning to use HTTPS as the transport :( 15:47:53 #info see devel@ thread "F21 Self Contained Change: Remote Journal Logging", https://lists.fedoraproject.org/pipermail/devel/2014-April/197980.html 15:47:55 * nirik just asked on list what ssl lib(s) they plan to use 15:48:05 if we ever want to build anything on this it seem to me it will require a ton of work on setup tools 15:48:38 I really wish we could influence them to not use such a broken protocol as HTTP 15:48:39 simo: Why is HTTPS a problem, in principle? You need to provision _some_ keys on both parties to get MITM protection 15:48:56 mitr: kerberos key are like 100 times easier to distribute an manage 15:49:09 and gssapi is not as broken as TLS when it comes to validation 15:49:21 most HTTPS client libraries screw up validation big time 15:49:29 #info simo has concerns about the security of the proposed design (suggests kerberos as a better framework than https) 15:49:33 And more generally, is there an reason to revisit https://fedoraproject.org/wiki/Server/Technical_Specification#Logging ? 15:49:39 let alone admins that seem to regularly fail to grasp how CAs work, let alone set up a personal PKI infrstructure 15:50:00 and I am saying this as one of the proponent of th domain controiller role that gives us a CA for free 15:50:20 adamw: not much about the scurty 15:50:25 but about usability 15:50:43 simo: Sorry, "HTTPS clients are generally screwed up" is too much of a stretch. I'm not saying it's easy or always used correctly but let's not be alarmist... 15:50:54 mitr: I think eventually forwarding structured logs will be a good idea 15:51:13 mitr: think about setting that thing up 15:51:18 what CA do you use ? 15:51:29 or do you excahnge fingerprints betwen client and server ? 15:51:41 it's hard 15:51:43 the domain-controller-created one would be natural (but not applicatble always) 15:51:58 mitr: I think what I want to say is that we should stick to rsyslog for the forseable future 15:52:04 simo: rsyslog can exract and forward the full journal data set (up to some message limit I suppose, not multi-gigabyte core files ) 15:52:25 but we should keep an eye on remote journald and provide feedback on the feasibility of configuring the mechanism they are proposing 15:52:42 mitr: the dc only proveds a CA 15:52:48 you cannot trust just any certificate though 15:52:49 simo: ok, I fully agree 15:52:55 LOGs are *very* sensistive 15:53:04 you need means to say *this* and only *this* server 15:53:21 and from the server's perspective you amy also wnat to say *these* and only *these* clients 15:53:29 to avoid being flooded by random clients 15:53:41 i gotta run in 5 15:53:48 and I see nothing about this stuff in the change proposal 15:53:55 I guess my proposal is: 15:54:40 proposal: due to configurability limits and unclear access control of current journald remote protocol the Server WG will stay on rsyslog for the foreseable future 15:55:09 that seems fine based on what I know now 15:55:11 *for remote loggin of course* 15:55:28 can we just make it 'will stay with rsyslog at least for Fedora 21 and then reconsider'? to make it less vague 15:55:29 I mean this change was just proposed, we don't need to jump on it's bandwagon for f21. ;) 15:55:42 adamw: until those concerns are resolved ? 15:55:55 simo: +1 to the conclusion, can't put my voice behind 'unclear access control" at this point because I really _am_ behind on fedora-devel 15:55:56 nirik: well they are proposing it for F21 15:55:59 sure I guess? 15:56:08 nirik: not sure what they mean by this proposal though 15:56:17 simo: as i read the proposal it's not to have it enabled by default or 'mandatory' in some way 15:56:22 "for the foreseeable future" is fine, I think - journal has AFAIK an explicit non-goal of being an enterprise logger with local filtering and the like 15:56:22 it's really a 'feature addition' 15:56:29 mitr: fair enough 15:56:32 +1 then 15:56:43 simo: yes, but I mean we don't need to change our plans based on some new experemental thing thats not even written yet. 15:56:55 but I share your concerns as well. 15:57:35 the "Scope" section of the Change proposal indicates quite well why it needs to be a change, I think - just for the fedora accounting 15:57:49 stuff like checking for conflicts with the default port, doing the user creation bureaucracy, etc etc. 15:58:29 * adamw counts votes 15:59:31 i can only count 4 +1s, really, and a couple of those are pretty vague - but it doesn't seem like we really need a formal agreement here anyway 16:00:00 #info it looks like we all have some reservations regarding the new proposed systemd remote logging functionality, and expect to stick with rsyslog for the foreseeable future for Server 16:00:04 is that good enough for now? 16:00:26 simo: ^^^ 16:00:32 and does anyone have anything else before end of meeting? 16:01:35 did I miss a zombie invasion or something?! 16:01:42 adamw: +1 16:01:52 adamw: braaaaaainz 16:01:58 I thi nk we are done for today 16:02:01 oh, you want brains? then I'm safe. whew. 16:02:05 we are lucky we pulled it off at all :) 16:02:27 adamw: lol, I need one, I want to know what it looks like, never tried myself ;-) 16:02:35 i hear they tickle 16:02:47 thanks for coming, folks! 16:02:49 #endmeeting