15:59:24 #startmeeting Server Working Group Weekly Meeting (2014-01-21) 15:59:24 Meeting started Tue Jan 21 15:59:24 2014 UTC. The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:24 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:59:27 #chair sgallagh mizmo nirik davidstrauss Evolution adamw simo tuanta mitr 15:59:27 Current chairs: Evolution adamw davidstrauss mitr mizmo nirik sgallagh simo tuanta 15:59:31 #topic roll call 16:00:41 ahoyhoy 16:00:56 .hellomynameis sgallagh 16:01:00 sgallagh: sgallagh 'Stephen Gallagher' 16:01:21 .hellomynameis adamwill 16:01:23 adamw: adamwill 'Adam Williamson' 16:02:10 Hello 16:03:08 * sgallagh hears mostly crickets 16:03:20 morning 16:03:58 sgallagh: bzzzzzzzzzzzzzzzzzzz...oh sorry, that's me. 16:04:16 .hellomynameis davidstrauss 16:04:17 davidstrauss: davidstrauss 'David Strauss' 16:06:30 We have minimum quorum at the moment. Perhaps we should wait a few extra minutes? 16:06:41 I'm in no rush this morning. 16:08:25 Ok, why don't we get started and see who shows up 16:08:30 #topic Post-PRD Steps 16:08:51 So, we need to start organizing how we're going to tackle the scoping and execution of our ambitious PRD efforts :) 16:09:18 Or, put another way: start assigning tasks to the people who haven't shown up today. 16:09:32 ha 16:10:23 Joking aside, in my opinion, the most immediate need is determining what we plan to have in a first release and how long we'll need to get there. 16:10:40 yep. 16:10:41 FESCo is going to need to make a decision on scheduling soon. 16:11:35 Related to that question is whether we plan for our first release to be something we intend to maintain a different lifecycle for than the traditional Fedora. 16:12:20 We have punted on the stable app runtimes, so perhaps we don't really need it 16:12:22 so ideally to me at least: we should try and use as much existing infrastructure as possible, and should strive to use upstream projects/code as much as possible and minimise the glue we custom make. 16:12:30 I'll state for the record that I absolutely don't want anything we ship in 2014 to have an expectation of long support 16:12:36 No way we'll be prepared for that yet. 16:12:41 :-) 16:13:09 I don't think we can even really look at that until we have some plan of implementation... 16:13:16 mitr: Sorry, remind me what "stable app runtimes" are again? (I can guess from the terms, but I'm not sure) 16:13:50 nirik: And yes, absolutely we should be building minimal glue. 16:14:14 I guess a good first question is "How ambitious do we want to be?" for this first release? 16:14:22 sgallagh: ability to run the same binary of an app for years even through Fedora upgrades 16:14:23 so, perhaps a wiki page would be of help, and today we brainstorm all the things we need to decide implementation on? then pick things and discuss. 16:14:53 To me, there's a spectrum from "Just provide a CD/DVD image with Server packages" all the way to "Complete everything in the PRD before we release." 16:15:45 first of all, pick a point you think you can achieve, and then divide it by three. ;)\ 16:16:09 several possible options: we could start at the 'easy' end and build... 16:16:10 adamw: That's actually not a bad way of approaching it 16:16:16 I think it would be neat to build/choose a way to ship apps and ship one or two. 16:16:47 Admins and the broader community may find lots of value in that. 16:17:06 Yeah, from my perspective I think our first release (ideally) should include (at least) one featured Role 16:17:17 I wouldn't be able to discuss scheduling without having at least a basic prototype (to see what already exists and what we need to write from scratch, and which parts that we need to write for scratch needs to be in the first release). 16:17:26 mitr: +1 16:17:41 so, we need to figure out how to implement a role and deliver it. 16:18:05 nirik: Perhaps we can take FreeIPA as the reference Role and plan for that to be our 1.0 deliverable? 16:18:12 Also, I'd suggest that we start with a small number of roles, and providing a stable/documented interface how to _use_ the roles, but for now keep the mechanism for _implementing_ them explicitly private and unstable 16:18:33 if there's people willing to work on it, sure... 16:18:58 mitr: seems reasonable, no promises. ;) 16:19:15 nirik: I suspect that there will be. And a lot of the work is already done there (since deployment scripts are already available) 16:19:36 I'd prefer to have simo weigh in on this, of course (being that he's upstream) 16:19:48 so, the big question is: how do you implement and deliver a role. 16:20:38 nirik: That is the biggest question. 16:20:53 1) A cohesive deployment strategy (the role must be possible to install with a single command and configure with an API) 16:21:04 lets talk details. 16:21:07 As noted in the Role requirements 16:21:08 Sure 16:21:39 So one possible answer is that a role must be packaged as a Software Collection with all of its dependencies (as a chain, not a single bundle) 16:21:55 sgallagh: IPA is a rather complex starting point, with all the components and orchestration (e.g. suppose we needed some packaging changes to every component). It would be a great thing to deliver in release 1, perhaps the prototype should be something much simpler (on the order of "FTP server", except that FTP server as such is a horrible idea) 16:21:56 I'm bullish on using Docker. We'd have two working groups invested in it, in addition to resources pushing from the RHEL side. 16:21:57 straw man: roles use rpms from the fedora collection, roles use comps groups, roles use a metarpm with the role information/glue in it. 16:22:14 Another choice is that it must be a standard set of packages (possibly with strictly-versioned Requires:) 16:23:03 so, we have a number of options, perhaps we could get interested parties to make mockups? 16:23:18 We might need the bits for two versions installed at the same time (for a major upgrade of a DB, we may need to install the new version to get a conversion tool, but we need a reasonable way to fall back if the conversion fails or is aborted for whatever reason). 16:24:15 To throw another wrench into the works: has anyone had a chance to look at the rpm-ostree stuff Colin Walters announced recently? 16:24:25 * nirik has not yet had time. 16:24:34 mitr: That's one possible approach to the major-upgrade problem. 16:24:43 was in django land all day yesterday. ;( 16:24:59 nirik: And you lived to tell of it? 16:25:02 mitr: perhaps lamp stack? people understand that and use it as a base for a lot of things... 16:25:03 sgallagh: alternatives? package the conversion tool as a separate rpm to be installed and then dropped? 16:25:22 mitr: That's certainly also an option 16:25:47 An alternative is to require everything in a role to come from Software Collections-style packages. 16:26:03 That would guarantee the ability to parallel-install the dependencies with other things on the system 16:26:04 davidstrauss: That was my first straw-man 16:26:17 so what does that buy us? 16:26:24 not being tied to package versions? 16:26:26 sgallagh: No, what you said is different. 16:26:27 Do we expect the current server roles to need parallel-installed dependencies? 16:26:44 mitr: As there are no "current" ones... no? 16:26:48 sgallagh: I'm saying the *dependencies* of a role may be required to come from an SC, not that the role is an SC. 16:26:52 I've seen earlier ostree builds; the major constraint is that it is a system-wide, all-or-nothing design; you can't switch to the new thing without a reboo 16:26:53 But going forward, absolutely 16:27:03 sgallagh: "whatever roles we are considering" 16:27:44 mitr: system-wide is no longer strictly true, but rebooting still mostly is. 16:27:51 SCs would, in a way, solve the requirement to have two major versions installed simultaneously (but that would require a _set_ of SCLs per ole) 16:27:58 But when used in conjunction with containers instead of the main OS... it gets interesting 16:28:00 I think SCLs could be great for possibly shipping a fedora server role on some EL os or something, but why wouldn't we use native packages? 16:28:57 Can we perhaps step away from RPMs for a moment (after all, (tar xf) would get bits to the disk as well), and instead talk about what bits do we want to install, and how, first? 16:28:58 nirik: Well, one thing it might gain us is the ability to have both stable and development roles in the stable repo 16:29:09 err s/stable repo/Fedora repo/ 16:29:28 sgallagh: sorry, got distracted, I am reading the scrollback and will answer whatever you asked 16:29:31 I suppose, is that a goal we have? 16:30:00 mitr: I guess, we have to decide a bunch of things, I don't care where we start. ;) 16:30:00 nirik: I think perhaps it is 16:30:16 We've stated categorically that we don't want a separate repo (other than install) for Server vs. the rest of Fedora 16:30:18 sgallagh: why can't development users install from rawhide/next-branched to test? 16:30:58 nirik: I see this as a potential way that we can potentially hang on to a stable Role longer than a single release cycle as well 16:31:36 sgallagh: What do you mean by that? After the base Fedora it's for isn't supported anymore? 16:31:37 i.e. MariaDB releases N+1.0 just in time for Fedora 22. We want to carry it, but not necessarily force an upgrade 16:31:55 sgallagh: MariaDB seems easily supportable with SC directly. 16:32:04 davidstrauss: Random example 16:32:07 * nirik would just do a compat package, but sure. 16:32:27 sgallagh: What happens in Fedora 23, when MariaDB N+1 is the mainline version? How do users get back to the mainline role? 16:32:44 nirik: implementation detail, I suppose. But it gets expensive if a lot of libraries change under the hood 16:32:45 Or, in the case of MariaDB, N+4.5 for the next major version. ;-) 16:32:52 Especially those that botch ABI regularly... 16:33:12 sgallagh: (I suppose this is all solvable, but not that different from just putting the alt version to a non-Fedora repo from a first glance) 16:33:17 mitr: My point is that we want to enable migrations separate from package installation 16:33:44 mitr: The parallel-installability can be leveraged for migrations. (Bingo!) 16:33:56 sgallagh: ... with an obsoletion mechanism for roles 16:34:21 mitr: Yes 16:34:33 I'm hoping more and more stuff moves to SC in Fedora to provide inter-release flexibility for major versions of things. 16:34:49 sgallagh: i was thinking we should aim for seamless and almost invisible migrations; so giving users a "migrate-from-mariadbN-to-mariadb" command would be ugly. Certainly something we can do for the complex case but the default case shouldn't ask the user to run an extra migration tool 16:34:54 I fear SCs will actually mean: people won't bother with upgrades anymore and users will always be forced to do manual migrations 16:35:05 s/to run/to have to decide to run/ 16:35:28 which might be ok, but I think we need to understand the consequences of heavily using SCs, especially in the loa dfor security releases that need to touch multiple SCs in the same way over and over again 16:36:11 also it seem to mean that if Fedora Server uses an SC and the faster fedoras sto pusing it because it is an old version then someone in the Fedora Server community needs to take over maintenance 16:36:15 simo: Ultimately the constraint to have only one version of a package available within Fedora is just too limiting, and we do need to either break the constriant or bypass it; currently SCLs are the only sem-sanctioned (well FPC is not moving towards approving them, but stilll) mechanism, so we see them more and more 16:36:43 mitr: I fully understand the problem and not opposing SCs in anyway 16:36:44 is it really the case that it's too limiting? 16:36:50 but you also need to analyze the consequences 16:36:51 nirik: absolutely 16:37:03 nirik: Yes, it's definitely too limiting. 16:37:05 if you start using SCs for Roles it means you are signing up to maintain the older SCs 16:37:07 * nirik looks at his f20 server. seems to be working fine. 16:37:19 Reminder: the two most common complaints about Fedora are "Fedora moves too fast" and "Fedora moves too slowly" 16:37:25 nirik: some things are *very* difficult 16:37:30 right, it's always a balancing act. 16:37:38 As for making stable/bleeding versions of the same role available at the same time, I'd be happy to punt implementing that to Server 3, as long as the mechanisms we chose will be able to do that by Server 3 16:37:40 (Deployers vs. Devolopers, respectively) 16:37:47 Developers* 16:37:53 There is a difference between using SC for the roles and using them as the building blocks to satisfy requirements. 16:38:01 nirik: especially in "Web-Land" where something built for django 1.4 cannot work with django 1.6 and you have 1 app that only works with 1.4 and one that only works with 1.6 and ytou need both 16:38:05 hilarity ensues 16:38:28 simo: VMs ensue 16:38:37 Or containers 16:38:39 davidstrauss: maybe for the deployer 16:38:41 sure, I know there are issues, but I don't know that replacing all packages with SCL's makes a supportable solution in the end. 16:38:57 davidstrauss: but the distribution ? you can't package both apps unless you have both django versions available 16:38:58 davidstrauss: which is a lot of CPU power to waste over something that should be solvable by just putting the two things into non-conflicting directories 16:39:08 nirik: To be clear, if we went this route, I'd be recommending that we only SCL-ify the divergence, not the entire stacks. 16:39:12 nirik: that is exactly my concern 16:39:15 mitr: You don't need to tell me that. ;-) 16:39:27 sgallagh: Not a bad idea. 16:39:33 I am trying to warn people that SCs are cool and all but overusing them is a huge maintenance burden 16:39:49 sgallagh: But that will mean we switch between using base and SCL packages depending on the Fedora release. 16:39:51 mitr: there is really no big cpu waste with containers 16:40:00 and even with VMs, today the overhead is really low 16:40:02 simo: I have this big SCL hammer... look! everything is a nail! 16:40:19 simo, nirik: Perhaps one way to think about it is that all that SCLs _really_ give us is the permission to use /opt/package-version as a %prefix; the separate repos and separate maintenance procedures are incidental and could be avoided 16:40:20 davidstrauss: Yes, whenever a backwards-incompatible change results in a common package having to be added to the SCL chain. 16:40:26 * nirik is intrigued by the docker idea. I wish docker was a bit more mature tho 16:40:29 davidstrauss: I *think* we can make that transparent to the user, though. 16:40:37 nirik: so, what's the alternative? 16:40:55 nirik: Docker isn't mature now, but it's on a very clear trajectory to being a solid foundation. 16:40:59 i think the problem we're talking around here is things that just don't package well 16:41:04 mitr: That's an interesting point... 16:41:07 nirik: Work is happening from so many directions. 16:41:21 sgallagh: note that we don't currently have a way to transparently start using a SCL from an application when the base library upgrades to an incompatible version; you need to start setting e.g. LD_LIBRARY_PATH 16:41:29 adamw: Yes, we call that "software" 16:41:34 nirik: we've been dealing with askbot in the last few days, for instance. it doesn't work well as a 'distribution package'. it would take a huge amount of work, probably non-upstreamable, to make it do so 16:41:43 and there are X thousand similar things out there that people want to deploy 16:41:53 sgallagh: Can be hidden in the role script, I suppose 16:41:54 adamw: I think you are over pessimistic on that, but sure, it would need work definitely. 16:42:00 mitr: Yeah, I was just about to say that 16:42:02 the high-level goal here is to allow people to deploy such things on fedora and, ideally, add some kind of distribution-y value to it, yes? 16:42:08 adamw: but we shouldn't produce a askbot role should we? 16:42:16 just give them the tools to 16:42:37 true 16:42:56 Yeah, we should try to be clear that a Role != an application 16:43:16 nirik: I hear "getting an existing server app with all its dependencies into Fedora can take two years" too frequently to believe these are rare cases 16:43:31 but I guess in an ideal world, someone would work on an askbot role, get it to do all the things we say roles should do, and then ask us to bless it as a featured role... 16:43:35 hell, we've had a review request open for *tiny tiny rss* for like three years. 16:43:38 As the PRDs are, web applications are explicitly and almost exclusively a realm of the Cloud product 16:43:40 anyhoo 16:43:47 ok so back to the point, I have no objections at using freeipa, but it is incorrect to say *I* am upstream anymore, although I do participate in the project still to some degree :) 16:44:23 I thnk the good thing about it is that it uses so much stuff it will give us a good testbed to understand how complex a role could be and what needs to be properly developed 16:44:29 if we're cutting off our work at the sort of 'platform' level, i'm rather with nirik in wondering whether it's really necessary to diverge so hard from distro packages 16:44:41 including iproving the systemd integration which is still somewhat funky in current freeipa 16:44:43 simo: And has already solved much of the glue work ahead of time 16:45:01 adamw: I think we should not diverge unless there is no other way 16:45:11 ie SCLs should be the last resort 16:45:29 do we need to deal with them at this point, then? 16:45:32 sgallagh: yes but you may not agree or want to preserve exactly all the glue work as it is 16:45:38 so we'll need to analyze some things 16:45:38 if our initial sort of 'proof of concept' is a freeipa role, for instance? 16:45:42 freeipa works fine from our distro packages 16:45:43 Sure, but we have already noted multiple examples in this conversation where parallel-installation is necessary and insufficient today 16:46:17 adamw: Sure, that's a case that generally works with distro-only stuff. 16:46:17 but in general I think it is in pretty good shape to just have us deal with the "role-ing" of it rather than have to also solve a ton of packaging and upstream issues first 16:46:19 adamw: So far my thinking is that we may need to diverge noticeably from the packaging guidelines, somewhat from the FHS, and in some areas from RPM assumptions. None of these automatically need to diverge us from the "Fedora universe" if they were adopted Fedora-wide, though. 16:46:23 simo: hah, yeah, that service is fun. 16:46:37 mitr: oh? why so? 16:47:25 adamw: as for "freeipa works fine" we haven't had a single GA where freeipa would work 16:48:00 you always have to wait a month for freeipa developers to fix some last minute brokenness introduced by one of the dependencies 16:48:00 simo: that wasn't really the context. 16:48:09 nirik: it's somewhat hazy and long to put in IRC, I should really prepare a prototype (which I may not have time to do :() Quick summary: two releases on disk at the same time, dedicated space for "users' data" kept across deployment and "configuration" mostly kept across deployment, ability to cleanly refuse a role upgrade if a migration fails 16:48:23 the context was whether we need to go outside the distro packaging framework. i am not aware of any way in which we need to do that in order to make freeipa work in fedora. 16:48:40 adamw: I know, but it is a clear thing that would need to be addressed for "featured roles" 16:48:42 the fact that sometimes they have bugs in the packages at release time isn't really relevant to that. 16:48:53 you can't feature something that doesn't work out of the box at GA :) 16:49:12 obviously we would need to make sure it does if it's a featured role. 16:49:17 mitr: i guess when you start bringing that kind of functionality in, what we have is a layering question 16:49:21 but that doesn't matter for implementation 16:49:37 nirik: I think it does 16:49:38 simo: Sure, but adamw makes a valid point. If we're going to decided that Fedora Server 1.0 is the platform and a single Role: FreeIPA Domain Controller, then we *could* punt on this part of the discussion for now 16:49:52 adamw: we can do these things _using_ RPM, sure; but not _only_ RPM and not the way we currently use RPM 16:50:12 nirik: for example if the "role packaging" (whatve that will be) constarins to use specific well know compatible versions it may give us a better experience 16:50:28 the flip of the coin though is that we may need more work to "approve" updated packages 16:50:29 right, so the question is, do we add functions like that by twiddling with the definition and capabilities of the distribution packaging layer? 16:50:43 so, it sounds like there's 3 possible options here: a) docker container b) normal rpm packages only + scripts/glue, c) normal rpms and SCL mix? 16:50:44 it depends how this role thing works out 16:50:59 * nirik sees pros and cons for all 16:51:01 i mean, we keep talking about docker - theoretically, can't the two releases be implemented at a container layer? 16:51:05 if it is something included in repodata that will be relatively easy as you update the compatibility matrix at "yum update" time 16:51:26 but I am not sure we will be allowed to add that kind of info at that level 16:51:42 Docker will, in the near future, allow superior security too. 16:52:03 selinux isolation will be at the container (role) level, not the daemon/executable/user level 16:52:09 sgallagh: I was pointing at it to come to the conclusion that roles may need this kind of features where data is dinamycally downloaded at "yum update" time 16:52:14 could be a yum plugin I guess 16:52:20 issues with docker: has to be post install (unless we teach anaconda to install roles), how about signing and updates? 16:52:23 adamw: 1) How do I ship the container to the host OS? (Solvable easily enough), 2) How do I let the "new" container run a binary to migrate data so far used by the "old" container, without copying the terabytes-large database? (No idea how easy this would be) 16:52:34 Two roles using MariaDB 5.5 would not share labels under a Docker/container model. 16:52:38 the nice thing is that such integration with yum would allow us to promote things to roles and make them listable at any time 16:52:52 so if we want to promote a role to featured in the middle of a server release we could 16:52:53 (They would each run MariaDB under their own labels) 16:53:11 and all a user would need to do is something like yum list roles 16:53:19 or yum roles list 16:53:30 or whatever command we'll use to update/list/handle roles 16:53:32 I'm a little concerned that we're trying to solve 100% of the general case in the first pass. 16:53:32 simo: yeah, or 'server-role' perhaps? 16:54:06 sgallagh: well the update system is fundamental to be able to maintain rols 16:54:12 * sgallagh nods 16:54:13 so I think we need to look at it from the start 16:54:22 sgallagh: yeah, although it could make a difference down the road to how hard something like an 'askbot' role would be to make... if it's a docker container it would be a lot easier, but has it's own downsides. 16:54:32 you do not want to find yuourself in trouble when it is time to actually try updates :) 16:54:53 I'm also interested in how we can inform the maturation of the Docker project's model for updates and distro-integration 16:54:58 In general, I think there are 3 major, independent, areas of concern: 1) deployment (installation/updates/migration); 2) APIs 3) resource control and monitoring 16:55:01 nirik: By the time people are using Docker, they're often using language-native installation tools anyway, or am I mistaken? 16:55:33 sgallagh: we ought to change that so you can use a centralized tool to update everything 16:55:34 sgallagh: currently. I was talking if we used them for roles... I'd assume we would add some glue on top/inside any such role containers. 16:55:45 mitr: +1 16:55:47 that's the value of bringing something like docker into a distribution IMO 16:56:56 right, but it will be a lot more work for us... new infra, new setup, etc. 16:56:57 anyway time is flying 16:57:04 is there any decision we need to take now ? 16:57:15 nirik: Right, I was just noting that the current method of operation there is to just trust upstream and pull in the exact versions of things they want you to 16:57:16 or do we need to start some threads on the mailing list 16:57:42 I still think the first thing we need to do is decide what deliverable we want for "Fedora 21" 16:57:50 also I want to warn I will probably not be bale to attend meetings the next 2 weeks as I will be traveling and/or in conference calls the whole time :/ 16:57:54 (Irrespective of time; no schedule is yet set) 16:57:56 FreeIPA in Docker? :-D 16:58:01 i was still working on the basis that we were 'distribution-izing' what you might call the platform layer - whatever you need to run your stuff on top of - and leaving the Stuff to cloud product or secondary dependency managers, but the discussion seems a bit confused 16:58:23 davidstrauss: we could discuss it 16:58:24 adamw: Well, we are to a certain degree 16:58:40 adamw: yeah, and perhaps thats more than enough for 21. ;) 16:58:41 adamw: But we're also talking about having one reference implementation of those secondary dependency managers 16:58:48 sgallagh: i agree, and i'm in favour of simplicity: as you suggested, a single PoC role and whatever base we need to implement that 16:59:07 sgallagh: what is an 'implementation' of a secondary dependency manager in this case? 16:59:24 (and what manager are we talking about if our PoC role is freeipa?) 16:59:25 adamw: An implemented Role 16:59:30 adamw: Our PRD says "we commit to produce ... roles" 16:59:46 adamw: Maybe I misunderstood your sentence. 16:59:54 yeah, i think we're talking past each other 16:59:57 We want to enable other groups to be able to create and promote roles 17:00:02 That's what I thought you meant. 17:00:13 no, i was talking about stuff like pip, composer etc. 17:00:39 Ok, I misunderstood you 17:00:46 so, abandon this fork: i agree with sgallagh that we should define what we're building 17:00:51 for f21 17:01:05 right, mailing list/#fedora-server ? 17:01:10 (rewind the conversation to XX:57 :>) 17:01:43 So shall we put "Fedora Server 1.0/21 will provide the FreeIPA Domain Controller Role and the infrastructure to enable it" as a straw-man to the mailing list? 17:01:53 +1 17:01:59 sure. 17:02:38 Ok 17:02:42 * davidstrauss thinks "straw man" here may not be the normal usage he thinks of. 17:03:02 davidstrauss: In the sense that if people who weren't at the meeting have a strong and valid argument to knock it over, great. 17:03:17 If not, straw men are still good scarecrows. 17:03:53 Ok, I'll write something short up and get the discussion started. 17:03:57 Okay 17:04:09 I'll be around for next week's meeting, but the following two are less certain 17:04:18 (Due to travel for FOSDEM and DevConf.cz) 17:05:09 Ok, anything further that we should discuss urgently? If not, I'll close out the meeting in two minutes. 17:07:12 Thanks for coming, folks. 17:07:15 #endmeeting