16:00:31 #startmeeting Server SIG Weekly Meeting (2016-01-26) 16:00:31 Meeting started Tue Mar 8 16:00:31 2016 UTC. The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:31 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:31 The meeting name has been set to 'server_sig_weekly_meeting_(2016-01-26)' 16:00:31 #meetingname ServerSIG 16:00:31 #chair sgallagh mizmo nirik stefw adamw simo danofsatx mhayden jds2001 16:00:31 #topic roll call 16:00:31 The meeting name has been set to 'serversig' 16:00:31 Current chairs: adamw danofsatx jds2001 mhayden mizmo nirik sgallagh simo stefw 16:00:37 * simo sine-waves 16:00:39 ahoyhoy 16:00:46 .hello adamwill 16:00:46 adamw: adamwill 'Adam Williamson' 16:00:48 .hello simo 16:00:49 simo: simo 'Simo Sorce' 16:00:50 .hello sgallagh 16:00:50 .hello dmossor 16:00:52 sgallagh: sgallagh 'Stephen Gallagher' 16:00:52 .hello kevin 16:00:53 .hello blc 16:00:55 danofsatx: dmossor 'Dan Mossor' 16:00:58 nirik: kevin 'Kevin Fenzi' 16:01:00 greetings and salutations, folks 16:01:00 mizmo: you are back? Welcome! 16:01:00 bconoboy: blc 'None' 16:01:06 .hello langdon 16:01:07 * nirik is here, but also in another meeting, will try and pay attention too tho 16:01:07 langdon: langdon 'Langdon White' 16:01:21 .hello duffy 16:01:21 .hello sct 16:01:21 mizmo: duffy 'Máirín Duffy' 16:01:24 sctw: sct 'Stephen Tweedie' 16:01:26 bconoboy: You may want to add your real name to FAS :) 16:01:54 * langdon thinks adamw needs to update zodbot to allow "ahoy" as an alias to hello 16:01:56 hi 16:02:03 it's 'ahoyhoy' 16:02:04 welcome back mizmo! 16:02:07 not a mere 'ahoy' 16:02:12 .hellomynameis jzb 16:02:13 jzb: jzb 'Joe Brockmeier' 16:02:17 adamw, that is too excited for this early in the morning 16:02:31 thanks nirik :) 16:02:40 langdon: if it's good enough for montgomery burns, it's good enough for me 16:02:41 sgallagh: I have, I don't know why it does that. 16:02:46 adamw, lol 16:02:56 bconoboy: You must have set your info private, then 16:03:01 .hello mattdm 16:03:02 mattdm: mattdm 'Matthew Miller' 16:03:05 .hello stefw 16:03:07 stefw: stefw 'Stef Walter' 16:03:27 mizmo hello! 16:03:47 .hello blc 16:03:48 bconoboy: blc 'Brendan Conoboy' 16:03:52 sgallagh: woot 16:04:04 hey mattdm 16:04:32 hi mizmo! 16:05:00 OK, let's get started. 16:05:05 #topic Agenda 16:05:05 #info Agenda Topic: Modularization 16:05:35 We're going to reserve the entire meeting for a single topic, unless there is a five-alarm (or more) fire. 16:05:46 .hello jstanley 16:05:47 jds2001: jstanley 'Jon Stanley' 16:05:55 /me pauses for 30s to listen for sirens. 16:06:17 OK, I hear two sirens, tops. 16:06:26 #topic Modularization 16:06:26 #info Guest Speaker: Langdon White 16:06:26 #chair langdon 16:06:26 Current chairs: adamw danofsatx jds2001 langdon mhayden mizmo nirik sgallagh simo stefw 16:06:34 This will be an informational meeting. 16:06:34 We intend to avoid deep implementation discussions today and focus on high-level goals. 16:06:34 I will be moderating the discussion to keep us on topic; if we get too far off course, please respect it if I ask you to take your thoughts to the mailing list. 16:06:41 * langdon notes in downtown boston, that isn't even worth remarking on 16:06:46 * danofsatx listens, saves questions for the end 16:07:08 And with that, I'll pass the mic over to langdon to give his overview. 16:07:37 ok.. despite having considered typing this all in advance decided to do real time.. hopes fingers are up to iit 16:08:17 I will attempt to #info key points as we go 16:08:21 so.. modularization.. in essence, open source has won.. and how does a distro keep up.. 16:09:57 modularization is the idea that we can start to work with modules as a larger granular unit than a package.. this does not mean "packages" go away.. but rather that we start to worry about the things at a higher level.. thereby allowing us to recognize some of the changes to the development world and packaging world.. and allow the tradeoffs between them as we goo 16:10:18 "as we goo", yes. 16:10:37 and.. to danofsatx's point.. please don't wait til the end of qs.. else my monolugue will make my fingers hurt 16:10:50 roger, ball 16:10:52 s/goo/go 16:10:56 :P 16:10:58 ohhhhhhhh :) 16:11:03 langdon: It might be useful to define some stopping points for questions, though. 16:11:07 * danofsatx thought the pun was intended 16:11:13 Lest we jump in right before you were about to cover the answer :) 16:11:42 sgallagh, meh.. i can always delete or mistime things a al my siren joke above 16:11:48 ill try to use ... as well 16:12:01 OK, please continue. 16:12:32 so.. to get started.. we want to consider a "core" module that is useful for physical, vm, and container images installs.. key being "binaries available but not all installed in all cases" 16:12:52 but we also want to consider a few simple modules that are basically just "applications".. 16:13:39 currently thinking httpd (in fedora but a desire for updates on their own cycle) .. howdoi ( a little python app not in fedora but fun and useful, also changes a lot) 16:13:41 langdon, would the core be the same for each use case or would it be tuned specifically via configs / etc for each case? 16:14:00 .oO(roles) 16:14:08 mizmo, the "module" would be the same... but the installation on demand would be different 16:14:14 mizmo: the core is a set of packages, which may or may not all be installed for each usecase 16:14:32 jwb, ack 16:14:36 okay so a module is just the software, the specific config is layered on top? 16:14:41 like... containers don't need a kernel. but the kernel is probably going to be in the core module 16:14:43 To visualize it, think about a DNF repo specifically for core that passes repoclosure. 16:14:44 do you install multiple modules? or just one module is created with all your needs? 16:14:53 Not necessarily for buildrequires, but just for runtime. 16:14:58 nirik: multiple. 16:15:12 nirik, what jwb said.. with, likely, dependencies.. 16:15:17 * nirik nods. 16:15:18 in some ways very like rpms.. 16:15:21 If you want to build something and call it Fedora, the foundational pieces would come from that repo 16:15:26 metarpms. ;) 16:15:51 nirik, or groups even.. we have many analogues.. but they are not "first order citizens" 16:16:03 in fact ... backing up a bit... 16:16:05 sgallagh: passes repoclosure on its own, or on top of some other defined set? 16:16:30 mattdm: Let's stay out of the implementation details. I was trying to use that as a handy visualization, rather than an implementation suggestion 16:16:33 i kinda glossed over some of the motivations.. basically.. look at puppet (which is in my forthcoming blog post).. 16:16:44 Also I was specifically talking about Core there 16:17:08 important to fedora .. but countless problems about maintaining the "correct" version of ruby for it (and I am sure many other examples).. 16:18:00 one of the main goals here is to allow for things to operate on a separate pace from the core OS.. in other words.. less like a "distro is released at point x" rather "an os is released at point x" and a bunch of apps are released on their own lifecycles.. 16:18:35 another motivation is "quality" a la rings.. but that isn't "quite" right.. as in the quality of a beta isn't "bad" it just isn't ready yet.. 16:18:52 the "quality" of puppet isn't bad.. it just hasn't updated to latest ruby yet.. 16:19:13 and the latest python isn't "bad" just the rest of the distro isn't ready for it yet.. 16:19:27 so .. these things need their own lifecycles.. modularity is attempting to support that 16:19:34 So different lifecycles this implies that some modules are incompatible with others, right? 16:19:49 So *given* different lifecycles ... 16:20:15 yes 16:20:22 stefw, that is a great question.. i think interactions between them might be "not supported" (in a sense).. but i think the long term goal is that they can coexist without hurting each other.. aside from deps... 16:20:34 stefw: unless you use a packaging mechanism that supports that??? 16:20:45 as in httpd-24-module deps core-f23 & core-f24 .. but not core-f22 16:20:53 stefw: not trying to go into implementation, but something like SCL's? 16:21:21 scls are another good example of a "solution" that was never a first-order citizen.. just a way to solve a problem 16:21:41 * stefw will ask about testing later :) 16:21:55 in other words.. we have lots of things that deal with this lifecycle symptom.. but not at the core (sorry to reuse the word) of the distro 16:22:57 okay another q.... modules == roles? 16:23:10 another motivation is the i included this package for my app to work but i am only supporting it for my app.. but, now a few apps use it.. and now we have another lifecycle problem.. does app-a or app-b get to decide on an update? what if we could declare a dep as a shared one or a private-to-app one.. 16:23:51 mizmo: roles are possibly one implementation or a subset of modules 16:23:58 mizmo, roles, as i understand them, are about getting something "running".. a role may require / use one or more modules. but modules are more about the logical set of things.. rather than the use of them.. 16:24:00 langdon: how do we avoid balkanization and having to maintain 15 different versions of the same package ? 16:24:41 and importantly, keeping things around that have known security issues? 16:24:43 simo: What are the problems with that? 16:24:53 simo: That's an open question and one that can *only* lead to complicated implementation discussions. Let's hold that for later. 16:24:58 (how do we even maintain 2 version in Fedora, given currently you cannot have more than one version per package in repos ?) 16:25:06 Or I guess we can discuss the high-level maintenance burden issues. 16:25:20 I am not asking about the technical solution 16:25:23 simo, righto.. that is a problem.. but one we currently have.. i think it is just not as explicit.. we need to still foster a community working together... but, enforcement via fiat seems to cause workarounds not solutions 16:25:26 simo: figuring out an implementation for that is part of the immediate work to do 16:25:28 but what is the plan to deal with thes issues 16:25:31 yes, a guess a question of "who is responsible"? 16:25:42 are modules "supported" (as in security) by Fedora itsealf? 16:25:59 I'd suggest that each module should probably have its own SIG maintaining it. 16:26:01 the "fedora modules" are .. but their would also be a copr concept as well 16:26:20 And if the SIG falls into a lack of maintenance, the module is retired. 16:26:22 sgallagh, maybe sets of modules.. otherwise there may be a LOT of sigs 16:26:30 People who need it for their module are responsible. If a version maintained to a high standard by someone else exists, great. If not, don't stop them from doing it as long as bar for security responsiblity is met 16:26:35 so therte is a strong incentive to Fedora to keep the modules it supports on the same schedule? 16:26:43 at least until/if tooling makes it a non-issue? 16:26:48 sgallagh: SIG is too big a construct 16:26:57 something smaller than SIG 16:26:58 stefw, not nec....i am not seeing the logic leap there 16:27:03 stefw, ohhh 16:27:03 mattdm: Perhaps, but we don't have a smaller one today. 16:27:08 maintainers/co-maintainers, like packages probibly 16:27:16 sgallagh: Sure we do. .... yeah what nirik said 16:27:17 stefw: Yes, very much so. Although we would at least have the ability to have differing schedules in special cases where we want it --- 16:27:28 ok.. lets be clear.. a REQUIREMENT of modularity is significant tooling change/automation/improvement.. 16:27:29 OK, I suppose that's true. 16:27:40 #info a REQUIREMENT of modularity is significant tooling change/automation/improvement 16:27:58 stefw: eg. if Fedora atomic host wants to move kube+docker faster, then that would be possible if we have a distinct "container support" module. 16:28:22 stefw: But still keep the core on the same schedule. 16:28:23 we need to have automated rebuilds of modules.. we need basic testing of a module on security updates automatically.. we need to build all that out for modules.. to a much greater degree than we do with rpms.. 16:28:23 It seems like another requirement (or perhaps consequence) is "more maintainers" or more interested parties? 16:28:41 because we don't want to update a module on the end system if it hasn't been tested as a unit 16:28:42 stefw: On "supported" — I'd like to have levels, ranging from "very centrally important -- critpath" to "gets security attention and help" to "copr-like edge" to "third party" 16:28:58 interesting 16:29:41 we also need to improve the "app -> module in fedora" path.. like really lower the barrier (of effort not quality) in an app developer maintaining their own module.. 16:30:06 On the tooling side: I'd say a prerequisite for this is going to be Continuous Deployment-based testing of modules right from the start. 16:30:08 +1, and to relate that to Simo's question earlier --- 16:30:14 langdon: do we need a new packaging system for that ? 16:30:20 stefw: A high-level hope is to move the focus of a lot of contributor activity away from working on individual tiny dep rpms to larger chunks. 16:30:20 but that is longer term.. for now.. we are just playing with modules as rpm repos.. the simplest case.. so a repo ~= module.. with some extra metadata and some client side tooling 16:30:23 sctw: What were you +1-ing? 16:30:52 langdon's app-to-module path. If a dependency is brought in purely to support one particular app, and the maintainer has no real interest in the dependency itself, 16:30:55 simo: Maybe new tooling, but there's some work underway to help with automating the creation of RPMs 16:30:59 make dealing with libraries and dependent modules less time consuming, freeing effort for applications people really care about 16:31:11 sgallagh: I am thinking of delivery 16:31:20 then it may actually be just _fine_ to support multiple instances of that. Mark those dependencies as belonging just to the app. 16:31:43 there probibly should be levels of importance inside modules... 16:31:45 And if somebody wants to maintain the dependency as an independent thing in its own right, then it belongs in some module of its own, not in the app module. 16:31:48 the problem with rpms is that we'll end up with app-a requiring foo-1.1 and app-b requiring foo1.2 16:31:51 simo: In my ideal world, I'd like to see this handled with RPMs and groups/metapackages. So we don't lose the flexibility we have with all the little pieces. 16:31:55 sctw, that is kinda what i am thinking (marking as "for that app") 16:32:01 They should just be the "expert-mode", not the standard way of doing things 16:32:04 the word everyone is avoiding here is bundling. there will be a form of bundling within modules 16:32:05 and if they are both build with standard rpms the 2 modules wll conflict 16:32:10 (But that's just my 2 cents) 16:32:15 so you won't be able to install both app-a and app-b 16:32:28 unless you allow relocation/embedding of packages 16:32:35 simo: No: figuring out how to cleanly parallel-install deps is a key point. 16:32:45 nirik: i don't follow your level of importance comment 16:32:52 jwb, well.. its bundling in the sense of being declarative about your deps.. but not in the sense an end system owner can't update the deps directly.. even if it may break an app on their machine 16:32:53 And yes, bundling/relocation are on the table here, with tooling to reduce the problems inherent there. 16:33:16 langdon: right. but module a and module b might both contain package C, at differing verions 16:33:18 simo: Indeed. So we may need to be able to package those bundled dependencies in a path that belongs to the app that wants them, and not have them share the system path. 16:33:22 (On the table != fait accompli, for the record) 16:33:24 jwb: I was not avoiding thee word, I just gave it for granted 16:33:26 :) 16:33:28 ie, the 'cockpit' module would have cockpit itself as critical, but 'libfoo' thats pulled in for something as much less so... 16:34:02 nirik: i dunno. if cockpit doesn't function without libfoo, then it's important. if it does, why is it in the cockpit module? 16:34:08 but if cockpit wont work without libfoo..... 16:34:13 jwb: I think he means "level of maintainer care" 16:34:13 also.. to simo's point (sorta) earlier.. i would like the language packaging mechanisms to be more directly supported.. reducing app dev effort, simplifying packaging, not creating a new packaging system, relying on app language unbundling, etc 16:34:22 sgallagh: still seems pretty high to me. 16:34:34 jwb: They care that it works *for their app* 16:34:37 jwb: well, it may only need some small part of libfoo and not care about anything else it does... 16:34:47 They may not care at all if it works for anyone else's app 16:34:52 so it could be upgraded and as long as that small part works, great 16:34:58 sgallagh: yes. but he said within modules. 16:35:01 Right, or only a subset of functionality, etc. 16:35:01 langdon: I am not against, that's why I was thinking a new module-evel packaging system may be required 16:35:12 jwb: I think he meant in terms of when to share and when not to 16:35:19 But now I'm putting words in his mouth, so I'll stop 16:35:20 or may be we simply build a big RPM with the contents fo all the RPMs used to generate the module 16:35:24 yeah, that. 16:35:25 nirik: oh, yes. but that doesn't mean we need to mark importance levels 16:35:33 a meta-rpm of sort 16:35:36 nirik, actually.. that is very much what i mean.. essentially a module only tests its deps for the api it needs.. if a dep becomes popular enough, it might get its own api test 16:35:43 importance might have been the wrong word... 16:35:48 that exposes only the deps you want to make public, and not every singlle package 16:35:56 but that would require some relocation/smarts 16:36:00 simo, i really don't want a new packaging method.. i would much rather embrace all the existing ones 16:36:30 langdon: it doesn't matter what packaging tool you use underneath, but you have to deliver a module and be able to coherently introspect it 16:36:53 simo: +1 16:36:56 call it a meta-package, you need info on all files, where they go, how to verify if they are still what was delivered, etc... 16:37:00 That *will* need new tooling. 16:37:01 simo, ahh i see what you are saying.. in that sense.. yes.. 16:37:39 but.. this "packaging" would not actually package the binary like rpm does.. we are imagining a split channel.. metadata that maps to binaries through two channels 16:38:13 simo: You might run into the issue that some "modules" have pieces that don't need to be installed (Like the base distribution module) whereas on the app spectrum it's more likely all the packages in the module need to be installed. 16:38:16 * langdon is impressed with his typing speed 16:38:18 I think it's best if we discuss only what information we need and stay away from the exact mechanism that writes bits to disk 16:38:40 i don't mean to discourage discussion, but does everyone understand the motivations here? 16:38:42 bconoboy: Well, that's not firm either. 16:38:48 Consider an Eclipse module. 16:38:48 if not, we should go back up a few thousand feet 16:38:54 sgallagh: Ah, true 16:39:01 jwb: +1 16:39:08 jwb: somewhat 16:39:20 jwb: but backing up might not be a bad idea 16:39:33 Yeah, let's enumerate the problems we believe we can solve 16:39:35 so motivations: 16:39:49 1) allow software to operate at a difference pace from OS 16:39:58 let me try and articulate what I've got: We need a way for applications to move indepentently of Fedora 16:40:04 2) allow for the latest version of foo to be available more easily 16:40:11 #info Motivation 1) allow software to operate at a difference pace from OS 16:40:27 mizmo: 2) I think is incomplete 16:40:30 3) solve the problem where someone pulls in some kind of dep and only that app uses it, then suddenly another app needs it, who gets to decide when to update 16:40:30 clarification on foo? 16:40:47 3) reworded: 16:40:52 langdon, eg puppet hasn't updated to latest ruby 16:41:23 3) make an explicit promise about whether a dep is for an app or meant to be shared 16:41:23 Reworded 2) Fedora needs to be more flexible about having newer or older versions of libraries, frameworks or applications available. 16:41:27 langdon, i guess the wording there was poor - 2) allow latest version of foo to be available without breaking apps that rely on older versions? 16:41:39 much better 3 :) 16:41:45 anyway, that's what i got re the motivations 16:41:50 ok.... trying 2 as well 16:42:34 4 (?) - move testing from component level to application level 16:42:39 2) allow an application to have an independent lifecycle, including its dependencies, from other applications and the OS but only branch on deps when necessary 16:42:48 might be a bit of 2&1 16:43:01 mattdm, 4 = module level 16:43:15 mattdm: Maybe 4) Make testing at a module level rather than individual or whole-distro level 16:43:32 4) move testing from implicit every package to explicit exposed public apis of a module 16:44:10 on 4) right now we implicitly promise an api/abi for every rpm for the lifecycle of the release.. but we don't really mean it 16:44:29 langdon: sure, module level. But that seems circular :) 16:44:33 i propose moving it to more explicit, limited api/abis at a less granular level 16:44:45 so how do modules map? or is that up to the maintainer(s)? ie, if there's fedora22/23/24 core modules, would there be just one httpd module that has 3 versions depending on which core? 16:45:15 nirik, hopefully up to the maintainer.. but in the short term, 1-1 .. moving very quickly to maintainer choice 16:45:51 * langdon hopes no one points out have more than 1 core on one system ;) 16:45:59 s/have/having 16:46:14 nirik: ideally a module can have a single source base that works on multiple versions of fedora 16:46:36 bconoboy, i am not sure i followed that... 16:47:18 Rather than having separate sources for fc22, and fc23, and fc24 you instead have sources for your module that get built on fc22, fc23, fc24. 16:47:25 right, so there might be httpd22 httpd24 and they would be built against all the various core/modules that they have as deps... 16:47:30 one source, many binaries 16:47:37 right.. yes 16:47:40 write once, run everywhere! 16:47:41 but the other way too 16:47:50 * nirik hands mattdm a cup of java 16:47:56 ohhh that would be nice 16:47:59 so versioned cores/bases... but rawhide modules all things? 16:48:05 * mattdm drinks that down. burns tongue. 16:48:13 as in .. core source may offer f22, f23, f24 apis .. even though it is only one source.. 16:48:33 langdon: do we have a plan on where to start and where we'll see the first artifacts ? 16:48:35 but... i think that is a ways away 16:48:51 simo, typing... 16:49:00 yes, definitely a ways off 16:49:37 simo, getting there.. we are updating the wiki pages right now (wiki/modularization), renewing the objective in a couple weeks, some prelim initial module attempts are getting written up now.. 16:49:45 also posting some blog posts.. 16:49:54 if i finish it today.. today.. else this week 16:50:19 langdon & mattdm happy to help with a user-facing post on Fedora Magazine. 16:50:25 if needed/desired 16:50:38 jzb, actually doing commblog.. as it seemed the more appropriate 16:50:51 i think when we have something "working" .. it would go to fed-mag 16:50:56 groovy 16:51:06 * nirik thinks this all sounds awesome. The devil is always in the details tho. 16:51:13 pshaw 16:51:34 nirik: sympathy for the devil will be required 16:51:47 well.. yes.. exactly.. thats in some ways why we have had trouble getting started and why we are starting with "simple" .. rpm-repo ~= module 16:52:08 nirik: Right, the major intention of this meeting was to get a feel for "Does this sound like something we want to do?" vs "This is a terrible idea, go home and leave us alone" 16:52:13 So is Fedora the first distro to try this with such a broad target (ie: not just "make everything into a container")? 16:52:14 langdon yeah I was just saying to jzb we could do a user-focused magazine post pointing to your commblog posts 16:52:22 or just link it in 5tftw 16:52:33 stefw: As always :-P 16:52:40 stefw, i think so.. and i think 'make it all a container' is not gonna work.. 16:52:43 * nirik wonders about some kind of rpm namespacing to support this, but thats way too into the details weeds 16:52:57 * langdon mutters relocatable rpms.. 16:53:23 ok.. come on.. more questions? 16:53:33 we haven't even used an HOUR 16:53:34 i have a summary i've almost got written up of this meeting, post to server-sig list? 16:53:52 We talked about automatically building the modules for each base, but is there value in trying to avoid doing that entirely when possible? 16:53:57 mizmo: you're too fast :D 16:54:01 langdon: so which WGs/SIGs are we going to ask to spearhead this? 16:54:18 For example, if we have modules with no shared deps, it makes sense to have the same exact module just tested atop F24 and F25 cores. 16:54:21 mizmo, mattdm or fedora-devel? or wait for the blog posts? 16:54:22 sgallagh: what about FTBFS against some base but not another? 16:54:31 mizmo++ 16:54:57 langdon: got that first blog post ready to go? 16:55:03 jds2001: Unclear today. 16:55:08 post it right now and add the url to mizmo's summary :) 16:55:13 sgallagh, i think it is too early to say about optimizing builds.. i think we just "build it all" until we can tell we can short circuit.. i know sct had some thoughts on where can do that... but i would punt for now 16:55:14 Needs investigation. I expect that's going to emerge quickly 16:55:31 I think this is important enough it should go to devel rather than just server 16:55:33 I think posting this to a broad audience without concrete examples is going to be premature 16:55:45 jds2001, what is ftbfs? 16:55:46 stefw: okay, that's a point too :) 16:55:54 mizmo: fail to build from source 16:56:00 thanks :) 16:56:28 langdon: Right. The main thing with builds is that I would like automation to try all builds and tell us what breaks; 16:56:29 langdon: Fair point. 16:56:36 jwb, part of the objective-renewal for the council is to ask base and e&s to make some changes to org and goals.. and then they would spearhead it 16:56:52 mattdm, langdon - i was thinking just server list since its the folks who were here today and isn't broader, and the notes could be used for anyone prepping something for a broader audience 16:56:55 regarding FTBFS across all bases, it may be ok if new packages fail to build on old cores, 16:56:57 Yeah, I'm going to reiterate that: 16:56:57 #info Automation is the linchpin for all of this 16:57:09 mizmo: sounds good 16:57:11 BTW, did we ever come up with a clean version of motivations 2+? 16:57:19 I only have 1 in the logs at the moment 16:57:31 i thought i had one.. but it really did 1&2 combined.. /me looks at scrollback 16:57:36 but the build system should tell us it fails and reject the build until we waive older builds. Ie. accepting failures should always require manual choice, the tools should tell us exactly what fails by default. 16:57:49 2) allow an application to have an independent lifecycle, including its dependencies, from other applications and the OS but only branch on deps when necessary 16:58:08 yeh i can paste all the cleans in 16:58:08 sec 16:58:09 sctw, +1 .. 16:58:15 sctw: yeah, and at that point you could decide that you don't want to keep supporting that old core or fix things so it works or whatever. 16:58:18 1) allow software to operate at a difference pace from OS 16:58:22 2) allow an application to have an independent lifecycle, including its dependencies, from other applications and the OS but only branch on deps when necessary 16:58:29 3) make an explicit promise about whether a dep is for an app or meant to be shared 16:58:33 4) move testing from implicit every package to explicit exposed public apis of a module 16:59:11 #info Motivation 2) allow an application to have an independent lifecycle, including its dependencies, from other applications and the OS but only branch on deps when necessary 16:59:19 #info Motivation 3) make an explicit promise about whether a dep is for an app or meant to be shared 16:59:31 #info Motivation 4) move testing from implicit every package to explicit exposed public apis of a module 16:59:34 i'm visualizing some kind of dashboard to be able to figure out what things ftbfs against which cores across different modules... 16:59:46 adamw can rewrite relval again :-P 17:00:00 a point i am not sure was clearly made.. we don't want to ship anything that hasn't passed testing as a module.. so for example, app-a builds successfully w/ dep-d.sec-patch but app-b does not.. so app-a ships an update.. but app-b does not.. 17:00:01 * mattdm is imaginging that dashboard all dark red :) 17:00:45 * adamw shoots sgallagh 17:00:51 langdon: as an end user, how do I know that my system is safe from the vulnerability fixed in sec-patch? 17:00:54 so app-b is still vulnerable to --dep-d.sec-patch but.. we need client tooling to allow 1) disable app-b 2) force patch app-b 3) ignore w/ risk 17:01:02 mattdm, ha.. was typing 17:01:13 * mattdm looks for more softballs to lob 17:01:15 from a UX pov, i think overall this is going to be a big win for users, who always want new hotness but can't without an OS update. i'm hoping that making it so you can build against older cores would make it easier to make new apps available for older cores? 17:01:37 * mizmo <= who builds inkscape from source to get new hotness on occasion 17:01:49 mizmo: Probably, but as Time Passes, more and more stuff will likely end up bundled. 17:01:57 app-b w/ risk seems like an ok idea on inward facing apps.. like .. what we also are supporting here is the end user system owner making explicit judgements about what risks they want to take.. 17:02:04 not to be awkward, but has anyone actually convincingly defined a 'core / apps' boundary yet? /me tends to find one person's core is another person's app. which is GTK+? 17:02:08 So it will result in an increasing maintenance burden over time. But with automation, maybe not too much 17:02:31 and modules likely will grow over time if they support older cores I bet. 17:02:32 it'll result in a lot more maintenance burden 17:02:34 adamw: We've been kind of actively avoiding talking about the desktop stack, because it's the hardest problem. 17:02:50 supporting against the versions of Fedora is already really insane ... eg: without automation Cockpit would be impossible 17:02:51 mizmo: I'd like to eventually get to the point where os core and modules can be mixed and match betweeen Fedora and CentOS/RHEL 17:02:54 It's one of the reasons we came to Server SIG first: the headless case is a lot more direct. 17:03:07 sgallagh: fine, which is python-requests ? 17:03:08 supporting against N modules * M cores will take the already insane state to the next level 17:03:11 adamw, i think this has been a huge problem.. so we are somewhat taking the approach of "pull all the things out we can, and what is left is the core" 17:03:22 GTK+ is "stacks" :) 17:03:34 stefw, but CI fixes ALL THE THINGS! 17:03:57 if only it was a requirement before pushing *any* update to *any* package 17:04:10 stefw: That's on the table for this effort. 17:04:18 stefw, that is, i think, the plan... if i am following you correctly 17:04:19 mattdm, im not following - frankenlinux? 17:04:41 mizmo: I think it's more Katamari Linux 17:04:43 four basic kinds of modules: base system; system services (e.g. logging); infrastructure (web, database, etc.); and then actual applications 17:04:45 sgallagh++ 17:04:49 * adamw looks forward to seeing the container truck fleet and helicopters turn up to deliver the CI cluster 17:04:57 i guess my concern is that we are forcing a tree of packages 17:05:03 er modules 17:05:06 and making a graph impossible 17:05:16 if fragmentation happens at each/many levels of the tree 17:05:21 mizmo: no, because the modules are big, clear deliniation points, rather than organs all sewn together 17:05:29 therhe is no hope for tools and projects that try to make the result usable, integrated, etc. 17:05:47 so there needs to be a balance found, and perhaps very strict CI would be that balance 17:05:48 stefw: If we can get to a place where the API/ABI boundary is really what we care about rather than the package versions, I think it gets more straightforward 17:05:54 * mattdm remembers something about voltron 17:06:09 sgallagh, indeed but in reality you have things constantly deprecating and then removing API 17:06:19 a good example is docker 17:06:30 dropping things out of the API (with fair warning mostly) 17:06:30 Docker is not a good example of anything -_- 17:06:32 from version to version 17:06:34 stefw, i guess i hope for automation to help with this.. as in, module owners have deps suggested to them when they work for them.. so the automation is actually trying to minimize the number of versions of deps 17:06:37 a good example of the problem 17:06:55 sgallagh: it is a good example of how people do things unfortunately 17:07:17 simo, stefw: Yes, I was being glib. 17:07:27 i guess if we allow modules to require certain packages or versions of API 17:07:33 mattdm, but i'm thinking the cores for centos vs fedora etc would be different? so how could you voltron modules 17:07:33 and as a result make them incompatible with other versions 17:07:40 then we'll at least model the issue quite nicely 17:07:44 and force fixes rather than ignore them 17:07:47 stefw: Yes, that's part of motivation 1, I think. 17:08:34 mizmo: bring along the appropriate runtime 17:09:00 mattdm, :( you lost me 17:09:04 mizmo: bundling 17:09:08 * stefw has gotta run 17:09:13 stefw, if we are constantly scanning for where modules' deps overlap, and testing versions for simplification, i think this will be better than you think.. software is much more sophisticated in recent years.. with much better backward/forward compat in libraries.. 17:09:14 This is easier in container world.... 17:09:18 in concrete terms: 17:09:27 with docker, use the fedora or centos base image 17:09:28 mizmo: I think he's talking about something similar to Docker, where you carry the entire user-space runtime with you 17:09:30 mattdm, jwb - but doesn't the runtime have to be built against a different core? or when you say module, you mean higher level not the binaries 17:09:33 with xdg-app, use the centos or fedora runtimes 17:09:40 langdon, yes it may have been worse ... but from working on Cockpit ... it's pretty bad 17:09:45 people are constantly pushing broken stuff into Fedora 17:09:47 stefw, lol 17:09:48 regressions 17:09:56 from systemd, to docker, to kubernetes, to NetworkManager, to lvm 17:10:12 stefw, right.. but we block that from coming in.. as in all the modules fail with new dep.. so no one picks it up 17:10:17 it's hard to think of a high level API that hasn't regressed at some point over the last couple years 17:10:32 mizmo: the other approach is rebuild, as people were talking about for f23/f24/f25 -- could be centos too 17:10:33 langdon, yes, that would be one way to fix this 17:10:33 stefw: nsswitch? 17:10:35 should be a requirement 17:10:46 mizmo: a la EPEL, really 17:10:48 stefw, ahh yes.. platform-api problem there.. i think that will be a harder case.. sorta.. 17:10:50 it's 2016 ... to me "API" means something remotable 17:11:05 like DBus, REST, spawning a process, etc. 17:11:14 stefw++ 17:11:15 sgallagh: Karma for stefw changed to 1 (for the f23 release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:11:16 stefw, yeah.. i just mean the libraries that you use.. local or remote 17:11:22 We need to harp on that wherever we can. 17:11:51 langdon, right, so you're right that this is a requirement of modularization 17:11:55 it should be a hard requirement 17:12:09 as in CI 17:12:21 otherwise we'd be building something that will get out of hand rapidly 17:12:29 stefw, i almost think it is a by product.. no module is updated unless it passes its tests.. so if you push a broken shared library.. no module will pick it up 17:12:39 mizmo: tbh, i don't think a module spanning across fedora and centos is an immediate target. but mattdm's explanations are one approach to doing that. 17:12:54 currently most of the tests are done by things that integrate other things 17:13:07 the libraries and modules themselves don't do near the amount of testing required to make this work 17:13:08 jwb: I agree. Let's try to keep the ocean at a low simmer for now. 17:13:09 mattdm, jwb: so by a module spanning fedora / centos you dont literally mean the same module, but the functionality / definition of the module 17:13:11 stefw, yeah... glossing over the current lack of tests ;) 17:13:29 I am partly Dreaming Big, here, not suggesting an immediate target, but I do want it kept in mind as we design 17:13:37 stefw, but, as with all things software these days, need to start fixing recipes not cakes 17:13:43 mizmo: with container approaches, maybe literally the same binary 17:13:52 mizmo: something like that, yes. it might be easier to think of it as "the source code for a module can be rebuilt on top of fedora and centos cores" 17:14:01 * stefw has really gotta run now ... 17:14:07 stefw, sorry ;) 17:14:08 mizmo: and then with containers, it might map to the same binaries too. 17:14:48 but again, that is all an eventuality we may not even try 17:15:15 although, if we can, it means we really did disconnect the os from the apps.. and honestly, for many apps built on scripting languages.. i bet running on multiple cores is not that hard even today 17:16:06 Or even running on golang and other bundle-the-world languages 17:16:08 mizmo are jwb and I making sense? :) 17:16:47 mattdm, not 100%, i mean, i'm filling in specific examples in my head and not quite getting it in terms of what it means for end use. 17:17:18 mattdm, eg, if inkscape is an app module - i can install that on epel or fedora today. the diff i guess is the epel version tends to be an older version of inkscape than available in fedora (and the newest version of fedora gets the newest version of inkscape) 17:17:32 so the big gain here is that you could have the latest version of inkscape on either centos or fedora? 17:17:43 mizmo: check this out: https://wiki.gnome.org/Projects/SandboxedApps/NightlyBuilds 17:17:52 includes inkscape as actual example :) 17:18:00 mizmo: I think the idea is that we would build tooling to bundle as much as you needed to from Fedora Latest into the EPEL module so that it would run 17:18:06 mizmo, from an end user perspective it is just, you have all current versions of an app avail for either cent or fedora.. 17:18:07 mattdm, yeh i have that inkscape and gimp running now 17:18:21 langdon, okay cool, that was the piece i was missing 17:18:26 \o/ 17:18:36 and yeah, latest build of inkscape *or* old version your institution has decided to standarize on 17:18:38 But they would be separate builds, not the same as installing inkscape.fc24.rpm on EPEL 17:18:45 but.. an app will start to have to seriously think about LTS style versions.. because the end user will start to care about their version stickiness more than the target OS 17:19:46 sgallagh, well.. from an end user perspective.. it doesn't matter.. dnf install inkscape.stable, inkscape.beta, inkscape.lts or whatever.. and you will get the right one.. 17:20:04 Well, hopefully :) 17:20:04 and if a patch comes out.. inkscape gets tested with the patch before the patch is shipped.. 17:20:07 (provided there are modules for all those maintained/available) 17:20:13 nirik, right 17:20:15 langdon, yes! 17:20:16 but there could be. ;) 17:20:36 langdon, i love how that would integrate with the systems that already exist rather than be yet another tool/system (which the currently nightly builds are) 17:20:52 mizmo, so.. inkscape may have the patch but openoffice may not.. because oo failed the new sec patch test... 17:21:00 could also help with the 'can you duplicate this bug against the latest upstream git snapshot?' problem... :) sure, let me install that module, yes, here's the error. 17:21:48 nirik: I hadn't thought of that. That would be brilliant. 17:22:12 nirik, OH YES.. i forgot that one! 17:22:14 ... 17:22:14 (provided the git snapshot module existed and passed builds/tests) 17:22:20 nirik, yes thats already helped with inkscape / gimp bugs 17:22:33 using alex's builds 17:22:39 Probably not in the first blush, but tools to hook up a git repo and do automated nightly builds would be fantastic too 17:22:47 we can also show to the app/module provider exactly what the end user is using when the bug was filed.. including force-patched fixes.. 17:23:07 sgallagh: aiming big: forget nightly. do it on each commit! 17:23:11 langdon: What is a force-patched fix? 17:23:29 mattdm: Probably unachievable for huge projects like kernel or glibc 17:23:40 sgallagh, that is my idea from above.. where oo failed the rebuild but the end user decided they wanted the security patch enough to force it on the module 17:23:55 sgallagh: pshaw, it's only hardware. ;) 17:24:07 ... 17:24:29 the kernel already has CI that does builds and testing on every commit 17:24:37 langdon: I think we'll have to come back to that one. I have concerns, but they're too low-level for this discussion 17:24:56 unfortunately, the kernel also supports so much diverse hardware that it is impossible to test all those CI builds on all the hardware, so bugs still happen a lot. 17:25:00 but.. suffice to say.. an end user can exactly report what binaries are in use in a particular module, you know that the module has been approved to use with those binaries (or are warned), and so bug reports are rarely "what the h*@&*@k did they have installed besides my app" 17:25:17 jwb: Doesn't building an armv7hl kernel take days? I know it took 22 hours for glibc on armv7hl this cycle... 17:25:34 sgallagh: the CI infrastructure is run upstream. it isn't a fedora/Red Hat thing 17:25:39 /me nods 17:25:51 sgallagh: iirc, it's Intel. i don't think ARM is tested, but i think it is built 17:26:17 sgallagh: and no, it doesn't take days even in fedora. it takes around 4-6 horus 17:26:21 OK, well it's a solvable problem if we get there. 17:26:30 okay guys i got a hard stop 17:26:32 we good we good? 17:26:43 ill send my summary to server list after i eat my food 17:27:04 * langdon jealous of mizmo with all her fancy "lunch" 17:27:34 heh all i have right now is hunger, i gotta figure out what "lunch" is before next meeting 17:27:59 I hear that. 17:28:07 mizmo, +1 17:28:09 OK, anyone have other questions? 17:28:25 Otherwise, we'll call it a day. Thank you very much for the participation. 17:28:32 this was really useful, thank you 17:29:47 #endmeeting