13:01:52 #startmeeting Env and Stacks (2014-09-16) 13:01:52 Meeting started Tue Sep 16 13:01:52 2014 UTC. The chair is hhorak. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:01:52 Useful Commands: #action #agreed #halp #info #idea #link #topic. 13:02:17 #chair pkovar tjanez samkottler bkabrda hhorak juhp mmaslano vpavlin sicampbell 13:02:17 Current chairs: bkabrda hhorak juhp mmaslano pkovar samkottler sicampbell tjanez vpavlin 13:02:36 hi 13:02:53 #topic init process 13:03:03 Hi! 13:03:18 hey! 13:03:29 hi 13:04:30 Regarding the ML discussion so far, I expect we'll have heated discussion, so let's start with it. 13:04:55 unless somebody wants to add something to the topics list.. 13:05:06 hhorak: i have one minor thing 13:05:30 bkabrda: what's that? 13:05:43 just before this meeting, I was ask by developers of Koschei, if we, as a group, could promote Koschei and recommend it as a good tool for all Fedora packagers 13:06:50 bkabrda: technically, I see it as a good idea, but do we have releng's POV if it does not break koji? (I mean so many scratch builds...) 13:07:21 CI? 13:07:24 hhorak: it does not, it's already been running for some time. it only runs scratch builds if overall Koji load is below a certain level 13:07:45 juhp_: sort of CI, yes 13:08:13 juhp_: it runs scratch builds of packages when their dependencies are updated, basically 13:08:23 #idea developers of Koschei would like to see Koschei promoted and recommend as a good tool for all Fedora packagers 13:08:45 thanks 13:09:06 #topic running Koschei for all Fedora packages (promote and recommend it) 13:09:08 sorry I was late folks - flaky internet :( 13:09:16 #link http://koschei.cloud.fedoraproject.org/ 13:09:23 #link https://github.com/msimacek/koschei 13:09:52 ncoghlan: hi, you're not late, we haven't started with repositories yet :) 13:10:25 hhorak: I don't think we should run it for all. we should just promote it so that packagers themselves suggest important packages to run there 13:10:53 the problem is that if too many packages are there, they'll be rebuilt in very long intervals not to overload koji 13:11:04 currently it is just run for http://koschei.cloud.fedoraproject.org/groups ? 13:11:06 so we should really stick to "the important" 13:11:14 bkabrda: so do we need more machines? 13:11:16 juhp_: afaik, yes 13:11:45 mmaslano: not right now, but if we were to add many more packages, then likely yes 13:12:05 let's add glibc ;oP just kidding 13:12:15 #info Koschei only runs scratch builds if overall Koji load is below a certain level, but too many packages would mean they'll be rebuilt in very long intervals not to overload koji 13:12:22 I guess it should work in a way "we recommend it, packagers start using it, we find out how much HW we need) 13:12:52 how do people opt in currently? 13:13:00 having fought those kinds of battles a fair bit, good metrics can be powerful for making the case for more machines 13:13:01 also, I think it's Koji that would need the new HW, not koschei server itself :) 13:13:24 bkabrda: right 13:13:59 juhp_: right now, I think you need to write to msimacek 13:14:07 (at redhat.com) 13:14:34 but AFAIK there are plans to create a dynamic web UI, where packagers would be able to add/remove packages themselves 13:14:37 * kushal is here too 13:14:49 bkabrda, okay - or pull request? :) I see 13:14:57 bkabrda: that sounds fine to me. Also manually acking packages seems fine to me, so we don't get into troubles in the beginning.. 13:15:06 sure 13:15:15 juhp_: yeah, I guess. I'm not *that* familiar with koschei 13:15:44 hhorak: yes, for the time being. I agree 13:15:52 #info there are plans to create a dynamic web UI, where packagers would be able to add/remove packages themselves 13:16:15 maybe I should mention it to the Workstation WG - sounds like it could also be useful for gnome etc - though they have their own CI too upstream I think 13:16:44 so my proposal is that people from our group should find out more about koschei; if everyone likes it, we can vote on some sort of formal recommendation next time 13:16:56 sounds good? 13:17:22 sure +1 sounds like a useful tool 13:17:25 bkabrda: yeah, making an AI for everybody.. 13:17:44 #action everbody should find out more about koschei; if everyone likes it, we can vote on some sort of formal recommendation next time 13:17:57 thanks :) 13:18:24 I don't think I/we can use it for Haskell packages though because ghc is too strict about version deps 13:19:04 juhp_: yeah, it really depends on the respective SIGs/packagers 13:19:06 (as we have to rebuild anyway) 13:19:14 sure 13:19:28 juhp_: if we identify packages not suitable, we should track them.. 13:19:35 right 13:19:45 anyway, ready for moving to the next topic? 13:20:04 hhorak: yep 13:20:56 #topic Language specific mirrors for Fedora Playground compliant packages 13:20:56 #info the idea was a bit discussed on ML already: https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-September/000507.html 13:21:32 OK, I'll start with a bit of context 13:21:46 ncoghlan: sure, go on 13:22:46 for several of Red Hat's internal tools, we currently deploy fully via RPM 13:22:46 for all the components of the web services 13:22:48 with more deployment options (like OpenShift) becoming available, we're reevaluating whether that makes sense 13:22:57 and for the projects where we run the one-and-only instance in the world, it really doesn't 13:23:21 we'd like to move those to a more software-as-a-service type deployment model 13:23:51 and in those cases, because it's the dev team responsibile for updates, the RPM format doesn't add any benefit at the application layer 13:24:06 what *is* still useful though is Fedora's review processes 13:24:19 when we go direct to upstream, we bypass all that 13:25:17 #info fully deployment in RPM for everything does not make sense in projects run one-and-only instance in the world 13:25:20 which would mean repeating the work 13:25:21 it would also mean when we review new packages, Fedora wouldn't benefit 13:25:23 hence the idea of being able to have a mirror where we could do the initial Fedora Playground checks 13:25:52 so you are after curated repos? 13:25:56 *without* the next step of setting up a COPR for the package 13:26:00 yeah 13:26:09 because we're going to have to do that work anyway 13:26:19 and I think it makes more sense to offer to do it upstream 13:26:26 rather than inside the firewall 13:27:08 we could seed the repos by flagging the packages that have already been accepted into Fedora 13:27:58 I'm confident we can make it work for at least Python, since devpi is well set up to handle a use case like this 13:28:28 #link http://doc.devpi.net/latest/ 13:29:17 I'm less sure of the mechanics when it comes to other languages 13:29:21 I have to admit that I've been playing with this idea for few weeks myself and I think there may be other benefits to it... and I agree that for Python, devpi is the way to go with this 13:29:38 I would be fine with trying it at least for languages, which make sense 13:29:51 if ncoghlan and bkabrda are saying it might work for Python, then let's try it 13:30:00 the other benefit I liked is a more incremental review path for new packages 13:30:15 I'm not sure about all languages, they have their specifics and not so linux based upstreams 13:30:16 language mirror -> Fedora Playground -> Fedora 13:30:39 Haskell has something similar to this too called Stackage (a subset of the main Hackage package source repo) 13:30:53 bkabrda, right - I think licensing is just a small component 13:32:03 consistent dependencies of package sets etc 13:32:17 so other benefit could be, that we would offer prebuilt binaries (the new Python packaging format called wheel) of upstream packages that have C extensions (and always fail installation from PyPI, since one needs gcc and *-devel packages) 13:32:34 making our devs happier about not having to learn RPM is a bonus on my end, but not one I'd expect to matter so much to the WG :) 13:33:24 is there an rpm packaging tool for python packages? 13:33:29 bkabrda: yes, that's on my list of benefits too 13:33:30 pretty sure it's a Python specific one, though 13:33:34 #info the idea is to be able to have a mirror of upstream repositories where we could do the initial Fedora Playground checks 13:33:34 #info for python devpi seems to be the way to go with this 13:33:34 #info we could seed the repos by flagging the packages that have already been accepted into Fedora, the incremental review model could look like language mirror -> Fedora Playground -> Fedora 13:33:35 #info other languages may have their specifics and not so linux based upstreams 13:33:56 juhp_: bkabrda's pyp2rpm 13:34:03 cool 13:34:13 ncoghlan: yes, but I'm assuming that every language/ecosystem will just have different set of advantages 13:34:22 so in theory the devpi -> COPR could be automated as well 13:34:33 ncoghlan, sure 13:34:44 ncoghlan: in theory and with some hints for pyp2rpm, yes :) 13:34:44 bkabrda: yeah 13:35:36 bkabrda: working on it! One day we'll get to the end of the metadata 2.0 saga and actually have a usable extension system :) 13:36:16 there is always problem with bundling in upstream repos, what would happen in this regard in the Fedora's repo? would we care or we would deliberately not care? 13:36:41 for the mirror & Playground, I'd say we wouldn't worry about it 13:36:54 hhorak: deliberately not care, I think. we don't care in COPR/Playground, either 13:37:08 fixing those kinds of issues is where the policy compliance step in getting into Fedora proper comes in 13:38:14 this is aimed at teams like mine where we're happy to handle the maintenance implications 13:38:25 #info un-bundling and similar maintaining work would be solved in the step 'playground -> Fedora' 13:38:42 ncoghlan: can you solve problems with depedencies outside of python modules? Let's say your module depends on some system library in exact version 13:38:44 rather than those where sysadmins are trying to deploy apps without the aid of a dedicated developer team 13:38:57 ncoghlan: many languages don't specify such data in their metadata 13:39:10 mmaslano: that's where the wheel binary format comes in 13:39:30 ncoghlan: magically solved, I understand :) 13:39:46 mmaslano: and some neat tricks in devpi that let us do the binaries on a Fedora/EPEL basis 13:40:09 while having a shared mirror for the source packages 13:40:36 longer term, we want to solve the external deps problem, but that's rather complicated :) 13:41:00 ncoghlan: I'd even say generally not solvable, but we can talk about that some other time 13:42:26 ncoghlan: I thought external deps, thanks 13:42:44 I guess remember a list of known external deps 13:42:45 so how would the source differ from upstream repo? would it actually differ or it would stay deliberately untouched like real mirror? 13:42:52 anyway, what I still haven't figured out (not even for myself) is whether we would want to keep older versions of packages... or just provide newest ones/just provide selected versions/... 13:42:56 bkabrda: there's a reason I'm starting with the distro specific repo approach instead :) 13:43:30 hhorak, bkabrda: I think those are open questions we don't need to decide here 13:43:45 hhorak: that's another fair question that I haven't really found out answer to (yet :)) 13:44:38 for tonight, I'd be looking to answer "What's the next step?" 13:44:40 a Python/devpi pilot as an F22 proposal? 13:45:13 ncoghlan: I think we need to have these things figure out to create a proposal out of it 13:45:15 we at least have PEP 440 approved now, so if we wait for pip 1.6, we'll be able to post patched packages without version conflicts with upstream 13:45:59 ncoghlan: IMO we should first create the devpi instance and start experimenting with different ways of using it; then when we figure out what/how we want to do, we can create a Fedora feature 13:46:51 bkabrda: OK, so maybe the next step is to create a draft for discussion at the next WG meeting? 13:47:28 ncoghlan: yep. and I guess I can request an instance in Fedora cloud to start playing around with devpi 13:47:43 bkabrda: sounds good 13:48:20 one question mark in my mind is how well it would cope with a 15k+ package white list 13:49:06 ncoghlan: I guess we'll find out (and improve where needed) :) 13:49:31 #info dependencies on some system libraries are complicated, could be partly solved by new format wheel and some tricks in devpi 13:49:32 #info we should first create the devpi instance and start experimenting with different ways of using it; then when we figure out what/how we want to do, we can create a Fedora feature 13:49:39 as I haven't looked into the details of how that is implemented 13:49:40 still, "try it and see" is likely the easiest way to answer that :) 13:49:57 ncoghlan: same here :) 13:50:08 #action bkabrda will request an instance in Fedora cloud to start playing around with devpi 13:50:28 if you have some plan, we could look at other languages 13:50:54 my main problem in those cases is I don't know the mirroring tools at all 13:52:01 I've started bugging some of the other devs in the Brisbane office to join the WG 13:52:02 or, rather, the mailing list 13:52:08 since we have Perl, Ruby and Java apps to consider as well 13:52:29 ncoghlan: around me are sitting people with knowledge of all these languages 13:52:42 ncoghlan: you can stop thinking about Java, msrb is looking at maven :) 13:52:49 and they have special needs 13:53:07 but other languages group can be inspired by your approach, or might not. 13:53:10 mmaslano: it's also in the other direction, though - these are the folks that will be *using* it to deploy internal apps 13:53:53 I guess more opinions from people with different perspectives won't hurt :) 13:53:54 ncoghlan: ok, they can talk :) 13:54:22 I know what will work for the Python apps 13:54:22 but don't have the same level of knowledge for our other apps 13:54:27 anyway, we have somewhere to start 13:54:57 if it expands beyond Python over time, cool :) 13:55:07 back to draft for discussion for some of the next WG meetings -- do we have a volunteer who will prepare something after some initial testing? 13:55:47 I'm happy to do a write-up after we experiment a bit with bkabrda's devpi instance 13:56:23 ncoghlan: bkabrda: great, thanks! 13:56:55 #action ncoghlan will do a write-up after we experiment a bit with bkabrda's devpi instance 13:57:02 well, I can start the write-up straightaway 13:57:03 and fill in more details as we try different things 13:57:13 ncoghlan: even better :) 13:59:35 So, I'm still thinking how this could work in practice.. I see why developers would like to work with our new shiny repos rather than with RPMs. 13:59:35 But I'm still uncertain why they would not use upstream repo directly? Or they would not notice? 14:00:16 hhorak: in our case, it's because we want the additional level of review that Fedora provides 14:00:28 hhorak: speaking for Python, one advantage would be that they would get precompiled C-extension packages 14:00:42 and yeah, the ability to post wheel files 14:00:46 so they wouldn't need gcc and "whatever-devel" packages on their machines 14:01:04 PyPI doesn't allow those for Linux yet, because we need to figure out the external dependencies problem 14:01:25 and that's a nightmare in the general case for Linux 14:01:33 but eminently solvable in the distro specific case 14:01:41 binaries? wouldn't they violate copr policy? 14:01:59 juhp_: for copr, we would rebuild the upstream source again 14:02:03 okay 14:02:16 at least that's my idea of how we would do that 14:02:24 bkabrda: yeah, agreed 14:02:43 devpi actually allows us to keep the source and wheel files in separate indices 14:03:13 okay cool 14:03:31 with the per-version wheel indices inheriting from a shared source index 14:04:28 wheels are distro specific? 14:04:34 #info reasons why developers would not use upstream repo directly and rather use Fedora's playground is because they might want the additional level of review that Fedora provides + they would get precompiled C-extension packages 14:05:00 juhp_: binary dependencies on Linux are distro specific 14:05:11 hhorak: I'm assuming that we can come up with more reasons, these are just some that occured right now :) 14:05:39 juhp_: and since the wheels can contain built binaries linked to distro binaries... 14:05:48 ok 14:06:37 bkabrda: I hope there are 14:07:48 anyway having a devpi instance for Fedora/EPEL sounds useful 14:08:05 hhorak: from my point of view, one reason is "we're going to set up a private PyPI mirror for Red Hat's internal use anyway, and hosting that in Fedora rather than inside the firewall means it might be useful to others" 14:08:06 the amount of work is basically the same for us either way 14:08:55 ncoghlan: right, sounds more than fine to me. 14:10:38 So, enough interest to discuss another topic or should we move it to the next meeting? (SCLs, building above them and their position in Fedora/EPEL) 14:10:55 ncoghlan, the problem may be more about getting consensus on the policy for the "mirror" 14:11:07 (I should ask first if we're done with the current topic already :) ) 14:11:45 it just ticked over to Wednesday morning here - did we want to move on to SCLs for a bit? 14:12:03 juhp_: aye, agreed 14:12:07 +1 for moving to SCL 14:12:21 let's move. we'll continue next week either way :) 14:14:45 #topic SCLs, building above them and their position in Fedora/EPEL 14:15:14 as far as I know some projects are not packaging their projects as SCL and they are using SCL 14:15:33 one reason is that they have to run their project on Fedora (no SCL) and on CentOS 14:16:10 SCL packaging doesn't have to be mandatory if you use smart enough wrapper script for setting up environment 14:17:00 right 14:17:52 it also works if even a container would be too heavy, but you want to decouple your language runtime upgrade cycles from the OS one 14:17:53 my recommendation would be not do SCL mandatory, it's up to users 14:18:25 mmaslano, I tend to agree 14:18:34 that is, still use an SCL on Fedora, but it might be *older* than the system runtime 14:19:28 I view the 4 approaches (parallel install, normal package depending on SCL, SCL depending on SCL, full container) as handling different use case 14:19:34 I know mainly about OpenShift use-case. They have lot of ifdefs for various systems.. 14:19:37 mmaslano: I totally agree, but it also kind of suggests supporting those wrappers, which already was discussed a bit, but not solved yet.. 14:20:04 I'm not sure how they are doing with containers, but they've planned to use wrapper inside the container to switch the version of language 14:20:14 I guess that didn't work well for Python 14:21:13 hhorak: it may also be a case where if 14:21:13 "normal packaging depending on SCL" gets unmaintainable 14:21:13 it's time to consider "SCL depending on SCL" or "container image" 14:22:10 yes 14:22:35 yeah, I'm kind of confused when thinking about a python app build for SCL that would not be SCL -- basically where the files would be stored? 14:22:48 I'm not sure - virtualenv does let you switch the Python runtime in such a way most Python software will adjust automatically 14:23:10 hhorak: consider a single file Python script installed into /usr/bin 14:23:32 hhorak: with the support files installed into the SCL 14:23:35 hm we discussed it many times and Python sounds hard, because you can't get rid off it from minimal install 14:23:40 of image 14:23:54 that's a "normal package 14:24:02 but you need the SCL for the Python runtime 14:24:34 it's a reasonable thing to want to do, but AFAIK the tools to get the shebang line and spec file right don't exist yet 14:24:59 ncoghlan: I could ask OpenShift guys how they've solved it if you want 14:25:14 well I ask anyway :) I'm curious how they are doing with SCL and images 14:25:32 note I'm also only considering the case where you *always* use the SCL, even on Fedora 14:25:49 yeah, I'd be interested to know 14:26:19 at the moment, we just run our command line clients in the system language runtimes 14:26:33 * juhp_ sometimes thinks we should only have SCLs ;) 14:26:37 #action mmaslano will figure out how OpenShift is using SCL in their images 14:27:28 juhp_: there are some spectacular rants online about Linux distros "ruining" dynamic languages by shipping old runtimes :) 14:27:40 :) 14:27:48 ncoghlan: that's old :) 14:28:06 * mmaslano needs to go to another meeting, bye for now 14:28:15 #info SCL packaging doesn't have to be mandatory if you use smart enough wrapper script for setting up environment 14:28:15 #info tools to get the shebang line and spec file right for non-SCL python packages depending on SCL might not exist yet 14:28:29 and I will head to bed - thanks folks, looking forward to working more with you! 14:28:52 ncoghlan: thanks for coming and see you! 14:29:27 ncoghlan, good night 14:29:46 I guess we can close the meeting, thanks everybody for joining! 14:29:46 Anybody interested in doing a chairman the next week? 14:31:35 Since nobody responds, the volunteer would probably be me :) Never mind, I can do it.. 14:31:35 Will close the meeting in 1 minute... 14:32:05 hhorak, sorry I rather wait for another week - next week not so good for me 14:32:18 juhp_: Sure, no problem. 14:32:36 hhorak, thanks 14:32:45 #action hhorak will be chairman for the next meeting 14:32:45 #endmeeting