12:01:52 #startmeeting Env and Stacks (2014-12-10) 12:01:52 Meeting started Wed Dec 10 12:01:52 2014 UTC. The chair is hhorak. Information about MeetBot at http://wiki.debian.org/MeetBot. 12:01:52 Useful Commands: #action #agreed #halp #info #idea #link #topic. 12:01:59 hi! 12:02:03 #meetingname env-and-stacks 12:02:03 The meeting name has been set to 'env-and-stacks' 12:02:07 #chair pkovar tjanez samkottler bkabrda hhorak juhp ncoghlan vpavlin sicampbell 12:02:07 Current chairs: bkabrda hhorak juhp ncoghlan pkovar samkottler sicampbell tjanez vpavlin 12:02:19 #topic welcoming... 12:02:21 Hi! 12:02:26 Hi 12:02:34 hi 12:02:37 evening all 12:03:52 #topic Follow-up: languages repositories 12:04:19 so, I finally got to experimenting with devpi 12:04:25 bkabrda: ++ 12:04:36 I've got good news and bad news (but nothing that can't be implemented) 12:04:45 let me start with the "bad" news 12:05:02 * vpavlin likes bad news first approach...:) 12:05:15 right now, devpi doesn't allow saying "these packages can't be mirrored" 12:05:40 or "just these packages can be mirrored" 12:06:28 I originally thought that the pypi_whitelist argument was used for this, but that was my misunderstanding of documentation 12:06:33 ummm, may I ask why are we starting with devpi and are not going directly with Pulp? I have read the log from the last meeting and didn't find this info 12:07:18 because I'm working on devpi and started speaking first :) I'm sure we'll get to pulp 12:07:33 bkabrda: fair enough :-) 12:07:43 jzeleny: I don't think there's a better reason than "we thought of devpi first" :) 12:07:57 the pulp idea came later 12:08:14 so as for the problem, I've already opened a bug upstream and I'll work with them to get this functionality of restricting mirrored packages to a specified set implemented 12:08:19 well, I was jsut curious if this time investment is wort it if we are going to drop it eventually :-) 12:08:24 also, both bkabrda & I misread the devpi docs and thought it already did what we were after 12:08:26 but ok, go on ... 12:08:35 ah, ok 12:08:44 jzeleny: we are going to drop it eventually? I was not aware of that 12:09:00 #info devpi was chosen for proof-of-concept just because it was first on mind, but it might be pulp in the future 12:09:02 so we are going to have both Pulp and devpi? 12:09:18 until someone comes up with an ecosystem *other* than Python where this model might work 12:09:41 devpi should suffice 12:09:47 I'm not sure, to be honest I haven't really compared pulp and devpi capabilities 12:10:07 I haven't either - I know the pulp_python plugin exists 12:10:15 but I haven't asked the Pulp folks how mature it is 12:10:18 or what it supports 12:10:38 ncoghlan: I saw a presentation couple months back 12:11:03 back then it was a work in progress but it's one of the few that are almost production-ready 12:11:27 it occurs to me that "lack of docs for Pulp's Python support" is an objective reason for why we started with devpi 12:11:36 it might be worth exploring this option before we get too deep with devpi, as it might block other languages to adopt this concept in the future 12:12:09 I don't think adopting devpi could block anyone... but I need to have a closer look at the pulp_python plugin, that's for sure 12:12:32 #info devpi doesn't allow saying "these packages can't be mirrored"; a bug upstream is opened, work in progress 12:12:57 we'd want to be wary of putting devpi specific assumptions into other tooling 12:13:11 bkabrda: well, if we use Pulp to manage python repos, it should be quite easy to extend this to other languages - just install more plugins 12:13:19 bkabrda: you can't do the same with devpi I assume 12:13:23 I am pretty sure we could switch to pulp after the PoC phase - at least we will have something to point at and say "Hey, we need these fetures in Pulp":) 12:13:36 jzeleny: no, but I don't see how using devpi would *block* someone 12:14:06 if we end up integrating with koji, for example, we'd want the upload code to be language neutral 12:14:06 ok, poor phrasing on my side ... not block, rather discourage 12:14:46 but vpavlin has a point - the PoC can be easily done with devpi so we can see what features do we require 12:15:17 OK, how does this work for a summary: 12:15:21 but I would like to revisit this discussion after that 12:15:28 well, the point is that I'd hate to spend X days working on devpi if we're going to throw it away eventually. it'd be better to work on pulp_python plugin from the start... 12:15:33 PoC with devpi makes sense, since it's already native to the Python ecosystem 12:16:06 and hence avoids Pulp plugin limitations as a potentially confounding factor 12:16:06 ncoghlan: pretty good ... as I said, I would just like to revisit the discussion after we have the PoC 12:16:34 my point being the potential for the future 12:16:54 sounds reasonable 12:16:59 ok, I'll keep working on the devpi pilot and also keep an eye on pulp_python 12:17:01 I don't think anyone will want to maintain repos for 5 languages that are using 5 different technologies :-) 12:17:30 bkabrda: ack 12:17:36 right, that's the train of thought where the "sustaining engineering" half of my brain went "run awaaay" :) 12:17:47 bkabrda: I will try to find out what's the status on the Pulp python plugin 12:18:07 #info PoC with devpi makes sense, since it's already native to the Python ecosystem and hence avoids Pulp plugin limitations as a potentially confounding factor; thus let's keep working on the devpi pilot and also keep an eye on pulp_python 12:18:08 jzeleny: good, if you send out some mails, please put me on CC 12:18:11 however, I think our limiting factor at this point is likely to be the fact most upstream ecosystems don't really support redistributors natively anyway 12:18:17 bkabrda: will do 12:18:21 thx 12:18:48 so let me get back to good news :) 12:20:32 good news is, that devpi lets you say that a package should only be installed from an uploaded archive, even if newer version is available on PyPI (i.e. you can "shadow" the PyPI package completely by uploading your package of the same name) 12:20:52 *uploaded archive == uploaded to your devpi instance 12:21:03 (not sure if I'm explaining this the right way...) 12:21:40 I think so - if something is *in* our devpi instance, devpi won't update it unless we tell it to 12:21:56 for example, let's say we want to mirror django only in versions that are in a fedora release - we just need to upload the tars in specific versions to our devpi instance 12:22:07 bkabrda: is it possible to revert this decision? 12:22:14 the other versions that are on pypi won't be offered by devpi for installation 12:22:29 jzeleny: yes. that is what the pypi_whitelist is for, actually :) 12:22:49 you just add a package to pypi_whitelist and suddenly all versions from pypi are mirrored again 12:23:22 cool 12:23:26 (and you can revert the reverted decision...) :) 12:23:27 #info good news is, that if something is *in* our devpi instance, devpi won't update it from pypi unless we tell it to using pypi_whitelist 12:23:38 bkabrda: I was just going to ask :-) 12:23:45 hhorak: nice wording! :) 12:23:46 or upload a newer version? :) 12:24:15 juhp_: yes. the point is that for uploaded packages you have a very good control 12:24:50 either you add the package to pypi_whitelist or you upload precisely the versions you want to be available 12:25:16 and that's about everything I have, except that information that my devpi21 SCL works just fine 12:25:24 bkabrda: so the only thing we're missing is the ability to have a global whitelist? 12:25:30 ncoghlan: yes, precisely 12:25:36 cool 12:25:44 ncoghlan: I opened a bug for that here: https://bitbucket.org/hpk42/devpi/issue/198/whitelisting-packages-that-can-be-mirrored 12:25:45 * hhorak thinks we should somehow track these requirements (for devpi functionality) so we have already written them for pulp or other similar tools in the future 12:26:17 #link https://bitbucket.org/hpk42/devpi/issue/198/whitelisting-packages-that-can-be-mirrored 12:26:52 hhorak: would listing them on the LSR project page be enough for now? 12:28:08 actually, it problem makes sense to make that one of the goals of the pilot - defining our requirements for a more permanent solution 12:28:31 ncoghlan: sure, that was what I wanted to propose 12:29:04 bkabrda: can I give you AI for adding the news into the wiki page? 12:29:36 hhorak: sure 12:30:03 #action bkabrda will list requirements/news learned for devpi on the LSR project page 12:30:35 I also think I'm getting a pretty good idea about what/how we want to mirror/prebuild, but that's for a longish discussion and I need to put it on a paper first. so maybe for next time I'll prepare a draft with proposal 12:31:04 bkabrda: greawt 12:32:01 #action bkabrda will prepare a draft document about what/how we want to mirror/prebuild on the pilot devpi instance 12:32:14 (just a reminder for myself :) 12:35:31 I also wanted to quickly summarize the short internal discussion we've had this week about LSR, about where are we going to with LSRs.. Very quick summary could be that eventually we aim to have most software delivered as images (I like Nick's comparison of images to walls) but we need to have bricks to build them (which will be packages in those repositories). 12:36:42 I really should flesh that analogy out into a blog post at some point 12:36:49 any other thought we should mention to get some public attention hopefully? 12:37:15 hhorak: just a note, I don't believe what we want are images per se 12:37:42 ncoghlan: +1, I think that's a great analogy. I think people are so crazy about containers that they don't realize they need to install their dependencies *somehow* into their containers 12:37:43 I would rather use the application analogy used by Gnome Software 12:38:27 bkabrda, right 12:38:43 ncoghlan, what's a wall? :) 12:39:17 now I'm trying to remember Langdon's comment that I was replying to :) 12:40:10 jzeleny: I'm afraid application may be easily messed up with a package, but generally agree with you 12:40:27 I think his point was that if everything is getting deployed as images or prebuilt ostree's, the ops side aren't going to be as worried about the dependency management systems 12:40:58 * langdon not getting enough context to help out ncoghlan 12:41:17 * langdon also still *always* reads "AI" as artificial intelligence 12:41:27 and my counter was that while that's true, the dev side still need a way to define those images and ostree's in the first place 12:42:00 langdon: that's what we're heading to, right? :) 12:42:31 so the analogy was that image/ostree = prefabricated wall, individual package = brick, dependency manager = an automated bricklayer :) 12:42:33 one more quick point about a pulp investigation, must ensure that devpi can mirror it in case a consumer wants to replicate the replication but only for python [insert language here] 12:42:47 ncoghlan, ah 12:43:16 langdon: good point. I'll find out whether that's possible 12:44:11 ncoghlan, ahh.. i don't think i saw that analogy yet.. but yes, i think that is not a bad one.. :) 12:44:23 langdon: just out of curiosity, would that still be the case if we used Pulp from the beginning? 12:44:31 #action bkabrda will find out whether devpi can mirror pulp's pulp_python repo 12:45:11 jzeleny: the important thing about devpi is that it's also supposed to be deployed on developer workstations to help them with certain development steps 12:45:19 jzeleny, i think so.. because consumers of the fedora replication (or whatever it is being called) may not want to use pulp.. that said.. it may not be a blocker.. but we should know the answer 12:45:46 jzeleny: so a company may use pulp with pulp_python as their *central* repo, but developers may want to use devpi as their local workstation mirrors 12:45:47 bkabrda: hm, I have probably missed that point ... is there some docs about this? 12:45:59 bkabrda: I'm also curious to know what the Pulp folks are using as a test suite - I know dstufft wants a clear implementation independent test suite for PyPI as part of Warehouse development, but I'm not sure if it exists yet 12:46:09 like we may have to say "to use this stuff you need to install pulp" but we want to know if "you can also do it with just devpi" 12:46:17 jzeleny: upstream docs about using it this way, if that's what you're interested in 12:47:09 bkabrda: well, I was rather interested in some background why would developers use devpi on their machines ... so if the rationale is there, I guess that's what I want to see :-) 12:47:35 jzeleny: yes, the rationale is there... let me find a link 12:48:00 jzeleny: local PyPI cache so they can work offline, local testing of package build and upload, that kind of thing 12:48:19 jzeleny: or maybe it's better to say that the rationale is obvious once you learn about all the cool features... like the ones ncoghlan just mentioned :) 12:48:22 jzeleny: http://doc.devpi.net/latest/quickstart-releaseprocess.html 12:48:48 bkabrda: thanks 12:53:35 #link http://doc.devpi.net/latest/quickstart-releaseprocess.html 12:54:19 * hhorak needs to leave in 7 minutes, can we shortly touch SCLs? 12:54:33 #topic Follow-up:SCLs 12:54:57 I planned to have already summarized until today all what was said, proposed and agreed on about SCLs in Fedora.. But.. There has been much more said than I thought so don't have it ready. 12:56:12 What I just wanted to quickly touch is some rough plan. 12:56:55 My simple proposal would be adopting the agreed names/prefixes/vendor and prepare some example collection (collections for fedora on softwarecollections.org do not use those) 12:58:25 aha 12:58:58 hhorak: just to make sure... where were the names/prefixes/vendor agreed on? 13:00:34 bkabrda: not sure if really agreed, but I have a feeling we got to some conclusion working for all, didn't we? like using /opt/fedora.. but that's still something I want to make sure.. 13:01:14 I'd be in favour of agreeing to something like that to get things moving forward 13:01:47 hhorak: yeah, I just wanted to know what precisely we're going to go with. there were lost of agreements in the past... ;) 13:02:28 * langdon wonders is bkabrda made a freudian slip 13:02:33 s/is/if 13:02:35 what I saw as a big issue was trying to include SCLs into base Fedora, which is something I'd like to avoid now, since it seems to produce many blockers... 13:03:18 langdon: I do these all the time ;) 13:03:57 hhorak: I'm not sure it even makes sense, as the main advantage of SCLs from my perspective is they're decoupled from the OS lifecycle 13:04:56 parallel language runtimes at the core OS layer is rather suboptimal (says the core Python developer...) 13:06:14 ncoghlan: that's what I agree with.. 13:06:21 well, I need to go now, so the important note from me is probably that I don't have anything ready now but I'm working on it :) 13:06:46 hhorak: I was agreeing with you - just puzzled by the original suggestion of doing it :) 13:07:03 ncoghlan: ah, ok :) 13:07:10 could anybody end meeting after there is nothing else to discuss? 13:07:15 ncoghlan, at the time there was "base" and "not fedora" ... nothing in between 13:07:23 * hhorak waiving! bye... 13:07:41 langdon: ah, OK - that makes more sense :) 13:07:43 personally i think one of envs&stacks primary "goals" is to provide options like this 13:08:28 langdon: +lots :) 13:08:28 not that i nec. agree with your base stmt ;) .. but that is for another, preferably beer-laden, day 13:08:44 heh 13:09:01 langdon, yes it is 13:11:20 SCLs was the last item on preplanned agenda 13:14:03 shall we finish then? 13:14:09 I think so 13:14:39 need a volunteer to chair next week, unless we volunteer hhorak in absentia :) 13:19:09 I guess it falls to hhorak by default... 13:20:01 anything else? Otherwise I'll end the meeting 13:20:41 OK - thanks folks, catch you next week 13:20:46 #endmeeting