12:01:52 <hhorak> #startmeeting Env and Stacks (2014-12-10)
12:01:52 <zodbot> Meeting started Wed Dec 10 12:01:52 2014 UTC.  The chair is hhorak. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:01:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
12:01:59 <bkabrda> hi!
12:02:03 <hhorak> #meetingname env-and-stacks
12:02:03 <zodbot> The meeting name has been set to 'env-and-stacks'
12:02:07 <hhorak> #chair pkovar tjanez samkottler bkabrda hhorak juhp ncoghlan vpavlin sicampbell
12:02:07 <zodbot> Current chairs: bkabrda hhorak juhp ncoghlan pkovar samkottler sicampbell tjanez vpavlin
12:02:19 <hhorak> #topic welcoming...
12:02:21 <hhorak> Hi!
12:02:26 <vpavlin> Hi
12:02:34 <nphilipp> hi
12:02:37 <ncoghlan> evening all
12:03:52 <hhorak> #topic Follow-up: languages repositories
12:04:19 <bkabrda> so, I finally got to experimenting with devpi
12:04:25 <hhorak> bkabrda: ++
12:04:36 <bkabrda> I've got good news and bad news (but nothing that can't be implemented)
12:04:45 <bkabrda> let me start with the "bad" news
12:05:02 * vpavlin likes bad news first approach...:)
12:05:15 <bkabrda> right now, devpi doesn't allow saying "these packages can't be mirrored"
12:05:40 <bkabrda> or "just these packages can be mirrored"
12:06:28 <bkabrda> I originally thought that the pypi_whitelist argument was used for this, but that was my misunderstanding of documentation
12:06:33 <jzeleny> ummm, may I ask why are we starting with devpi and are not going directly with Pulp? I have read the log from the last meeting and didn't find this info
12:07:18 <bkabrda> because I'm working on devpi and started speaking first :) I'm sure we'll get to pulp
12:07:33 <jzeleny> bkabrda: fair enough :-)
12:07:43 <ncoghlan> jzeleny: I don't think there's a better reason than "we thought of devpi first" :)
12:07:57 <ncoghlan> the pulp idea came later
12:08:14 <bkabrda> so as for the problem, I've already opened a bug upstream and I'll work with them to get this functionality of restricting mirrored packages to a specified set implemented
12:08:19 <jzeleny> well, I was jsut curious if this time investment is wort it if we are going to drop it eventually :-)
12:08:24 <ncoghlan> also, both bkabrda & I misread the devpi docs and thought it already did what we were after
12:08:26 <jzeleny> but ok, go on ...
12:08:35 <jzeleny> ah, ok
12:08:44 <bkabrda> jzeleny: we are going to drop it eventually? I was not aware of that
12:09:00 <hhorak> #info devpi was chosen for proof-of-concept just because it was first on mind, but it might be pulp in the future
12:09:02 <jzeleny> so we are going to have both Pulp and devpi?
12:09:18 <ncoghlan> until someone comes up with an ecosystem *other* than Python where this model might work
12:09:41 <ncoghlan> devpi should suffice
12:09:47 <bkabrda> I'm not sure, to be honest I haven't really compared pulp and devpi capabilities
12:10:07 <ncoghlan> I haven't either - I know the pulp_python plugin exists
12:10:15 <ncoghlan> but I haven't asked the Pulp folks how mature it is
12:10:18 <ncoghlan> or what it supports
12:10:38 <jzeleny> ncoghlan: I saw a presentation couple months back
12:11:03 <jzeleny> back then it was a work in progress but it's one of the few that are almost production-ready
12:11:27 <ncoghlan> it occurs to me that "lack of docs for Pulp's Python support" is an objective reason for why we started with devpi
12:11:36 <jzeleny> it might be worth exploring this option before we get too deep with devpi, as it might block other languages to adopt this concept in the future
12:12:09 <bkabrda> I don't think adopting devpi could block anyone... but I need to have a closer look at the pulp_python plugin, that's for sure
12:12:32 <hhorak> #info devpi doesn't allow saying "these packages can't be mirrored"; a bug upstream is opened, work in progress
12:12:57 <ncoghlan> we'd want to be wary of putting devpi specific assumptions into other tooling
12:13:11 <jzeleny> bkabrda: well, if we use Pulp to manage python repos, it should be quite easy to extend this to other languages - just install more plugins
12:13:19 <jzeleny> bkabrda: you can't do the same with devpi I assume
12:13:23 <vpavlin> I am pretty sure we could switch to pulp after the PoC phase - at least we will have something to point at and say "Hey, we need these fetures in Pulp":)
12:13:36 <bkabrda> jzeleny: no, but I don't see how using devpi would *block* someone
12:14:06 <ncoghlan> if we end up integrating with koji, for example, we'd want the upload code to be language neutral
12:14:06 <jzeleny> ok, poor phrasing on my side ... not block, rather discourage
12:14:46 <jzeleny> but vpavlin has a point - the PoC can be easily done with devpi so we can see what features do we require
12:15:17 <ncoghlan> OK, how does this work for a summary:
12:15:21 <jzeleny> but I would like to revisit this discussion after that
12:15:28 <bkabrda> well, the point is that I'd hate to spend X days working on devpi if we're going to throw it away eventually. it'd be better to work on pulp_python plugin from the start...
12:15:33 <ncoghlan> PoC with devpi makes sense, since it's already native to the Python ecosystem
12:16:06 <ncoghlan> and hence avoids Pulp plugin limitations as a potentially confounding factor
12:16:06 <jzeleny> ncoghlan: pretty good ... as I said, I would just like to revisit the discussion after we have the PoC
12:16:34 <jzeleny> my point being the potential for the future
12:16:54 <juhp_> sounds reasonable
12:16:59 <bkabrda> ok, I'll keep working on the devpi pilot and also keep an eye on pulp_python
12:17:01 <jzeleny> I don't think anyone will want to maintain repos for 5 languages that are using 5 different technologies :-)
12:17:30 <jzeleny> bkabrda: ack
12:17:36 <ncoghlan> right, that's the train of thought where the "sustaining engineering" half of my brain went "run awaaay" :)
12:17:47 <jzeleny> bkabrda: I will try to find out what's the status on the Pulp python plugin
12:18:07 <hhorak> #info PoC with devpi makes sense, since it's already native to the Python ecosystem and hence avoids Pulp plugin limitations as a potentially confounding factor; thus let's keep working on the devpi pilot and also keep an eye on pulp_python
12:18:08 <bkabrda> jzeleny: good, if you send out some mails, please put me on CC
12:18:11 <ncoghlan> however, I think our limiting factor at this point is likely to be the fact most upstream ecosystems don't really support redistributors natively anyway
12:18:17 <jzeleny> bkabrda: will do
12:18:21 <bkabrda> thx
12:18:48 <bkabrda> so let me get back to good news :)
12:20:32 <bkabrda> good news is, that devpi lets you say that a package should only be installed from an uploaded archive, even if newer version is available on PyPI (i.e. you can "shadow" the PyPI package completely by uploading your package of the same name)
12:20:52 <bkabrda> *uploaded archive == uploaded to your devpi instance
12:21:03 <bkabrda> (not sure if I'm explaining this the right way...)
12:21:40 <ncoghlan> I think so - if something is *in* our devpi instance, devpi won't update it unless we tell it to
12:21:56 <bkabrda> for example, let's say we want to mirror django only in versions that are in a fedora release - we just need to upload the tars in specific versions to our devpi instance
12:22:07 <jzeleny> bkabrda: is it possible to revert this decision?
12:22:14 <bkabrda> the other versions that are on pypi won't be offered by devpi for installation
12:22:29 <bkabrda> jzeleny: yes. that is what the pypi_whitelist is for, actually :)
12:22:49 <bkabrda> you just add a package to pypi_whitelist and suddenly all versions from pypi are mirrored again
12:23:22 <jzeleny> cool
12:23:26 <bkabrda> (and you can revert the reverted decision...) :)
12:23:27 <hhorak> #info good news is, that if something is *in* our devpi instance, devpi won't update it from pypi unless we tell it to using pypi_whitelist
12:23:38 <jzeleny> bkabrda: I was just going to ask :-)
12:23:45 <bkabrda> hhorak: nice wording! :)
12:23:46 <juhp_> or upload a newer version? :)
12:24:15 <bkabrda> juhp_: yes. the point is that for uploaded packages you have a very good control
12:24:50 <bkabrda> either you add the package to pypi_whitelist or you upload precisely the versions you want to be available
12:25:16 <bkabrda> and that's about everything I have, except that information that my devpi21 SCL works just fine
12:25:24 <ncoghlan> bkabrda: so the only thing we're missing is the ability to have a global whitelist?
12:25:30 <bkabrda> ncoghlan: yes, precisely
12:25:36 <ncoghlan> cool
12:25:44 <bkabrda> ncoghlan: I opened a bug for that here: https://bitbucket.org/hpk42/devpi/issue/198/whitelisting-packages-that-can-be-mirrored
12:25:45 * hhorak thinks we should somehow track these requirements (for devpi functionality) so we have already written them for pulp or other similar tools in the future
12:26:17 <hhorak> #link https://bitbucket.org/hpk42/devpi/issue/198/whitelisting-packages-that-can-be-mirrored
12:26:52 <ncoghlan> hhorak: would listing them on the LSR project page be enough for now?
12:28:08 <ncoghlan> actually, it problem makes sense to make that one of the goals of the pilot - defining our requirements for a more permanent solution
12:28:31 <hhorak> ncoghlan: sure, that was what I wanted to propose
12:29:04 <hhorak> bkabrda: can I give you AI for adding the news into the wiki page?
12:29:36 <bkabrda> hhorak: sure
12:30:03 <hhorak> #action bkabrda will list requirements/news learned for devpi on the LSR project page
12:30:35 <bkabrda> I also think I'm getting a pretty good idea about what/how we want to mirror/prebuild, but that's for a longish discussion and I need to put it on a paper first. so maybe for next time I'll prepare a draft with proposal
12:31:04 <hhorak> bkabrda: greawt
12:32:01 <bkabrda> #action bkabrda will prepare a draft document about what/how we want to mirror/prebuild on the pilot devpi instance
12:32:14 <bkabrda> (just a reminder for myself :)
12:35:31 <hhorak> I also wanted to quickly summarize the short internal discussion we've had this week about LSR, about where are we going to with LSRs.. Very quick summary could be that eventually we aim to have most software delivered as images (I like Nick's comparison of images to walls) but we need to have bricks to build them (which will be packages in those repositories).
12:36:42 <ncoghlan> I really should flesh that analogy out into a blog post at some point
12:36:49 <hhorak> any other thought we should mention to get some public attention hopefully?
12:37:15 <jzeleny> hhorak: just a note, I don't believe what we want are images per se
12:37:42 <bkabrda> ncoghlan: +1, I think that's a great analogy. I think people are so crazy about containers that they don't realize they need to install their dependencies *somehow* into their containers
12:37:43 <jzeleny> I would rather use the application analogy used by Gnome Software
12:38:27 <juhp_> bkabrda, right
12:38:43 <juhp_> ncoghlan, what's a wall? :)
12:39:17 <ncoghlan> now I'm trying to remember Langdon's comment that I was replying to :)
12:40:10 <hhorak> jzeleny: I'm afraid application may be easily messed up with a package, but generally agree with you
12:40:27 <ncoghlan> I think his point was that if everything is getting deployed as images or prebuilt ostree's, the ops side aren't going to be as worried about the dependency management systems
12:40:58 * langdon not getting enough context to help out ncoghlan
12:41:17 * langdon also still *always* reads "AI" as artificial intelligence
12:41:27 <ncoghlan> and my counter was that while that's true, the dev side still need a way to define those images and ostree's in the first place
12:42:00 <hhorak> langdon: that's what we're heading to, right? :)
12:42:31 <ncoghlan> so the analogy was that image/ostree = prefabricated wall, individual package = brick, dependency manager = an automated bricklayer :)
12:42:33 <langdon> one more quick point about a pulp investigation, must ensure that devpi can mirror it in case a consumer wants to replicate the replication but only for python [insert language here]
12:42:47 <juhp_> ncoghlan, ah
12:43:16 <bkabrda> langdon: good point. I'll find out whether that's possible
12:44:11 <langdon> ncoghlan, ahh.. i don't think i saw that analogy yet.. but yes, i think that is not a bad one.. :)
12:44:23 <jzeleny> langdon: just out of curiosity, would that still be the case if we used Pulp from the beginning?
12:44:31 <bkabrda> #action bkabrda will find out whether devpi can mirror pulp's pulp_python repo
12:45:11 <bkabrda> jzeleny: the important thing about devpi is that it's also supposed to be deployed on developer workstations to help them with certain development steps
12:45:19 <langdon> jzeleny, i think so.. because consumers of the fedora replication (or whatever it is being called) may not want to use pulp.. that said.. it may not be a blocker.. but we should know the answer
12:45:46 <bkabrda> jzeleny: so a company may use pulp with pulp_python as their *central* repo, but developers may want to use devpi as their local workstation mirrors
12:45:47 <jzeleny> bkabrda: hm, I have probably missed that point ... is there some docs about this?
12:45:59 <ncoghlan> bkabrda: I'm also curious to know what the Pulp folks are using as a test suite - I know dstufft wants a clear implementation independent test suite for PyPI as part of Warehouse development, but I'm not sure if it exists yet
12:46:09 <langdon> like we may have to say "to use this stuff you need to install pulp" but we want to know if "you can also do it with just devpi"
12:46:17 <bkabrda> jzeleny: upstream docs about using it this way, if that's what you're interested in
12:47:09 <jzeleny> bkabrda: well, I was rather interested in some background why would developers use devpi on their machines ... so if the rationale is there, I guess that's what I want to see :-)
12:47:35 <bkabrda> jzeleny: yes, the rationale is there... let me find a link
12:48:00 <ncoghlan> jzeleny: local PyPI cache so they can work offline, local testing of package build and upload, that kind of thing
12:48:19 <bkabrda> jzeleny: or maybe it's better to say that the rationale is obvious once you learn about all the cool features... like the ones ncoghlan just mentioned :)
12:48:22 <bkabrda> jzeleny: http://doc.devpi.net/latest/quickstart-releaseprocess.html
12:48:48 <jzeleny> bkabrda: thanks
12:53:35 <hhorak> #link http://doc.devpi.net/latest/quickstart-releaseprocess.html
12:54:19 * hhorak needs to leave in 7 minutes, can we shortly touch SCLs?
12:54:33 <hhorak> #topic Follow-up:SCLs
12:54:57 <hhorak> I planned to have already summarized until today all what was said, proposed and agreed on about SCLs in Fedora.. But.. There has been much more said than I thought so don't have it ready.
12:56:12 <hhorak> What I just wanted to quickly touch is some rough plan.
12:56:55 <hhorak> My simple proposal would be adopting the agreed names/prefixes/vendor and prepare some example collection (collections for fedora on softwarecollections.org do not use those)
12:58:25 <juhp_> aha
12:58:58 <bkabrda> hhorak: just to make sure... where were the names/prefixes/vendor agreed on?
13:00:34 <hhorak> bkabrda: not sure if really agreed, but I have a feeling we got to some conclusion working for all, didn't we? like using /opt/fedora.. but that's still something I want to make sure..
13:01:14 <ncoghlan> I'd be in favour of agreeing to something like that to get things moving forward
13:01:47 <bkabrda> hhorak: yeah, I just wanted to know what precisely we're going to go with. there were lost of agreements in the past... ;)
13:02:28 * langdon wonders is bkabrda made a freudian slip
13:02:33 <langdon> s/is/if
13:02:35 <hhorak> what I saw as a big issue was trying to include SCLs into base Fedora, which is something I'd like to avoid now, since it seems to produce many blockers...
13:03:18 <bkabrda> langdon: I do these all the time ;)
13:03:57 <ncoghlan> hhorak: I'm not sure it even makes sense, as the main advantage of SCLs from my perspective is they're decoupled from the OS lifecycle
13:04:56 <ncoghlan> parallel language runtimes at the core OS layer is rather suboptimal (says the core Python developer...)
13:06:14 <hhorak> ncoghlan: that's what I agree with..
13:06:21 <hhorak> well, I need to go now, so the important note from me is probably that I don't have anything ready now but I'm working on it :)
13:06:46 <ncoghlan> hhorak: I was agreeing with you - just puzzled by the original suggestion of doing it :)
13:07:03 <hhorak> ncoghlan: ah, ok :)
13:07:10 <hhorak> could anybody end meeting after there is nothing else to discuss?
13:07:15 <langdon> ncoghlan, at the time there was "base" and "not fedora" ... nothing in between
13:07:23 * hhorak waiving! bye...
13:07:41 <ncoghlan> langdon: ah, OK - that makes more sense :)
13:07:43 <langdon> personally i think one of envs&stacks primary "goals" is to provide options like this
13:08:28 <ncoghlan> langdon: +lots :)
13:08:28 <langdon> not that i nec. agree with your base stmt ;) .. but that is for another, preferably beer-laden, day
13:08:44 <ncoghlan> heh
13:09:01 <juhp_> langdon, yes it is
13:11:20 <ncoghlan> SCLs was the last item on preplanned agenda
13:14:03 <juhp_> shall we finish then?
13:14:09 <ncoghlan> I think so
13:14:39 <ncoghlan> need a volunteer to chair next week, unless we volunteer hhorak in absentia :)
13:19:09 <ncoghlan> I guess it falls to hhorak by default...
13:20:01 <ncoghlan> anything else? Otherwise I'll end the meeting
13:20:41 <ncoghlan> OK - thanks folks, catch you next week
13:20:46 <ncoghlan> #endmeeting