12:00:23 <hhorak1> #startmeeting Env and Stacks (2015-06-18)
12:00:23 <zodbot> Meeting started Thu Jun 18 12:00:23 2015 UTC.  The chair is hhorak1. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
12:00:40 <hhorak1> #meetingname env-and-stacks
12:00:40 <zodbot> The meeting name has been set to 'env-and-stacks'
12:00:44 <hhorak1> #chair bkabrda hhorak juhp ncoghlan vpavlin sicampbell walters ttomecek phracek
12:00:44 <zodbot> Current chairs: bkabrda hhorak hhorak1 juhp ncoghlan phracek sicampbell ttomecek vpavlin walters
12:01:04 <hhorak1> #topic init process
12:01:22 <hhorak1> hey, anybody here?
12:01:28 * vpavlin hides
12:01:49 <hhorak> vpavlin hide better, I see you pretty well
12:03:06 * ttomecek waves
12:06:17 <ttomecek> hhorak, I got to admit I haven't read ML for a couple of weeks :/
12:07:33 <hhorak> ttomecek: I'm pretty sure you're not boring anyway.. :)
12:07:51 * langdon lurks
12:08:12 <hhorak> langdon: hi
12:08:22 <hhorak> well, not too many people so far, but let's try to beign..
12:08:44 <langdon> hhorak, hi
12:09:30 <hhorak> #topic Fedora Docker images
12:10:53 <ttomecek> layered?
12:11:05 <hhorak> yes :)
12:11:09 <hhorak> #undo
12:11:09 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x136c3090>
12:11:18 <hhorak> #topic Fedora layered docker images
12:11:27 <hhorak> So, the dockerfiles at https://github.com/fedora-cloud/Fedora-Dockerfiles/ are not perfect... yet..
12:11:39 <hhorak> #link  https://github.com/fedora-cloud/Fedora-Dockerfiles/
12:12:05 <hhorak> one thing that sgallagh point me at yesterday, we should squash them..
12:12:23 <vpavlin> What do you mean by squash?
12:12:23 <hhorak> ttomecek: the squashing in osbs is done by dock?
12:13:26 <hhorak> #link http://jasonwilder.com/blog/2014/08/19/squashing-docker-images/
12:13:33 <hhorak> vpavlin: something like that ^
12:13:59 <bkabrda> hi everyone, sorry I'm late
12:14:14 <ttomecek> I would be very careful with using tool docker-squash
12:14:15 <hhorak> bkabrda: hi
12:14:17 <ttomecek> look at issues
12:14:34 <ttomecek> #link https://github.com/goldmann/docker-scripts
12:14:38 <ttomecek> we use ^
12:14:48 <vpavlin> imo we have no way how to squash the images until we setup OSBS for Fedora...
12:14:52 <ttomecek> hhorak, and yes, dock calls the docker-scripts
12:15:20 <ttomecek> vpavlin, well, we could use the tool directly
12:15:42 <langdon> personally, squashing is a nice to have.. lets get images building, then layered, then squashing...
12:15:43 <vpavlin> ttomecek: How? Images are (afaik) build on Docker Hub
12:16:22 <hhorak> ttomecek: so the docker-scripts have no issues like docker-squash, do I understand it correctly?
12:16:31 <langdon> vpavlin, is that the "plan" per this topic? or still considering copr adding docker build?
12:17:21 <vpavlin> langdon: Last I talked to msuchy, the plan was to teach copr how to talk to Docker Hub and submit builds there, so still nothing we can influence very well...No idea if that changed..
12:17:25 <ttomecek> #link https://github.com/docker/docker/pull/13929
12:17:38 <ttomecek> current upstream PR related to squashing built images
12:17:46 <hhorak> langdon: I understood we'd have both at some point, osbs and copr-docker
12:17:55 <ttomecek> hhorak, you do; docker-scripts are being used in production
12:18:38 <vpavlin> hhorak: correct, at some point, we might be able to swap calling Docker Hub api with calling osbs api
12:19:01 <vpavlin> ttomecek: Do you have news about this? Copr vs. Docker images?
12:19:01 <ttomecek> I would prefer setting up osbs in Fedora and teaching it to cooperate with whole Fedora Infra, rather then implementing docker build in copr
12:19:23 <ttomecek> vpavlin, I don't
12:19:33 <hhorak> vpavlin: if it is what we want -- it kind of makes sense to call docker hub directly for copr projects.. or maybe have a choice between osbs & docker hub
12:19:38 <langdon> ttomecek, that is actually what i meant.. user "sees" copr though.. maybe.. and copr talks to osbs over fedmsg
12:19:55 <ttomecek> langdon, this
12:20:27 <langdon> hhorak, still route to d-hub or osbs over fedmsg... then there is a natural redirect point without copr changes
12:20:34 <vpavlin> hhorak: I think the agreemen was to do it the easier way - use Docker Hub for now...but as I said, I have no idea if plans changed anyhow
12:20:45 <hhorak> #info docker-squash have issues, while docker-scripts doesn't and it is proved in practice already
12:20:56 <vpavlin> langdon: yes
12:21:14 <langdon> and.. d-hub (and lack of osbs in fed-infra) = no layers anyway
12:21:31 <langdon> s/layers/layered-builds
12:21:47 <ttomecek> #link https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
12:22:02 <ttomecek> mattdm wants me to sign under it ^
12:22:25 <langdon> isn't the priority copr->fedmsg & fedmsg->d-hub? then stand up osbs .. then fedmsg-osbs? then, some day, squashing?
12:22:44 <vpavlin> langdon: What do you mean by "no layered-builds anyway"?
12:22:56 <vpavlin> ttomecek: Guess you are happy to get more work to do:-P
12:23:25 <langdon> vpavlin, d-hub doesn't provide layered builds the way "we" define it anyway, right? we could simulate it by chaining builds to d-hub .. but not directly...
12:23:54 <hhorak> langdon: with osbs there will be squashing solved (it uses dock, which uses docker-scripts), so we don't have to bother with squashing anymore..
12:24:13 * langdon is not holding his breath ... ;)
12:24:46 <vpavlin> langdon: Ah..yeah, well..Docker Hub does not have OpenShift's "Image Stream"..so, yeah, you would have to do the chaining manually
12:25:08 <ttomecek> vpavlin, exactly; I'm not going to be responsible for another osbs instance
12:25:32 <ttomecek> build chaining is on our roadmap
12:25:34 <hhorak> langdon: vpavlin: I still don't understand what is different in docker hub -- what is OS's image stream?
12:26:14 <hhorak> is it about automatic rebuilding after base image gets rebuilt?
12:26:36 <vpavlin> hhorak: Image Stream is a "thing" that keeps track of dependencies  between images wrt layering
12:26:58 <vpavlin> So that you can set automatic trigger for rebuild if your base image change
12:27:19 <vpavlin> "base" being FROM (.*)
12:27:33 <hhorak> vpavlin: ok
12:27:40 <vpavlin> hhorak: ah...yeah, what you said is correct)
12:27:40 <vpavlin> :)
12:28:11 <vpavlin> ttomecek: So what about being a consultant for someone who would set up tha OSBS and work on the integration with infra
12:28:20 <vpavlin> No idea if we have someone like that...but...:)
12:28:32 <langdon> hhorak, the idea of layered builds is that you build 1 dfile, then build another based on it, then another based on that
12:28:32 <langdon> so then you can swap out intervening layers with new ones if lower parts of the stack change
12:28:32 <langdon> well
12:28:33 <langdon> then you can identify only the parts you have to rebuild..
12:28:35 <langdon> not explaining that terribly well....
12:28:38 <langdon> i am not seeing a serious amount of value for fedora users.. for a company, it means all their products can be based on ever increasingly big, well known builds.. for fedora, does it help that much? i am not sure
12:29:08 <ttomecek> vpavlin, I've already offered that
12:29:16 <ttomecek> vpavlin, wanna more e-mails?
12:30:09 <vpavlin> ttomecek: Sure, feel free to get me involved
12:30:10 <ttomecek> langdon, I think we want d-hub being swarmed by fedora based docker images
12:31:00 <vpavlin> ttomecek: Yeah, we want that, but then it does not make much sense to build our stuff in OSBS and serve it in our own registry, does it?:)
12:31:02 <langdon> ttomecek, i am not sure if you are kidding.. but i want all the d-hub images to be fed (or at least cent).. but not sure how layered builds come in to that... too much..
12:34:51 <vpavlin> If we want to have fedora on Docker Hub, we should use what Docker Hub provides for builds and so on..
12:34:56 <hhorak> so, going one step back, osbs for copr projects is more nice to have than a must, for more serious layered builds for fedora the osbs is still must.
12:35:22 <ttomecek> from my PoV: connect copr with the build system: basically, once upstream releases new version of something, do rpm build in copr asap, create docker image, put it to d-hub
12:35:30 <ttomecek> cutting edge images
12:37:53 <langdon> ttomecek, i guess i don't see where a need for osbs comes in there.. aside from having more control over the build.. to be really useful, we need "something" to also know all the things that were built based on a particular package.. then trigger rebuilds throughout the tree.. then push the new builds to d-hub
12:39:00 * langdon would like to see copr building from arbitrary git repo (vs srpm) as a priority over triggering d-hub
12:40:04 <hhorak> any volunteer to talk to copr guys? ^ (check what their plans are and share the ideas mentioned here)
12:41:07 <langdon> well.. and as far as copr is concerned, all it really needs to do is trigger a fedmsg when build is complete (if it doesn't already).. to trigger d-hub, we just need a fedmsg listener
12:41:56 <langdon> well.. glossing over a ui change "somewhere" to say "i want my copr repo to build docker images using xyz dockerfile"
12:43:17 * langdon wonders if he is not the only one without enough coffee
12:44:40 <hhorak> langdon: you're definitely not the only one :)
12:44:56 <bkabrda> (+1)
12:45:06 <ttomecek> I thought that similar feature is already on their roadmap
12:46:13 <hhorak> well, we miss msuchy and cwalters in this discussions anyway, so what we could do is to summarize the questions and try to find answer on ML..
12:46:23 <hhorak> any volunteer for that? ^
12:47:12 <langdon> i could write it.. but I don't think i will have any real cycles until mid/late next week
12:47:47 <langdon> and, for me, this would require research, e.g. research roadmap, ask intelligent qs
12:48:28 <hhorak> langdon: I see, same for me, so let's make a competition, whoever makes it first, wins :)
12:48:55 <langdon> hhorak, ha.. ok.. or we can volunteer bkabrda .. he loves writing emails... and research
12:49:21 * vpavlin already talked to msuchy about docker images...
12:49:33 <bkabrda> langdon: I wonder where you got that ;)
12:49:33 <hhorak> there was another thing I say in the fodora dockerfiles that actually influence image size more than missing squashing -- `yum update` -- we should not run that in docker files, correct?
12:49:38 <vpavlin> So I might be able to pick it up and revisit what changed..
12:49:58 <hhorak> vpavlin: thanks!
12:50:01 <ttomecek> I'm planning pto next week: so, no time from me; doing TL;DR of that ^, writing it to our ML, invite copr guys and start discussions sounds like a good idea
12:50:18 <vpavlin> hhorak: poke me if I don't deliver in reasonable time;)
12:50:27 <langdon> hhorak, ttomecek vpavlin  does any of the "squashing" code review the dockerfile itself? or is it all on the "binaries"?
12:52:18 <hhorak> langdon: well, my guess is binaries only, but hopefully ttomecek knows better :)
12:52:57 <ttomecek> binaries
12:53:38 <ttomecek> input = {tar,tag}, output = {tar,image in engine}
12:54:09 <langdon> so.. vpavlin .. may want to add to the discussion, "approval by docker lint before building"?
12:54:22 <langdon> or is that an "acceptance to fedora" criteria?
12:54:39 <ttomecek> langdon, sounds like a good idea to me
12:55:24 <vpavlin> #info Use dockerfile_lint prior the build to check the Dockerfile
12:55:35 <vpavlin> +1
12:56:03 * bkabrda has to go for another meeting. sorry for not being very helpful today...
12:56:05 <hhorak> well, the discussion went a bit further from squashing than I expected, so let's touch the other docker-related topic a bit as well..
12:56:13 <hhorak> #topic testing of fedora layered docker images -- where to keep tests
12:56:27 <hhorak> I think we kind of agreed on using https://github.com/Containers-Testing-Framework/ for testing images.. but where the fedora tests should live?
12:56:45 <vpavlin> In the image directory?
12:57:13 <hhorak> vpavlin: which is?
12:57:21 <hhorak> vpavlin: where the dockerfiles live?
12:57:21 <vpavlin> https://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/mariadb - mkdir tests
12:58:40 <hhorak> vpavlin: that makes sense for specific tests for mariadb.. but the concept is designed to be able to have common tests for specific domain (e.g. databases)
12:58:46 <hhorak> or daemons
12:59:08 <vpavlin> hhorak: good point..then I have no idea
12:59:08 <hhorak> those could rather live near the framework itself..
12:59:09 <vpavlin> ;-)
12:59:49 <vpavlin> hhorak: They already sort of do live there:) (common-features, common-steps)
12:59:57 <langdon> hhorak, does this assume "unit"/"real"/"regular" tests have already been run on the "rpm"? like not quite sure how to ask this. but is this testing "integration with being in a docker image"? or the "thing" itself?
13:00:13 <vpavlin> I think it's valid use case for pull request to add "common-database-features/steps"
13:00:41 <vpavlin> langdon: We should test only explicit use cases described by behave features
13:00:44 * langdon notes this also gets in to the "fedora modularization" discussion...
13:01:18 <vpavlin> langdon: I wouldn't re-test mariadb rpm itself just because it's now installed in container
13:01:19 <langdon> vpavlin, well.. that isn't quite what i mean.. like when/where do the "tests provided by maria" come in?
13:01:43 <vpavlin> langdon: Tests for mariadb rpm, or tests for mariadb contianer?:)
13:02:05 <langdon> vpavlin, ok.. so .. "build rpm" -> "run unit tests" -> "run integration tests"? -> build image -> run generic db tests?
13:02:06 <vpavlin> You can expect the rpm will work the same..you just need to check the "API" of container gives you what you expect..
13:02:20 * langdon ran out of quotes steam
13:03:05 <hhorak> langdon: yes, testing of rpms would be run before testing the image
13:03:14 <langdon> vpavlin, definitely x-ing in to the fedora-modularization space.. (not a bad thing, just noting it)
13:03:40 <vpavlin> langdon: "build rpm" -> run all the test you would run without any knowledge about containers -> build image -> run tests to check if db is reachable and usable when container is started
13:03:51 <vpavlin> langdon: sure
13:04:30 <vpavlin> Sorry, container-tools standup..I need to go
13:04:37 <langdon> vpavlin, hhorak  ok.. so ... do you see the tests for no-sql to be different from sql databases?
13:06:49 <hhorak> langdon: I expect all db-tests could some common tests/steps
13:07:30 <hhorak> langdon: not sure if that is answer for your question :)
13:07:53 <hhorak> #info at least for start we'll have the general tests (or meta tests and steps) under https://github.com/Containers-Testing-Framework/ and package-specific parts near the dockerfiles (https://github.com/fedora-cloud/Fedora-Dockerfiles/), in tests sub-directory
13:08:39 <hhorak> #info docker images tests will test just the image itself, rpms will be tested before image build already..
13:08:51 <hhorak> #topic Open Floor
13:08:51 <hhorak> if anybody is still here by any chance :)
13:09:11 <langdon> hhorak, well... i guess i am just thinking.. are there really any different tests for db-subtypes (sql, no-sql, object, n/v pair), then why are there any tests that a different for any container? like if the container says.. "something should be listening on port x" then the testing doesn't care what is listening
13:10:53 <hhorak> langdon: there will be specific tests e.g. for mariadb, since for example environment variables which the image accepts to initialize the database are specific for every database
13:11:21 <hhorak> langdon: or mongo has different authentication mechanism than postgresql/mariadb
13:11:42 <hhorak> which needs to be mirrored in the image's API (which is environment variables)
13:12:08 <hhorak> #undo
13:12:08 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x172a1e50>
13:13:09 <langdon> hhorak, ok.. i guess those stmts make me think there do need to be specific maria-in-a-container tests that are beyoond db-in-a-container tests.. or we need to think closely about if we *could* have generic .. which would help the modularization effort..
13:14:27 <hhorak> langdon: if we're able to make it generic, then it would be great. I don't think it will be possible in all the cases though.
13:15:36 <hhorak> #info there will probably be specific tests e.g. for mariadb, since for example environment variables which the image accepts to initialize the database are specific for every database (another example is mongodb that has different authentication mechanism than postgresql/mariadb, which needs to be mirrored in the image's API)
13:15:36 <hhorak> #info but we should try to make the tests be generic as much as possible
13:16:36 <hhorak> #topic Open Floor
13:16:40 <hhorak> second call :)
13:18:29 <hhorak> it seems nobody left, so closing :)
13:18:32 <hhorak> #endmeeting