12:00:23 #startmeeting Env and Stacks (2015-06-18) 12:00:23 Meeting started Thu Jun 18 12:00:23 2015 UTC. The chair is hhorak1. Information about MeetBot at http://wiki.debian.org/MeetBot. 12:00:23 Useful Commands: #action #agreed #halp #info #idea #link #topic. 12:00:40 #meetingname env-and-stacks 12:00:40 The meeting name has been set to 'env-and-stacks' 12:00:44 #chair bkabrda hhorak juhp ncoghlan vpavlin sicampbell walters ttomecek phracek 12:00:44 Current chairs: bkabrda hhorak hhorak1 juhp ncoghlan phracek sicampbell ttomecek vpavlin walters 12:01:04 #topic init process 12:01:22 hey, anybody here? 12:01:28 * vpavlin hides 12:01:49 vpavlin hide better, I see you pretty well 12:03:06 * ttomecek waves 12:06:17 hhorak, I got to admit I haven't read ML for a couple of weeks :/ 12:07:33 ttomecek: I'm pretty sure you're not boring anyway.. :) 12:07:51 * langdon lurks 12:08:12 langdon: hi 12:08:22 well, not too many people so far, but let's try to beign.. 12:08:44 hhorak, hi 12:09:30 #topic Fedora Docker images 12:10:53 layered? 12:11:05 yes :) 12:11:09 #undo 12:11:09 Removing item from minutes: 12:11:18 #topic Fedora layered docker images 12:11:27 So, the dockerfiles at https://github.com/fedora-cloud/Fedora-Dockerfiles/ are not perfect... yet.. 12:11:39 #link https://github.com/fedora-cloud/Fedora-Dockerfiles/ 12:12:05 one thing that sgallagh point me at yesterday, we should squash them.. 12:12:23 What do you mean by squash? 12:12:23 ttomecek: the squashing in osbs is done by dock? 12:13:26 #link http://jasonwilder.com/blog/2014/08/19/squashing-docker-images/ 12:13:33 vpavlin: something like that ^ 12:13:59 hi everyone, sorry I'm late 12:14:14 I would be very careful with using tool docker-squash 12:14:15 bkabrda: hi 12:14:17 look at issues 12:14:34 #link https://github.com/goldmann/docker-scripts 12:14:38 we use ^ 12:14:48 imo we have no way how to squash the images until we setup OSBS for Fedora... 12:14:52 hhorak, and yes, dock calls the docker-scripts 12:15:20 vpavlin, well, we could use the tool directly 12:15:42 personally, squashing is a nice to have.. lets get images building, then layered, then squashing... 12:15:43 ttomecek: How? Images are (afaik) build on Docker Hub 12:16:22 ttomecek: so the docker-scripts have no issues like docker-squash, do I understand it correctly? 12:16:31 vpavlin, is that the "plan" per this topic? or still considering copr adding docker build? 12:17:21 langdon: Last I talked to msuchy, the plan was to teach copr how to talk to Docker Hub and submit builds there, so still nothing we can influence very well...No idea if that changed.. 12:17:25 #link https://github.com/docker/docker/pull/13929 12:17:38 current upstream PR related to squashing built images 12:17:46 langdon: I understood we'd have both at some point, osbs and copr-docker 12:17:55 hhorak, you do; docker-scripts are being used in production 12:18:38 hhorak: correct, at some point, we might be able to swap calling Docker Hub api with calling osbs api 12:19:01 ttomecek: Do you have news about this? Copr vs. Docker images? 12:19:01 I would prefer setting up osbs in Fedora and teaching it to cooperate with whole Fedora Infra, rather then implementing docker build in copr 12:19:23 vpavlin, I don't 12:19:33 vpavlin: if it is what we want -- it kind of makes sense to call docker hub directly for copr projects.. or maybe have a choice between osbs & docker hub 12:19:38 ttomecek, that is actually what i meant.. user "sees" copr though.. maybe.. and copr talks to osbs over fedmsg 12:19:55 langdon, this 12:20:27 hhorak, still route to d-hub or osbs over fedmsg... then there is a natural redirect point without copr changes 12:20:34 hhorak: I think the agreemen was to do it the easier way - use Docker Hub for now...but as I said, I have no idea if plans changed anyhow 12:20:45 #info docker-squash have issues, while docker-scripts doesn't and it is proved in practice already 12:20:56 langdon: yes 12:21:14 and.. d-hub (and lack of osbs in fed-infra) = no layers anyway 12:21:31 s/layers/layered-builds 12:21:47 #link https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service 12:22:02 mattdm wants me to sign under it ^ 12:22:25 isn't the priority copr->fedmsg & fedmsg->d-hub? then stand up osbs .. then fedmsg-osbs? then, some day, squashing? 12:22:44 langdon: What do you mean by "no layered-builds anyway"? 12:22:56 ttomecek: Guess you are happy to get more work to do:-P 12:23:25 vpavlin, d-hub doesn't provide layered builds the way "we" define it anyway, right? we could simulate it by chaining builds to d-hub .. but not directly... 12:23:54 langdon: with osbs there will be squashing solved (it uses dock, which uses docker-scripts), so we don't have to bother with squashing anymore.. 12:24:13 * langdon is not holding his breath ... ;) 12:24:46 langdon: Ah..yeah, well..Docker Hub does not have OpenShift's "Image Stream"..so, yeah, you would have to do the chaining manually 12:25:08 vpavlin, exactly; I'm not going to be responsible for another osbs instance 12:25:32 build chaining is on our roadmap 12:25:34 langdon: vpavlin: I still don't understand what is different in docker hub -- what is OS's image stream? 12:26:14 is it about automatic rebuilding after base image gets rebuilt? 12:26:36 hhorak: Image Stream is a "thing" that keeps track of dependencies between images wrt layering 12:26:58 So that you can set automatic trigger for rebuild if your base image change 12:27:19 "base" being FROM (.*) 12:27:33 vpavlin: ok 12:27:40 hhorak: ah...yeah, what you said is correct) 12:27:40 :) 12:28:11 ttomecek: So what about being a consultant for someone who would set up tha OSBS and work on the integration with infra 12:28:20 No idea if we have someone like that...but...:) 12:28:32 hhorak, the idea of layered builds is that you build 1 dfile, then build another based on it, then another based on that 12:28:32 so then you can swap out intervening layers with new ones if lower parts of the stack change 12:28:32 well 12:28:33 then you can identify only the parts you have to rebuild.. 12:28:35 not explaining that terribly well.... 12:28:38 i am not seeing a serious amount of value for fedora users.. for a company, it means all their products can be based on ever increasingly big, well known builds.. for fedora, does it help that much? i am not sure 12:29:08 vpavlin, I've already offered that 12:29:16 vpavlin, wanna more e-mails? 12:30:09 ttomecek: Sure, feel free to get me involved 12:30:10 langdon, I think we want d-hub being swarmed by fedora based docker images 12:31:00 ttomecek: Yeah, we want that, but then it does not make much sense to build our stuff in OSBS and serve it in our own registry, does it?:) 12:31:02 ttomecek, i am not sure if you are kidding.. but i want all the d-hub images to be fed (or at least cent).. but not sure how layered builds come in to that... too much.. 12:34:51 If we want to have fedora on Docker Hub, we should use what Docker Hub provides for builds and so on.. 12:34:56 so, going one step back, osbs for copr projects is more nice to have than a must, for more serious layered builds for fedora the osbs is still must. 12:35:22 from my PoV: connect copr with the build system: basically, once upstream releases new version of something, do rpm build in copr asap, create docker image, put it to d-hub 12:35:30 cutting edge images 12:37:53 ttomecek, i guess i don't see where a need for osbs comes in there.. aside from having more control over the build.. to be really useful, we need "something" to also know all the things that were built based on a particular package.. then trigger rebuilds throughout the tree.. then push the new builds to d-hub 12:39:00 * langdon would like to see copr building from arbitrary git repo (vs srpm) as a priority over triggering d-hub 12:40:04 any volunteer to talk to copr guys? ^ (check what their plans are and share the ideas mentioned here) 12:41:07 well.. and as far as copr is concerned, all it really needs to do is trigger a fedmsg when build is complete (if it doesn't already).. to trigger d-hub, we just need a fedmsg listener 12:41:56 well.. glossing over a ui change "somewhere" to say "i want my copr repo to build docker images using xyz dockerfile" 12:43:17 * langdon wonders if he is not the only one without enough coffee 12:44:40 langdon: you're definitely not the only one :) 12:44:56 (+1) 12:45:06 I thought that similar feature is already on their roadmap 12:46:13 well, we miss msuchy and cwalters in this discussions anyway, so what we could do is to summarize the questions and try to find answer on ML.. 12:46:23 any volunteer for that? ^ 12:47:12 i could write it.. but I don't think i will have any real cycles until mid/late next week 12:47:47 and, for me, this would require research, e.g. research roadmap, ask intelligent qs 12:48:28 langdon: I see, same for me, so let's make a competition, whoever makes it first, wins :) 12:48:55 hhorak, ha.. ok.. or we can volunteer bkabrda .. he loves writing emails... and research 12:49:21 * vpavlin already talked to msuchy about docker images... 12:49:33 langdon: I wonder where you got that ;) 12:49:33 there was another thing I say in the fodora dockerfiles that actually influence image size more than missing squashing -- `yum update` -- we should not run that in docker files, correct? 12:49:38 So I might be able to pick it up and revisit what changed.. 12:49:58 vpavlin: thanks! 12:50:01 I'm planning pto next week: so, no time from me; doing TL;DR of that ^, writing it to our ML, invite copr guys and start discussions sounds like a good idea 12:50:18 hhorak: poke me if I don't deliver in reasonable time;) 12:50:27 hhorak, ttomecek vpavlin does any of the "squashing" code review the dockerfile itself? or is it all on the "binaries"? 12:52:18 langdon: well, my guess is binaries only, but hopefully ttomecek knows better :) 12:52:57 binaries 12:53:38 input = {tar,tag}, output = {tar,image in engine} 12:54:09 so.. vpavlin .. may want to add to the discussion, "approval by docker lint before building"? 12:54:22 or is that an "acceptance to fedora" criteria? 12:54:39 langdon, sounds like a good idea to me 12:55:24 #info Use dockerfile_lint prior the build to check the Dockerfile 12:55:35 +1 12:56:03 * bkabrda has to go for another meeting. sorry for not being very helpful today... 12:56:05 well, the discussion went a bit further from squashing than I expected, so let's touch the other docker-related topic a bit as well.. 12:56:13 #topic testing of fedora layered docker images -- where to keep tests 12:56:27 I think we kind of agreed on using https://github.com/Containers-Testing-Framework/ for testing images.. but where the fedora tests should live? 12:56:45 In the image directory? 12:57:13 vpavlin: which is? 12:57:21 vpavlin: where the dockerfiles live? 12:57:21 https://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/mariadb - mkdir tests 12:58:40 vpavlin: that makes sense for specific tests for mariadb.. but the concept is designed to be able to have common tests for specific domain (e.g. databases) 12:58:46 or daemons 12:59:08 hhorak: good point..then I have no idea 12:59:08 those could rather live near the framework itself.. 12:59:09 ;-) 12:59:49 hhorak: They already sort of do live there:) (common-features, common-steps) 12:59:57 hhorak, does this assume "unit"/"real"/"regular" tests have already been run on the "rpm"? like not quite sure how to ask this. but is this testing "integration with being in a docker image"? or the "thing" itself? 13:00:13 I think it's valid use case for pull request to add "common-database-features/steps" 13:00:41 langdon: We should test only explicit use cases described by behave features 13:00:44 * langdon notes this also gets in to the "fedora modularization" discussion... 13:01:18 langdon: I wouldn't re-test mariadb rpm itself just because it's now installed in container 13:01:19 vpavlin, well.. that isn't quite what i mean.. like when/where do the "tests provided by maria" come in? 13:01:43 langdon: Tests for mariadb rpm, or tests for mariadb contianer?:) 13:02:05 vpavlin, ok.. so .. "build rpm" -> "run unit tests" -> "run integration tests"? -> build image -> run generic db tests? 13:02:06 You can expect the rpm will work the same..you just need to check the "API" of container gives you what you expect.. 13:02:20 * langdon ran out of quotes steam 13:03:05 langdon: yes, testing of rpms would be run before testing the image 13:03:14 vpavlin, definitely x-ing in to the fedora-modularization space.. (not a bad thing, just noting it) 13:03:40 langdon: "build rpm" -> run all the test you would run without any knowledge about containers -> build image -> run tests to check if db is reachable and usable when container is started 13:03:51 langdon: sure 13:04:30 Sorry, container-tools standup..I need to go 13:04:37 vpavlin, hhorak ok.. so ... do you see the tests for no-sql to be different from sql databases? 13:06:49 langdon: I expect all db-tests could some common tests/steps 13:07:30 langdon: not sure if that is answer for your question :) 13:07:53 #info at least for start we'll have the general tests (or meta tests and steps) under https://github.com/Containers-Testing-Framework/ and package-specific parts near the dockerfiles (https://github.com/fedora-cloud/Fedora-Dockerfiles/), in tests sub-directory 13:08:39 #info docker images tests will test just the image itself, rpms will be tested before image build already.. 13:08:51 #topic Open Floor 13:08:51 if anybody is still here by any chance :) 13:09:11 hhorak, well... i guess i am just thinking.. are there really any different tests for db-subtypes (sql, no-sql, object, n/v pair), then why are there any tests that a different for any container? like if the container says.. "something should be listening on port x" then the testing doesn't care what is listening 13:10:53 langdon: there will be specific tests e.g. for mariadb, since for example environment variables which the image accepts to initialize the database are specific for every database 13:11:21 langdon: or mongo has different authentication mechanism than postgresql/mariadb 13:11:42 which needs to be mirrored in the image's API (which is environment variables) 13:12:08 #undo 13:12:08 Removing item from minutes: 13:13:09 hhorak, ok.. i guess those stmts make me think there do need to be specific maria-in-a-container tests that are beyoond db-in-a-container tests.. or we need to think closely about if we *could* have generic .. which would help the modularization effort.. 13:14:27 langdon: if we're able to make it generic, then it would be great. I don't think it will be possible in all the cases though. 13:15:36 #info there will probably be specific tests e.g. for mariadb, since for example environment variables which the image accepts to initialize the database are specific for every database (another example is mongodb that has different authentication mechanism than postgresql/mariadb, which needs to be mirrored in the image's API) 13:15:36 #info but we should try to make the tests be generic as much as possible 13:16:36 #topic Open Floor 13:16:40 second call :) 13:18:29 it seems nobody left, so closing :) 13:18:32 #endmeeting