17:02:33 #startmeeting fedora_atomic_wg 17:02:33 Meeting started Wed Oct 18 17:02:33 2017 UTC. The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:02:33 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:02:33 The meeting name has been set to 'fedora_atomic_wg' 17:02:35 #topic roll call 17:02:39 .hello dustymabe 17:02:40 dustymabe: dustymabe 'Dusty Mabe' 17:02:41 .hello jasonbrooks 17:02:42 .hello smilner 17:02:43 jbrooks: jasonbrooks 'Jason Brooks' 17:02:45 ashcrow: smilner 'None' 17:02:51 .hello maxamillion 17:02:51 maxamillion: maxamillion 'Adam Miller' 17:02:58 * maxamillion jumped the gun :) 17:04:01 .hello jberkus 17:04:02 jberkus: jberkus 'Josh Berkus' 17:04:13 .hellomynameis kushal 17:04:14 kushal: kushal 'Kushal Das' 17:05:54 #topic previous meeting action items 17:06:07 jberkus i'm going to start with yours 17:06:12 * jberkus to follow-up on adding/removing Atomic WG members via 17:06:14 atomic-devel 17:06:15 ok 17:06:16 * jberkus to identify and create issue with list of blog posts we want 17:06:18 for f27 release 17:06:20 * jberkus to follow up on images from Flock container workshop 17:06:22 * jberkus to follow up on docs from Flock workshop 17:06:24 * jberkus to raise kube installation issues with openshift installer 17:06:26 team 17:06:50 .hello ttomecek 17:06:51 ttomecek: ttomecek 'Tomas Tomecek' 17:07:02 davdunc: around? 17:07:08 atomic WG members: we did add/remove a few, I need to put out a general call, though 17:07:22 list of blog posts .... that will be under meeting issues, later 17:07:30 those issues are all filed and I tagged them #meeting 17:08:05 docs/containers still in progress. docs is blocked on building infra, currently in progress 17:08:16 I haven't heard back from installer folks yet 17:08:33 jberkus: what do we need to re-action? 17:08:47 containers, docs, installer 17:09:23 #action jberkus to follow up on images from Flock container workshop 17:09:27 jberkus: is there anything in the realm of nagging the openshift installer folks I can help with or are you just waiting on an email/ 17:09:35 #action jberkus to follow up on docs from Flock workshop 17:09:45 #action jberkus to raise kube installation issues with openshift installer team 17:09:52 maxamillion: waiting on an email. if you're in a meeting with the right folks, then you can ask in person 17:10:15 maxamillion: really we're trying to figure out which of our various kube installer options is the most likely to converge with their efforts 17:10:46 ok other action items 17:11:10 * dustymabe to update the rolling release plan with info about f27ah 17:11:12 * dustymabe to update the ticket regarding rawhide atomic-ci stream 17:11:16 i need to reaction one of them 17:11:21 #action dustymabe to update the rolling release plan with info about f27ah 17:11:28 jberkus: rgr 17:11:52 I did update the atomic CI ticket for dennis gilmore 17:11:54 https://pagure.io/atomic-wg/issue/349#comment-472028 17:12:07 i'll follow up with him and close it 17:12:13 ok other action items: 17:12:16 jberkus: if I can, I'd like to be roped into that discussion ... my focus when I join the Ansible Team officially will be the intersection of ansible and k8s/openshift in various capacities so I have a long-term vested interest in that 17:12:25 * davdunc to blog AWS MP availability of FAH/Cloud. 17:12:31 * strigazi to do some testing around upgrades from f26 to f27 with 17:12:32 maxamillion: I'll send a new email 17:12:32 kubernetes 17:12:37 jberkus: thank you 17:12:38 * jbrooks to email fedora-devel with issue of needed deps for 17:12:40 asciibinder container 17:13:19 dang, I haven't done that yet -- I'll send a message after the meeting 17:13:32 #action jbrooks to email fedora-devel with issue of needed deps for asciibinder container 17:13:42 is strigazi or davdunc around? 17:13:44 The question there is basically, hey, does anyone care if we just use gems in the atomic doc pipeline vs rpms? 17:14:01 #chair maxamillion jbrooks ashcrow jberkus kushal ttomecek 17:14:01 Current chairs: ashcrow dustymabe jberkus jbrooks kushal maxamillion ttomecek 17:14:21 jbrooks: yeah I think the broader question to fedora devel is: 17:14:50 "hey i want to package this thing but there a bunch of deps that are missing and I don't want to be a maintainer for them all, what do you do in a situation like this"? 17:15:34 All right, and, does it really matter is asciibinder is packaged? 17:15:37 the current answer is "spend a lot of time writing a tool like gofed to automate all of the steps we've designed to be manual by default" 17:15:39 for reference, the current "official" asciibinder container, maintained by the openshift docs team, uses gems 17:15:42 on CentOS 17:15:42 I mean, I personally don't care 17:15:59 rather than e.g. driving autogenerating specs into the default pipeline 17:16:30 walters: if that's the answer, then our answer will be to use the CentOS-based container 17:17:07 what's funny is i think the copr guys have automated building rpms from gems 17:17:14 but we should get that answer officially 17:17:31 anywho, the discussion on fedora devel certainly won't hurt 17:17:34 dustymabe, not just copr, there's a *vast* set of these tools 17:17:38 many people feel this pain 17:17:54 I was able to build the deps easily in coprs 17:17:58 we can't use copr? 17:18:06 But officially maintaining them is a whole other thing 17:18:30 jberkus: sure we can use whatever we want. but this is a general problem that blocks software from getting into fedora 17:18:34 But if no rpms, gems alone works... I don't see the value of rpms there 17:18:38 it's worth a discussion 17:18:50 re-actioning other items now 17:18:52 OK, agreed 17:18:58 #action strigazi to do some testing around upgrades from f26 to f27 with kubernetes 17:19:06 #action davdunc to blog AWS MP availability of FAH/Cloud. 17:19:35 #topic re-evaluate atomic working group meeting time 17:19:41 #link https://pagure.io/atomic-wg/issue/346 17:19:53 so we have some times that have most people able to be there 17:20:15 i'm thinking maybe create a new doodle with just a few options and have people vote on that one 17:20:29 i.e. narrow it down to 4 options and then we can go from there 17:20:34 objections? 17:20:43 do you also want to prioritize? 17:20:55 in what way? 17:20:58 Do we not have a leader from the existing doodle? 17:20:59 e.g. the time should probably fit the WG members 17:21:09 ahh i see 17:21:19 so care more about the votes from actual members of the WG 17:21:59 i was considering most people eqal at this point 17:22:13 I'm not seeing any consensus in that doodle 17:22:31 jberkus: there are a few times that have 'all but three people' 17:22:40 ok 17:22:58 and we don't really know if the "no" is a "I absolutely can't make it" or a "I can make it, but I'd prefer not" 17:23:09 dustymabe: yah, that's a limitation of doodle 17:23:23 We can say, these are the top three times, vote 17:23:27 I figure the next doodle i make we can ask people to weigh that 17:23:31 jbrooks: right 17:23:36 ok that's enough on that topic 17:23:44 Doing another doodle seems to risk losing some people who just don't fill it out again 17:23:57 #action dustymabe to make new doodle with top 3 or 4 possible times and have people re-vote 17:24:34 jbrooks: same issue if you ask them to vote on it any other way 17:24:46 #topic The Future of AutoCloud 17:24:52 #link https://pagure.io/atomic-wg/issue/361 17:25:55 so.. 17:26:17 autocloud is in a bit of a questionable state 17:26:34 luckily it has been running fine lately, and I guess sayan is maintaining it 17:26:50 with roshi gone I'm not too confident about it moving to taskotron 17:27:18 s/roshi gone/roshi not as involved in Fedora/ 17:27:54 My thoughts are that the Fedora CI objective is where autocloud like tests should live in the future 17:28:09 agreed 17:28:10 there is a gap there, which is autocloud was being refitted to be able to test AWS instances 17:28:12 .hello davdunc 17:28:13 davdunc: davdunc 'David Duncan' 17:28:25 that makes sense 17:28:27 I think the CI effort is supposed to supercede the efforts of AutoCloud 17:28:29 so we could ask the Fedora CI effort about that 17:28:37 Ack dustymabe 17:28:59 davdunc: i re-actioned. if you have an update we'll get it from you in open floor :) 17:29:27 kushal: sayan: do you have any input about AutoCloud and the Fedora CI efforts? 17:29:57 I think with all the upstream-first test stuff that's going on, it just makes sense to join that effort 17:30:03 dustymabe: which autocloud tests? 17:30:09 https://upstreamfirst.fedorainfracloud.org/ 17:30:20 tflink: all of them?? 17:30:31 https://apps.fedoraproject.org/autocloud/compose 17:30:35 IIRC, roshi migrated most of the tests a while back but there's a remaining snag that hasn't been figured out yet 17:31:04 ie, the fedmsg is emitted before the images are actually available and the tests fail because they can't find an image 17:31:22 jberkus, Present 17:31:48 yay 17:31:51 dwalsh: welcome. we'll get to the issue in just a moment 17:32:32 tflink: ok. the question is, does having autcloud around at all any more help us *after* the Fedora CI pipeline is set up ? 17:32:44 coudn't tell you 17:32:55 tflink: ok 17:33:05 i think that is the question maxamillion is asking in the ticket 17:33:20 maxamillion: i think we know the answer to that? 17:33:22 tflink: which fedmsg? 17:33:41 I don't recall off the top of my head 17:33:58 dustymabe: do we? can Fedora CI run the AutoCloud tests? do we need to migrate them? what's that scope of work? 17:34:34 maxamillion: i think the Fedora CI should be able to run the equivalent (or better) of the autocloud tests 17:34:47 we will need to make sure that we have feature parity between the two 17:35:04 but that should answer the question of "what is the future of Autocloud" 17:35:27 unless I am missing anything 17:35:47 it's one of the fedmsgs which indicates that the images are done, I'll look for the exact fedmsg. it's been a while since I was looking into this 17:35:51 dustymabe: agreed, I just want to make sure this isn't hand-waived over and then it falls off the map without anyone doing the work 17:36:11 well until the work is done, we still have autocloud running 17:36:34 correct, but the tests still aren't authoritative which is unfortunate 17:37:22 org.fedoraproject.prod.pungi.compose.status.change is the fedmsg that we listen for 17:37:27 sure 17:37:35 maxamillion: do you want to update the ticket, or do you want me to? 17:38:07 Sorry I'm late. Double-booking is the worst. 17:38:19 #chair dwalsh gholms 17:38:19 Current chairs: ashcrow dustymabe dwalsh gholms jberkus jbrooks kushal maxamillion ttomecek 17:38:24 dustymabe: do we have a conclusion to add to the ticket as an update? 17:40:15 #info While autocloud has served us well in the past we don't have anyone working on new features for it and if it were to need serious maintainence we'd probably have trouble managing it. The Fedora CI pipeline (from the Fedora CI objective) should provide feature parity with Autocloud in the future and we should leverage that pipeline as soon as it does. This can eventually become our 17:40:16 authoritative source of test results 17:40:39 objections? 17:41:19 +1 17:41:27 tflink: kushal ? 17:41:43 sayan? thoughts/concerns? 17:42:43 moving to next topic in 20 17:42:58 that sounds like the plan, yes 17:43:09 #topic Decide strategy for including container runtimes in Atomic Host 17:43:11 dustymabe, Where are the topics 17:43:15 #link https://pagure.io/atomic-wg/issue/360 17:43:22 dwalsh: this one 17:43:42 I don't expect we'll fully decide on a strategy today, but it's worth a discussion 17:44:18 dwalsh: walters jberkus jbrooks - weigh in :) 17:44:38 I'm +1 to runtimes via system container 17:44:46 Perhaps pre-pulled, for convenience 17:44:56 the question is, do we want to include container runtimes (other than runc) in the base image, or pull them by system cotnainer 17:44:58 dustymabe: thanks 17:45:02 the pros and cons are outlined in that issue 17:45:08 Ok, when talking to potential users of Atomic Host, the topic of minimization always comes up. People want less on Atomic Host. Currently I believe we ship docker/etcd/flanneld and perhaps kubernetes? 17:45:12 that feeds directly into "include CRI-O in atomic host" 17:45:22 dwalsh: we removed kubernetes recently 17:45:28 Yes, but in 27 etcd flannel and kube are going 17:45:52 I would only ship atomic and runc on the base image. 17:46:31 I think atomic should be Container Runtime agnostic. There is a potential issue around not having the docker CLI, which could be problematic. 17:46:32 One issue is that we need to be really confident about FLIBS 17:46:45 The releases have to be very predictable 17:47:01 dwalsh: with docker and cri-o being provided via system containers if/when needed? 17:47:07 yes 17:47:24 jbrooks: can you explain a bit more about that point? 17:47:25 dwalsh: we install other clis with system containers 17:47:44 the upstream cri-o CI jobs seem to install directly on the host 17:47:49 The problem we have now is not just docker, but you also need etcd and flannel, or some kind of VPN tool to setup networks installed and running before the container runtime. 17:47:50 https://github.com/kubernetes-incubator/cri-o/blob/master/.travis.yml#L13 17:48:00 ashcrow, The FLIBS is very manual at this point, and maxamillion is the point person for it, and he's moving depts 17:48:27 I'm currently transitioning that to someone in Fedora RelEng, we'll have a new point person in the coming weeks 17:48:29 If containers are the way we deliver core parts of the atomic host, we need that delivery to be solid 17:48:53 jbrooks / maxamillion: OK I follow now, thanks! 17:48:53 It's just something we must consider, because we'll be leaning harder on it 17:48:56 jbrooks: agreed 17:48:58 jberkus, Ok, if you can install the docker CLI from the docker runtime onto some place like /usr/local/bin/docker, And similarly /usr/local/bin/kpod from crio 17:49:13 dwalsh, That should totally work 17:49:55 Allowing system containers makes it less opinionated and more flexible going forward. 17:50:10 +1 17:50:15 ok, i have some concerns 17:50:25 - first off, I feel like system containers are a bit of an "advanced user" feature 17:50:30 walters: Are you noting that our CI jobs are not testing cri-o the way we'd be using it in Atomic? 17:50:38 Someone might want to install a complete stack based on Origin that brings in etcd, openshift-vpn, CRI-O and origin as system containers. 17:50:39 ashcrow, AFAICS, yes 17:50:51 if someone wants to spin up atomic host and run an apache container 17:51:01 i don't think they should have to find a docker system container first 17:51:17 dustymabe, Couldn't this be handled via playbooks? 17:51:22 dustymabe: why not? if someone wants to do this on Fedora Workstation, they have to install docker first. 17:51:24 i also don't think we have the story of 'updating system containers' very polished 17:51:40 dustymabe: how different is that from someone having to install docker via yum on Fedora? 17:51:41 does anyone know where the cri-o jenkins job definitions live? 17:51:46 It's actually pretty smooth, just not well documented 17:51:57 ashcrow: it's different because you aren't using "yum" to install it 17:51:57 (for obvious reasons this is really hard to google) 17:52:01 which is what people are used to 17:52:01 :-) 17:52:30 they have to search for 'how do i install docker on atomic host' which is an OS designed to run containers 17:52:31 walters, Aren't they in the github? 17:53:03 dustymabe, atomic install docker versus dnf install docker. 17:53:24 dustymabe: that sounds like a "getting started docs" problem, not a technical problem 17:53:28 dwalsh: and for upgrades? 17:53:34 atomic upgrade docker 17:53:38 dwalsh, where? 17:53:41 now they are decoupled, good for power users, bad for beginners 17:54:13 if we do choose system container as the route. I'll vote for baking a system container into the media we ship 17:54:14 walters, Where is the github? https://github.com/kubernetes-incubator/cri-o 17:54:45 but then we'll essentially make our media quite a lot bigger 17:54:50 dwalsh, where are the jenkins jobs that power https://github.com/kubernetes-incubator/cri-o/pull/1029#issuecomment-337663158 17:54:53 so there are no easy answers here 17:55:11 dustymabe: I'm not buying the idea that adding "atomic install docker" as a *single step* is significantly harder for new users 17:55:33 It's not harder, but it's not what people are used to doing 17:55:38 I think that's more dustymabe's point 17:55:40 ah found them 17:55:52 To the extent that people are used to running atomic 17:55:56 ashcrow: they're used to doing it on every other OS 17:55:58 ashcrow: it also cannot tied to any other update mechanism 17:56:05 It's status quo for regular fedora or centos or ubuntu, etc 17:56:11 dwalsh, yep, looks like these tests aren't running on AH: https://github.com/kubernetes-incubator/cri-o/blob/master/contrib/test/integration/system.yml#L3 17:56:22 Yes under contrib/test 17:57:10 now, upgrade issues are potentially a significant problem, in that we lose some of the atomicity. 17:57:22 walters, Yes we are just getting to the point of packaging in an RPM, next step is to try to test the build artifacts. 17:57:44 an "rpm-ostree rollback" won't roll back an "atomic docker upgrade" 17:57:58 Right those are independent. 17:58:02 but that is balanced against breaking our deadlock around docker versions 17:58:08 But I think you can roll back a system container. 17:58:08 jberkus, *that* is a whole big thing... 17:58:10 which is a significant problem right now 17:58:16 here's the thing though 17:58:34 i don't see why we can't bake in docker, but allow people who want independence to run a system container 17:58:43 Yes, you can roll back a system container ... atomic containers rollback $NAME 17:58:53 flexibility + batteries included 17:59:06 We should bake in crio if we're baking in docker 17:59:07 dustymabe: because baked-in docker as we have it now is broken for users 17:59:13 why? 17:59:19 why is that broken for users? 17:59:39 dustymabe: because you need different versions depending on what you want to put on top of it, and we provide no way to do that 17:59:48 jberkus: system containers 17:59:58 It's not so much broken as that some systems pin to docker versions and the version that comes with AH may or may not be what they want. System Containers gets round it either way. 17:59:59 we did that with kube in fedora 26 18:00:01 dustymabe, Maybe do it in stages, where you ship docker system container preinstalled. 18:00:23 dwalsh: +1 18:00:33 jberkus: so we have baked in docker (which is what we test with) and then if someone wants something else we have system containerized docker for other versions 18:00:33 Docker preinstalled is not the norm 18:00:38 Let's not forget that 18:00:43 Ppl expect to install docker 18:00:58 unless it's an operating system for running containerized workloads 18:01:06 i know people who only use atomic because it's already installed 18:01:16 The real solution is to have dnf understand (system) containers. 18:01:20 just had this conversation with vbatts today 18:01:35 dustymabe: wait, so you're suggesting that we have one version of docker baked in, and tell the user to install a 2nd version if they need somethign different? With the understanding that 70% of users *will* need something different? That doesn't seem user-friendly to me. 18:01:37 Having dnf install docker on an atomic host, prefer a docker system container. 18:01:37 dwalsh: yes, something like that would give us a better story 18:01:51 dustymabe: what are those people using docker for? 18:02:01 Has anyone looked into a plugin for dnf to handle this? 18:02:24 oh man - looks like we have run off the rails a bit 18:02:25 dnf install docker hits the plugin which ends up doing a atomic install docker, and now dnf would list the package. 18:02:30 we're over time 18:02:35 oh, yeah, we're over schedule 18:02:53 i really want to pick up this discussion again 18:03:03 should we schedule a one off meeting or something? 18:03:05 dwalsh: https://github.com/ashcrow/dnf-plugin-container <-- the closest I've seen is what I wrote the other day but it's not the same thing 18:03:05 well, let's continue it on #atomic 18:03:07 we could take this to the mailing list? 18:03:26 yeah, mailing list or the issue 18:03:28 ok there were other meeting tickets 18:03:31 anything urgent? 18:03:34 #topic open floor 18:04:36 anyone with anything urgent for open floor 18:04:42 will close meeting in 2 minutes if not 18:05:50 #endmeeting