14:34:09 #startmeeting CentOS Atomic SIG 14:34:09 Meeting started Tue Oct 7 14:34:06 2014 UTC. The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:34:09 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:34:09 Meeting started Tue Oct 7 14:34:09 2014 UTC. The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:34:09 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:34:24 * Evolution here 14:34:30 #chair quaid imcleod jbrooks Evolution gregdek walters 14:34:30 Current chairs: Evolution gregdek imcleod jbrooks jzb quaid walters 14:34:30 Current chairs: Evolution gregdek imcleod jbrooks jzb quaid walters 14:34:38 .hellomynameis imcleod 14:34:39 imcleod: imcleod 'Ian McLeod' 14:34:49 .hellomynameis jzb 14:34:51 jzb: jzb 'Joe Brockmeier' 14:35:20 .hellomynameis jasonbrooks 14:35:21 jbrooks: jasonbrooks 'None' 14:35:32 #topic Hardware for bare metal work 14:35:54 quaid: I think this is still in process, but the request has been filed, yeah? 14:36:11 kbsingh: you had some questions on this? 14:36:24 jzb: does it need to be bare metal, or is a vm acceptable? 14:36:50 Evolution: this was for Anaconda work, IIRC 14:36:59 Hello 14:37:05 Evolution: walters was saying he needed bare metal 14:37:19 dwalsh: welcome 14:37:24 Evolution, the compose scripts use (libguestfs | ImageFactory) which both spawn VMs, so if the compose server is a VM, you're in vm-in-vm land which is slow. 14:37:26 walters, jzb: FWIW, we have been able to use VMs with nested-virt enabled for ImageFactory and Oz development. Perf is slightly degraded but not horrible. Far better than full emulation. 14:37:36 okay. that should be doable. a vm is easiest to provision, but we should be able to allocate bare metal as well. 14:37:44 walters: I can point you at an internal host to fiddle on if you'd like to try. 14:38:08 OTOH-I fully support HW purchases. Always..... 14:38:40 Evolution: I think you're looped in on the email thread about this, but if not I will loop you in. 14:39:07 jzb: you and I each filed a bug on this, so mostly we just need to make sure we're saying the right things. I linked mine to yours. 14:39:08 imcleod, last i tried it was ~10x perf hit 14:39:20 though things are worse with libguestfs 14:39:24 hi guys 14:39:38 kbsingh: we're on the hardware topic 14:39:51 kbsingh: maybe we can get that hammered out now? 14:39:59 sounds good 14:40:23 kbsingh: what do we need to settle to get that provisioned ASAP? 14:40:29 so what do we need the hardware machine for - the anaconda dev team has a rather large lab of their own 14:40:40 which also includes a mainframe iirc 14:41:12 to be clear if a VM is easiest to quickstart i'm fine with that 14:41:30 the treecompose part *doesn't* need baremetal, and that's most critical to run often, the images are a periodic thing 14:41:55 I am still struggling to figure out what the plan or intention is 14:42:05 kbsingh, building trees and images 14:42:23 jbrooks: so you need access to a git repo ? 14:42:42 kbsingh, a machine, preferably bare metal 14:42:47 is what Colin needs 14:43:00 (and docker base and prototyping out layered images are an important part of this) 14:43:36 walters, kbsingh: Ideally, we'd get some additional baremetal available to the CBS koji instance and start building there, in a tracked and shareable way. 14:43:44 Similar to what we are doing for the Fedora builds. 14:43:47 That's my thought anyway. 14:44:38 imcleod: thats my understanding as well, its what i sort of mentioned in the cbs meeting on Monday - if were building official images, and if ImgFac is able to do that now, then the builders should come up in cbs 14:44:51 which is also why i am struggling to workout what this baremetal machine is needed for. 14:45:23 we have a few machine ( 6 to be exact ) that can be used for SIG work, and i can likely find resources on a machine - eg the one that virt sig is using for some of their stuff ) 14:45:29 I see walters saying enormous perf problems with vm-on-vm 14:46:08 but at the moment, i dont think anyone actually has either a plan or a direction they want to go into mapped out 14:46:15 so maybe first step might be to work that out 14:46:27 We talked about this last meeting 14:46:37 ( either that, or i am missing something... in which case, tell me.. and we can just go find a machine for 45 days, and revisit then ) 14:47:17 walters requested a machine, and getting that machine is what we're talking about -- maybe Colin can provide more detail? 14:47:37 * jbrooks has done all my stuff in VMs so far 14:48:19 lets start at the top 14:48:30 we have an atomic build out there, should we get that updated real quick ? 14:48:35 kbsingh: is a plan as simple as, "We need a machine for 60 days to do development work on X, with a goal of picking out the best pieces of that infra and putting them in cbs.centos.org." 14:48:35 I've taken down the AMI already 14:48:50 kbsingh, If that's going to be the build we'll use, we really need the json 14:49:00 jbrooks: right, so lets work on that 14:49:05 Like, in an email, or anything 14:49:28 then, in the longer term ( ie. a week from now ) - how are we going to be building images ? so far, I hvent seen an authoritative answer on use-anaconda or use-toolbox 14:49:39 kbsingh: is the system you've used to generate the existing image somewhere we can turn over to Ian? 14:50:07 jzb: i think its just the jason 14:50:07 erm 14:50:20 json that needed, i can put that into a git repo and setup a pull from there to build 14:50:36 That would be good 14:50:43 if you guys want to skip to next point, i can go do the git thing now, and we can cover it before EOF 14:51:57 That's fine w/ me 14:52:02 sorry, got pulled 14:52:07 ah, cool 14:52:09 basically we can skip the baremetal if necessary 14:52:17 most important thing is to get the sources in centos 14:52:18 walters: what do we lose if we skip that? 14:53:47 speed, up to 10x, anecdotally, right? 14:54:40 yes just speed 14:55:08 walters: "just speed" - is this going to inhibit the ability to get things done, though? 14:55:39 probably not 14:55:39 walters: e.g., if you can do builds at regular speed will we gain days getting this out the door? 14:56:14 And we can expect bare metal on the horizon when we mash this into koji 14:56:20 cbs 14:56:38 imcleod: anything to add / any thoughts here since this is also going to be on your plate? 14:57:06 jzb: Just that I think Colin should get whatever HW he wants, but I'll try to help speed up his VM performance in the meantime ;-). 14:57:17 imcleod: +1 14:57:51 excellent 14:58:07 welp, I'm apparently terrible at telling UTC time and totally missed this one ... sorry all :( 14:58:14 kbsingh: here's my suggestion, if we have the ability to provision this for a short term (e.g. 45 days) 14:58:20 kbsingh: we should do so. 14:58:32 kbsingh: if we can't, then let us know today and we'll move forward in VMs 14:58:39 but I'd like to get this item decided 14:59:24 * dustymabe is just as bad as maxamillion 14:59:27 jzb: so here is what i think we can do 15:00:01 lets get the json files up, lets do a build or a few, lets get a good image we can promote at people in a way that they can use it - that should be fairly straightforward with whats already in place 15:00:33 once we have an image set we are happy with, we can reprovision the box that is being used as the builderbox now, i can remove all non atomic stuff from there, and that can be the dev machine 15:00:57 kbsingh: builderbox for Atomic? 15:01:03 that also gives us the room to keep doing side builds, which dont need to make it to the release / dev cloud.centos.org setup 15:01:05 kbsingh: does imcleod have access to that? 15:01:54 no, noone has access to that apart from me and johnny, its part of the centos buildservices ( the private one ) 15:02:00 jzb: what's builderbox? 15:02:14 maxamillion: ask kbsingh :-) 15:02:26 kbsingh: what's builderbox? 15:02:43 arbitary container that runs builds :) 15:02:47 (not trying to derail the meeting, just curious ... I can follow up later if that's preferred) 15:02:54 can be baremetal or vm or cluster of vm's 15:03:03 kbsingh: ah 15:03:05 in this case, its a blade in a bladecenter 15:03:20 kbsingh: to be clear, my understanding is that imcleod will be taking on getting the Atomic builds done. 15:03:36 kbsingh: but it's sounding like this setup will still depend on you? 15:03:52 hence my instance on a plan for it to not keep depending on me 15:04:22 imcleod: can you and kbsingh get together and set up a timeline for transitioning this? 15:04:23 where 'it' => the atomic image we ship for centos atomic 15:04:43 transition to what ? 15:05:20 kbsingh: where imcleod is getting the builds done 15:05:21 kbsingh: Transition to me being responsible for building it, and having access to the needed resources to do so. 15:05:26 jzb: ^^^ - correct? 15:05:43 imcleod: yes 15:05:52 kbsingh: Sound good? 15:07:14 to me this is more of a tooling thing than a resource thing, but i seem to be missing something here, so lets move on. 15:07:38 is anaconda able to build atomic images at this point ? 15:07:54 kbsingh: I'm not sure that's the case. 15:08:14 kbsingh: what I'm trying to establish is at what point will imcleod be able to generate a build. 15:08:31 walters: Correct me if I'm wrong. The answer on Anaconda is "yes in F21, yes in RHEL, not in CentOS at the moment" 15:09:16 right 15:09:29 kbsingh: This will get at another agenda item for my discussion with you. Namely, assuming we have Anaconda patches to create an Atomic-capable CentOS 7 installer, I'm not sure how we would turn that into a composed CentOS 7 tree. 15:09:34 Is this rhel 7.1's anaconda that we need? 15:10:01 kbsingh: The CBS is very clearly for generating additional packages. The core build and compose are done on a different system in a non-public way, correct? 15:10:01 kbsingh, https://copr.fedoraproject.org/coprs/rvykydal/anaconda-el7-atomic/ 15:10:05 so then the 'transition' that is needed is to get the toolbox stuff deprecated out and replaced with an anaconda build able to generate the images we need, right / 15:10:36 I think the transition is that the current test image can't be reproduced by anyone else 15:10:41 imcleod: right, the distro is still built with the older system, as are the isos and images - eventually we will end up with one system 15:10:49 And we need to transition away from that 15:11:20 jbrooks: hopefully that will happen in a few, i just need to relocate to a diff machine and tunnel into machine to get content 15:11:33 kbsingh, sweet 15:12:03 Then, we can build images in the deprecated way. But if we're to start building the new way, we need documentation on that 15:12:16 Is that all documented by the Fedora peeps now? 15:12:57 kbsingh, only kind of...the toolbox code now contains a wrapper for imagefactory in addition to the old libguestfs way 15:13:11 jbrooks: It's as documented as any of the fairly deep components of the build system, which is to say "slightly, and at least a bit by oral tradition". 15:13:35 jbrooks: But trying to duplicate the process in a parallel CentOS koji instance will certainly help identify doc gaps. 15:13:47 imcleod, cool 15:13:57 So for now, we proceed in the old way? 15:14:00 Or both 15:14:20 I want to see updated images and updates avail 15:14:43 me too =) 15:15:28 We can take the draft atomic definition that jzb put together and use it to update the json 15:15:29 jbrooks: FWIW-I'm prepared to move forward either way, or both ways in parallel. Right now the images are ad-hoc, by one person. I don't think we need the next formal step to be "fully automated inside of a merged CentOS koji/core build system" 15:15:39 Agreed 15:15:51 jbrooks: "Partially automated on a sidecar build host" sounds just fine. 15:16:03 i'm going back to step 0: figure out where to build packages for centos, and is it OK if they override (e.g. systemd) 15:16:20 walters: I believe we can do that in the CBS koji instance with appropriate tags. 15:16:27 walters, I think that it's ok, by definition (of a SIG) that pkgs overlap 15:16:32 Where to build is a good Q 15:16:44 walters: With a potential backup of continuing with COPR. 15:16:56 COPR now, CBS soon? 15:17:07 at this point I'm happy to just put down my CC for an AWS instance or instances to set up a build system 15:17:09 The current test image uses copr pkgs 15:17:30 if it's what it takes to get moving. 15:17:31 jzb "CC" ? 15:17:37 imcleod: credit card 15:17:56 jzb: Unfortunately, the one place I can guarantee poor "VM in VM" perf is inside of any public cloud. 15:18:13 At least one important pkg for us, docker 1.2, is currently built in CBS 15:18:25 imcleod: just expressing my impatience 15:18:27 jzb: Now, if you're willing to put your CC down for ebay, I have some ideas. 15:18:28 yes, it's not just about the ostree part of atomic - need to figure out the docker interlock 15:18:48 I feel like we've still not reached a plan here. 15:18:56 also we are trying to do geard -> kubernetes, which is now built in fedora 15:19:33 OK, kbsingh is putting the json in a git repo, and we can use that to update the image 15:19:45 W/ the update in the deprecated way to start 15:20:02 jbrooks: The deprecated way meaning toolbox, yes? 15:20:05 new pkgs can come from a combo of copr and cbs, w/ the intention to move to all cbs soon 15:20:10 imcleod, Yep 15:20:14 https://github.com/CentOS/sig-atomic-buildscripts coming here 15:20:35 we can also setup a sig-atomic team there to run that, otherwise Evolution and I can do pull req's in the interim 15:20:48 jbrooks: ^ 15:20:50 thanks, kbsingh 15:20:51 kbsingh: please add me and jbrooks 15:20:56 and imcleod 15:21:01 imcleod: what's your GH username? 15:21:21 kbsingh: (I'm jzb) 15:21:22 i'm cgwalters 15:21:26 mine is jasonbrooks 15:21:28 jzb: I am imcleod 15:21:33 Evolution: ^ 15:21:45 jbrooks: I always look for "jbrooks" first. 15:22:00 * jbrooks not into the whole brevity thing, on GH 15:22:34 * number80 suggests that we add github handles in FAS3 15:23:01 good idea 15:23:10 OK, once we have the json - do we have a plan for next steps? 15:23:16 imcleod: ^^ 15:23:40 kbsingh: jperrin --^ gh username 15:23:41 is sig-atomic-buildscripts the equivalent of https://fedorahosted.org/fedora-atomic/ ? ie the definition of the host? 15:23:58 Evolution: erm, i meant could you please do the team thing :) 15:24:21 oh, certainly. wasn't paying attention and missed the context in scrollback 15:24:30 thanks 15:24:40 I'm not sure how user teams work without ading them to all of /CentOS 15:24:41 jzb: I will attempt to build with the JSON on HW I already have access to. 15:25:11 jzb: Then formulate a plan for getting the Docker and Atomic updates into either COPR or CBS on an ongoing basis, in collaboration with walters and dan. 15:25:29 jzb: Then formulate a plan for building regularly using that update stream. 15:25:51 jzb, kbsingh: Do we have AWS credentials for the CentOS project proper? I can look at auto-updating AMIs in that case. 15:26:04 #action imcleod Attempt to set up private builds, then a plan for getting Docker + Atomic updates into COPR or CBS 15:26:32 #action imcleod work on plan to create regular Atomic builds 15:26:35 imcleod: you'll need that from the project atomic side of things 15:26:47 #action Evolution add SIG members to github repo 15:27:01 #action kbsingh upload JSON files for Atomic to https://github.com/CentOS/sig-atomic-buildscripts 15:27:05 did I miss anything? 15:27:34 kbsingh: By that you mean that project Atomic has its own credentials? (Or needs them?) 15:28:19 okay. invites sent. 15:28:48 imcleod: i'd guess so 15:28:58 we're approaching the 1 hour mark, and I know several folks have a meeting 15:29:00 although we have a marketplace setup etc, there is no 'centos' account in AWS 15:29:15 kbsingh: How have you uploaded AMIs? PErsonal credentials? 15:29:19 kbsingh: how do you upload the CentOS images? 15:29:22 what imcleod said. 15:29:37 you dont need to upload AMI's in the AMP 15:30:03 this is the decade of the cloud, these things happen over shared excel spreadsheets and raw backing files over http 15:30:21 ( not kidding.. ) 15:30:23 o.O 15:31:04 kbsingh: I've never touched the marketplace stuff, only the public API to make public AMIs. Anyhew, I guess we can pick this up on the next call, by voice. 15:31:07 overall, I would love to have publicly available CentOS Atomic hosts with online updates by AWS re:invent (Nov 11) 15:31:50 walters: thats a good target 15:32:08 walters: just for completeness I'll give you that AI 15:32:08 :-) 15:32:35 #action walters work with imcleod to see if we can get CentOS Atomic AMIs by AWS re:Invent. 15:32:37 the online updates is a very important aspect of this discussion - it's not just about uploading an image once, we need a flow of trees (and ideally, regularly updated docker images built from centos content too) 15:32:44 walters: +1 15:33:30 OK - any last items before we adjorn? 15:33:50 in rpm-ostree-toolbox we've been prototyping picking up package build notifications from a QPID instance after Koji builds 15:34:26 but if we just get the packages built that's a good first start 15:34:35 can someone link me to the docker CBS builds? 15:34:50 jzb, nothing else from me beyond that last q 15:35:01 walters: OK, thanks@ 15:35:03 er, thanks! 15:35:11 #action jzb get link to Docker CBS builds for walters 15:35:23 OK, winding it up -see you all on the mailing lists. 15:35:26 #endmeeting