15:03:16 #startmeeting RDO packaging meeting (2015-06-03) 15:03:17 Meeting started Wed Jun 3 15:03:16 2015 UTC. The chair is apevec. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:17 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:03:29 #chair gchamoul number80 rbowen jpena 15:03:29 Current chairs: apevec gchamoul jpena number80 rbowen 15:03:36 #topic roll call 15:03:42 \o 15:03:48 \o 15:03:53 o/ 15:03:54 o/ 15:04:03 o/ 15:04:04 o/ 15:04:46 o/ 15:04:54 rbowen won't join us as he is having a company call right now 15:04:56 \o 15:05:09 o/ 15:05:27 ok 15:05:44 agenda is at https://etherpad.openstack.org/p/RDO-Packaging 15:05:55 currently only one topic: 15:06:04 #topic Post Kilo retrospective https://trello.com/c/oGSEYj8J/53-post-kilo-retrospective 15:06:30 thanks everybody who added their inputs in https://trello.com/c/oGSEYj8J/53-post-kilo-retrospective 15:06:55 looks like common theme is better planning :) 15:07:09 #info we need better planing 15:07:24 o/ 15:07:55 #chair chandankumar 15:07:55 Current chairs: apevec chandankumar gchamoul jpena number80 rbowen 15:08:06 Delorean should help us detected new deps but package review process should start at the same time when new package is added to rdoinfo 15:08:23 +1 15:08:33 #action apevec to expand master packaging add-new-package procedure 15:08:43 currently it is: 15:09:05 https://www.rdoproject.org/packaging/rdo-packaging.html#_how_to_add_a_new_package_to_rdo_master_packaging 15:09:37 related isssue is how to maintain maintainers 15:09:48 metamaintainers! 15:09:49 separate pkgdb2 instance was suggested 15:09:49 yes, we need more input from them 15:09:52 * jruzicka ducks 15:10:42 +1 15:10:42 number80, jruzicka - what do you think about my suggestion to add status flag to rdoinfo? 15:11:09 it's another (mis)use of rdoinfo, polluting jruzicka's clean design :) 15:11:11 apevec: it'll help 15:11:30 rdoinfo currently drives Delorean directly 15:11:31 we could even leverage this on board-like web page 15:11:46 with status flags we could create subsets 15:12:07 BTW we also have ownership info in CBS Koji and Fedora Koji 15:12:19 Fedora Koji is synced with Fedora pkgdb? 15:12:30 I think I need to map the current workflow 15:12:31 apevec: yes, 15:12:34 create diagram 15:12:44 see what's happening, what's the structure of everything 15:12:49 collect missing features 15:12:57 jruzicka: +1 15:13:08 and redesign rdoinfo in as backward compatible way as possible 15:13:11 diagram++ 15:13:15 but better clean with temporary breakage 15:13:23 then cr4p forevah 15:13:25 number80, and in CBS Koji it's whatever we put when doing cbs add-pkg ... i.e. not update after that 15:14:01 apevec, how would it look? 15:14:20 jruzicka, ok, so currently we start with rpm-master in github/openstack-packages 15:14:35 anyway, I'm all for having all the moving parts described declaratively 15:14:41 (NB there's upstream Packaging as project proposal which might change that) 15:14:49 but danger is to have too much redundancy 15:14:59 jruzicka, that's the trouble all parts are moving :) 15:15:04 hehe, right 15:15:12 apevec: yeah, atm, I don't pay attention to this but it will be important for our big tent packaging goals 15:15:14 so I say, pollute the rdoinfo for now 15:15:15 situation normal 15:15:57 and I'll redesing it in the near future once I myself know wtf is going on 15:15:57 jruzicka, for backward compat we just need to keep current field, adding should be fine 15:16:53 so back to rpm-master: we track upstream changes there, but at some point we have discontinuity(sp?) 15:17:10 when we need to move to dist-git to make "real" builds 15:17:38 so suggestion was to avoid this singular point, 15:17:45 IOW let's see what the real needs do to rdoinfo, and I'll try to clean it up afterwards 15:17:46 and just ship Delorean builds in RDO repos 15:18:05 I've trouble with that b/c of build reproducibility 15:18:22 jruzicka, ack 15:18:32 apevec: I think most of the mentioned issues are getting sorted, at least on the CentOS side 15:19:05 making Delorean builds suitable for production (signing, reproducibility) would be a lot of efforts to end up stumbling on the very same issues ... 15:19:20 yeah, we would reimplement Koji... 15:19:32 which is actually being rewritten :) 15:20:09 heh 15:20:35 why not make delorean *use* koji/cbs? 15:21:14 #info CentOS SIGs side is sorting out publishing pipeline 15:21:39 mburned, yeah, that one idea, question is about accounts 15:21:55 currently all account have human accountable for it 15:22:11 I'm not sure bot account in the buildsystem would be allowed 15:22:17 need rel-eng input on that 15:22:26 for Fedora, it's likely to be no 15:22:31 apevec: if we can build to a non-production tag... 15:22:43 number80, yeah, bots cannot sign CLA :) 15:22:46 or even scratch builds that we extract into our own repos... 15:22:56 :) 15:23:06 mburned, right, so that would be still 2nd class repos 15:23:09 then we have something a human could import and build into the correct tags 15:23:13 for production you'd still need real thing 15:23:29 mburned, yep, that's critical 15:23:30 apevec: yes, there is a manual step, but we'd have srpms we can immediately import 15:23:35 how to automate import 15:23:54 mburned, but I don't think scratch koji builds would help 15:24:07 they'd be still from master packaging templates 15:24:21 for real builds you need immutable NVR i.e. commit in dist-git 15:24:28 at least that's current Koji setup 15:24:30 moreover, delorean builds don't have changelogs 15:24:34 maybe that could be improved? 15:24:51 number80, yeah, but I think we should follow suse there i.e. separate file for changelog 15:24:58 +1 15:25:10 apevec: are fedora sigs the same? 15:25:20 I'm pretty sure that nobody would really complain if we start doing it though 15:25:22 use the same koji, same rules, same tags, etc? 15:25:35 also in upstream "no-more-stable-point-releases" discussion automatic changelog generation was suggested 15:25:47 with specially formatted commit messages 15:26:16 mburned: not at all, they're informal groups 15:26:19 mburned, fedora sigs (or actually workgroups?) require package to be in Fedora 15:26:23 i.e. there isn't CBS 15:26:30 ok... 15:26:37 I'm trying to champion CentOS SIG-like in fedora for a while :) 15:26:39 closest for Fedora would be copr 15:26:41 that doesn't help then... 15:26:51 yeah, 15:27:25 so current suggestion would be to improve centos SIG pipeline, solve import to dist-git 15:27:39 and eventually have production RDO repos only at mirror.centos.org 15:27:52 for Fedora would offer Delorean repos only 15:28:01 apevec: +2 15:28:09 apevec: so drop packages from fedora proper? 15:28:20 mburned: we still need the reviewing process 15:28:22 mburned, eventually yes, missing piece is centos dist-git 15:28:38 so we need to use fedora dist-git for now 15:28:39 number80: sure, but we can enforce guidelines as part of rdo 15:28:49 mburned: yeah 15:28:53 we don't *need* fedora builds for that 15:29:00 apevec: sure 15:29:01 mburned, yes, but would be reinventing the process 15:29:16 mburned, instead why not push Fedora to improve review process 15:29:28 mburned, note that we don't want _all_ deps in RDO 15:29:36 we do want to push general ones to Fedora 15:29:40 apevec: you're already proposing moving our packages out of fedora 15:29:42 like most of python-* 15:29:50 apevec: sure, deps should go in fedora 15:30:06 mburned, only openstack-* 15:30:18 mburned, we do want clients (and oslo as their deps) in Fedora 15:30:36 so that Fedora can be used as a client to openstack based clouds 15:30:42 jruzicka, ^ 15:30:56 agreed 15:31:05 apevec: we still get into trouble there... 15:31:07 with clients 15:31:22 yeah, upstream created stable/* branches for clients 15:31:24 mburned, upstream started using stable/RLS for clients as well 15:31:25 delorean repo could be used for people who want using fedora to build small PoC and development station so it should be good enough 15:31:33 but those are not really useful 15:31:35 jruzicka, ^ 15:31:41 right, so we have a Fedora --> Openstack mapping 15:31:49 even though we don't want it 15:31:50 mburned, now is first cycle, so let's see how stable client branches change the sad client scene 15:32:02 jruzicka, it made it worse :( 15:32:03 Is there any change on the rdo installer? After executing "yum install -y openstack-packstack" I got this error: https://paste.fedoraproject.org/228500/ 15:32:11 e.g. heatclient team wasn't aware 15:32:20 it asks me for python-docutils 15:32:21 so current heatclient stable/kilo is missing Kilo features... 15:32:23 if we have clients separate in rdo, then we can have a kilo-clients repo and a liberty-clients repo 15:32:23 quite some responsibility moved from us to upstream, which is convenient... but also loss of control 15:32:33 yesterday worked fine 15:32:58 both for the same fedora version 15:33:01 cucumber_, that seems to very old version , which repos you're using 15:33:04 apevec, I see. So we need to pressure upstream to take client versioning seriously 15:33:13 cucumber_, but please wait until meeting is over... 15:33:31 http://rdo.fedorapeople.org/rdo-release.rpm 15:33:34 jruzicka: versioning of clients forces mapping of fedora and openstack version unless the clients live outside... 15:33:35 oh.. ok, sorru 15:33:37 sorry 15:33:38 it's great all these magnificient features hit the core services, but that's not all that great when you can't talk to it 15:34:06 mburned, some clients are better than none. They still ought to be backward compat 15:34:22 jruzicka, yeah, that's what heatclient team promised: 15:34:33 master will work with Kilo deps and be backward compatible 15:34:51 still forces people to update fedora if they want latest... 15:34:55 None of promises about clients were met 15:34:55 so they should be safe to ship in Kilo repo 15:35:02 all clients should be backward compatible 15:35:09 so let's see this cycle... 15:35:10 but we don't have that guarantee for all clients 15:35:13 or we need to only ship latest clients regardless of version 15:35:20 trown, should, yes, but no guarantees :( 15:35:43 yes, it's this "one nice lady told me it should be like that" 15:35:51 mburned, if they work in Kilo repo, but how do we ensure 15:35:54 but mechanisms still missing 15:36:00 apevec: exactly my point... 15:36:05 anyway, that's upstream problem now 15:36:10 and more reason to have them outside fedora 15:36:12 so let's solve it upstream 15:36:21 the openstack client should also help wrt outliers in backward compatibility 15:36:33 we can bring added value by providing client that actually work with latest features 15:36:34 since that would be handled there 15:36:41 as we did till now 15:36:48 but we should pressure upstream to take care of that 15:37:06 jruzicka, you already had to backport one feature to OSC Kilo 15:37:34 yeah, yeah, there are stable/kilo branches 15:37:40 but it looks exactly the same to me 15:37:53 not too many fscks given about poor clients 15:38:16 well people who actually use them will hit issues, report issues, someone's gonna fix them... 15:38:48 jruzicka, mburned, my concern is that master clients could get new dep from Liberty reqs 15:38:54 Hmm, I need to go to next summit. 15:39:14 apevec: yes, we need to watch that 15:39:16 unless client team actively watches requirements updates (those are automatic from global reqs) 15:39:40 apevec: so far, they seem mostly safe 15:39:50 i looked at heat yesterday, only concerning one was pbr 15:40:00 mburned, only heatclient team (shardy actually) commited doing that in https://review.openstack.org/182672 15:40:15 mburned, pbr should be fine, I'm updating it 15:40:43 jruzicka, ^ have a look at that review BTW, to see the upstream stable clients story 15:40:53 unfortunately we have to let in new PBR so the gate works 15:41:21 shardy, ^ but it's still not completely clarified in https://review.openstack.org/182672 ? 15:41:39 ryansb, update PBR is actually better 15:41:52 it even dropped bogus pip dep from requirements.txt 15:42:03 ah, well that's good 15:42:36 ok, we went into weeds, let's get back to strategy level :) 15:43:03 #info Kilo bootstrap repo was bad combination of Juno and new deps in RDO Kilo GA location 15:43:24 #action apevec to start Liberty bootstrap with new deps in _testing_ location 15:43:32 and make Delorean Trunk use that 15:44:17 I think combining RDO Juno + boostrap Kilo created issues when doing real RDO repos 15:44:50 I'm here now, if the meeting is still ongoing. 15:44:52 i share that opinion 15:44:58 rbowen: you're more than welcome 15:45:01 I see that it is. :-) 15:45:04 also new-pkg procedure mentioned earlier, will require new package to be added to RDO Liberty via rdopkg update 15:45:20 rbowen, yes, I think we'll use full hour for sure 15:45:48 ok next was about doing bigger pkg changes, like splitting packages 15:46:10 #info Changes in packaging which required changes in the Puppet module... 15:46:19 apevec: will we expect ci for `rdopkg update` for liberty builds? or will those get a pass while we are building up those repos? 15:46:24 this needs to be done very early in the cycle 15:46:53 we need to have puppeteers reviewing these changes too 15:46:57 eggmaster, TBD - I'd sanity check, e.g. yum install *rpm works 15:47:03 seemed like for new kilo builds we had no ci support (at least none that was working) 15:47:03 I'd like... 15:47:28 jpena: you'd be ok, if I add you as reviewer when needec? 15:47:33 apevec: actually, changing packaging very early in the process will not allow us to have it fixed in the puppet side, because stable branches take some time to be ready after release 15:47:34 eggmaster, yeah, that's what we'd need to add for Liberty, before liberty-1 15:47:44 number80: you can count me in as well! 15:47:50 gchamoul: \o/ 15:47:54 There's not going to be a Liberty-1 15:47:54 gchamoul++ 15:47:55 jpena, ah good point 15:47:55 number80: sure, no problem. Feel free to add me. 15:48:05 rbowen, what do you mean? 15:48:22 jpena, and actually that brings the good point: 15:48:30 packaging changes must keep backward compat! 15:48:32 Unless I misunderstood Theirry's message, we're no longer going to do the point releases. 15:48:39 #info packaging changes must keep backward compatibility 15:48:45 Looking for the blog post. One second. 15:48:47 rbowen, that's for stable 15:48:55 apevec: +1 to that 15:48:57 liberty-1 is devel milestone, that's still there 15:48:59 Oh. I guess I did misunderstand, then. 15:49:23 here is L schedule https://wiki.openstack.org/wiki/Liberty_Release_Schedule 15:49:40 apevec, jpena: in some case, that's not possible 15:49:53 number80, then you'll get revert from dprince :) 15:49:59 like for glance 15:50:18 yeah, but we need at some point being able to do such changes :/ 15:50:21 this one https://review.gerrithub.io/230453 15:50:48 number80, it's not just puppet where we have folks participating 15:50:55 that's exactly what I want to avoid in the future :) 15:50:58 it's chef/ansible/bash scripts/whatnot 15:51:26 number80, default should be to keep backward compat 15:51:28 well, the problem will be allowing finer-grained deployment too 15:51:34 agreed on default 15:51:37 jpena, and puppet could support both ? 15:51:51 i.e. take advantage if subpkg split is available 15:51:58 or is that too much to expect? 15:52:29 number80, split into new subpkgs and keep meta subpkg for backward compat? 15:52:37 apevec: what do you mean? Supporting both the old and new packaging schemes? It looks difficult to me :-/ 15:52:38 if we can't change, then we should also be stricter during package reviews 15:52:51 jpena, yeah, I'd guess so 15:53:00 apevec: yes, we need to enforce meta-subpkg too 15:53:08 number80, you can't know in advance... 15:53:25 apevec: we can enforce having meta subpkg for all service 15:53:39 apevec: I can't think of an easy way to support two types of packages for the same platform 15:54:18 jpena, yeah, you'd need to query is some package avialable or not... 15:54:31 jpena: we couldn't have puppet-modules starting to support a newer type starting a specific upstream release? 15:55:05 number80, so shall we test drive such changes asap: repropose glance api/registry and proposed sahara split 15:55:15 and see first if backward compat is doable 15:55:30 number80: yes, that is doable (we did that for neutron lbaas/fwaas in this cycle, for example). The only issue is timing: we cannot do anything liberty-specific that breaks backwards compatibility until the stable/kilo branch is ready 15:55:35 apevec: for glance it's not possible, we had conflicting needs 15:55:47 jpena: I see 15:56:08 apevec, jpena: let's reschedule this discussion as soon as we have liberty branch then 15:56:19 number80, I think there was some combination which should work, but forgot... let's just start it again 15:56:29 ok 15:56:35 number80, for puppet-* ? 15:56:45 jpena, what is ^ that expected? 15:56:47 apevec: yes, it seems to be easier 15:56:57 actually, stable/kilo for puppet-* ? 15:57:07 and master would be then open for L changes 15:57:19 from my understanding master is currently kilo for puppet- 15:57:25 yep 15:57:34 number80: yes! 15:57:42 so question is when stable/kilo for puppet-* is expected? 15:58:14 apevec: I'm not aware of a set date, from https://etherpad.openstack.org/p/liberty-summit-design-puppet-master-branch they say "master breaks when there is packaging from upstream", which in RDO means.. now 15:58:41 right 15:58:42 gchamoul: do you know if there is already a defined date for stable/kilo branch in puppet-* ? 15:58:58 apevec: AFAIK I lost the argument and should abandon that patch 15:59:13 jpena: no date yet! and the PTL is on PTO ... 15:59:14 apevec: maybe confirm with stevebaker, but I belive we plan to just keep releasing from the master branch 15:59:42 shardy, ok, but you commit to keep deps Kilo-compatible? 16:00:19 stevebaker, ^ 16:00:40 shardy, otherwise we can publish master relase in RDO Kilo repo 16:00:49 ahem, cannot ! 16:01:11 apevec: well, we can, but people won't be too happy... 16:01:32 mburned, what do you mean? 16:01:46 apevec: we need to have that discussion with stevebaker, my personal opinion is yes, but stevebaker normally handles releases for heatclient, so ultimately he gets the final say (also he's PTL atm.. ;) 16:02:38 shardy, ack 16:02:50 ok, time is up, let's quickly review remaining low points... 16:03:10 #info Improving reviewing process 16:03:19 apevec: i mean we could physically ship the package, but it would break users 16:03:26 so they'd be mad if we did update it 16:03:32 * mburned was trying to be glib... 16:03:59 mburned, that's what I'm trying to ensure w/ shardy / stevebaker - we don't want to break anything 16:04:10 ensure backward compatible deps is first step 16:04:13 I crossed the points already dealt in the trello card 16:04:13 apevec: yes, i agree 16:04:20 but also keeping backward compat CLI, API etc 16:04:28 number80, thanks 16:04:50 #info Identifying maintainers for every component 16:05:14 I'd like we encourage community members here 16:05:18 also touched, I'll try to expand rdoinfo to be our "pkgdb" for everything we ship 16:05:25 yes! 16:05:54 #info delorean/master stopped or stuck 16:06:14 that's under discussion, we're looking for better hosting 16:06:38 also we shall puppetize it, so we can rebuild delorean server easily 16:06:54 #info delorean notification 16:07:00 apevec: I take this task! 16:07:05 should be fixed w/ better hosting and puppetizations 16:07:08 gchamoul, thanks! 16:07:31 now quickly + points ( yes, we have them! :) 16:07:34 gchamoul++ 16:07:34 gchamoul: if you need help or a betatester, let me know ;) 16:07:49 jpena: ack! 16:08:02 #info Delorean Trunk worked fine (we it was working) 16:08:31 #info and last but not least: More community participation <3 16:08:41 apevec: number80: And any packaging experts, not sure if you've noticed this response yet upstream -- http://lists.openstack.org/pipermail/openstack-dev/2015-June/065679.html 16:08:47 thanks everybody who participated in Kilo cycle, more fun stuff is coming! 16:08:54 thanks to alphacc without whom we couldn't have shipped this release in time \o/ 16:08:58 thx everybody! 16:08:58 Err sorry, didn't notice the meeting in progress :-( 16:09:11 #topic open floor 16:09:26 kashyap, it's just appropriate for open floor, lemme have a look... 16:09:27 apevec: Okay, I think I arrived at right time. 16:09:29 kashyap: Interesting. That's actually very relevant to the meeting, it seems. 16:09:31 Exactly. 16:09:59 rbowen: I was looking at scroll, and totally forgot meeting has started. As I'm switching between Nova channel and here. 16:10:17 apevec: Yeah, especally the last paragraph. 16:10:19 important 16:10:38 apevec: What is the hurry for Thomas, I don't see. 16:11:01 I do not get it too :/ 16:11:05 Hmm. Having Debian/Ubuntu as part of the "official" packaging, and CentOS/Fedora/SUSE *not* makes a pretty obvious statement. 16:11:08 me neither, I guess he wants to offload work, 16:11:17 it was all him 16:11:17 that is, Ubuntu is the right place to deploy OpenStack. 16:11:22 He seems to be in a big hurry. 16:11:23 and Ubuntu was just taking his work 16:11:37 So, someone might want to comment and ask about what's the hurry 16:11:42 So I'm just releasing rdopkg-0.28 with `rdopkg reqquery` 16:11:59 jruzicka++ !!! 16:12:03 which is able to query requires versions across rdo/fedora repos 16:12:19 source can be requirements.txt, $THAT from git, .spec file 16:12:28 number80, we need to have a look at opensuse specs and see how we could work with them 16:12:30 jruzicka: There's another topic in discussion. 16:12:34 :-) 16:12:43 open floor is open :) 16:12:48 sry 16:13:01 apevec: I suggest that we split this in two part: gerrit integration for review as we do for delorean and figure out the tooling part 16:13:02 Open floor == one topic at a time. 16:13:13 *later 16:14:11 number80, right, that's what derekh suggested: we donate openstack-packages namespace so we can get deb/rpm spec reviews going in the open 16:14:27 and infra tooling could be developed at its own pace 16:14:41 we have delorean which is fine for now 16:15:08 and deb could have whatever they want, if they don't like delorean design 16:15:15 sbuild or whatnot 16:15:30 I'll ping opensuse folks btw 16:15:48 number80, we already got pinged, I'll fwd you email from derek 16:15:58 ok 16:16:22 but I wanted to quickly review their specs before replying 16:16:59 I did in the paste and noticed separate changelog file which I liked 16:17:02 past 16:17:22 oh yes, separate changelog sounds good to me 16:17:35 I forgot where's their dist-git ... 16:18:12 anyway, we'll followup on that 16:18:28 meeting is running over, let's release the channel 16:18:39 last chance for meeting minutes! 16:18:46 3 16:18:57 2 1 16:19:02 thanks everybody! 16:19:05 #endmeeting