15:31:09 <dgilmore> #startmeeting RELENG (2015-07-06)
15:31:10 <zodbot> Meeting started Mon Jul  6 15:31:09 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:31:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:31:19 * pingou o/
15:31:23 <dgilmore> #meetingname releng
15:31:24 <zodbot> The meeting name has been set to 'releng'
15:31:24 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:31:24 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:31:26 <dgilmore> #topic init process
15:31:31 * nirik waves
15:31:41 * maxamillion is here
15:32:25 * pbrobinson waves
15:32:52 <dgilmore> #topic #6164 bodhi2 status update requested
15:32:58 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164
15:33:07 <dgilmore> lmacken: where is bodhi2 at currently?
15:33:34 <lmacken> dgilmore: working on the staging setup, working through some proxy issues today.
15:33:52 <lmacken> will be hacking with threebean in person today, and will hopefully get through these last deployment roadblocks
15:33:54 <dgilmore> lmacken: okay, is there things we can and should be poking at?
15:34:27 <lmacken> dgilmore: not right now, but soon we'll be able to point our tools at it and start testing pushes
15:34:39 <pingou> cool :)
15:34:43 <dgilmore> lmacken: okay, that would be good
15:35:04 <dgilmore> #info lmacken and threebean working together in person today
15:35:18 <dgilmore> #info soon we'll be able to point our tools at it and start testing pushes
15:35:30 <dgilmore> thanks lmacken, looking forward to testing it out
15:35:42 * nirik is also happy to help any way I can
15:35:46 <dgilmore> #topic #6200 Cannot mirror Fedora drpms using OpenAFS
15:35:50 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6200
15:36:21 <dgilmore> so I am not opposed to putting the deltas in hashed directories
15:36:32 <pbrobinson> yea, I think that makes sense
15:36:43 <pbrobinson> same as we did for the standard rpms
15:36:45 <dgilmore> not sure it really will do much and it seems using openafs for mirroring is a plain bad idea
15:36:46 <nirik> sure, or we could keep fewer?
15:36:57 <nirik> agreed.
15:37:02 <dgilmore> nirik: not sure we really can
15:37:47 <nirik> yeah, I suppose not.
15:38:12 <nirik> since people do fresh installs and update (so that needs the base->newest) and we want to keep the last -> newest
15:38:55 <dgilmore> yep
15:39:07 <dgilmore> what ones we should keep is kinda hard to really know
15:39:23 <dgilmore> speaking of what to keep
15:39:49 <dgilmore> I think i closed a bug as wontfix that was requestiong we never remove rpms from updates and updates-testing repos
15:41:07 <nirik> I think thats fine. I don't think we want to keep all packages for all time.
15:41:17 <dgilmore> I do not think so either
15:41:22 <nirik> I personally wouldn't mind keeping N and N-1 but thats just me it seems
15:42:37 <dgilmore> does anyone want to try write a patch for createrepo and createrepo_c to hash the deltas?
15:43:09 <dgilmore> createrepo is actually pretty broken upstream in regards to delta support right now
15:43:35 <dgilmore> I am not sure that createrepo_c even supports hashing the rpms
15:43:48 <dgilmore> nirik: I think that is just you
15:44:16 <nirik> seems like. ;)
15:44:58 <pbrobinson> hehe
15:44:58 <dgilmore> #info someone needed to look at adding support to hash deltarpms and make sure createrepo_c actually supports hashing rpms
15:45:28 <dgilmore> #topic Secondary Architectures updates
15:45:29 <dgilmore> #topic Secondary Architectures update - ppc
15:45:32 <maxamillion> nirik: I'm with you on N and N-1 actually :)
15:45:38 <dgilmore> pbrobinson: how is ppc?
15:45:57 <pbrobinson> not bad
15:46:11 <pbrobinson> there's an issue with nightly composes
15:46:27 <pbrobinson> randomly shitting themselves with next to no mash debug
15:46:38 <dgilmore> always fun
15:46:39 <pbrobinson> so I need to look @ them closer this evening
15:46:57 <dgilmore> okay
15:46:58 <pbrobinson> other than that mass rebuild is tinkering along
15:47:10 <dgilmore> cool.
15:47:25 <dgilmore> how is the rebuilding of infra going?
15:48:15 <pbrobinson> focus on this week, was actually poking at ansible and friends earlier
15:48:46 <dgilmore> awesome, I would like to see us get everything done before branching
15:48:50 <dgilmore> which is 3 weeks ago
15:48:52 <dgilmore> away
15:49:12 <pbrobinson> that's the plan
15:49:20 <dgilmore> cool
15:49:29 <dgilmore> okay lets move on unless you have something else
15:49:40 <dgilmore> #topic Secondary Architectures update - s390
15:49:42 <pbrobinson> also need to poke mike about getting a new koji release done too
15:49:57 <pbrobinson> no Czech Rep people today
15:49:57 <dgilmore> pbrobinson: yes we do, for primary and secondaries
15:50:29 <dgilmore> #info 15:50 < pbrobinson> no Czech Rep people today
15:50:32 <pbrobinson> dgilmore: yes, even as it stands I believe we have enough for a .10 release even if we delay other features until .11
15:50:38 <dgilmore> #topic Secondary Architectures update - arm
15:50:43 <dgilmore> pbrobinson: right
15:50:50 <dgilmore> hows aarch64?
15:51:14 <pbrobinson> so here we've now started to move forward on mass rebuild now the qemu/tooolchain issue is resolved/understood
15:51:35 <pbrobinson> we're around 9K builds behind but that should pick up considerably this week
15:51:54 <dgilmore> awesome
15:51:56 <nirik> pbrobinson: when were we going to move the builders?
15:52:04 <dgilmore> are we also working to rebuild the infra?
15:52:22 <pbrobinson> nirik: I'm not sure who is shipping what from where
15:52:48 <nirik> pbrobinson: Smooge will be in phx2 the week of the 27th.
15:52:49 <pbrobinson> I need to dig back into email to ensure we have the 6 or so aarch64 boxes we're moving around in place time for smooge
15:52:56 <nirik> so ideally we could ship them the week before...
15:52:58 <pbrobinson> nirik: yep
15:53:08 <nirik> hopefully this week and next is enough to catch up on mass rebuild?
15:53:49 <pbrobinson> if no we'll pause it or use some other temp HW in the interim, we have 2 builders already in PHX, that will allow to at least keep moving along slowly
15:53:57 <nirik> ok. ;)
15:55:08 <pbrobinson> and once we have them there it should be all straightforward
15:55:16 <pbrobinson> update firmwares and install
15:55:18 <dgilmore> #info will have some new builders in phx2 soon
15:55:48 <pbrobinson> nirik: btw what's the status  of the new s390 hub/db, I need to review the secondary groups/ACLs
15:57:03 <nirik> pbrobinson: I've not done anything with it recently. as I recall it should be all setup, but we couldn't run kojid on it because we needed to add the new hub hostname as a builder... and sharkcz wanted to implement a 'shared shadow' setup...
15:57:15 <nirik> if you or he could move that forward that would be lovely.
15:57:29 <nirik> I can also make arm ones anytime you like.
15:58:20 <pbrobinson> nirik: OK, I'll liaise with sharkcz when he's back
15:58:39 <pbrobinson> seems like I'm 110% secondary infra this week ;-)
15:58:40 <nirik> cool. I'm happy to help with ansible side if any changes needed there.
15:58:59 <pbrobinson> cool
15:59:01 * dgilmore would be happy if we get all hubs installed and configured the same
15:59:10 * nirik too, we will get there.
15:59:24 <dgilmore> anything else?
15:59:31 <pbrobinson> not from me
15:59:55 <dgilmore> #topic Open Floor
16:00:06 <dgilmore> does anyone have anything for open floor?
16:00:13 <maxamillion> I do
16:00:25 <dgilmore> maxamillion: go for it
16:00:27 * nirik too, can go after maxamillion
16:00:36 <maxamillion> I wanted to bring up pungi4 ... what all is it going to block and when do we have to have it in working condition?
16:00:52 <maxamillion> (I have a couple others so nirik can go next and I'll follow up after)
16:01:13 <dgilmore> maxamillion: we can continue to run it in teh way we have
16:01:27 <dgilmore> maxamillion: so if we do not get it 100% we are not totally screwed
16:01:31 <nirik> AFAIK, the run-root stuff should be ready to use now (wasn't it going to use that?)
16:01:37 <dgilmore> but it will block taking a step forward for atomic
16:01:43 <pbrobinson> maxamillion: I'm going to follow up with dmach tomorrow morning EU time
16:01:46 <maxamillion> dgilmore: alright, ideally when should we have it done then?
16:01:50 <maxamillion> pbrobinson: +1 - thanks
16:01:51 <dgilmore> afaik we should be able to run it as is
16:01:55 <pbrobinson> to see if we can't get some focus
16:02:07 <maxamillion> dgilmore: wait, should be able to run what as is?
16:02:15 <pbrobinson> dgilmore: sure but we still need to try and keep it moving forward in parallel
16:02:16 <dgilmore> but we will have to move /mnt/fedora_koji/compose to /mnt/fedora_koji/koji/compose
16:02:40 <dgilmore> maxamillion: the release compose as we did in f22 should still just work
16:02:49 <maxamillion> dgilmore: ah, gotchya
16:02:58 <dgilmore> pbrobinson: I do not want to do it that way
16:03:13 <maxamillion> dgilmore: if we can pull off getting pungi4 in good shape though, when should that be done by?
16:03:16 <dgilmore> but I think at this point we need to test doing everything
16:03:25 <dgilmore> withing the confines of /mnt/koji
16:03:32 <dgilmore> which is a bug in pungi
16:03:42 <dgilmore> that we can not use paths outside of it
16:03:50 <dgilmore> maxamillion: 3 weeks
16:03:56 <maxamillion> dgilmore: oh wow ... alright, good to know
16:04:06 <pbrobinson> dgilmore: what way, I'm not suggesting by the statement above doing it any way, but rather if need be we leave it as it is we still should be pushing forward getting pungi4 into a good state
16:04:27 <maxamillion> alright, thanks ... that's all I had for pungi4, go for it nirik
16:04:29 <dgilmore> pbrobinson: I do not want to do composes the same way we did in f22
16:04:57 <dgilmore> pbrobinson: at this point we need to test doing a compose within the known limitations of pungi
16:05:09 <dgilmore> and see what we get
16:05:16 <pbrobinson> is it in a state where it's even working yet?
16:05:27 <dgilmore> but we do need to work to remove the limitations and do things better all around
16:05:28 <maxamillion> pbrobinson: I've yet to see it work
16:05:33 <dgilmore> pbrobinson: unknown
16:05:41 <dgilmore> pbrobinson: we have not tried
16:06:16 <dgilmore> runroot is setup and tested working
16:06:25 <dgilmore> but we need a new koji to actually use it
16:07:14 <dgilmore> I will bug mikem to get a koji release out
16:07:36 <dgilmore> and if he can not do it today or tomorrow I will pull a new git snapshot only into rawhide
16:07:55 <dgilmore> then we need to run a full compose and see what happens
16:08:28 <dgilmore> sound good to everyone?
16:08:29 <pbrobinson> agreed I think with that and the URL fixes for cloud images plus other fixes there's more than enough for a release, and I don't think the dnf patches are ready yet to merge
16:08:39 <nirik> I wanted to ask about builders. Currently they are all f21. Do we want to try and move them to 22? That was waiting on some dnf stuff I think? is that sorted? or should we keep them on 21 for a while more?
16:08:41 <pbrobinson> yep, seems reasonable
16:08:54 <dgilmore> nirik: they have to stay f21 for now
16:08:56 <nirik> ha. ok, that answers that I guess. ;)
16:09:04 <pbrobinson> nirik: yes, I think that needs a lot more testing yet :)
16:09:17 <nirik> ok, fair enough. done. maxamillion...
16:09:25 <maxamillion> dgilmore: why so? (I don't know the motivation behind them needing to stay f21)
16:09:28 <pbrobinson> nirik: sounds like something for staging once the koji dnf patch set lands
16:09:51 <nirik> sure, if staging is in shape to test (which I hope it will be, threebean has been working on that)
16:09:52 <pbrobinson> because of dnf being default and not working with koji yet
16:10:16 <dgilmore> maxamillion: because koji doesnt know how to deal with using different packaging backends, and the way that mock deals with yum/yum-deprecated/dnf is broken
16:10:40 <maxamillion> dgilmore: ah
16:10:50 <dgilmore> the move to dnf as default and taking over /usr/bin/yum breaks many things
16:11:33 <dgilmore> does anyone else have anything they want to bring up?
16:11:39 <maxamillion> I do
16:11:46 <pbrobinson> don
16:11:57 <pbrobinson> don't we have a limit per person ;-)
16:12:12 <dgilmore> maxamillion: go ahead
16:12:23 <maxamillion> I wanted to bring up the Layered Build Image change (https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service) ... I'm working right now to see if I can make this work on ARM to make sure that we can satisfy all primary architectures but I think there will be challenges there because of upstream docker and kubernetes not having the best ARM suppport ... buuuut, the reason I w
16:12:29 <maxamillion> anted to bring it up is that I think we need to get someone from the Infrastructure team involved because I suspect they are going to be left with a brunt of the work long term to keep the lights on
16:12:33 <maxamillion> (sorry, was typing)
16:12:57 <dgilmore> maxamillion: :)
16:13:08 <maxamillion> I'd like to propose this to the Infrastructure group and request some amount of "buy in" from an interested party
16:13:20 <dgilmore> maxamillion: so right now we do have the issue of being unable to build the base image inside of fedora infra
16:13:39 <maxamillion> I don't want to just do all this stuff and then be like "here, take this and deal with 3am pager duty if it blows up"
16:13:51 <maxamillion> dgilmore: oh?
16:13:52 <nirik> we don't want that either. (speaking for infrastructure)
16:14:09 <dgilmore> maxamillion: I built the f22 images at my house
16:14:24 <maxamillion> nirik: right, that'd be a total asshole move to expect that of the Infra team
16:14:41 <maxamillion> nirik: so I want to sort out how we should go about this and get everyone involved and on the same page
16:14:47 <nirik> maxamillion: people try and do that kind of thing all the time to us. ;)
16:14:49 <dgilmore> maxamillion: which i could add a node to koji to build the docker base image, but I am not 100% sure we should
16:14:59 <maxamillion> nirik: :(
16:15:07 <nirik> anyhow, we have a process for this... called request for resources.
16:15:08 <dgilmore> it would be a bit of overhead to make sure it has access to internal infra
16:15:28 <maxamillion> dgilmore: what's to missing magic bits to build a base image in koji?
16:15:35 <maxamillion> s/to/the
16:15:43 <nirik> maxamillion: I really appreciate you doing the investigation legwork on this. ;)
16:15:50 <dgilmore> maxamillion: we have no arm hardware in phx2 that has virt support
16:16:59 <maxamillion> nirik: absolutely, and OpenShift V3 is kind of a beast of a system to go from zero to production on in terms of background knowledge .... I definitely don't want to attempt to just drop this on the Infra team, I'm also discussing alternatives for a more simple backend to the build tooling .... there's a thread here: https://lists.fedoraproject.org/pipermail/rel-eng/2015-July/020417.html
16:17:03 <nirik> would it be good for me to install openshiftv3 somewhere and see how much pain? or it might be we can get away without the entire OSv3 to do this?
16:17:08 <maxamillion> dgilmore: ahhhh, that'll do it
16:17:16 <nirik> maxamillion: yep. I read it. ;)
16:17:31 * nirik is hoping we can get away without entire OSv3 for this.
16:17:52 <maxamillion> nirik: yeah, basically I'd like to not need OSv3 for this .... running an entire PaaS just for build tooling seems like the wrong solution
16:18:17 <nirik> yes.
16:18:39 <maxamillion> nirik: I think using atomic-reactor and some automation around it would be much better, so I'm hoping we can go down that route .... at least then we'd have a nice simple solution and we aren't re-inventing the wheel, just attaching it to a smaller and more simple cart
16:18:49 * nirik nods.
16:19:10 <nirik> is there anything I can/should do to help this now? or wait and see if this atomic-reactor thing will pan out?
16:19:11 <maxamillion> alright, cool so I'll start the process for Request for Resources today, thanks
16:19:27 <maxamillion> nothing that I can think of in the immediate term
16:20:07 <maxamillion> nirik: if I think of anything, I'll ping/nag you about it... I mostly wanted to at least get it in the radar of the Infra team and make sure it the Infra team's concerns are kept in focus as we work through things
16:20:34 <maxamillion> nirik: effectively, I don't want to blind side or dump work on the Infra Team
16:20:44 <nirik> ok. I think we should decide what we are going to do first, then talk infrastructure.
16:21:01 <maxamillion> nirik: sounds good, I'll go that route then. Thanks
16:21:12 <maxamillion> ok, I have one more thing for open floor if that's cool
16:21:14 <maxamillion> :D
16:21:46 <dgilmore> maxamillion: pushing it :P but okay
16:22:27 <maxamillion> does anyone know if there's any sort of point of contact for the MIPS revival? I keep seeing things about Fedora MIPS show up in my news feed but I've not seen much on our side of things ... am I just missing it somewhere? A mailing list I'm not on? $other?
16:22:51 <pbrobinson> there has been nothing
16:23:03 <nirik> maxamillion: mtoman was working on that, but I have seen 0 in the way of communication on it.
16:23:16 <dgilmore> maxamillion: I know they exist but that is about it
16:23:19 <pbrobinson> I've mostly written an email to reach out to them to actually find out what was happening from a secondaries PoV
16:23:47 <maxamillion> pbrobinson: ah ok, that was going to be my follow up question: has anyone reached out to them?
16:23:54 <maxamillion> pbrobinson: any word? still radio silence?
16:23:54 <pbrobinson> I'll let people know if I get a response but I want them to actually participate
16:24:02 <maxamillion> pbrobinson: +1 thanks
16:24:06 <nirik> yeah, it sounds like they are in early days.
16:24:09 <maxamillion> alright, that's all I had :)
16:24:23 <nirik> but still, early communication could save a lot of wasted efforts.
16:24:30 <pbrobinson> ultimately in it's current state it's not even a remix so they really shouldn't be even using the name
16:24:32 <maxamillion> I just have a strange soft spot for MIPS because of some time I spent coding for it in assembly in college
16:24:33 <dgilmore> we do need them to work with us. It will make everything much easier
16:24:39 <maxamillion> nirik: +1
16:28:15 <dgilmore> nirik: indeed
16:28:25 <dgilmore> especially if they go and fork a bunch of packages
16:28:34 <dgilmore> getting patches in after does not work well
16:28:40 <dgilmore> we had that with arm years ago
16:28:52 <dgilmore> if there is nothing else I will wrap up
16:29:00 <pbrobinson> over 5 years later I'm still not over those scars!
16:29:07 <pbrobinson> did nirik have something?
16:29:18 <nirik> just the builders thing...
16:29:19 <dgilmore> pbrobinson: I thought he asked it
16:29:38 <pbrobinson> ah :)
16:31:06 <dgilmore> #endmeeting