15:30:42 <dgilmore> #startmeeting
15:30:42 <zodbot> Meeting started Mon Jun  9 15:30:42 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:30:48 <dgilmore> #meetingname releng
15:30:48 <zodbot> The meeting name has been set to 'releng'
15:31:41 <dgilmore> #topic === Releng Roll Call ===
15:31:41 <dgilmore> #chair dgilmore dwa nirik tyll sharkcz bochecha
15:31:41 <zodbot> Current chairs: bochecha dgilmore dwa nirik sharkcz tyll
15:31:48 <nirik> morning
15:33:55 <dgilmore> anyone else here?
15:34:16 <pbabinca> well, I'm here too. Just silently watching. :)
15:34:54 * masta is here
15:35:28 <dgilmore> #chair masta
15:35:28 <zodbot> Current chairs: bochecha dgilmore dwa masta nirik sharkcz tyll
15:37:26 <dgilmore> lets get started
15:37:53 <dgilmore> #topic Tickets https://fedorahosted.org/rel-eng/report/10
15:38:08 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5877
15:38:21 <dgilmore> okay so the mass rebuild happened over the weekend
15:38:36 <dgilmore> #info massrebuild releng passthrough completed
15:38:44 <dgilmore> #info bugs being filed now
15:39:07 <dgilmore> on the bug side im having some issues where it seems bugzilla is randomly logging me out
15:39:55 <dgilmore> we had 39 x86 builders and 48 arm
15:40:04 <dgilmore> mass rebuild was faster than last time
15:40:07 <dgilmore> but not a lot
15:40:42 <dgilmore> one thing that I think we need to work on going forward is to add native logging to the mass rebuild script and threading
15:41:00 <dgilmore> we were slow to get going because koji was building faster than we cold submit
15:41:34 <nirik> dgilmore: threading would be nice...
15:41:36 <dgilmore> if we broke up the package list and ran 4 or 5 threads submitting builds at the same time we should get full faster
15:41:48 <nirik> it shouldn't be bad to just submit everything and have it process it down.
15:42:04 <dgilmore> perhaps another way to do it would be to submit the tasks that take the longest to complete first
15:42:15 <dgilmore> rather than 0-9 and a-z
15:42:42 <nirik> order by build time? yeah, could be good... might not be easy to script tho
15:43:07 <dgilmore> nirik: main issue is it takes on average about 8 seconds to clone, bump, commit, push and build a package
15:43:24 <dgilmore> some packages more some less
15:43:41 <masta> I bet the clone takes the bulk of that time
15:43:43 <dgilmore> maybe we could do bare clones or something else to speed it up a little
15:44:01 <nirik> or could we run on pkgs01 against the bare repos?
15:44:19 <nirik> I guess we need a checkout to bump.
15:44:25 <dgilmore> nirik: would still need to clone
15:44:53 <dgilmore> my desktop where i ran it has /home on a pair of 256gb ssd's in a raid 1
15:45:01 <dgilmore> and a ton of memory
15:45:11 <nirik> yeah, was just trying to eliminate the ssh/network part
15:45:31 <dgilmore> ssh is not that bad
15:45:53 <dgilmore> i create a open socket to pkgs and keep the session open
15:46:02 <masta> perhaps a pre-clone run, or even a thread to clone ahead of the anticipated activities of another thread that bumps and pushes
15:46:10 <dgilmore> so its all reusing an existing connection
15:46:20 <dgilmore> masta: you cant
15:46:37 <dgilmore> you would need to do a git pull in every repo
15:46:51 <dgilmore> incase someone did something in the repo
15:47:16 <dgilmore> you need to do each package in a sweep
15:47:39 <dgilmore> I think the best thing to do is make the script multithreaded
15:47:50 <dgilmore> I ended up running 3 instances
15:48:07 <dgilmore> where 2 of them i only did packages starting with certain letters
15:48:46 <nirik> I'd say do a lot more next time... it's ok for the queue to be long on the build side I would think.
15:48:50 <nirik> better than it being idle.
15:49:58 <dgilmore> nirik: right. I nearly think just fork a thread for the first letter in every package
15:50:07 <dgilmore> which would be about 40-50 threads
15:50:09 * nirik nods
15:50:19 <dgilmore> though some would only do one or two
15:50:25 <dgilmore> while others would do hundreds
15:50:42 <dgilmore> but it should get us started with a ton of builds
15:51:33 <dgilmore> #action rewrite the mass rebuild script to be multithreaded
15:52:15 <dgilmore> anyone want to bring up anything else about the mass rebuild?
15:53:02 <nirik> there was some buildroot breakage...
15:53:10 <dgilmore> pbabinca: that was your fault
15:53:10 <nirik> lvm2 and then rpkg
15:53:20 <dgilmore> the rpkg one
15:53:33 <pbabinca> dgilmore, well. Only today I pushed updated version there.
15:53:37 <nirik> lvm2 spec has Release: for all it's subpackages, and the bump script didn't grok it.
15:53:37 <dgilmore> pbabinca: we need to get a new update of rpkg done
15:53:42 <masta> I think aarch64 is still catching up on the rebuild =)
15:54:03 <dgilmore> nirik: I think we need to clean up the lvm2 spec
15:54:12 <pbabinca> dgilmore, Should I wait for mass rebuild to end or may I build new rpkg now?
15:54:18 <dgilmore> pbabinca: its done
15:54:19 <masta> any enhancement to our rebuild script would impact the shadow builds of secondaries too
15:54:27 <dgilmore> pbabinca: so go ahead
15:54:28 <nirik> dgilmore: I should file a bug on it... will do so now.
15:54:39 <dgilmore> masta: no it wont
15:54:57 <dgilmore> masta: there would be zero impact on secondaries
15:55:33 <masta> dgilmore: I'll yield to your greater experience here, but I would imagine that following the order of builds would change on shadow builds... it's trivial perhaps?
15:56:15 <dgilmore> masta: koji shadow makes a repo for each and every single build
15:56:40 <dgilmore> masta: the order of teh builds on primary does absolutely nothing on the secondariues
15:56:55 * masta stands corrected
15:56:59 <masta> ok
15:57:02 <dgilmore> koji-shadow will process the builds in teh order it gets them
15:57:27 <dgilmore> it will do exactly the same as it always does
15:58:02 <dgilmore> it would  be kinda nice to work out a way to make a superrepo and send all the builds built with it
15:58:20 <dgilmore> so you could use the one repo for 100 200 or even 1000 builds
15:58:31 <dgilmore> but even then things wouldnt change
15:59:02 <dgilmore> packages available to the buildroot do not change much
15:59:17 <dgilmore> as we tag the builds into a side tag that does not popukate the buildroot
15:59:30 <dgilmore> anyway thats a tangent
15:59:43 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5870
15:59:55 <dgilmore> not sure we have much progress here
16:00:20 <dgilmore> we need to make sure the autosign box can talk to the secondsary hubs and sigul
16:00:31 <nirik> I have a ticket in for that.
16:00:52 <dgilmore> cool
16:01:07 <dgilmore> i believe tyll can sign rawhide builds on primary
16:01:16 <dgilmore> and is
16:01:36 <dgilmore> though packages like texlive make life difficult
16:01:43 <nirik> yeah, there's some issues with hanging sigul too
16:02:19 <dgilmore> :( yeah
16:02:34 <dgilmore> #topic
16:02:38 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5914
16:02:52 <dgilmore> lets move on
16:03:05 <dgilmore> nirik: do we have somewhere where we want to run this?
16:03:07 <nirik> haven't had a chance to do much on that one.
16:03:30 <nirik> we could do it on releng04... or our 'sundries' boxes (misc services that don't need their own instances)
16:03:39 <dgilmore> I think tyll had some questions over what to use as a trigger
16:03:57 <dgilmore> releng04 is probably fine
16:04:21 <dgilmore> we could reuse one of the existing certs
16:04:28 <dgilmore> masher or some other user
16:04:35 <dgilmore> or we could make a new one
16:06:08 <dgilmore> lets move on
16:06:11 <dgilmore> #topic secondary arches
16:06:18 <dgilmore> #topic secondary arches - ppc
16:06:54 <dgilmore> so dwa left Red Hat friday and will be taking time off but wants to come back and be involved down the road
16:07:09 <dgilmore> not sure anyone else from the ppc sig is here
16:07:49 <masta> kinda
16:07:51 <dgilmore> there is an open req to replace him. I have put in a suggestion on who the replacement should be and they have applied, time will tell
16:08:07 <dgilmore> not sure how ppc is doing in following the mass rebuild
16:08:13 * masta is covering his ppc tasks until somebody else is assigned the role
16:09:16 <nirik> I've not seen ppc builds much today...
16:09:17 <masta> I think they are waitting to rebuild, I did not see any ppc64 shadow builds, or perhaps just did not see them when looking
16:09:28 <nirik> oh, I take that back, they are flowing now
16:10:09 <dgilmore> any other ppc updates?
16:10:47 <dgilmore> i thought that ppc was holding off mass rebuild until ppc64le merged into the hub
16:11:24 <masta> Just that Nirik and I got the keys and passwords from dwa, so if anybody needs something, poke me or nirik
16:12:11 <masta> dgilmore: I thought so too, but it appears Kersten is building things in a way that looks mass-rebuild'ish
16:12:27 <dgilmore> masta: he is building thinsg
16:12:45 <dgilmore> i just asked and the plan is to wait to merge in ppc64le
16:13:07 <dgilmore> masta: i got the smae email also
16:13:16 <dgilmore> #topic secondary arches - s390
16:13:25 <masta> dgilmore: well... I was under the same impression as you, and right now I see a bunch of builds.... they are perhaps not mass builds...
16:13:48 <dgilmore> masta: just checked some are f20 mass rebuild
16:14:39 <masta> dgilmore: ppc64le is moving along well, fyi...
16:14:50 <dgilmore> #undo
16:14:50 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0xab7f890>
16:15:09 <dgilmore> hopefully this week they will merge
16:15:14 <dgilmore> #topic secondary arches - s390
16:15:21 <dgilmore> sharckcz is not here
16:15:48 <dgilmore> but it doesnt look like they have started the mass rebuild
16:15:54 <dgilmore> #topic secondary arches - arm
16:15:56 <masta> I don't see sharkcz here, any other s390 guru's about?
16:16:06 <dgilmore> f19 updates creep along slowly
16:16:17 <dgilmore> ma	there is no other s390 peopel
16:16:19 <masta> ok, arm... the aarch64 rebuild is still going along. look for Peter's name on those
16:17:14 <dgilmore> aarch64 started following the mass rebuild as it happened. its hampered by one of the data centres where the builders are hosted being offline all weekend
16:17:42 <masta> (power outage)
16:17:44 <dgilmore> todays rawhide report had
16:17:46 <dgilmore> Summary:
16:17:46 <dgilmore> Added Packages: 64
16:17:46 <dgilmore> Removed Packages: 0
16:17:47 <dgilmore> Modified Packages: 3458
16:17:54 <dgilmore> which is pretty good
16:18:02 <dgilmore> about 1/3rd of the mass rebuild
16:18:12 <masta> yes, that is good actually
16:18:20 <masta> give the numbers involved
16:18:39 <masta> I want to move these builders to phx2
16:19:21 <masta> but that can wait...
16:19:24 <dgilmore> masta: as soon as its possible we will
16:19:49 <dgilmore> 3.16 should work on the hardware and we would be able to use rawhide or likely f21 brancheds
16:19:52 <dgilmore> branched
16:20:26 <nirik> if setup is going to be tricky, smooge and I are planning to be out at phx2 at the end of july...
16:20:53 <masta> There is alot of behind the scene things going on with aarch64, and honestly I'm not sure what I can even say given NDA... so I won't, except to say things will improve greatly soon.
16:21:26 <dgilmore> nirik: not overly tricky, the issue is the hardware has uefi and the support is still on its way upstream
16:21:37 <masta> nirik: that is a great idea... we can probably make them happy before they land in phx2, but would be nice to have you around for the racking.
16:21:37 * nirik nods.
16:22:40 <dgilmore> nirik: they do not have ipmi at all. but they do have serial and with pdu power control we will be able to do everything needed
16:22:57 <dgilmore> will have to use cyclades and the pdu
16:22:57 <nirik> ok
16:23:25 <dgilmore> they use grub2 to boot
16:23:33 <dgilmore> and pxe boot is done by grub2
16:24:36 <dgilmore> we can probably cover this all in infra space
16:24:45 <dgilmore> lets move on
16:24:51 <dgilmore> #topic open floor
16:25:01 <dgilmore> anyone want to bring anything up
16:25:33 * nirik has nothing
16:25:45 <masta> is there anything we can do to help the folks with failed builds?
16:26:42 <dgilmore> masta: not really, i guess you could triage the bugs
16:26:58 <dgilmore> the maintainer needs to fix whatever the cause is
16:27:06 <dgilmore> seems java and ruby are pretty unhappy
16:27:30 <masta> ok
16:27:58 <dgilmore> I need to fix the scripts that just broke
16:28:09 <dgilmore> not all bugs are filed
16:29:06 <pbabinca> rpkg rebuilt. Hopefully this time it won't break buildroot. If builds start to fail more often feel free to untag that build and let me know. I'll will watch that closely in my timezone (Europe/CEST).
16:29:28 <dgilmore> pbabinca: cheers
16:30:03 <dgilmore> okay if nothing else ill wrap up
16:36:43 <dgilmore> #endmeeting