15:32:29 <dgilmore> #startmeeting RELENG (2016-04-18)
15:32:29 <zodbot> Meeting started Mon Apr 18 15:32:29 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:32:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:32:29 <zodbot> The meeting name has been set to 'releng_(2016-04-18)'
15:32:29 <dgilmore> #meetingname releng
15:32:29 <zodbot> The meeting name has been set to 'releng'
15:32:29 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:32:29 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:32:32 <dgilmore> #topic init process
15:32:33 <maxamillion> .hello maxamillion
15:32:34 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
15:32:48 <nirik> .hello kevin
15:32:49 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
15:32:59 * sharkcz is here
15:33:03 <pbrobinson> o/
15:34:03 <dgilmore> lets make this a short one
15:34:15 <dgilmore> there is no tickets with meeting keyword
15:34:30 <dgilmore> #topic koji status
15:34:43 <dgilmore> nirik: lets quickly go over the db issues of the lastw eek
15:35:05 <nirik> sure, shall i?
15:35:31 <dgilmore> we seem to be hitting some issues on koji.fp.o where there is some locks happening in the db and not getting released, causing things to back up and be slow
15:35:33 <dgilmore> nirik: sure
15:35:49 <nirik> actually: https://fedoraproject.org/wiki/Infrastructure/KojiDBIssues
15:35:52 <nirik> has the info on it. ;)
15:36:10 <nirik> it seems pretty clear to me now that koji-gc is the part of koji to blame.
15:36:29 <dgilmore> things were fine over the weekend?
15:36:41 <dgilmore> I have not yet looked at teh db load the last few days
15:36:45 * masta is here, lurking
15:36:59 <nirik> well, I also migrated the db-koji01 instance
15:37:34 <nirik> the last time the script that kills long locks killed something was:
15:37:37 <nirik> 2016-04-15 15:20:01.224433+00
15:38:01 <dgilmore> which is awhile ago
15:38:10 <nirik> yep. before you disabled koji-gc
15:39:01 <dgilmore> looks like the db system load massively dropped after what you did this morning
15:39:58 <nirik> yeah, the new bvirthost01 is one of the new dells... so it's local storage instead of iscsi and a faster set of cpus and nothing else is on that bvirthost yet. ;)
15:40:46 <masta> It usually grown load over times, so will be interesting to watch
15:41:02 <nirik> so, really we need to fix koji-gc
15:41:16 <dgilmore> nirik: seem we went from load of 20 or so to about 3
15:41:16 <nirik> and then confirm it's fixed and then we can drop the script that kills locks. ;)
15:41:28 <nirik> well, some of the load was likely due to the virsh migrate too
15:41:40 <dgilmore> okay
15:42:21 <nirik> I tried to live migrate and it mostly worked, but I had to stop postgres for about 20seconds this morning to get it to finish
15:43:03 <nirik> anyhow, I think we are in ok shape for beta freeze
15:43:11 <dgilmore> nirik: excellent
15:44:29 <dgilmore> #info db-koji01 moved to a different host and off of iscsi
15:44:39 <dgilmore> #info need to get koji-gc fixed
15:45:09 <dgilmore> #topic Fedora 24 Beta status
15:45:21 <dgilmore> so change freeze is upon us
15:45:28 <dgilmore> I have a pungi update I need to build
15:45:36 <dgilmore> for a fix in ostree
15:45:45 <pbrobinson> freeze is tomorrow right?
15:46:13 <dgilmore> the last stable push needs to happen after 00:00 UTC today
15:46:15 <nirik> yep
15:46:23 <dgilmore> but yes freeze date is tomorrow
15:46:27 <nirik> masta: I think you are on pushes this week ^
15:46:34 <dgilmore> .pushduty
15:46:35 <zodbot> dgilmore: The following people are on push duty: Fedora Release Engineering, Jon
15:46:37 <zodbot> dgilmore: - https://apps.fedoraproject.org/calendar/release-engineering/
15:46:49 <masta> nirik, oh thanks
15:47:00 <masta> I'll get on that soon
15:47:25 <dgilmore> masta: f24 stable updates need to happen some time after midnight UTC
15:47:25 <masta> so for free, don't push the stable, right?
15:47:33 <nirik> I guess you could do a full push now, and then if it's all done a f24 stable beofre 00;00
15:47:33 <dgilmore> after that 24 updates-testing only
15:47:53 <dgilmore> nirik: well after
15:47:53 <masta> err ...s/free/freeze
15:48:05 <dgilmore> people have until 00:00 to get things pushed to stable
15:48:06 <nirik> sure
15:48:23 <masta> hrm... about 8 hours to zero UTC
15:48:28 <dgilmore> masta: right no f24 stable
15:48:31 <masta> I'll try to go quick
15:49:13 <dgilmore> I think we have all the deliverables for Beta sorted out
15:51:04 <nirik> FYI, I updated all the primary builders and rebooted them over the weekend.
15:51:16 <nirik> so they should be all pretty upda teo date and in sync
15:51:17 <dgilmore> nirik: cool
15:52:00 <dgilmore> I am anticipating Beta being fairly smooth
15:52:04 <pbrobinson> I updated all of aarch64 builders last week, rebuilt most of the ppc builder too, just finishing the last of them today. Will have more details laster in the meeting
15:52:08 <dgilmore> well as smooth as usual
15:52:21 <pbrobinson> ha!
15:54:40 <dgilmore> anything anyone wants to bring up for Beta?
15:55:48 <dgilmore> #topic Secondary Architectures updates
15:55:49 <dgilmore> #topic Secondary Architectures update - ppc
15:56:19 <pbrobinson> so 3 out of 4 of the Power8 hosts have now been rebuilt
15:56:33 <dgilmore> great
15:56:45 <pbrobinson> the last one had a anaconda crash I have to collect some bits from before I can kick it again and finish it
15:56:49 <pbrobinson> that will be done shortly
15:57:00 <dgilmore> how does that place us for moving the hub to be ansibilised?
15:57:31 <pbrobinson> I'll be pushing an initial build of the DB/hub VMs shortly
15:58:01 <pbrobinson> and then I'll hand that over to nirik to poke at and then it's when do we want to move it
15:58:28 <nirik> Probibly after beta at this point?
15:58:34 <nirik> unless we can get it done today?
15:58:35 <dgilmore> okay, I expect that will be after Beta is done
15:58:37 <pbrobinson> so probably, at a guess, right after beta
15:59:19 <dgilmore> #info ppc koji to move after Beta is done
16:00:46 <pbrobinson> other than that all is looking reasonable
16:00:49 <pbrobinson> for pcc
16:00:54 <pbrobinson> ppc even
16:01:13 <dgilmore> pbrobinson: any new deliverables for Beta?
16:01:31 <pbrobinson> nope, none that I'm aware of. We have docker/qemu
16:01:51 <pbrobinson> there's likely a few package fixes we still need but they'll go through the standard process
16:02:02 <pbrobinson> and I'm not convinced some of those are blockers
16:02:17 <dgilmore> cool, just be sure to file bugs early if needing FE's
16:02:49 <pbrobinson> yes, well there's two already there (firefox/389-ds-base)
16:03:00 <pbrobinson> the later is fixed, not sure of the status of the former
16:03:03 <dgilmore> I think 389-ds-base is already fixed
16:03:30 <dgilmore> if we need to escalate the firefox bug lets do that now
16:03:36 <pbrobinson> yes, it's in testing with a req for stable so should go stable today
16:04:18 <dgilmore> cool, anything else?
16:04:46 <pbrobinson> nope
16:04:52 <dgilmore> #topic Secondary Architectures update - s390
16:05:07 <dgilmore> sharkcz: how is s390 looking for Beta?
16:05:08 <pbrobinson> myself and sharkcz will be sorting out the compose this week
16:05:14 <dgilmore> okay
16:05:39 <sharkcz> we are looking fairly good
16:05:51 <pbrobinson> The Everything repo has been going out for a bit for both branched and rawhide
16:06:42 <sharkcz> and I'm working on email announcing dropping 32-bit support in f25 (not sure about f24), we had a discussion with IBM engineering and they agree to drop it
16:06:54 <nirik> finally. ;)
16:06:58 <dgilmore> okay, just the runroot issue to sort out?
16:07:06 <pbrobinson> yep
16:07:11 <dgilmore> sharkcz: great news
16:07:23 <dgilmore> #info 31 bit to be removed from f25
16:07:35 <dgilmore> sharkcz: I think we should drop it from f24 also
16:08:10 <sharkcz> dgilmore: that's my thinking too, not to have to keep s390 for another year
16:08:22 <pbrobinson> yep, agreed
16:09:00 <dgilmore> sharkcz: its not like people have been able to run s390 builds on a s390 machibe
16:09:03 <dgilmore> machine
16:09:18 <dgilmore> it has existed purely for multilib
16:09:29 <dgilmore> lets just remove it entirely
16:10:12 <sharkcz> yep, there is a guy doing pure 32-bit rebuild of fedora, but it shouldn't affect him much as we won't add any artificial blockers in the packages
16:11:17 <pbrobinson> does he have one in his basement?
16:11:44 <dgilmore> proposal #agreed we will remove s390 builds from f24 and f25 and not ship them after f23
16:11:51 <sharkcz> pbrobinson: dunno :-)
16:12:08 <dgilmore> sharkcz: that will have the effect of freeing builder resources
16:12:18 <sharkcz> yes
16:12:40 <dgilmore> anyone disagree with my proposal?
16:12:58 <pbrobinson> nope, just as long as IBM is happy with it going in f24
16:13:09 <sharkcz> no
16:15:09 <dgilmore> #agreed we will remove s390 builds from f24 and f25 and not ship them after f23
16:15:21 <dgilmore> sharkcz: that is excellent news
16:15:36 <dgilmore> it leaves x86_64 the only user of multilib
16:15:51 * masta does not disagree
16:15:54 <dgilmore> anything else?
16:16:06 <sharkcz> no
16:16:08 <dgilmore> #topic Secondary Architectures update - arm
16:16:14 <dgilmore> pbrobinson: how is aarch64?
16:16:16 <pbrobinson> looking reasonable here
16:16:32 <pbrobinson> we now have nightly cloud/docker builds
16:16:49 <dgilmore> #info aarch64 nightly docker and cloud images now available
16:16:52 <nirik> hopefully soon we can get pbrobinson enough time to mess with the moonshot. ;)
16:16:54 <pbrobinson> I want to get images in place for SBCs
16:17:26 <pbrobinson> nirik: still awaiting the final details of the last of the kernel bits but i have a whole bunch of stuff in place
16:17:27 <dgilmore> pbrobinson: okay, main thing being that they will use u-boot and not uefi?
16:17:48 <pbrobinson> dgilmore: TBD
16:18:02 <nirik> pbrobinson: cool.
16:18:07 <pbrobinson> dgilmore: I need to get usable images so I can hack on them to see what's needed
16:18:12 <pbrobinson> and for that I need time!
16:18:30 <dgilmore> pbrobinson: or people to work with us to get it done
16:18:41 <dgilmore> though I am not sure people will do that
16:18:41 <pbrobinson> u-boot 2016.05rc1 has uEFI services emulation in place so it might be we can use that
16:18:56 <pbrobinson> and have _BOTH_ u-boot and uEFI ;-)
16:19:18 <dgilmore> :)
16:19:52 <pbrobinson> I have 3 SBCs and now I have a lot of infra boxes ticked I'm hoping to have time to get them working
16:20:36 <pbrobinson> well a bit more time at least. I have a PINE54, a geekbox and a Jetson TX1
16:20:44 <dgilmore> #info work to happen to enable aarch64 support
16:21:04 <pbrobinson> but all the rest of the deliverables for f24 are now pretty much in place there too
16:21:35 <pbrobinson> and the builds overall for aarch64 are looking pretty good
16:21:49 <dgilmore> excellent news
16:22:11 <dgilmore> #info aarch64 deliverables in place and builds are looking pretty good
16:22:19 <dgilmore> anything else or open floor?
16:22:37 * nirik has one item for open floor
16:22:49 * maxamillion has one item for open floor also
16:22:58 <dgilmore> #topic Open Floor
16:23:03 <dgilmore> nirik: you are up
16:23:59 <nirik> someone asked on devel list how to get their package off 'critpath'. I can reply to them, but it got me thinking about critpath. It's a mess. ;) Do we want to ask fesco if we should just drop it? or fix it so it's close to what we care about again?
16:24:21 <dgilmore> critpath generation is broken right now
16:24:26 <nirik> right now it's not been updated in a long time, updates are manual. We have critpath groups for all the desktops...
16:24:32 <dgilmore> basically you can't
16:24:35 <nirik> and yeah, could well be just broken
16:24:49 <nirik> well, there is a sop / process.
16:25:01 <dgilmore> the critpath generation fails
16:25:13 <dgilmore> somoen needs to sit down and fix the code
16:25:15 <nirik> it does use yum I see
16:25:26 <dgilmore> it does use yum
16:25:56 <nirik> but it has no reflection on the products we ship these days.
16:26:03 <dgilmore> the way it is supposed to work is it takes the comps groups, resolves all deps, thats what is critpath
16:26:08 <nirik> except by accident
16:26:29 <dgilmore> if your package is in a comps group you can get it removed  from the comps group
16:26:48 <dgilmore> if its pull in as deps then you need to make what is requiring it not require it
16:26:53 <pbrobinson> nirik: the package in question on the list I believe is shipped in workstation so I think it's still correct based on that def
16:26:56 <nirik> sure, but that doesn't update critpath until someone updates it.
16:27:23 <dgilmore> nirik: saying that you do not want your package to be critpath is not a maintainers decision
16:27:48 <dgilmore> nirik: the process is not automatable
16:28:04 <nirik> note that it doesn't actually use the comps groups people change...
16:28:05 <dgilmore> unless we can use ssl cert auth or something like it for pkgdb
16:28:23 <dgilmore> nirik: right it is a different group
16:28:27 <nirik> ie, critical-path-gnome is the one it looks at...
16:28:37 <nirik> and people likely ignore that when they update things.
16:28:41 <dgilmore> we likely need to sit down and revisit critpatha nd how it works
16:28:46 <nirik> right.
16:29:09 <nirik> one thing we (well, fesco I guess) could do is tell bodhi to ignore it... then nothing would be using it until we can sort it out.
16:29:19 <dgilmore> but it will not change the fact that if a package is critpath or not is not a decision a maintainer can make
16:29:54 <nirik> sure, but they could see why...
16:30:12 <dgilmore> that would be a useful improvement
16:30:43 <nirik> anyhow, I can file a fesco ticket I guess and see what people will want to do here.
16:31:07 <dgilmore> sure
16:31:16 <dgilmore> we do need to fix the generation
16:31:22 <dgilmore> and run it more frequently
16:31:49 <dgilmore> perhaps we can talk to puiterwijk or pingou over ways to automate the auth
16:32:06 <dgilmore> or perhaps to use fedmsg to trigger updates automatically
16:32:26 <nirik> yep.
16:32:37 <dgilmore> we publish a message that its updated and pkgdb goes and pulls in the results
16:32:52 <nirik> oh, reminds me. I ran out of time again to move comps/spin-kickstarts to pagure. ;( will look at it after beta...
16:33:07 <dgilmore> we really need critpath generation and updating to be automatically done
16:33:27 <dgilmore> nirik: okay
16:33:50 <dgilmore> nirik: I will reply to the email on devel@
16:33:57 <dgilmore> anything else?
16:34:01 <nirik> should I file with fesco now? or do want to discuss things more out of meeting first?
16:34:12 <dgilmore> nirik: lets wait
16:34:31 <dgilmore> we can take it to them if its going to take too long to fix
16:34:42 <nirik> ok
16:34:56 <nirik> releng ticket to track then? or ?
16:35:08 <dgilmore> yeah
16:35:14 * nirik can file one
16:35:17 <dgilmore> thanks
16:35:34 <dgilmore> nirik: does that cover it?
16:35:42 <nirik> yep
16:35:49 <dgilmore> maxamillion: the floor is yours
16:36:07 <maxamillion> I just wanted to give a quick update on osbs in stage, the registry is successfully deployed, osbs is really close to working but I've found a bug in python-docker-py and have some folks from the osbs team helping me track down a fix ... hopefully get that fixed soon, I think it's the last thing we need to fix
16:36:29 <dgilmore> cool, so prod shortly after Beta is done
16:36:36 <maxamillion> ttomecek actually just ping'd me, he found the issue and is working on a patch now, said he will have it for me today
16:36:52 <dgilmore> #info osbs and layered images almost complete in stg
16:36:59 <dgilmore> excellent
16:37:00 <maxamillion> dgilmore: hopefully, yes ... I'd like to completely destroy the osbs machine at least once after we get everything fixed and do a full redeploy from ansible
16:37:07 <dgilmore> maxamillion: cool
16:37:08 <maxamillion> and test to make sure
16:37:15 <maxamillion> but yes, we're *finally* close to done
16:37:31 <dgilmore> maxamillion: I would really like us to figure out how to make docker search work with our registry
16:38:07 <dgilmore> maxamillion: and we need to figure out the process for importing the base images
16:38:07 <maxamillion> dgilmore: yeah, that's on my list
16:38:21 <maxamillion> dgilmore: also on the list
16:38:25 <dgilmore> :)
16:38:29 <dgilmore> anything else?
16:39:25 <dgilmore> #endmeeting