15:37:11 <dgilmore> #startmeeting
15:37:11 <zodbot> Meeting started Mon May 19 15:37:11 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:37:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:37:17 <dgilmore> #meetingname releng
15:37:17 <zodbot> The meeting name has been set to 'releng'
15:38:03 <dgilmore> #chair tyll_ masta dwa sharkcz nirik
15:38:03 <zodbot> Current chairs: dgilmore dwa masta nirik sharkcz tyll_
15:38:08 <nirik> morning
15:38:19 <dgilmore> #topic rollcall
15:38:20 <sharkcz> hi
15:38:22 <dgilmore> whos here
15:38:57 <tyll_> Hi there
15:40:14 <dgilmore> lets get started,
15:40:25 <dgilmore> #topic congrats masta
15:40:39 <dwa> ahoyhoy
15:40:44 <dgilmore> so for those that do not know masta and his wife had a baby last week
15:41:19 <tyll_> masta: Congratulations!
15:41:35 <nirik> congrats!
15:41:41 <dgilmore> a little boy
15:41:56 <dgilmore> #topic tickets
15:41:57 <dgilmore> https://fedorahosted.org/rel-eng/report/10
15:42:16 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5846
15:42:54 <dgilmore> not heard anything about if this is in stage yet
15:43:17 <dgilmore> and bochecha is not here today
15:43:35 <dgilmore> ive not seen him online since he left singapore and moved back to france
15:43:49 <nirik> I think he was active on irc the other day...
15:43:52 <nirik> might still be moving tho
15:44:04 <dgilmore> yeah not really sure
15:44:17 <dgilmore> need to get it in stage and get some testing done
15:44:28 * nirik nods.
15:44:34 <dgilmore> #info need to get it in stage and get some testing done.
15:45:00 <dgilmore> #info will remove from meeting and once its had some testing we can add it back
15:45:47 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5870
15:45:54 <dgilmore> rawhide signing
15:46:02 <dgilmore> i had a thought on the weekend
15:46:09 <nirik> oh?
15:46:32 <dgilmore> we could add a .repo file to a fedora-release-rawhide-koji package
15:47:02 <dgilmore> and people that are complaining about rawhide being too slow can get a firehose fix
15:47:28 <nirik> it's a thought...
15:47:37 <dgilmore> then we could add signing and moving of the rawhide packages to the push process
15:48:02 <dgilmore> so when you push updates you sign rawhide and move the builds from a staging tag
15:48:15 <dgilmore> not really sure i love the idea still
15:48:22 <nirik> yeah, would likely mean rawhide (and branched) land later in the day...
15:48:35 <dgilmore> i forgot to get the security guys to join us
15:48:45 <dgilmore> yeah
15:49:01 <dgilmore> we could sign move and kickoff rawhide and branched
15:49:19 <dgilmore> just means its no longer an automated process
15:49:23 <nirik> yeah would mean uncertenty as to when it fires off.
15:49:30 <nirik> but that might be ok... not sure.
15:49:37 <dgilmore> not sure either
15:49:40 <dwa> un-automating processes makes me sad :(
15:49:48 <dgilmore> it would probably mean that some days it wont run
15:49:53 <dgilmore> dwa: indeed
15:50:08 <dgilmore> its more of an impact on secondaries than primary
15:50:25 <dgilmore> since secondaries dont always push updates every day
15:50:33 <dwa> would secondaries necessarily need to sign rawhide if they didn't want to?
15:50:43 <dgilmore> they would have to
15:50:56 <dgilmore> whatever we do we need to do it on primary and secondaries
15:51:02 * nirik nods.
15:51:32 <dwa> I would be unhappy with that since secondaries aren't generally overflowing with manpower
15:51:39 <sharkcz> +1
15:51:49 <dwa> ('that' = signing rawhide, not sticking to what primary does)
15:52:17 <dgilmore> so some users have complained rawhide is not signed
15:52:53 <dgilmore> and the PackageKit gnome-software folks are complaining because rawhide is using different code paths due to not being signed
15:52:55 <nirik> I guess we keep pondering on it and see if we can come up with a better solution
15:53:52 <dgilmore> i personally do not really like any of the options
15:54:16 <dwa> signing rawhide in generally seems like a decent enough thing to try for, but making it a manual process doesn't seem like a good idea. just my 2c
15:54:19 <nirik> yeah, none of them stand out as nice.
15:54:36 <dgilmore> dwa: there is no good way to automatically sign it
15:54:54 <dwa> yeah
15:54:57 <nirik> the best of them is the signer program that interfaces with sigul... IMHO... but that needs coding
15:55:08 <dgilmore> every option has some big downsides
15:55:18 <dgilmore> nirik: yeah
15:55:36 <dgilmore> i think thats the best option
15:55:55 <dgilmore> it would mean that the box has passphrases for unlocking the keys
15:56:05 <dgilmore> so we would need to be extra careful with it
15:56:07 <nirik> yeah. ideally only in mem
15:56:26 <nirik> and it could only have the needed ones for rawhide
15:56:33 <dgilmore> yeah
15:57:02 <dgilmore> dwa's koji-stalk might be a good starting point
15:57:36 <dgilmore> instead of firing off koji-shadow it could run sigul to sign the builds
15:58:01 * masta is here
15:58:10 <masta> sry... was deep into some work.
15:58:19 <dgilmore> masta: not good enough :P
15:58:32 <dwa> masta: we'll just assume you're deep in dirty diapers unless you say otherwise for the next few months ;)
15:59:15 <dgilmore> does anyone want to write a tool to try sign the builds?
15:59:41 <masta> hehe
15:59:56 <tyll_> Is there a test system that can be used to develop it?
16:00:06 <masta> dgilmore: to replace the existing sigul stuff?
16:00:12 <dgilmore> masta: no
16:00:27 <masta> ok sry (still reading up)
16:00:32 <nirik> tyll_: sadly we don't have a staging sigul setup. ;(
16:00:36 <dgilmore> masta: something to listen on fedmsg and run sigul to sign the build
16:00:39 <tyll_> I would like to take a look but I fear that it might take a long time to get a development enviroment ready before I can start with something
16:00:45 <masta> ah!
16:01:28 <dgilmore> nirik: wouldnt be the end of the world to do it on production
16:01:38 <dgilmore> the worst would be builds getting signed
16:02:25 <nirik> makes me nervos, but ok
16:03:03 <dgilmore> my biggest concern is things locking up because of that bug in sigul
16:04:16 <nirik> true... I've not see it do that in a while actually...
16:04:23 <nirik> but I could be misremembering
16:04:50 <dgilmore> ive seen it recently
16:04:59 <dgilmore> it is less frequent though
16:05:34 <dgilmore> anyway
16:05:54 <dgilmore> #action dgilmore and tyll_ to work together to get somewhere to test
16:06:30 <dgilmore> okay lets move on
16:06:57 <dgilmore> #topic move tyll_'s script to block retired packages into infra
16:07:16 <dgilmore> so i think we can say its working pretty well
16:07:24 <dgilmore> and we should move it into infra
16:07:39 <nirik> cool. where is it now?
16:07:51 <tyll_> It is running on my system here
16:07:52 <dgilmore> tyll_: you run it from home right?
16:07:57 <nirik> ah, ok.
16:08:03 <nirik> sure, we can move it in
16:08:23 <dgilmore> not sure if its best to run it on an existing releng box
16:08:32 <dgilmore> or if we should setup a vm just for it
16:08:41 <tyll_> it might need some kind of adjustments due to pkgdb2, I want to check it the next days btw
16:08:51 <dgilmore> tyll_: what os are you running it on?
16:09:08 <dgilmore> tyll_: pkgdb2 doesnt allow retirement of stable releases
16:09:47 <dgilmore> im going to get them to remove retirement from the web
16:10:04 <dgilmore> fedpkg needs to call something different to retire packages now
16:10:15 <tyll_> dgilmore: it is currently running on Fedora 19
16:10:24 <danofsatx-work> I'm here, although I don't think I need to be....
16:10:31 * danofsatx-work is late anyhow
16:10:56 <tyll_> But it should be easy to run it on CentOS6 I guess
16:11:10 <danofsatx-work> whoops, wrong meeting.....sorry guys
16:11:56 <dgilmore> tyll_: should be as long as it has everything you're using
16:12:26 <tyll_> eventually it could just use the dead.package commit to identify retired packages
16:13:46 <dgilmore> we really only should block packages in rawhide and branched
16:14:00 <tyll_> this is already what is happening
16:14:02 <dgilmore> which is what pkgdb supports
16:14:12 <dgilmore> pkgdb2
16:16:38 <tyll_> So, how do we proceed?
16:17:27 <dgilmore> nirik: where should we run it?
16:17:55 <dgilmore> tyll_: we either package it as a package that we can install, or add it to the releng git repo
16:18:10 <nirik> we could do releng04, or I have some 'sundries' servers for misc cron jobs, apps... could live there I guess.
16:18:27 <dgilmore> its a constantly running process
16:18:47 <dgilmore> I guess we could put it in ansible also
16:19:22 <dgilmore> tyll_: does it run as a service? or do you run it on screen?
16:19:46 <dgilmore> in
16:20:02 <tyll_> dgilmore: currently it runs on a screen, but I believe it can someone hook into the fedmsg to be started as part of other fedmsg services, but I did not try it
16:20:17 <dwa> (ditto for koji-stalk, FYI.)
16:20:22 <nirik> would be good for it to be unattended.
16:20:47 <dgilmore> would be kinda nice to run as a service
16:20:55 <dgilmore> so it just runs when the box boots
16:21:01 <dgilmore> we could work it out
16:21:04 <tyll_> this should be no problem I guess
16:21:24 <dgilmore> I am not sure it really makes sense to package and put into fedora and epel
16:21:46 <dgilmore> as it would only be useful if someone was using koji and pkgdb
16:22:00 <dgilmore> basically mimicing our setup
16:22:08 <dgilmore> but i do want it to be open
16:23:16 * nirik nods
16:23:20 <dgilmore> #action nirik dgilmore and tyll_ to work together to get it up and running on releng04
16:23:45 <dgilmore> #topic secondary arches
16:23:49 <dgilmore> #topic secondary arches - ppc
16:23:54 <dwa> whee
16:23:55 <dgilmore> dwa: you're up
16:24:17 <dwa> #info ppc32 is dead as of last friday on rawhide & later
16:24:41 <dwa> (there are some ... creatively motivated ... people who would like to continue it but I think best to ignore them and not engage)
16:25:07 <dwa> #info We're working on getting ppc64le merged into ppc.koji as soon as possible
16:25:42 <dwa> #info ppc64 is catching up to primary - we just now unblocked webkitgtk, which blocks a stupid amount of packages
16:26:01 <dgilmore> dwa: until they actually do something yeah
16:26:25 <dwa> One thing that I'd like a few thoughts on - sharkcz this morning requested that I move some/all of the builders from RHEL6 to F20. I'm grumpy about it because it's more work and makes them need more active maintenance
16:26:26 * dgilmore wonders why browsers are such huge bits of ugly code
16:26:43 <dwa> but it's because ruby (and a couple other packages?) are dependent on the host kernel
16:26:53 <dgilmore> dwa: we run f20 on the primary builders
16:26:59 <dwa> yeah
16:27:04 <dwa> we've been on RHEL6 forever
16:27:08 <dgilmore> its not really much more work
16:27:10 <sharkcz> so my other argument was consistency
16:27:12 <dwa> and they require very little maintenance, which I like :)
16:27:22 <dwa> dgilmore: you guys regularly re-image your builders, right?
16:27:30 <dgilmore> dwa: yeah
16:27:34 <nirik> irregularly, but yeah.
16:27:35 <dwa> how often?
16:27:37 <dgilmore> we do reinstalls
16:27:44 <dgilmore> every month or two
16:27:55 <dgilmore> dwa: we dont update them daily
16:28:13 <dgilmore> our idea was to never update
16:28:24 <dgilmore> just rebuild with updates enabled
16:28:52 <nirik> right. Or if there's some issue... like createrepo was broken for a while, we had to downgrade it.
16:28:54 <dwa> nod... with ours being RHEL it was just real easy to watch for critical updates & .x releases and update them then
16:29:18 <dgilmore> dwa: we shoudl at least be doing monthly updates of them
16:29:39 <nirik> ideally we would make a process that reaps and rebuilds them
16:29:47 <sharkcz> there is very little what runs them, so the risks are small
16:30:07 <sharkcz> s/them/there/
16:30:09 <nirik> sharkcz: also, they should be isolated from the net... only the hub should be able to talk to them
16:30:09 <dgilmore> nirik: ideally yeah
16:31:10 <dgilmore> i know that the aarch64 builders are not running rhel6 or fedora
16:31:42 <sharkcz> yep, but it's quite close to fedora
16:31:58 <dgilmore> dwa: so there will likely be a point where we can not use rhel6
16:32:01 <dwa> dgilmore: would you say running the builders on RHEL was a /bad/ idea that we should fix post-haste or just merely not optimal?
16:32:34 <dgilmore> dwa: well, its not a bad idea
16:32:44 <dgilmore> we used to use it for all fedora builders
16:32:48 <dwa> (just so I know where I want to prioritize doing that, if I should do them all asap or just a couple to unblock packages immediately, etc)
16:32:54 <dgilmore> and we still have 4 that are rhel6
16:33:01 <dgilmore> 2xppc and 2xx86
16:33:28 <dwa> not even used fedora for power on the ppc builders? Feh ;)
16:33:31 <dgilmore> dwa: being consistent across the arches is a good thing
16:33:45 <dgilmore> dwa: well the 2 power builders only build epel
16:35:02 <dgilmore> dwa: theres not really any good way to make sure some builds land only on some builders
16:35:13 <dwa> dgilmore: couldn't do it with channels?
16:35:15 <nirik> all the armv7 ones are f20... no rhel for them. ;)
16:35:28 <dgilmore> dwa: you can but its uber ugly
16:35:34 <dgilmore> as its manually maintained
16:36:02 <dwa> nod
16:36:12 <dgilmore> nirik: its only the kernel builders on rhel6 for fedora right?
16:36:18 <nirik> yep.
16:36:26 <nirik> I have on my list to move them too, but haven't gotten to it.
16:36:40 <dwa> alright, that's all I've got for today
16:36:46 <dgilmore> dwa: cool.
16:36:56 <dgilmore> so its not end of the world not to do it
16:36:59 * nirik had 2 small items for open floor when we get there.
16:37:05 <dgilmore> but its likely for the best to do so
16:37:11 <dwa> nod
16:37:24 <dgilmore> #topic secondary arches - s390
16:37:32 <dgilmore> sharkcz: you're up
16:37:37 <sharkcz> s390 is catching up primary, after fixing problems here and there
16:37:57 <dgilmore> okay.
16:38:06 <dgilmore> anything exiting to add?
16:38:07 <sharkcz> we can get rid of fake-build-provides as the new kernel package provides kernel also for s390
16:38:19 <dgilmore> sharkcz: interesting
16:38:33 <dgilmore> does it build a vmlinuz?
16:38:37 <sharkcz> yeah, it's the kernel package reorg jwb did and added a little fix
16:38:54 <sharkcz> no, it provides package called kernel on all build arches
16:38:59 <dgilmore> could we make a 31 bit compose?
16:39:07 <dgilmore> ahh okay
16:39:14 <sharkcz> I wouldn't try :-)
16:39:31 <dgilmore> would be kinda nice to drop 31 bit support
16:39:41 <sharkcz> yep, I'll ask about it
16:40:00 <dgilmore> anything else?
16:40:04 <sharkcz> nope
16:40:08 <dgilmore> okay
16:40:16 <dgilmore> #topic secondary arches - arm
16:40:28 <dgilmore> so build issues are getting fixed up
16:40:47 <dgilmore> java 1.7.0 and 1.8.0 need some work but its being worked on upstream
16:41:19 <dgilmore> I ran pungi last week and was able to build a fedora install tree
16:41:28 <dgilmore> but im not able to test it
16:41:56 <dgilmore> not really much to add
16:42:02 <dgilmore> #topic open floor
16:42:10 <dgilmore> nirik: you said you had some things
16:42:21 <nirik> yeah, real quick...
16:42:37 <nirik> 1. the mash patch to keep only recent drpms isn't working. ;(
16:43:24 <nirik> 2. mass rebuild. we should see if we can think of anything we need to do before that happens.... perhaps we could speed up submitting the builds, etc... and should look at any bugs reported in the past we should fix before we fire it off.
16:43:51 <dgilmore> submitting the builds is pretty quick
16:44:15 <dgilmore> but there is some bugs in the reporting scripts that we should look to fix
16:44:28 <nirik> ok. yeah, I recall there were some things the last few times.
16:44:34 <nirik> but we always forget them the next time we run it. ;)
16:45:01 <nirik> doesn't need to be now, but perhaps next meeting we could figure those out
16:45:08 <sharkcz> and maybe check the "tagging back after rebuild" script
16:45:19 <dgilmore> since we moved to git submitting the builds is really fast
16:45:40 <dgilmore> sharkcz: for what?
16:45:58 <dgilmore> nirik: sure
16:46:07 <dgilmore> we are over an hour already
16:46:23 <sharkcz> IIRC there used to few issues with thing tagged wrongly? but not sure if it can be checked in advance
16:46:33 <nirik> oh...
16:46:40 <nirik> also, we need to do the orphan roundup?
16:46:50 <dgilmore> yeah we do
16:47:08 <dgilmore> i know gmime22 has failed to build since f18
16:47:24 <dgilmore> it should be removed
16:47:41 <dgilmore> need to clean up long term ftbfs and orphabns
16:47:44 <dgilmore> orphans
16:47:49 <nirik> yep. before mass rebuild.
16:48:06 <dgilmore> who normally does that?
16:48:11 <dgilmore> i think its notting
16:48:26 <nirik> yeah, althought tyll_ might have done it last cycle?
16:49:25 <dgilmore> #action need to run ftbfs and orphan cleanup
16:50:03 <dgilmore> #action all, look at scripts used for mass rebuild and reporting and fix bugs
16:50:36 <dgilmore> i think there was some bugs with the script that reported ftbfs bugs
16:51:33 <dgilmore> anyone have anything else?
16:52:23 <dgilmore> #endmeeting