15:33:17 <dgilmore> #startmeeting RELENG (2016-04-04)
15:33:17 <zodbot> Meeting started Mon Apr  4 15:33:17 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:33:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:33:17 <zodbot> The meeting name has been set to 'releng_(2016-04-04)'
15:33:17 <dgilmore> #meetingname releng
15:33:17 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:33:17 <zodbot> The meeting name has been set to 'releng'
15:33:17 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:33:20 <dgilmore> #topic init process
15:33:23 <nirik> morning
15:34:16 <masta> hey gang
15:34:26 <kushal> hello
15:34:41 <maxamillion> .hello maxamillion
15:34:43 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
15:34:55 <maxamillion> brb, need a cup of coffee
15:35:19 <dgilmore> maxamillion: I may need to brew more
15:35:23 <dgilmore> or make some espresso
15:35:38 <dgilmore> pbrobinson: here?
15:35:57 <dgilmore> #topic #1107 auto-cleanup rawhide trees
15:36:02 <dgilmore> https://fedorahosted.org/rel-eng/ticket/1107
15:36:16 * sharkcz is here
15:36:27 <dgilmore> we need to come up with a few things
15:36:39 * pbrobinson waves
15:36:51 <dgilmore> namely Agreement on policy for cleanup
15:37:22 <pbrobinson> I have some ideas around this with the new pungi, just not had time to hack up some scripts
15:37:24 <dgilmore> I know in the past we wanted to make sure we kept an install tree
15:37:26 <nirik> I know in the past QA has wanted all the tc/rc's to stay around until after the GA release.
15:37:46 <dgilmore> given how pungi works now if we dont have a install tree the compose fails
15:37:57 <dgilmore> nirik: this is a different thing
15:38:08 <nirik> ok. wasn't sure if we were just talking rawhide or all of them.
15:38:16 <dgilmore> this is about how many branched/rawhide composes to keep
15:38:31 <nirik> ah, ok.
15:38:56 <dgilmore> I believe lmacken is making bodhi keep the last 3 updates pushes trees
15:39:17 <maxamillion> annnnd back
15:39:19 <nirik> last 3 that worked? or last three period?
15:39:24 <dgilmore> I think we can keep with manually removing TC/RC's
15:39:37 <maxamillion> dgilmore: +1 more coffee
15:39:42 <dgilmore> I think last three that worked
15:39:52 <nirik> that sounds reasonable.
15:39:55 <pbrobinson> yea, they're in a separate directory structure so straight forward enough
15:39:56 <dgilmore> but would need to check with lmacken
15:40:20 <maxamillion> how do programmatically know which ones "worked"?
15:40:22 <dgilmore> I think for branched and rawhide 14 days would be okay
15:40:33 <maxamillion> like, how is that quantified?
15:40:33 <pbrobinson> maxamillion: check the STATUS file?
15:40:36 <maxamillion> pbrobinson: ah
15:40:57 <pbrobinson> maxamillion: it has a one word status in there that should be easy enough to parse
15:41:02 <maxamillion> pbrobinson: I didn't know if something was running tests against it or if it was literally just "the compose worked"
15:41:13 <dgilmore> that status file is only pungi
15:41:17 <dgilmore> not bodhi
15:41:22 <dgilmore> bodhi uses mash
15:41:26 <nirik> 14 days is fine with me. If we go more than 3-4days broken things are not fun. ;(
15:41:34 <dgilmore> and at some point will use koji signed repos
15:41:41 <dgilmore> nirik: right
15:42:23 <dgilmore> I want to turn the fedmsg off in #fedora-releng for when rawhide/branched start and finish
15:42:24 <pbrobinson> well presumably then we can use the standard koji garbage collection to deal with that?
15:42:36 <dgilmore> and for when bodhi finishes
15:42:38 <nirik> oh good, mm doesn't completely blow up when there's not been a rawhide compose in >4 days
15:42:45 <dgilmore> and only have status updates for when things fail
15:43:03 <dgilmore> pbrobinson: afaik no,
15:43:13 <dgilmore> pbrobinson: kojira manages it
15:43:30 <dgilmore> I think it actually defaults to keeping them all
15:43:32 <pbrobinson> dgilmore: the signed repos or the GC?
15:43:44 <dgilmore> pbrobinson: koji-gc can not do signed-repos
15:44:12 <pbrobinson> dgilmore: as in clean up old signed repos after X days?
15:44:16 <dgilmore> pbrobinson: yes
15:44:28 <dgilmore> koji-gc has zero knowledge of signed repos
15:44:42 <pbrobinson> dgilmore: any reason why it can't be added, it seems like the right spot to do it if it cleans up everything else
15:44:47 <nirik> sidenote: any chance we could get some dnf folks help and make signed repos use dnf so we can use rich deps? or way too much work?
15:44:55 <dgilmore> pbrobinson: because kojira manages them
15:45:05 <dgilmore> and by default I think it keeps them all
15:45:22 <dgilmore> nirik: pungi still uses yum and not dnf
15:45:31 <dgilmore> nirik: none of the compose tooling uses dnf
15:45:39 <nirik> dgilmore: but it doesn't seem to care about rich deps for some reason
15:45:45 <nirik> but I could be wrong
15:45:48 <dgilmore> nirik: no idea there
15:45:56 <pbrobinson> dgilmore: what ever, my point was surely koji (which ever bit, I really don't give a shit) would be the right thing to track and do garbage collection/cleanup
15:45:57 <nirik> just by luck I am sure.
15:46:02 <dgilmore> but it uses yum to resolve deps for multilib
15:46:23 <nirik> ok. thats a side track anyhow, sorry...
15:46:40 <dgilmore> pbrobinson: somthing needs to
15:46:58 <dgilmore> what that is I am not sure
15:47:12 <dgilmore> Jay was talking about having koji keep all signed repos forever
15:47:26 <dgilmore> I know there was some kojira integration
15:47:31 <dgilmore> its very early days
15:47:49 <nirik> the docs say kojira can delete them
15:48:00 <nirik> "hey are maintained by the Kojira service, which is responsible for regenerating them as needed when the content of a build tag changes (including inheritance changes), and for deleting them when they become stale"
15:48:34 <nirik> https://fedoraproject.org/wiki/Koji/Repositories
15:49:02 <nirik> anyhow, +1 to 14 days of rawhide and branched kept
15:49:25 <dgilmore> does anyone object to removing any branched/rawhide compose older than 14 days?
15:49:36 <sharkcz> nope
15:49:43 <pbrobinson> nope
15:50:06 <dgilmore> #agreed we will remove any branched/rawhide compose older than 14 days
15:50:14 <dgilmore> cool. that should be easy to implement
15:50:15 <masta> no objection
15:50:38 <dgilmore> #action dgilmore to implement rawhide/branched cleanup in nightly.sh
15:50:57 <dgilmore> #topic Secondary Architectures updates
15:51:09 <dgilmore> any generic secondary arch things to cover?
15:51:27 <pbrobinson> not really, various infrastructure changes all over them this week
15:51:32 <maxamillion> dgilmore: why turn off the fedmsg?
15:51:35 <nirik> there was an unfortunate switch change last week that knocked ppc offline. ;(
15:51:45 <maxamillion> dgilmore: wow, nvm ... sorry ... my irc client froze in the backscroll
15:51:46 <pbrobinson> nirik: and arm too
15:52:14 <nirik> oh, I thought it just hit ppc? huh... oh, the arm pungi4 composer is in ppc... right
15:52:24 <pbrobinson> nirik: yep, for the moment it is
15:52:42 <nirik> anyhow, hopefully thats sorted now
15:52:54 <dgilmore> cool
15:52:55 <dgilmore> #topic Secondary Architectures update - ppc
15:53:01 <dgilmore> pbrobinson: how is ppc?
15:53:02 <lmacken> n/win bal
15:53:32 <pbrobinson> ppc is good, alpha went out on Thurs, we now have koji build qemu/docker nightlies all live
15:53:56 <pbrobinson> so overall it's looking pretty good
15:54:14 <dgilmore> nice
15:54:20 <pbrobinson> I'm upgrading the compose boxes now to get the latest pungi and have all the changes we needed in and testing them
15:54:25 <dgilmore> #info f24 alpha for ppc shipped
15:54:27 <pbrobinson> so hoping there's no regressions
15:54:32 <sharkcz> build-wise the ghc stack is broken because of a silent ABI hash change, will be fixed with next ghc rebase, there is no simple solution for that :-(
15:54:41 <dgilmore> #info now have nightly cloud images and docker base image
15:55:01 <dgilmore> sharkcz: thats ghc being broken
15:55:22 <dgilmore> the number of cores on the build box changing can change the abi
15:55:40 <pbrobinson> and I'll be doing various infrastructure rebuilds this week in the last push to clean ppc all up, nirik will be bothering you about koji hubs at some point
15:55:45 <dgilmore> at least in the past
15:55:47 <sharkcz> yep, seems related to the move to the new p8 builders
15:55:53 <pbrobinson> oh yay!
15:56:21 <dgilmore> sharkcz: sounds joyous
15:56:23 <nirik> pbrobinson: sounds good. ;)
15:56:42 <dgilmore> pbrobinson: ppc hub is teh last thing to be ansibilised right?
15:56:48 <pbrobinson> yep!
15:57:13 <dgilmore> awesome
15:57:18 <dgilmore> that is for ppc?
15:57:19 <pbrobinson> I'm rebuilding the underlying P8s from ppc64 -> ppc64le first so they're fully supportable
15:57:27 <dgilmore> awesome
15:57:32 <dgilmore> fully supportable is good
15:57:35 <pbrobinson> then we'll move the hub over
15:57:55 <dgilmore> you are going to move the hub to ppc64le from ppc64 right?
15:58:01 <pbrobinson> and the puiterwijk and I will start dealing with the ppc cloud boxes for copr
15:58:18 <pbrobinson> dgilmore: that is the plan, yes
15:58:18 <dgilmore> so it may be a bit bigger outage as we will have to dump and restore the db rather than rsyncing it over
15:58:46 <pbrobinson> possibly, was going to chat wil nirik about that once I was closer to ready
15:59:08 <nirik> the secondary db's have not been that big... not next to primary anyhow. ;)
15:59:30 <pbrobinson> because we'll need to dump as a char not binary dump for BE -> LE reasons
16:00:17 <dgilmore> #info need to work out be to le db conversion plan
16:00:59 <dgilmore> #topic Secondary Architectures update - s390
16:01:04 <dgilmore> sharkcz: how is s390x?
16:01:21 <pbrobinson> so we should have network changes in place for compose boxes there shortly too
16:01:49 <sharkcz> looking good, the weak deps resolving likely has a known cause - needs  F-23+ based builder for newRepo tasks
16:01:51 <pbrobinson> and I'll build x86 bits for that so we can move over repo creation and pungi control etc
16:02:08 <dgilmore> sharkcz: what are you using for the builders?
16:02:23 <pbrobinson> dgilmore: the hub is currrently used for newRepo
16:02:23 <sharkcz> dgilmore: builders are F-23, but the hub with /mnt/koji is EL-7
16:02:35 <dgilmore> sharkcz: ahh okay
16:02:54 <dgilmore> sharkcz: but should be using createrepo_C right?
16:03:05 <sharkcz> dgilmore: it is
16:03:08 <dgilmore> since kojid.conf in ansible is using it
16:03:36 <dgilmore> sharkcz: so the createrepo_c in el7 does not support weakdeps ?
16:03:50 <dgilmore> i.e. it does not put them into the repodata>
16:04:07 <sharkcz> dgilmore: the underlying rpm doesn't, so the weak/rich deps are silently ignored
16:04:38 <dgilmore> sharkcz: well thats crappy
16:04:46 <sharkcz> dgilmore: yep ...
16:04:47 <pbrobinson> yep, hence the reason we'll put a x86 f23 builder online in PHX, we have compute resources, and do it that way.
16:05:48 <sharkcz> dgilmore: https://bugzilla.redhat.com/show_bug.cgi?id=1300669#c33
16:06:05 <dgilmore> I need to put out a koji update this week
16:06:18 <dgilmore> FPC decided that weakdeps need to not be installed
16:06:27 <dgilmore> in the buildroot
16:06:36 <dgilmore> people need to explicitly BR them
16:06:43 <dgilmore> so I do not think its a huge deal
16:06:58 <dgilmore> as it will be the default everywhere soon
16:08:05 <sharkcz> probably, the question is how it will work with "weak" Requires of 1st level BuildRequires, in the deps tree
16:09:14 <dgilmore> I do think that weak deps and rich deps have not been well thought out in practice
16:09:43 <dgilmore> and there hasd not been enough effort from the packaging team to make sure the tooling supported them
16:09:44 <sharkcz> yes, IMHO there are hidden consequences
16:10:24 <dgilmore> indeed
16:10:33 <dgilmore> but its not totally our problem to solve
16:10:48 <dgilmore> anything else to cover?
16:10:55 <sharkcz> nope
16:10:58 <dgilmore> #topic Secondary Architectures update - arm
16:11:04 <dgilmore> pbrobinson: how is aarch64?
16:11:14 <pbrobinson> looking good, we got alpha out
16:11:26 <pbrobinson> will be adding qemu/docker bits in the next week or two
16:11:33 <pbrobinson> before beta freeze
16:11:36 <dgilmore> #info Alpha for f24 shipped
16:11:38 <pbrobinson> and also disk images
16:11:56 <dgilmore> #info work underway to add cloud images and docker base image to composes
16:12:24 <dgilmore> pbrobinson: do we have a full list of planned deliverables for F24 for aarch64?
16:12:27 <pbrobinson> over all it's looking pretty good and we should be able to have shipping "firmware" for aarch64 uEFI soon now the license issues are resolved
16:12:35 <dgilmore> probably should actually check that for all arches
16:12:57 <dgilmore> nice
16:13:02 <pbrobinson> dgilmore: sort of, we have a list, but it's not really documented
16:13:05 <dgilmore> I missed that had been resolved
16:13:16 <pbrobinson> dgilmore: yes, contribution by Intel
16:13:41 <dgilmore> awesome
16:13:54 <maxamillion> nice
16:14:01 <pbrobinson> I think it's landed upstream, I've reached out to virt team to work out the consumable time frame
16:14:23 <dgilmore> https://github.com/tianocore/tianocore.github.io/wiki/edk2-fat-driver
16:14:37 <pbrobinson> people on PTO I think so not got a response yet but I'm hoping it'll land in F-24, its self contained so I'm hopeful
16:14:54 <pbrobinson> complete response...
16:15:41 <masta> I think the edk2 in rawhide still points at sourceforge
16:16:20 <masta> so hopefully that links-up with github
16:16:56 <dgilmore> pbrobinson: please keep us in the loop as you hear from people
16:17:34 <pbrobinson> dgilmore: of course
16:18:05 <dgilmore> anything else aarch64 related we need to cover?
16:18:34 <dgilmore> #topic Open Floor
16:18:46 <dgilmore> does anyone have anything they want to bring up?
16:18:54 <maxamillion> I have something
16:19:01 <pbrobinson> I do to
16:19:02 <pbrobinson> too
16:19:07 <dgilmore> maxamillion: you are up?
16:19:54 <maxamillion> I just wanted to update everyone on the Layered Image Build Service ... things are working now and I'll be deploying OSBS to it's proper stage instance this afternoon, patches have been submitted upstream --> https://bugzilla.redhat.com/show_bug.cgi?id=1243736#c20
16:20:31 <nirik> cool.
16:20:47 <dgilmore> maxamillion: nice. when are we going to deploy in prod?
16:21:14 <maxamillion> dgilmore: maybe next week? ... I still have some things around the docker registry I need to sort out
16:21:39 <dgilmore> maxamillion: okay cool. Let me or anyone else know if you need help getting things done
16:21:45 <maxamillion> dgilmore: +1 - will do, thanks
16:21:46 * nirik notes he reinstalled koji01.stg a bit ago.
16:22:03 <maxamillion> nirik: +1 thanks
16:22:10 <maxamillion> nirik: that's done and I can fiddle with it now?
16:22:14 <nirik> yep.
16:22:26 <nirik> I didn't sync the db... that takes a long long time... 6hours?
16:22:30 <maxamillion> nirik: were you going to re-install buildvm-01.stg also?
16:22:37 <nirik> I can if you like
16:22:40 <maxamillion> nirik: no worries, I don't need a resync db
16:22:52 <maxamillion> nirik: I don't mind either way, I don't know if it needs it or not
16:22:52 <dgilmore> maxamillion: anything else?
16:23:00 <maxamillion> dgilmore: no
16:23:10 <dgilmore> pbrobinson: the floor is yours
16:23:37 <pbrobinson> so has there been changes in pushing of updates that haven't been reflected in docs?
16:23:45 <pbrobinson> I've getting "IOError: [Errno 13] Permission denied: '/etc/bodhi/production.ini'"
16:24:00 <dgilmore> pbrobinson: you need to sudo to apache not masher
16:24:03 <pbrobinson> and it's owned by apache with 600 perms
16:24:07 <dgilmore> I thought the docs got updated for that
16:24:36 <pbrobinson> I didn't see it when I checked in pagure this morning but maybe I was being blind
16:24:49 <dgilmore> bodhi was changed to run as apache so that it can hardlink the rpms to /mnt/koji/packages
16:25:01 <dgilmore> pbrobinson: maybe it got forgotten :(
16:25:15 <masta> yeah, we sudo -u apache now
16:26:29 * masta updates the wiki to reflect the change
16:26:39 <maxamillion> wait, what wiki page?
16:26:41 <dgilmore> masta: the wiki is dead
16:26:45 <maxamillion> the releng docs aren't in the wiki anymore
16:26:50 <dgilmore> masta: the docs are all in pagure
16:27:01 <dgilmore> pbrobinson: the docs did not get updated sorry
16:27:16 <masta> https://fedoraproject.org/wiki/Pushing_updates_SOP  <-- that one
16:27:27 <pbrobinson> I was checking docs in pagure
16:27:44 <pbrobinson> masta: as it says in the header that one is dead
16:27:53 <masta> okay
16:27:57 <dgilmore> masta: look at teh top
16:28:08 <dgilmore> pbrobinson: I just updated the docs
16:28:23 <dgilmore> #info everyone please when changing processes update teh docs in pagure
16:28:42 <pbrobinson> dgilmore: I did say "<pbrobinson> I didn't see it when I checked in pagure this morning but maybe I was being blind" above ;-)
16:29:35 <dgilmore> pbrobinson: :) you were not blind
16:29:47 <dgilmore> https://fedoraproject.org/wiki/Pushing_updates_SOP now has no content
16:29:53 <pbrobinson> although I often still use the wiki docs for some things because pagure doesn't have describing URLs I can quivkly access via history and end up having to poking through pagure each time
16:29:59 <dgilmore> and has a link to the pagure docs page
16:30:55 <dgilmore> pbrobinson: https://docs.pagure.org/releng/sop.html#standard-operating-procedures
16:31:33 <nirik> yeah, if you go via the main site -> docs it's in a frame
16:32:01 <dgilmore> nirik: right, it is a lot less useful at pagure.io
16:32:18 <nirik> there's a thing in the upper right to 'open in a new tab'
16:32:23 <dgilmore> https://docs.pagure.org/releng/
16:32:31 <pbrobinson> ah, nice, I'd not managed to find that set of links
16:34:02 <dgilmore> I need to update the sudo rules
16:34:13 <dgilmore> for teh apache change
16:36:25 <dgilmore> anyone have anything else?
16:37:02 <pbrobinson> not from me
16:37:08 <nirik> nada
16:37:10 <dgilmore> #endmeeting