15:36:46 <dgilmore> #startmeeting RELENG (2017-02-13)
15:36:46 <zodbot> Meeting started Mon Feb 13 15:36:46 2017 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:36:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:36:46 <zodbot> The meeting name has been set to 'releng_(2017-02-13)'
15:36:46 <dgilmore> #meetingname releng
15:36:46 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
15:36:46 <zodbot> The meeting name has been set to 'releng'
15:36:46 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
15:36:49 <dgilmore> #topic init process
15:36:59 <dgilmore> #chair puiterwijk
15:36:59 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
15:37:06 <mboddu> .hello mohanboddu
15:37:07 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
15:37:11 <sharkcz> .hello sharkcz
15:37:13 <zodbot> sharkcz: sharkcz 'Dan Horák' <dan@danny.cz>
15:37:14 <marc84> hi everyone
15:37:29 <pbrobinson> o/
15:37:34 <puiterwijk> Hello
15:37:39 <puiterwijk> .hello puiterwijk
15:37:40 <zodbot> puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' <puiterwijk@redhat.com>
15:39:11 <nirik> morning
15:39:28 <dgilmore> lets get started
15:39:43 <dgilmore> #topic #6626 Consider adding f26-pending the f26-build inheritance chain
15:39:50 <dgilmore> https://pagure.io/releng/issue/6626
15:40:22 <dgilmore> I am against changing this at this point in time
15:40:30 <dgilmore> branching is not far away
15:40:40 <puiterwijk> Right. I meant this more in general for rawhide
15:40:53 <dgilmore> if we added f26-pending into the buildroot
15:41:02 <nirik> so the unsigned ones also add to the buildroot?
15:41:08 <puiterwijk> nirik: correct. That was my idea
15:41:19 <dgilmore> we would have to redo a bunch of inheritance, that would have to be undone at branching or Alpha Freeze
15:41:23 <nirik> note that if we ever decide to enable gpgcheck=1 that will quash that ideas
15:41:35 <dgilmore> nirik: enable it where?
15:41:42 <puiterwijk> nirik: well, only for building. Not for stuff that gets into the repo.
15:41:46 <nirik> on builders
15:41:59 <dgilmore> nirik: not possible without massive amounts of change in koji
15:42:03 <puiterwijk> dgilmore: right. So this ticket was more for the general rawhide case than for the specific f26 one.
15:42:13 <nirik> ok.
15:42:24 <dgilmore> puiterwijk: I would rather just wait until post branching.
15:42:32 <nirik> Not a big deal, but I think it's still a good goal to someday have https and gpgcheck everywhere.
15:42:48 <dgilmore> puiterwijk: because we are going to have to make all sorts of changes if/when no more alpha change is accepted
15:42:49 <puiterwijk> dgilmore: right. I'd totally be fine with that.
15:42:54 <dgilmore> as we will have other gating
15:43:16 <dgilmore> and would want to have builds that are not ready for consumption in the buildroot to build against
15:43:51 * nirik is happy to punt that idea, just something to consider
15:43:57 <dgilmore> my idea for it is that only after testing has passed does a build get signed
15:43:57 <bowlofeggs> .hello bowlofeggs
15:43:58 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
15:44:30 <dgilmore> nirik: not sure it is an idea that fits into future changes at all. but  we can look at it down the road
15:44:57 <nirik> sure
15:45:08 * puiterwijk notes that this is why my ticket title says "Consider" and not "please do" :)
15:45:17 * dgilmore was wanting packages to be signed before landing in rawhide, abd that things with breakage or not passed tests live in a unsigned world
15:45:24 <dgilmore> puiterwijk: sure
15:45:28 <dgilmore> puiterwijk: we are considering
15:45:32 <puiterwijk> +1
15:46:00 <puiterwijk> So, how about we say "punt to post-branch", and if anyone in the meantime has any strong feelings, they can add it to the ticket?
15:46:11 <dgilmore> puiterwijk: thats fine with me
15:47:22 <dgilmore> #agreed we will punt until after branching and evaluate with f27 changes
15:47:52 <dgilmore> #topic #6603 Enable rhel-kvm-rpms on EPEL7 aarch64 buildroots
15:47:59 <dgilmore> https://pagure.io/releng/issue/6603
15:48:09 <dgilmore> pbrobinson: how is this coming along?
15:48:20 <dgilmore> probably does not need to be in the meeting
15:49:00 <dgilmore> #info will ask pbrobinson for status out of meeting
15:49:12 <dgilmore> #topic #6601 Migrate the rel-eng tickets from (old) trac
15:49:20 <dgilmore> https://pagure.io/releng/issue/6601
15:49:39 <dgilmore> nirik: any update here?
15:50:03 <nirik> nope. I should ping pingou again on it. ;)
15:50:28 <dgilmore> I just asked him to joing and give us an update
15:50:44 <nirik> ok, I just asked too.
15:51:05 <dgilmore> #info awaiting feedback from pingou
15:51:30 <dgilmore> #topic #6600 Mock and sigined rawhides
15:51:40 <dgilmore> https://pagure.io/releng/issue/6600
15:51:51 <dgilmore> I have thought about this some
15:52:06 <dgilmore> and I really do not think there is a good answer
15:52:19 <dgilmore> there has always been issues around branching
15:52:29 <dgilmore> and this is just another snag
15:52:45 <nirik> well...
15:53:01 <dgilmore> this will break rawhide mock builds
15:53:15 <nirik> if we make the f27 key now and sign things and mock adds the f27 key now, at branching time won't it just work?
15:53:16 <dgilmore> but there is always the delay in getting the new releases configs in place
15:53:33 <dgilmore> maybe
15:54:35 <nirik> There may be some short gap, but it should be small... unless mock doesn't want to update before branching
15:54:40 <dgilmore> nirik: the mock config will have to be updated every release
15:54:53 <dgilmore> which he asks to avoid
15:54:56 <nirik> yeah... well, it has to update at branch anyhow right?
15:55:03 <dgilmore> right
15:55:19 <dgilmore> 3) ideally it does not change rawhide mock config every release (so update does not create .rpmnew files).
15:55:42 <nirik> yeah, no way there. Unless mock moved to a 'get mock configs from the net' model or something
15:55:45 <dgilmore> mock also does not use our copies of the keys
15:55:59 <dgilmore> it uses copies from a different package
15:56:20 <nirik> oh yeah. ;(
15:56:39 <masta_> that is unfortunate
15:56:42 <nirik> anyhow, I saw we make f27 key now, sign everything with both it and f26 key... at branching we are ready a
15:56:56 <nirik> mock can add the f27 key now or wait until branching
15:57:22 <dgilmore> we will need to make the f27 key in the next week or so
15:57:29 <dgilmore> and start signing everything
15:57:36 <nirik> yep
15:57:38 <dgilmore> so that at branching rawhide will be signed
15:57:48 <dgilmore> an option is to have a rawhide key
15:58:01 <dgilmore> and then prior to branching create and use the release key
15:59:06 <masta> that seems nice too
15:59:39 <nirik> I kinda like using the next fedora key... then it gets swapped out as time goes on.
16:01:22 <masta> I like both ideas.
16:01:31 <nirik> so anyhow, I'd say on this ticket we tell him our plans, say we can't do anything to solve all the needs, do the best we can
16:01:44 <dgilmore> nirik: there is perks to switching the key
16:01:46 <masta> I mean, a rawhide key sorta make sense... things stop being rawhide when they are signed with the fedora key.
16:01:53 <dgilmore> it never ends up weak for instance
16:02:13 <dgilmore> nirik: sounds good
16:02:30 <nirik> yeah, we always get a new shiny one
16:02:33 <dgilmore> #info Will update the ticket with our plans.
16:03:00 <dgilmore> #topic #6577 Fedora 26 mass rebuild
16:03:07 <dgilmore> https://pagure.io/releng/issue/6577
16:03:18 <dgilmore> so we finally ran the mass rebuild
16:03:24 <dgilmore> we twice hit snags
16:03:37 <dgilmore> one was due to a bug in fedpkg
16:03:57 <dgilmore> well really rpkg
16:04:04 <dgilmore> https://pagure.io/rpkg/issue/197
16:04:42 <dgilmore> #info issue filed to fix the breakage and pain we saw with fedpkg/rpkg
16:04:48 <nirik> The arm01 builders were flaky and I removed them almost from the start...
16:04:56 <nirik> but otherwise everything did very well
16:05:01 <dgilmore> the other issue I am not sure why we hit it exactly
16:05:16 <dgilmore> the kerberos tgt expired
16:05:30 <dgilmore> we should have been using a keytab
16:05:41 <puiterwijk> dgilmore: that totally depends on how you ran the koji commands
16:05:54 <puiterwijk> I am not 100% sure that fedpkg uses the keytabs, due to that same bug
16:05:57 <dgilmore> puiterwijk: well fedpkg does not use koji
16:06:13 <dgilmore> it uses the api and implements its own koji cli
16:06:17 <puiterwijk> Right.
16:06:30 <puiterwijk> And then it would need to actually parse the keytab arguments to pass to koji
16:06:31 <dgilmore> i ended up kiniting and it moved on
16:06:43 <dgilmore> puiterwijk: likely not doing so
16:06:52 <puiterwijk> dgilmore: nope. It doesn't
16:07:10 <puiterwijk> So again, due to rpkg implementing its own koji config parsing it didn't pass required stuff on (this time keytab)
16:07:27 <dgilmore> http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-need-rebuild.html
16:07:29 <puiterwijk> (just verified in the source code)
16:07:34 <dgilmore> http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-failures.html
16:07:41 <puiterwijk> So, can we clear the "I have no idea how it happened" flag? :)
16:07:53 <dgilmore> #info currently 1363 failed builds
16:08:13 <dgilmore> #info currently 1781 packages needing rebuilding
16:08:39 <mboddu> dgilmore: since we are here, I am getting https://paste.fedoraproject.org/556932/14870020/ messages when I tried signing f26-rebuild
16:08:47 <dgilmore> #action mboddu to send out email with mass rebuild status and updates to devel-announce@lists.fp.o
16:08:59 <dgilmore> I think we forgot to send an email saying we were starting
16:09:21 <dgilmore> mboddu: so signing failed
16:09:32 <dgilmore> and it can not write out the signed copy
16:09:37 <mboddu> dgilmore: yes
16:10:00 <dgilmore> mboddu: thats not very helpful. the siging failure was previous in the logs
16:10:22 <dgilmore> It probably needs debugging on the bridge
16:10:34 <dgilmore> mboddu: but it is offtopic for the meeting
16:10:35 <mboddu> dgilmore: we can move it to #fedora-releng
16:10:41 <puiterwijk> Before you do that, try to run a single thing via sigul sign-rpm...
16:10:43 <mboddu> dgilmore: yes
16:11:38 * nirik notes related rawhide is failing with a pretty strange failure. ;)
16:13:01 <dgilmore> #info once we have signed all of the mass rebuild it will be merged into rawhide
16:13:52 <dgilmore> #info as there is not supposed to be any soname bumps, etc changing as teh mass rebuild does not handle them we do not expect breakage in deps to change
16:15:22 <dgilmore> #topic Broken Rawhide
16:15:35 <dgilmore> lets cover this real quickly since it came up
16:15:53 <dgilmore> my guess is that either a new rpm or nss showed up which breaks things
16:15:59 <nirik> sure. I looked, but it was odd and I didn't have any immediate ideas...
16:16:17 <nirik> it's crashing in doing a 'rpm -qa' at the end of the cloud install.
16:17:48 <nirik> there is a new nss... might be the culprit
16:18:01 <dgilmore> nirik: its build install
16:18:02 <nirik> "Disable TLS 1.3, following the upstream change"
16:18:04 <dgilmore> not cloud install
16:18:10 <dgilmore> buildinstall is lorax
16:18:10 <nirik> sorry, yeah, misspoke
16:18:20 <dgilmore> not making a cloud image
16:18:48 <dgilmore> maybe lorax changed
16:19:17 <dgilmore> there was a lorax build on the 6th
16:19:56 <dgilmore> though the changelog does not look suspicous
16:20:39 <nirik> I am suspecting the nss build on the 9th, but I would think it would have caused problems here for me.:)
16:21:34 <dgilmore> nirik: not necessarily
16:21:50 <dgilmore> if its only initiating a new environment you would not hit it
16:22:14 <dgilmore> #info debugging and fixing rawhide breakages have to be high priority
16:22:37 <nirik> yeah.
16:22:41 <dgilmore> rawhide in the last month has been too broken for my liking
16:22:58 <dgilmore> lets take debugging out of the meeting
16:23:02 <nirik> agreed, but I think we should be more clear when we talk about broken
16:23:18 <dgilmore> nirik: more clear in what way?
16:23:30 <nirik> composes are broken vs install path is broken vs day to day use broken
16:23:57 <nirik> "rawhide is broken" makes people think it's unusable for anything and doesn't boot or work at all
16:24:04 <dgilmore> sure
16:24:24 <dgilmore> by rawhide being broken I am talking only about the compose and install path being broken
16:24:27 <nirik> I agree composes and install path is too broken lately and we need to up our game
16:24:36 * nirik nods
16:24:38 <puiterwijk> The lorax stuff is curl/nss
16:25:04 <puiterwijk> dgilmore, nirik: initiating a new NSS environment is often problematic within Koji builds, because NSS doesn't work well with multiple libraries using it in the same process
16:25:26 <dgilmore> #info rawhide broken in teh releng context is to do with broken composes and install path
16:25:33 <dgilmore> #undo
16:25:33 <zodbot> Removing item from minutes: INFO by dgilmore at 16:25:26 : rawhide broken in teh releng context is to do with broken composes and install path
16:25:38 <puiterwijk> I've got the Lorax stuff on my todo list. My Oz patch is awaiting upstream review
16:25:40 <dgilmore> #info rawhide broken in the releng context is to do with broken composes and install path
16:25:53 <nirik> puiterwijk: sure, but it used to work here, so I think the new nss update changed something
16:26:19 <puiterwijk> nirik: either that, or the fact that we are now forcing everything over https. But yeah, likely new NSS broke more then before.
16:26:29 <puiterwijk> I just try to move everything away from curl when I see it
16:26:40 <dgilmore> the last working rawhide compose was the 8th of Feb
16:26:49 <dgilmore> which lines up with the nss update
16:27:00 <puiterwijk> Right. So that'd say that NSS broke even more. Fun
16:27:09 <dgilmore> puiterwijk: and lorax does not use curl
16:27:29 <dgilmore> not in teh context of processing its templates
16:27:34 <nirik> https://kojipkgs.fedoraproject.org//work/tasks/3674/17833674/runroot.log is the log FWIW
16:27:35 <puiterwijk> dgilmore: it uses NSS somewhere.
16:27:48 <puiterwijk> But right. What I saw is that it errored in the execution of the installtree template
16:27:56 <nirik> rpm requires nss...
16:28:07 <nirik> which was running rpm -qa
16:28:09 <puiterwijk> nirik: yep. And it even initializes it :/
16:28:34 <puiterwijk> Right. So likely something left NSS in a bad state, and then something else (rpm?) didn't like the current state
16:28:45 <dgilmore> https://github.com/rhinstaller/lorax/blob/master/share/templates.d/99-generic/runtime-postinstall.tmpl#L120
16:28:53 <dgilmore> thats the failing line of the template
16:29:23 <puiterwijk> dgilmore: the problem is that the NSS state is process-shared, and traverses to childs. Meaning that even with fork()'s, it will still have the broken NSS state
16:29:35 <dgilmore> puiterwijk: sure
16:29:52 <puiterwijk> So, it might have broken the state at the start, and we might then not see it untuil the very end
16:30:21 <dgilmore> anyway we are up on time
16:30:32 <dgilmore> lets do alt arches and open floor really quickly
16:30:35 <puiterwijk> Anyway, it sounds like we're going to blame NSS folks. +1 from me, just saying it'll likely be hard to debug
16:30:51 <dgilmore> #topic Alternative Architectures updates
16:31:01 <dgilmore> puiterwijk++
16:31:01 <nirik> I guess a first stab would be to try untaggng the nss build... barring any good way to debug
16:31:11 <dgilmore> nirik: indeed
16:31:18 <dgilmore> that will let us test
16:31:27 <dgilmore> pbrobinson: sharkcz: any updates?
16:31:40 * nirik is reminded to go ask about s390 again)
16:31:44 * dgilmore has a quick thing
16:31:50 <sharkcz> waiting for the internal s390 infra changes
16:32:01 <dgilmore> lastw eek in some meetings I had Alt arches came up
16:32:01 <sharkcz> otherwise no issues
16:32:24 <dgilmore> as a result of the discussions.
16:32:27 <dgilmore> https://fedoraproject.org/wiki/Architectures
16:32:35 <dgilmore> I updated the architecture wiki page
16:32:57 <dgilmore> it still had i686 as primary and did not at all reflect the current state
16:33:10 <dgilmore> #info Architecture wikli page has been updated
16:33:24 <dgilmore> #action all please review and update or raise concerns
16:33:54 <dgilmore> #action dgilmore to put out a call for a x86 sig to be formed by interested parties
16:34:25 <nirik> pbrobinson: any news on the armv7 buildvms on aarch64? I would be happy to spend a few cycles on it if that would be of help
16:34:37 <dgilmore> there is probaly updates needed over the new koji setup
16:34:52 <pbrobinson> nirik: getting there
16:35:25 <nirik> cool. I'd love to stop power cycling and dealing with the calxeda. ;)
16:35:27 <pbrobinson> sorry, just mostly currently attempting to remain upright with a stomach cramp
16:35:34 <nirik> ouch. ;(
16:35:37 <dgilmore> :(
16:35:43 <dgilmore> #topic open floor
16:35:46 <bowlofeggs> Announcement: There's a bodhi-2.4.0 beta deployed at https://bodhi.stg.fedoraproject.org and you can try the CLI from https://copr.fedorainfracloud.org/coprs/bowlofeggs/bodhi-pre-release/ . If i don't hear about issues, I'd like to deploy to production this week.
16:35:49 <dgilmore> does anyone have anything?
16:35:55 <dgilmore> bowlofeggs: cool
16:36:18 <dgilmore> #info < bowlofeggs> Announcement: There's a bodhi-2.4.0 beta deployed at https://bodhi.stg.fedoraproject.org and you can try the CLI from https://copr.fedorainfracloud.org/coprs/bowlofeggs/bodhi-pre-release/
16:36:35 <bowlofeggs> release notes: https://github.com/fedora-infra/bodhi/blob/develop/docs/release_notes.rst
16:37:55 <adamchy> Hello, I have one more thing
16:38:03 <adamchy> I hope this is the right place to start
16:38:11 <adamchy> I would like to help develop fedora from releng perspective. Where should I start, with who should I speak?
16:41:08 <nirik> adamchy: well, this meeting and the folks in #fedora-releng. ;) Welcome by the way...
16:44:23 <dgilmore> adamchy: jump in #fedora-releng talk to folks, look at the easyfix issues in pagure
16:44:29 <dgilmore> adamchy: but welcome
16:44:39 <dgilmore> adamchy: what are your skills?
16:44:57 <adamchy> I'm C++ developer
16:45:10 <dgilmore> adamchy: okay, how is your python?
16:45:16 <adamchy> I know python also
16:45:21 <dgilmore> adamchy: all of our tools are python and bash
16:45:38 <adamchy> That's great
16:46:10 <dgilmore> adamchy: :) cool
16:46:24 <dgilmore> we ahng out and do most of our work from #fedora-releng
16:46:50 <dgilmore> there is docs https://docs.pagure.org/releng/
16:46:53 <adamchy> Ahh, ok
16:47:32 <dgilmore> https://docs.pagure.org/releng/contributing.html
16:47:47 <dgilmore> https://docs.pagure.org/releng/sop_adding_new_release_engineer.html
16:47:57 <dgilmore> probably some good places to get started
16:48:28 <dgilmore> though getting all the permissions takes some time
16:48:39 <dgilmore> if nothing else I will end the meeting
16:48:43 <adamchy> ok
16:48:53 <adamchy> I'll read the docs
16:48:53 <dgilmore> #endmeeting