15:31:54 <dgilmore> #startmeeting RELENG (2017-01-30)
15:31:54 <zodbot> Meeting started Mon Jan 30 15:31:54 2017 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:31:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:31:54 <zodbot> The meeting name has been set to 'releng_(2017-01-30)'
15:31:54 <dgilmore> #meetingname releng
15:31:54 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
15:31:54 <zodbot> The meeting name has been set to 'releng'
15:31:54 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
15:31:57 <dgilmore> #topic init process
15:32:12 <mboddu> .hello mohanboddu
15:32:13 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
15:32:22 <dgilmore> hey
15:32:33 <nirik> .hello kevin
15:32:34 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
15:33:42 * masta is here
15:34:06 * pbrobinson is here, a little late
15:35:33 <dgilmore> lets get moving
15:35:49 <dgilmore> https://pagure.io/releng/issues?status=Open&tags=meeting
15:38:02 <dgilmore> #topic #6603 Enable rhel-kvm-rpms on EPEL7 aarch64 buildroots
15:38:12 <dgilmore> https://pagure.io/releng/issue/6603
15:39:28 <pbrobinson> I will do that this week, I need to review ppc64le too as part of it to ensure they're all the same
15:39:36 <dgilmore> pbrobinson: so the virt bits are not in teh base right
15:39:38 <pbrobinson> should close it out tomorrow or wed
15:40:03 <dgilmore> so we will need to add a empty repo for all the other arches
15:40:04 <pbrobinson> dgilmore: correct, only because they ship a newer qemu as 1.5.x didn't have LE/aarch64 and it was too hard to back port
15:40:13 <pbrobinson> correct and that's what I was going to do
15:40:14 <dgilmore> okay
15:40:27 <dgilmore> cool. just wanted to go over it
15:40:42 <dgilmore> #action pbrobinson to impleement this week
15:41:11 <dgilmore> #info aacrh64 virt is in a different channel, will make empty repos for other epel arches
15:41:30 <dgilmore> #info #6601 Migrate the rel-eng tickets from (old) trac
15:41:41 <dgilmore> https://pagure.io/releng/issue/6601
15:42:02 <nirik> not sure where pingou is. Can talk with him once he's done with travel
15:42:10 <dgilmore> nirik: how is dealing with the bugs going?
15:42:14 <dgilmore> okay
15:42:21 <dgilmore> he is in Prague today
15:42:43 <dgilmore> #info nirik to chat with pingou on status
15:42:44 <nirik> ok.
15:43:01 <dgilmore> #info #6600 Mock and sigined rawhides
15:43:02 <nirik> I'm happy to help, but he was going to work on it after some import patches...
15:43:12 <dgilmore> https://pagure.io/releng/issue/6600
15:43:45 <dgilmore> nirik: thanks
15:43:56 <dgilmore> so this issue is kinda messy
15:44:21 <dgilmore> there will nearly always be timing issues at branching
15:45:02 <masta> maybe create the gpg well ahead of time?
15:45:16 <dgilmore> we can, and it may help some
15:45:37 <dgilmore> we also likely have changes coming in the next few releases
15:45:47 <dgilmore> we can also have a long lived rawhide key
15:46:00 <dgilmore> but that would mean resigning everything at branching
15:48:33 <masta> That is a lot of signing. My concern is that signing load would impact the bodhi updates work flow of previous releases...
15:48:36 <nirik> well, we are going to need to resign everything at branch anyhow
15:49:13 <dgilmore> right
15:49:39 <dgilmore> #undo
15:49:39 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x41528990>
15:49:55 <dgilmore> #topic #6600 Mock and sigined rawhides
15:49:57 <pbrobinson> can we temp add the f26 key into the f27 repo files for a while post branching given they're all signed with that key?
15:50:17 <pbrobinson> not sure if you can do 2 keys on one repo
15:50:23 <pbrobinson> just thinking out loud
15:51:53 <nirik> dunno off hand either
15:52:31 <pbrobinson> just thought that might be an option while signing caught up
15:52:33 <dgilmore> pbrobinson: it might help some yeah
15:52:42 <dgilmore> at least with the transition
15:52:55 <pbrobinson> yep, and then flip it a few weeks later
15:53:05 <dgilmore> we will need to adjust the pungi configs also
15:53:38 <dgilmore> but whenever we flip there will be issues
15:53:52 <dgilmore> and mock does not use our copy of the gpgkeys
15:54:05 <dgilmore> it uses the copy that is in the ditribution-keys package
15:54:06 <nirik> so we have the f27 key now right? can't we sign in advance?
15:54:28 <nirik> like resign after mass rebuild (with both f26 and f27)
15:54:41 <dgilmore> nirik: we do not have a f27 key
15:54:46 <dgilmore> but we could make one
15:55:01 <masta> dgilmore, yep... the gpg would have to be created very early. mock & pungy, and everything else be updated very early too.
15:55:02 <nirik> I think we should and sign with both after mass rebuild...
15:55:10 <nirik> then the transition should be smooth
15:55:38 <dgilmore> should be yeah
15:55:50 <dgilmore> #info we should make the f27 key now
15:56:36 <nirik> wonder if we can get autosign to do both...
15:56:40 <dgilmore> #info we will need to update schedules
15:57:03 <dgilmore> #action dgilmore to ask puiterwijk about signing packages with two keys
15:57:30 <dgilmore> I do wonder if we should not go to legal and see about different options for signing keys
15:58:03 <puiterwijk> dgilmore: that's quite simple. Do we want that, then I'll just add that code to robosignatory
15:58:26 <dgilmore> puiterwijk: it may be useful
15:58:42 <dgilmore> lets think about it some more and review again next week
15:58:55 <dgilmore> see if some briliance hits us
15:59:15 <pbrobinson> signing with two keys isn't the problem, it's the dnf/pungi support for 2 keys
15:59:37 <dgilmore> pbrobinson: acording to muschy it supports two keys
15:59:39 <nirik> well, we don't need to worry about that if we have them all signed by two
15:59:53 <puiterwijk> dgilmore: I'll add the code, so we can just configure it
16:00:01 <nirik> because branched pungi will gather all the signed by one and rawhide pungi will gather all the signed by the other.
16:00:18 <dgilmore> but if we get them all signed and the handoff right
16:00:39 <dgilmore> there will be some benefits
16:00:54 <dgilmore> but users with mock will always be kinda clunky
16:01:01 <nirik> yeah
16:01:17 <nirik> it always has been in the past. ;)
16:01:49 <dgilmore> yeah
16:02:06 <dgilmore> changing at rawhide was simpler
16:02:14 <dgilmore> signing complicates it
16:02:17 <dgilmore> anyway
16:02:36 <dgilmore> #topic #6587 PDC ownership in Fedora
16:02:44 <dgilmore> https://pagure.io/releng/issue/6587
16:03:14 <dgilmore> mboddu: you wanted to take this
16:03:22 <mboddu> dgilmore: yes
16:03:38 <dgilmore> #info mboddu to own fedora PDC
16:04:06 <mboddu> dgilmore: I need to talk to you more about it later
16:04:34 <dgilmore> mboddu: okay, any simple questions
16:04:37 <dgilmore> ?
16:04:55 <nirik> we should try and get more things using PDC...
16:05:23 <dgilmore> nirik: we need a lot more in it
16:05:41 <dgilmore> we need to get a lot more data into pdc
16:05:51 <mboddu> dgilmore: nothing right now, but need to understand what we want from it or what we want it to do for us
16:06:08 <dgilmore> whats in the livecd's, cloud images, docker base image, anaconda runtime etc
16:06:15 <dgilmore> mboddu: okay
16:06:59 <dgilmore> #topic #6577 Fedora 26 mass rebuild
16:07:08 <nirik> coming up soon. ;)
16:07:14 <dgilmore> https://pagure.io/releng/issues?status=Open&tags=meeting
16:07:15 <mboddu> dgilmore: I think we will work on it?
16:08:16 <dgilmore> gahh
16:08:25 <dgilmore> https://pagure.io/releng/issue/6577
16:08:51 <dgilmore> so i think gcc7 landed
16:08:58 <dgilmore> not sure its 100% ready
16:09:15 <nirik> yep. gcc7 is landed
16:09:22 <nirik> that dnf update is applied
16:09:29 <nirik> pkg-config is landed
16:09:41 <pbrobinson> anything needed from Alt Arch stuff?
16:09:42 <nirik> boost is still going... but hopefully they are done today?
16:09:52 <dgilmore> I know that jwb really wanted to try get s390 merged first
16:10:00 <dgilmore> we need boost to be done
16:10:02 <nirik> yeah, I was hoping for that too.
16:10:04 <pbrobinson> all the builders are now f25 and clean so any updates there should apply cleanly across
16:10:50 <dgilmore> nirik: any idea whats in the road other than just importing rawhide?
16:10:57 <dgilmore> is the infra side ready?
16:11:21 <nirik> no.
16:11:29 <nirik> RHIT hasn't finalized the firewall setup yet...
16:11:33 <pbrobinson> s390 really needs to be done in ansible so they're even
16:11:34 <dgilmore> okay
16:11:47 <pbrobinson> nirik: ah, so the firewall change is actually going ahead?
16:11:49 <nirik> stickster and I bugged them last week... I can ping today and see where we are.
16:11:59 <nirik> pbrobinson: so far as I know...
16:12:00 <pbrobinson> \o/
16:12:03 <dgilmore> nirik: so RHIT I assumme will not be ready this week
16:12:11 <nirik> dunno. It's unclear. ;(
16:12:22 <dgilmore> boo
16:12:41 <pbrobinson> based on the alt arch vlan.... next week means in 4 months time
16:12:45 <dgilmore> I think we can bump starting a few days in order to bring s390 in
16:12:57 <dgilmore> but probably not more than that
16:13:51 <dgilmore> of course if boost takes longer then we will have to wait
16:14:20 <dgilmore> #info gcc7 landed but there has not been an update saying it is ready for us to go
16:14:33 <dgilmore> #info boost side tag needs to be merged first
16:14:36 <pbrobinson> and obv need confirmation that gcc7 is abi/api stable and frozen
16:14:47 <dgilmore> #info pkgconf has been handled
16:14:53 <dgilmore> right
16:15:05 <dgilmore> I had asked jakub to give us teh green light
16:15:14 <nirik> were there any other changes pending for the mass rebuild
16:15:34 <dgilmore> I was just going to look
16:16:20 <pbrobinson> maybe reach out to python team to make sure they're good
16:16:31 <nirik> parallel installable debuginfo?
16:18:16 <dgilmore> https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
16:18:21 <dgilmore> it says not
16:18:27 <dgilmore> but it is confusingly worded
16:18:37 <dgilmore> This feature doesn't need a mass rebuild. But before being able to install parallel installable debuginfo packages old debuginfo packages need to be upgraded to a new parallel installable version. The dnf debuginfo-install plugin should help with that. It would be good to have a way to detect old vs new debuginfo packages so dnf can provide clear warnings/feedback. An open question is whether or not dnf upgrade should handle debuginfo packages
16:19:13 <dgilmore> it sounds like it really does need a mass rebuild
16:19:50 <dgilmore> but things would work but not be parallel installable if the package is not rebuilt
16:19:58 <masta> yeah, it does seem so
16:20:52 <dgilmore> https://fedoraproject.org/wiki/Changes/StaticLibraryDebuginfo
16:20:59 <dgilmore> that needs a full mass rebuild
16:21:29 <dgilmore> #info need to make sure that rpm is able to handle static library debuginfo
16:22:09 <dgilmore> https://fedoraproject.org/wiki/Changes/Fedora26CFlags
16:22:26 <dgilmore> #info need to ensure that the CFLAGS have been updated
16:23:09 <dgilmore> https://fedoraproject.org/wiki/Changes/aarch64-48bitVA
16:23:19 <dgilmore> possibly needs a mass rebuild
16:23:42 <dgilmore> pbrobinson: does the 48bit VA change need full or partial rebuild?
16:24:44 <dgilmore> I think that is all the changes for mass rebuild
16:25:30 <dgilmore> #info will check with all owners for status updates and see how things come along
16:27:01 <dgilmore> #info in order to help make sure we know all changes needing our assistance the change template has been changed to require a link to a releng issue where a discussion has been started on the change
16:27:14 <dgilmore> just to ensure we always know what things need our help
16:27:32 <dgilmore> and in theory a change can not go forward without the issue
16:28:08 <dgilmore> appparently some changes requiring releng assistance got misscommunicated
16:28:33 <dgilmore> people talked to others who they thought were comminicating with us or were us
16:29:08 <dgilmore> and so they thought that they had done the right thing and I got blindsided by new deliverables for f26 on Friday
16:29:20 <dgilmore> #topic open floor
16:29:35 <dgilmore> does anyone have anything?
16:30:07 <nirik> I was going to bring up possibly setting updates to go on a cron, but we can discuss that down the road... doesn't need to be today and we are low on time
16:30:48 <dgilmore> okay
16:31:11 <dgilmore> I wnted to go over the rawhide failures for the last nearly week
16:31:36 <nirik> I looked at it, but was too busy to dig much
16:31:52 <nirik> well, I fixed some I guess and more came in
16:32:07 <dgilmore> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170130.n.0/logs/x86_64/buildinstall-Cloud.x86_64.log
16:32:31 <dgilmore> so that failed with 2017-01-30 06:55:53,688: /usr/bin/perl, needed by /var/tmp/lorax.nam76v22/installtree/usr/bin/rxe_cfg, does not exist
16:32:34 <dgilmore> /usr/bin/perl, needed by /var/tmp/lorax.nam76v22/installtree/usr/bin/rxe_cfg, does not exist
16:33:08 <masta> ugh...
16:33:10 <nirik> dgilmore: I wanted to know the workflow for updating tools on composers/builders? should we always build for fedora ? and koji is in a state I am unsure of right now. :)
16:33:15 <dgilmore> extrating the package names from the last working compose to the current had a package list change of
16:33:23 <dgilmore> about 15 packages
16:34:00 <nirik> idea: could pungi make those diffs for each compose? might help us track down failures faster.
16:34:05 <dgilmore> perl dropped out entirely
16:34:48 <masta> that sounds good, perl dropping out... unless it breaks things (and it seems it does).
16:35:17 <dgilmore> and after some digging into what provides /usr/bin/rxe_cfg in the koji buildroot I discovered the libibverbs version in the repo is very different to the version in koji
16:35:45 <dgilmore> I ended up finding out that a package got added rdma-core https://bugzilla.redhat.com/show_bug.cgi?id=1404043
16:36:04 <dgilmore> it is providing much never versions of some tools
16:36:13 <dgilmore> and it seems has completely broken packaging
16:36:53 <dgilmore> I filed https://github.com/rhinstaller/lorax/issues/180 to try help in future debugging
16:37:21 <dgilmore> I have untagged the rdma-core build and updated the bug
16:37:47 <dgilmore> #info rdma-core was added and had majoring packaging bugs and cause the rawhide breakages
16:37:58 <dgilmore> we need to get onto and fix breakages like it quicker
16:38:15 <dgilmore> at the least do a root cause analyssis and get people fixing bugs
16:38:24 <dgilmore> nirik: okay onto you
16:38:44 <dgilmore> what do you mean by should we always build for fedora?
16:39:12 <nirik> so. several things came up... first: I reinstalled rawhide-composer as f25
16:39:39 <nirik> rawhide failed to compose because koji on that box didn't understand --can-fail=whatever passed to livecd tasks
16:39:39 <dgilmore> okay
16:40:05 <nirik> so I found tha patch that added that to koji, added it and built it in f25-infra updated rawhide-composer and it worked
16:40:06 <dgilmore> we have a timining issue there
16:40:14 <dgilmore> and need to actually test that functionality
16:40:19 <nirik> but then aarch64 builders picked up that build and it broke things there.
16:40:45 <nirik> also, there was a pungi patch needed because rawhide broke after that
16:41:03 <dgilmore> we need to do all testing in stag e
16:41:05 <dgilmore> stage
16:41:43 <nirik> how can we test rawhide compose in stg?
16:42:11 <dgilmore> nirik: something that really confused me, the infra builds have nothing to differenciate from a fedora build
16:42:35 <dgilmore> as far as tooling goes testing f25 would be fine
16:42:47 <dgilmore> we need to make sure the tooling changes are tested
16:42:48 <nirik> in any case, perhaps this could be a SOP?
16:42:57 <nirik> where you need to test what you need to build where?
16:42:59 <dgilmore> if rawhide breaks because of changes we have no gating yet
16:43:21 <dgilmore> we possibly need infra tags for testing and prod
16:43:27 <nirik> I know you dislike hotfixes, so we should say that too...
16:43:30 <dgilmore> so we can build, put in stg and test
16:43:35 <dgilmore> then promote to prod
16:43:36 <nirik> yeah, possibly
16:44:33 <dgilmore> nirik: sadly at the moment the main thing is being aware of all teh code changes going on. and the status of them
16:44:37 <dgilmore> we have to do better
16:44:43 <dgilmore> and get better testing in place
16:44:50 <dgilmore> and make it as hands off as possible
16:44:53 <nirik> anyhow, I know how to make things work, I just don't want to change them in a way no one can track or likes. ;)
16:44:59 <nirik> yeah.
16:45:09 <nirik> more frequent koji releases should help
16:45:24 <dgilmore> I have asked mikem to do a 1.11.1
16:45:30 <dgilmore> with the changes to date
16:45:39 <dgilmore> so hopefully we can actually get that done
16:46:48 <dgilmore> we need to get more frequent koji relases and probably make sure that we do not push pungi or other changes before a koji change lands
16:47:15 <dgilmore> I think we really just needed som ebuilds of the code to allow us to test the feature in stage
16:47:36 <nirik> and stg could always be better
16:47:59 <dgilmore> its so we can say that workstation and kde livecd, base close image etc can fail on 32 bit x86 but have to complete on x86_64
16:48:33 <dgilmore> #action dgilmore to work on a sop for updating tools
16:49:04 <dgilmore> I have one quick announcement
16:49:44 <dgilmore> after talking with adamw on the weekend, I announced in my talk yesterday that we will be pushing to and plan to not do alpha releases after Fedora 26
16:50:23 <nirik> alrighty.
16:51:03 <adamw> yaaay
16:51:51 <dgilmore> will be filing a change this week
16:52:37 <dgilmore> and if nothing else we will close up in 30
16:52:41 <dgilmore> 29
16:52:43 <dgilmore> 28
16:52:49 <dgilmore> ....
16:53:04 <dgilmore> 4
16:53:07 <dgilmore> 3
16:53:08 <dgilmore> 2
16:53:10 <dgilmore> 1
16:53:15 <dgilmore> #endmeeting