15:31:54 #startmeeting RELENG (2017-01-30) 15:31:54 Meeting started Mon Jan 30 15:31:54 2017 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:31:54 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:31:54 The meeting name has been set to 'releng_(2017-01-30)' 15:31:54 #meetingname releng 15:31:54 #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu 15:31:54 The meeting name has been set to 'releng' 15:31:54 Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll 15:31:57 #topic init process 15:32:12 .hello mohanboddu 15:32:13 mboddu: mohanboddu 'Mohan Boddu' 15:32:22 hey 15:32:33 .hello kevin 15:32:34 nirik: kevin 'Kevin Fenzi' 15:33:42 * masta is here 15:34:06 * pbrobinson is here, a little late 15:35:33 lets get moving 15:35:49 https://pagure.io/releng/issues?status=Open&tags=meeting 15:38:02 #topic #6603 Enable rhel-kvm-rpms on EPEL7 aarch64 buildroots 15:38:12 https://pagure.io/releng/issue/6603 15:39:28 I will do that this week, I need to review ppc64le too as part of it to ensure they're all the same 15:39:36 pbrobinson: so the virt bits are not in teh base right 15:39:38 should close it out tomorrow or wed 15:40:03 so we will need to add a empty repo for all the other arches 15:40:04 dgilmore: correct, only because they ship a newer qemu as 1.5.x didn't have LE/aarch64 and it was too hard to back port 15:40:13 correct and that's what I was going to do 15:40:14 okay 15:40:27 cool. just wanted to go over it 15:40:42 #action pbrobinson to impleement this week 15:41:11 #info aacrh64 virt is in a different channel, will make empty repos for other epel arches 15:41:30 #info #6601 Migrate the rel-eng tickets from (old) trac 15:41:41 https://pagure.io/releng/issue/6601 15:42:02 not sure where pingou is. Can talk with him once he's done with travel 15:42:10 nirik: how is dealing with the bugs going? 15:42:14 okay 15:42:21 he is in Prague today 15:42:43 #info nirik to chat with pingou on status 15:42:44 ok. 15:43:01 #info #6600 Mock and sigined rawhides 15:43:02 I'm happy to help, but he was going to work on it after some import patches... 15:43:12 https://pagure.io/releng/issue/6600 15:43:45 nirik: thanks 15:43:56 so this issue is kinda messy 15:44:21 there will nearly always be timing issues at branching 15:45:02 maybe create the gpg well ahead of time? 15:45:16 we can, and it may help some 15:45:37 we also likely have changes coming in the next few releases 15:45:47 we can also have a long lived rawhide key 15:46:00 but that would mean resigning everything at branching 15:48:33 That is a lot of signing. My concern is that signing load would impact the bodhi updates work flow of previous releases... 15:48:36 well, we are going to need to resign everything at branch anyhow 15:49:13 right 15:49:39 #undo 15:49:39 Removing item from minutes: 15:49:55 #topic #6600 Mock and sigined rawhides 15:49:57 can we temp add the f26 key into the f27 repo files for a while post branching given they're all signed with that key? 15:50:17 not sure if you can do 2 keys on one repo 15:50:23 just thinking out loud 15:51:53 dunno off hand either 15:52:31 just thought that might be an option while signing caught up 15:52:33 pbrobinson: it might help some yeah 15:52:42 at least with the transition 15:52:55 yep, and then flip it a few weeks later 15:53:05 we will need to adjust the pungi configs also 15:53:38 but whenever we flip there will be issues 15:53:52 and mock does not use our copy of the gpgkeys 15:54:05 it uses the copy that is in the ditribution-keys package 15:54:06 so we have the f27 key now right? can't we sign in advance? 15:54:28 like resign after mass rebuild (with both f26 and f27) 15:54:41 nirik: we do not have a f27 key 15:54:46 but we could make one 15:55:01 dgilmore, yep... the gpg would have to be created very early. mock & pungy, and everything else be updated very early too. 15:55:02 I think we should and sign with both after mass rebuild... 15:55:10 then the transition should be smooth 15:55:38 should be yeah 15:55:50 #info we should make the f27 key now 15:56:36 wonder if we can get autosign to do both... 15:56:40 #info we will need to update schedules 15:57:03 #action dgilmore to ask puiterwijk about signing packages with two keys 15:57:30 I do wonder if we should not go to legal and see about different options for signing keys 15:58:03 dgilmore: that's quite simple. Do we want that, then I'll just add that code to robosignatory 15:58:26 puiterwijk: it may be useful 15:58:42 lets think about it some more and review again next week 15:58:55 see if some briliance hits us 15:59:15 signing with two keys isn't the problem, it's the dnf/pungi support for 2 keys 15:59:37 pbrobinson: acording to muschy it supports two keys 15:59:39 well, we don't need to worry about that if we have them all signed by two 15:59:53 dgilmore: I'll add the code, so we can just configure it 16:00:01 because branched pungi will gather all the signed by one and rawhide pungi will gather all the signed by the other. 16:00:18 but if we get them all signed and the handoff right 16:00:39 there will be some benefits 16:00:54 but users with mock will always be kinda clunky 16:01:01 yeah 16:01:17 it always has been in the past. ;) 16:01:49 yeah 16:02:06 changing at rawhide was simpler 16:02:14 signing complicates it 16:02:17 anyway 16:02:36 #topic #6587 PDC ownership in Fedora 16:02:44 https://pagure.io/releng/issue/6587 16:03:14 mboddu: you wanted to take this 16:03:22 dgilmore: yes 16:03:38 #info mboddu to own fedora PDC 16:04:06 dgilmore: I need to talk to you more about it later 16:04:34 mboddu: okay, any simple questions 16:04:37 ? 16:04:55 we should try and get more things using PDC... 16:05:23 nirik: we need a lot more in it 16:05:41 we need to get a lot more data into pdc 16:05:51 dgilmore: nothing right now, but need to understand what we want from it or what we want it to do for us 16:06:08 whats in the livecd's, cloud images, docker base image, anaconda runtime etc 16:06:15 mboddu: okay 16:06:59 #topic #6577 Fedora 26 mass rebuild 16:07:08 coming up soon. ;) 16:07:14 https://pagure.io/releng/issues?status=Open&tags=meeting 16:07:15 dgilmore: I think we will work on it? 16:08:16 gahh 16:08:25 https://pagure.io/releng/issue/6577 16:08:51 so i think gcc7 landed 16:08:58 not sure its 100% ready 16:09:15 yep. gcc7 is landed 16:09:22 that dnf update is applied 16:09:29 pkg-config is landed 16:09:41 anything needed from Alt Arch stuff? 16:09:42 boost is still going... but hopefully they are done today? 16:09:52 I know that jwb really wanted to try get s390 merged first 16:10:00 we need boost to be done 16:10:02 yeah, I was hoping for that too. 16:10:04 all the builders are now f25 and clean so any updates there should apply cleanly across 16:10:50 nirik: any idea whats in the road other than just importing rawhide? 16:10:57 is the infra side ready? 16:11:21 no. 16:11:29 RHIT hasn't finalized the firewall setup yet... 16:11:33 s390 really needs to be done in ansible so they're even 16:11:34 okay 16:11:47 nirik: ah, so the firewall change is actually going ahead? 16:11:49 stickster and I bugged them last week... I can ping today and see where we are. 16:11:59 pbrobinson: so far as I know... 16:12:00 \o/ 16:12:03 nirik: so RHIT I assumme will not be ready this week 16:12:11 dunno. It's unclear. ;( 16:12:22 boo 16:12:41 based on the alt arch vlan.... next week means in 4 months time 16:12:45 I think we can bump starting a few days in order to bring s390 in 16:12:57 but probably not more than that 16:13:51 of course if boost takes longer then we will have to wait 16:14:20 #info gcc7 landed but there has not been an update saying it is ready for us to go 16:14:33 #info boost side tag needs to be merged first 16:14:36 and obv need confirmation that gcc7 is abi/api stable and frozen 16:14:47 #info pkgconf has been handled 16:14:53 right 16:15:05 I had asked jakub to give us teh green light 16:15:14 were there any other changes pending for the mass rebuild 16:15:34 I was just going to look 16:16:20 maybe reach out to python team to make sure they're good 16:16:31 parallel installable debuginfo? 16:18:16 https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo 16:18:21 it says not 16:18:27 but it is confusingly worded 16:18:37 This feature doesn't need a mass rebuild. But before being able to install parallel installable debuginfo packages old debuginfo packages need to be upgraded to a new parallel installable version. The dnf debuginfo-install plugin should help with that. It would be good to have a way to detect old vs new debuginfo packages so dnf can provide clear warnings/feedback. An open question is whether or not dnf upgrade should handle debuginfo packages 16:19:13 it sounds like it really does need a mass rebuild 16:19:50 but things would work but not be parallel installable if the package is not rebuilt 16:19:58 yeah, it does seem so 16:20:52 https://fedoraproject.org/wiki/Changes/StaticLibraryDebuginfo 16:20:59 that needs a full mass rebuild 16:21:29 #info need to make sure that rpm is able to handle static library debuginfo 16:22:09 https://fedoraproject.org/wiki/Changes/Fedora26CFlags 16:22:26 #info need to ensure that the CFLAGS have been updated 16:23:09 https://fedoraproject.org/wiki/Changes/aarch64-48bitVA 16:23:19 possibly needs a mass rebuild 16:23:42 pbrobinson: does the 48bit VA change need full or partial rebuild? 16:24:44 I think that is all the changes for mass rebuild 16:25:30 #info will check with all owners for status updates and see how things come along 16:27:01 #info in order to help make sure we know all changes needing our assistance the change template has been changed to require a link to a releng issue where a discussion has been started on the change 16:27:14 just to ensure we always know what things need our help 16:27:32 and in theory a change can not go forward without the issue 16:28:08 appparently some changes requiring releng assistance got misscommunicated 16:28:33 people talked to others who they thought were comminicating with us or were us 16:29:08 and so they thought that they had done the right thing and I got blindsided by new deliverables for f26 on Friday 16:29:20 #topic open floor 16:29:35 does anyone have anything? 16:30:07 I was going to bring up possibly setting updates to go on a cron, but we can discuss that down the road... doesn't need to be today and we are low on time 16:30:48 okay 16:31:11 I wnted to go over the rawhide failures for the last nearly week 16:31:36 I looked at it, but was too busy to dig much 16:31:52 well, I fixed some I guess and more came in 16:32:07 https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170130.n.0/logs/x86_64/buildinstall-Cloud.x86_64.log 16:32:31 so that failed with 2017-01-30 06:55:53,688: /usr/bin/perl, needed by /var/tmp/lorax.nam76v22/installtree/usr/bin/rxe_cfg, does not exist 16:32:34 /usr/bin/perl, needed by /var/tmp/lorax.nam76v22/installtree/usr/bin/rxe_cfg, does not exist 16:33:08 ugh... 16:33:10 dgilmore: I wanted to know the workflow for updating tools on composers/builders? should we always build for fedora ? and koji is in a state I am unsure of right now. :) 16:33:15 extrating the package names from the last working compose to the current had a package list change of 16:33:23 about 15 packages 16:34:00 idea: could pungi make those diffs for each compose? might help us track down failures faster. 16:34:05 perl dropped out entirely 16:34:48 that sounds good, perl dropping out... unless it breaks things (and it seems it does). 16:35:17 and after some digging into what provides /usr/bin/rxe_cfg in the koji buildroot I discovered the libibverbs version in the repo is very different to the version in koji 16:35:45 I ended up finding out that a package got added rdma-core https://bugzilla.redhat.com/show_bug.cgi?id=1404043 16:36:04 it is providing much never versions of some tools 16:36:13 and it seems has completely broken packaging 16:36:53 I filed https://github.com/rhinstaller/lorax/issues/180 to try help in future debugging 16:37:21 I have untagged the rdma-core build and updated the bug 16:37:47 #info rdma-core was added and had majoring packaging bugs and cause the rawhide breakages 16:37:58 we need to get onto and fix breakages like it quicker 16:38:15 at the least do a root cause analyssis and get people fixing bugs 16:38:24 nirik: okay onto you 16:38:44 what do you mean by should we always build for fedora? 16:39:12 so. several things came up... first: I reinstalled rawhide-composer as f25 16:39:39 rawhide failed to compose because koji on that box didn't understand --can-fail=whatever passed to livecd tasks 16:39:39 okay 16:40:05 so I found tha patch that added that to koji, added it and built it in f25-infra updated rawhide-composer and it worked 16:40:06 we have a timining issue there 16:40:14 and need to actually test that functionality 16:40:19 but then aarch64 builders picked up that build and it broke things there. 16:40:45 also, there was a pungi patch needed because rawhide broke after that 16:41:03 we need to do all testing in stag e 16:41:05 stage 16:41:43 how can we test rawhide compose in stg? 16:42:11 nirik: something that really confused me, the infra builds have nothing to differenciate from a fedora build 16:42:35 as far as tooling goes testing f25 would be fine 16:42:47 we need to make sure the tooling changes are tested 16:42:48 in any case, perhaps this could be a SOP? 16:42:57 where you need to test what you need to build where? 16:42:59 if rawhide breaks because of changes we have no gating yet 16:43:21 we possibly need infra tags for testing and prod 16:43:27 I know you dislike hotfixes, so we should say that too... 16:43:30 so we can build, put in stg and test 16:43:35 then promote to prod 16:43:36 yeah, possibly 16:44:33 nirik: sadly at the moment the main thing is being aware of all teh code changes going on. and the status of them 16:44:37 we have to do better 16:44:43 and get better testing in place 16:44:50 and make it as hands off as possible 16:44:53 anyhow, I know how to make things work, I just don't want to change them in a way no one can track or likes. ;) 16:44:59 yeah. 16:45:09 more frequent koji releases should help 16:45:24 I have asked mikem to do a 1.11.1 16:45:30 with the changes to date 16:45:39 so hopefully we can actually get that done 16:46:48 we need to get more frequent koji relases and probably make sure that we do not push pungi or other changes before a koji change lands 16:47:15 I think we really just needed som ebuilds of the code to allow us to test the feature in stage 16:47:36 and stg could always be better 16:47:59 its so we can say that workstation and kde livecd, base close image etc can fail on 32 bit x86 but have to complete on x86_64 16:48:33 #action dgilmore to work on a sop for updating tools 16:49:04 I have one quick announcement 16:49:44 after talking with adamw on the weekend, I announced in my talk yesterday that we will be pushing to and plan to not do alpha releases after Fedora 26 16:50:23 alrighty. 16:51:03 yaaay 16:51:51 will be filing a change this week 16:52:37 and if nothing else we will close up in 30 16:52:41 29 16:52:43 28 16:52:49 .... 16:53:04 4 16:53:07 3 16:53:08 2 16:53:10 1 16:53:15 #endmeeting