14:38:54 #startmeeting RELENG (2016-07-25) 14:38:54 #meetingname releng 14:38:54 #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu 14:38:54 Meeting started Mon Jul 25 14:38:54 2016 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:38:54 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:38:54 The meeting name has been set to 'releng_(2016-07-25)' 14:38:54 The meeting name has been set to 'releng' 14:38:54 Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll 14:38:57 #topic init process 14:39:15 * pbrobinson o/ 14:39:19 morning 14:40:57 .hello maxamillion 14:40:58 maxamillion: maxamillion 'Adam Miller' 14:41:16 .hello bowlofeggs 14:41:18 bowlofeggs: bowlofeggs 'Randy Barlow' 14:42:16 #topic #6420 Hypothetical maximum speed mirroring 14:42:24 https://fedorahosted.org/rel-eng/ticket/6420 14:42:33 not sure there is really much to cover here 14:42:45 we have ssecondary integration to go 14:43:08 and figure out how to ensure we send fedmsg updates for all pushes 14:43:21 yeah. 14:43:39 perhaps we can finish that up at a flock workshop... just needs doing 14:46:19 #info follow up in person at flock over getting https://pagure.io/quick-fedora-mirror support 100% done 14:46:37 #topic Secondary Architectures updates 14:46:37 #topic Secondary Architectures update - ppc 14:46:55 pbrobinson: how is ppc this week? 14:47:05 few issues come up recently with some glibc changes, mostly I think we're getting on top of them 14:47:29 some build VMs upgraded to F-24, hoping to get the rest done this week 14:48:33 an issue with the compose report I need to look into, composes themselves look to be running OK 14:48:56 that's about it really 14:49:20 pbrobinson: is the issue due to the symlink being updated differently? 14:49:42 dgilmore: no idea, when would that have changed? 14:49:44 I had fixed a bug in pungi not dealing with releases only having 24 or Rawhide etc 14:49:57 there was some code expecting everything to be 24.,0 etc 14:50:11 pbrobinson: depends when you updated pungi to whats in rawhide 14:50:34 I do not think that pungi in f24 ever got fixed 14:51:23 sounds like it may be related 14:51:37 http://ppcpkgs.fedoraproject.org/compose/rawhide/ 14:52:00 there is a latest-Fedora- and latest-Fedora-Rawhide there 14:52:21 so likely 14:52:49 dgilmore: so given I never heard about this change what needs to be done to fix it? 14:52:54 pbrobinson: the version of pungi in rawhide also has a lot of simplification of the config in it 14:53:40 https://pagure.io/pungi-fedora/c/6ef9598cd461d68656e0083c97fdf242d225b1b3?branch=master 14:56:03 anything else? 14:56:08 #topic Secondary Architectures update - s390 14:56:14 sharkcz: hows s390? 14:56:48 I guess he is not here? 14:56:58 #topic Secondary Architectures update - arm 14:57:04 hows aarch64 pbrobinson 14:57:05 ? 14:57:18 mostly OK, mostly up to date 14:57:41 cool 14:57:48 we have a fixed lorax that is getting the compose further but the one I kicked off this morning failed in other ways 14:58:10 so I have to try and find time to debug that tomorrow along with all the other stuff that will be happening this week 14:58:13 I saw the email from the guy that built a tianocore image for armv7 14:58:23 all the builders are now on F-24 14:58:28 cool 14:58:39 pbrobinson: want me to have a look at the logs? 14:59:14 yea, but it's useless on it's own without all the other supporting stuff like grub2/anaconda support etc and it's a small corner case compared to other devices so I don't see it's of much use 14:59:28 pbrobinson: that was my thoughts also 14:59:52 and without hardware that runs it I see much less value in doing the work to make it work 14:59:57 dgilmore: would love you to if you've got time, it's complaining of non bootable stuff so I suspect it's still a lorax problem, that's as far as I got 15:00:16 pbrobinson: will make some time to go over it 15:02:17 anything else arm wise to cover? 15:02:36 * nirik notes we should look at primary images too... a bunch seem to have fallen out today I think... would be nice to get them all back working... 15:02:40 and vendor confirmed what I suspected that moonshot has same firmware issue to mustang, awaiting a fix for both 15:02:58 awesome 15:03:04 hopefully the fix comes quickly 15:03:26 nirik: yeah. lorax seems to be particulary fragile right now 15:03:49 they seem to have done a bunch of hack and slash without a lot of testing 15:03:53 did the workaround for dnf barfing on unavailable in comps ever go out? anyhow, can look out of meeting 15:04:02 #info vendor confimred moonshot has same firmware bug as mustang 15:04:13 nirik: yeah 15:04:30 ok. 15:04:47 nirik: but I suspect that dnf made packages listed in comps strict again 15:05:04 though I think anaconda explictly sets strict to false 15:05:21 but we need to get the whole support of arches in comps sorted 15:05:26 did the fix for the random failing lives ever land in lorax (I seem to remember hearing the issue was discovered, but maybe I dreamt that) 15:05:40 pbrobinson: it was an issue in dracut 15:05:45 and I am not 100% sure 15:06:17 I never heard that it got discovered. 15:06:26 they made the package support strict in lorax explicitly, it was the cause of issues in aarch64 due to lack of kexec support 15:06:30 nirik: yeah adamw foun dit 15:06:31 workstation today failed because libreoffice is busted. 15:06:32 found it 15:06:38 and it was in dracut 15:06:58 the weird fsck/umount thing? 15:07:04 or are we talking about something else? 15:08:04 https://bugzilla.redhat.com/show_bug.cgi?id=1282981 15:08:19 nirik: /dev/shm being left open 15:08:19 oh yeah, that... 15:08:30 nirik: which is the fsck/umount 15:08:30 thats not the same bug 15:08:52 dracut when making the initramfs was opening /dev/shm and it was not getting freed up 15:09:05 resulting in it being unable to be unmounted 15:09:08 https://bugzilla.redhat.com/show_bug.cgi?id=1315541 15:09:57 https://bugzilla.redhat.com/show_bug.cgi?id=1347436 may be related also 15:10:16 it was due to fcoe being included in the initramfs 15:10:50 huh. fun times. 15:10:55 anyhow.. 15:11:51 #info looks like the dracut bug causing lives to randomly fail is not yet fixed 15:12:11 pbrobinson: to my quick glance it is not fixed 15:12:26 looking today all the lives that failed failed for deps. 15:12:32 we have been having lots of weird issues causing compose failures 15:12:33 (which doesn't prove anything really) 15:13:07 nirik: yeah the fcoe bits may have changed or something causing a different code path to be followed 15:13:25 I guess we need to hassle harold about it 15:13:46 #topic Open Floor 15:13:51 he says it needs to be fixed elsewhere... but sure. 15:13:59 oh nice: "ERR anaconda: You have specified that the package 'lv2-fil-plugins' should be installed. This package does not exist. Would you like to ignore this package and continue with installation?" 15:14:07 (from todays failed jam-kde) 15:14:14 reminder I will be on PTO from tomorrow without computer access for the rest of the week 15:14:27 nirik: thats fun 15:14:41 nirik: who is he? 15:14:49 harold. 15:15:06 f25 branching is tomorrow. 15:15:28 and I have scheduled 2 outages this week... later today buildsystem and qa, wed for everything else. 15:15:38 f25 branching is tomorrow 15:15:40 pbrobinson: any issues with updating/rebooting arm/ppc/s390 hubs, etc? 15:16:00 nirik: nope, I did ppc late last week 15:16:14 so it probably doesn't need to be done 15:16:16 cool. I would expect things to be smooth... 15:16:30 will check if there's anything new since then, but probibly not 15:17:01 nirik: I was doing the builders that are on the same host so I did the underlying hypervisor, and the ppc koji/db 15:17:09 I think mboddu has all of the steps for mass branching okay, he needs to work with pingu on the pkgdb side 15:17:16 pbrobinson: cool. Thanks. 15:17:41 oh, I have almost all primary builders on f24 now. 15:17:52 nirik: I need to rebuild ppc8-0[23] due to them not having a resilient /boot (I forgot to update the kickstart when I'd verified the firmware issue had been fixed) 15:17:54 There's one buildhw that needs a new drive first... (Hopefully today) 15:18:15 but had issues with getting on the ipmi SoL console, not got back to them 15:18:16 and bkernel02 needs doing after I am sure bkernel01 is ok on building/signing 15:18:22 ok. 15:18:26 nirik: okay 15:19:03 note that alpha freeze is right after flock 15:19:06 2016-08-09 15:19:41 nirik: yep 15:20:02 pbrobinson: how late are you stying up? 15:20:17 dgilmore: why? 15:20:38 We could setup the f24 branched configs in pungi-fedora if you're around for awhile 15:20:43 so they are ready to go 15:20:56 f25 you mean? 15:20:57 then the cronjob stuff needs to be sorted tomorrow 15:21:01 yeah f25 15:21:06 too many releases 15:21:30 f25 branched configs... 15:21:36 for pungi 15:21:53 or I could send in a PR for them 15:22:00 that you can review tomorrow 15:22:05 yea, I think I'm mostly good with that, was going to do them tomorrow morning my time over coffee 15:22:18 okay 15:22:34 the branched one should come from rawhide 15:22:56 as there was quite a bit of simplification of the config that happened in pungi 15:23:03 I mean it's mostly just branching off rawhide and updating both master and f25 with appropriate bits 15:23:14 yeah 15:23:42 so I don't forsee any issues there, I'm mostly up to date with the pungi bits 15:23:56 pbrobinson: found out with fedora-release and fedora-repos on friday that you can not send in a pr that adds a new branch 15:23:58 maybe just atomic that I'd have queries about 15:24:01 * nirik can enable the cron job after branching is done 15:24:06 the branch has to be made on the master repo 15:24:11 unless someone else is 15:24:26 nirik: either you or pbrobinson can 15:24:31 nirik: cron job in ansible is easy too ;-) 15:24:45 yeah, it's easy to do... 15:25:26 does anyone have any questions on branching? 15:25:31 dgilmore: is there anything special needed for Atomic, or do we not enable that straight away and leave it to maxamillion for the 2 week stuff? 15:25:49 pbrobinson: it is enabled from day 1 15:26:02 pbrobinson: should only need a tweak in the config 15:26:17 pbrobinson: it will need a new branch and config in the atomic config repo 15:26:25 do we have to create empty atomic repos or will pungi deal with that stuff too 15:26:32 ah, yep, cool 15:26:35 can deal with that 15:26:38 pungi is supposed to deal with it 15:26:52 pbrobinson: walters maintains the configs 15:27:02 I will bug him to get the branch done 15:27:10 I do not have access to do so afaik 15:27:31 +1 15:27:32 OK 15:28:38 easy enough to chase him if it's not there when everything else is ready to go 15:28:41 pbrobinson: https://pagure.io/fedora-atomic 15:28:46 only walters has access 15:29:30 that's a bit of a SPOF that should likely be fixed :) 15:29:49 yes 15:30:09 it is his baby 15:30:57 anyone have anything else on branching? 15:31:31 don't think so 15:31:35 #info discussion on the steps needed for branching 15:31:46 what time will the core bit be kicked off? 15:31:57 paul frields complained the other week there was not enough use of #info etc 15:32:06 mboddu: ^ 15:32:43 I am thinking of starting the process probably at 9 15:33:07 14:00 UTC 15:33:11 yep 15:33:30 hopefully the pkgdb side is much faster than last time 15:33:42 hopefully so 15:33:44 f24 branching pkgdb was about 8 or 9 hours 15:33:48 mboddu: cool, just let me know of delays/progress etc on #fedora-releng 15:33:56 but pingu thinks he fixed that 15:34:08 pbrobinson: will do, thanks 15:34:09 * nirik has to run and grab more coffee. 15:34:30 #info mass branching to start around 14:00 UTC 15:34:58 I am going to cancel next weeks meeting now 15:35:13 as at the least I and likely others will be on the way to flock 15:36:10 yep 15:38:41 I have nothing else 15:39:05 I will close up in 2 minutes unless tehre is something someone wants covered 15:42:47 #endmeeting