14:38:54 <dgilmore> #startmeeting RELENG (2016-07-25) 14:38:54 <dgilmore> #meetingname releng 14:38:54 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu 14:38:54 <zodbot> Meeting started Mon Jul 25 14:38:54 2016 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:38:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:38:54 <zodbot> The meeting name has been set to 'releng_(2016-07-25)' 14:38:54 <zodbot> The meeting name has been set to 'releng' 14:38:54 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll 14:38:57 <dgilmore> #topic init process 14:39:15 * pbrobinson o/ 14:39:19 <nirik> morning 14:40:57 <maxamillion> .hello maxamillion 14:40:58 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com> 14:41:16 <bowlofeggs> .hello bowlofeggs 14:41:18 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <bowlofeggs@electronsweatshop.com> 14:42:16 <dgilmore> #topic #6420 Hypothetical maximum speed mirroring 14:42:24 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6420 14:42:33 <dgilmore> not sure there is really much to cover here 14:42:45 <dgilmore> we have ssecondary integration to go 14:43:08 <dgilmore> and figure out how to ensure we send fedmsg updates for all pushes 14:43:21 <nirik> yeah. 14:43:39 <nirik> perhaps we can finish that up at a flock workshop... just needs doing 14:46:19 <dgilmore> #info follow up in person at flock over getting https://pagure.io/quick-fedora-mirror support 100% done 14:46:37 <dgilmore> #topic Secondary Architectures updates 14:46:37 <dgilmore> #topic Secondary Architectures update - ppc 14:46:55 <dgilmore> pbrobinson: how is ppc this week? 14:47:05 <pbrobinson> few issues come up recently with some glibc changes, mostly I think we're getting on top of them 14:47:29 <pbrobinson> some build VMs upgraded to F-24, hoping to get the rest done this week 14:48:33 <pbrobinson> an issue with the compose report I need to look into, composes themselves look to be running OK 14:48:56 <pbrobinson> that's about it really 14:49:20 <dgilmore> pbrobinson: is the issue due to the symlink being updated differently? 14:49:42 <pbrobinson> dgilmore: no idea, when would that have changed? 14:49:44 <dgilmore> I had fixed a bug in pungi not dealing with releases only having 24 or Rawhide etc 14:49:57 <dgilmore> there was some code expecting everything to be 24.,0 etc 14:50:11 <dgilmore> pbrobinson: depends when you updated pungi to whats in rawhide 14:50:34 <dgilmore> I do not think that pungi in f24 ever got fixed 14:51:23 <pbrobinson> sounds like it may be related 14:51:37 <dgilmore> http://ppcpkgs.fedoraproject.org/compose/rawhide/ 14:52:00 <dgilmore> there is a latest-Fedora- and latest-Fedora-Rawhide there 14:52:21 <dgilmore> so likely 14:52:49 <pbrobinson> dgilmore: so given I never heard about this change what needs to be done to fix it? 14:52:54 <dgilmore> pbrobinson: the version of pungi in rawhide also has a lot of simplification of the config in it 14:53:40 <dgilmore> https://pagure.io/pungi-fedora/c/6ef9598cd461d68656e0083c97fdf242d225b1b3?branch=master 14:56:03 <dgilmore> anything else? 14:56:08 <dgilmore> #topic Secondary Architectures update - s390 14:56:14 <dgilmore> sharkcz: hows s390? 14:56:48 <dgilmore> I guess he is not here? 14:56:58 <dgilmore> #topic Secondary Architectures update - arm 14:57:04 <dgilmore> hows aarch64 pbrobinson 14:57:05 <dgilmore> ? 14:57:18 <pbrobinson> mostly OK, mostly up to date 14:57:41 <dgilmore> cool 14:57:48 <pbrobinson> we have a fixed lorax that is getting the compose further but the one I kicked off this morning failed in other ways 14:58:10 <pbrobinson> so I have to try and find time to debug that tomorrow along with all the other stuff that will be happening this week 14:58:13 <dgilmore> I saw the email from the guy that built a tianocore image for armv7 14:58:23 <pbrobinson> all the builders are now on F-24 14:58:28 <dgilmore> cool 14:58:39 <dgilmore> pbrobinson: want me to have a look at the logs? 14:59:14 <pbrobinson> yea, but it's useless on it's own without all the other supporting stuff like grub2/anaconda support etc and it's a small corner case compared to other devices so I don't see it's of much use 14:59:28 <dgilmore> pbrobinson: that was my thoughts also 14:59:52 <dgilmore> and without hardware that runs it I see much less value in doing the work to make it work 14:59:57 <pbrobinson> dgilmore: would love you to if you've got time, it's complaining of non bootable stuff so I suspect it's still a lorax problem, that's as far as I got 15:00:16 <dgilmore> pbrobinson: will make some time to go over it 15:02:17 <dgilmore> anything else arm wise to cover? 15:02:36 * nirik notes we should look at primary images too... a bunch seem to have fallen out today I think... would be nice to get them all back working... 15:02:40 <pbrobinson> and vendor confirmed what I suspected that moonshot has same firmware issue to mustang, awaiting a fix for both 15:02:58 <dgilmore> awesome 15:03:04 <dgilmore> hopefully the fix comes quickly 15:03:26 <dgilmore> nirik: yeah. lorax seems to be particulary fragile right now 15:03:49 <pbrobinson> they seem to have done a bunch of hack and slash without a lot of testing 15:03:53 <nirik> did the workaround for dnf barfing on unavailable in comps ever go out? anyhow, can look out of meeting 15:04:02 <dgilmore> #info vendor confimred moonshot has same firmware bug as mustang 15:04:13 <dgilmore> nirik: yeah 15:04:30 <nirik> ok. 15:04:47 <dgilmore> nirik: but I suspect that dnf made packages listed in comps strict again 15:05:04 <dgilmore> though I think anaconda explictly sets strict to false 15:05:21 <dgilmore> but we need to get the whole support of arches in comps sorted 15:05:26 <pbrobinson> did the fix for the random failing lives ever land in lorax (I seem to remember hearing the issue was discovered, but maybe I dreamt that) 15:05:40 <dgilmore> pbrobinson: it was an issue in dracut 15:05:45 <dgilmore> and I am not 100% sure 15:06:17 <nirik> I never heard that it got discovered. 15:06:26 <pbrobinson> they made the package support strict in lorax explicitly, it was the cause of issues in aarch64 due to lack of kexec support 15:06:30 <dgilmore> nirik: yeah adamw foun dit 15:06:31 <nirik> workstation today failed because libreoffice is busted. 15:06:32 <dgilmore> found it 15:06:38 <dgilmore> and it was in dracut 15:06:58 <nirik> the weird fsck/umount thing? 15:07:04 <nirik> or are we talking about something else? 15:08:04 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1282981 15:08:19 <dgilmore> nirik: /dev/shm being left open 15:08:19 <nirik> oh yeah, that... 15:08:30 <dgilmore> nirik: which is the fsck/umount 15:08:30 <nirik> thats not the same bug 15:08:52 <dgilmore> dracut when making the initramfs was opening /dev/shm and it was not getting freed up 15:09:05 <dgilmore> resulting in it being unable to be unmounted 15:09:08 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1315541 15:09:57 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1347436 may be related also 15:10:16 <dgilmore> it was due to fcoe being included in the initramfs 15:10:50 <nirik> huh. fun times. 15:10:55 <nirik> anyhow.. 15:11:51 <dgilmore> #info looks like the dracut bug causing lives to randomly fail is not yet fixed 15:12:11 <dgilmore> pbrobinson: to my quick glance it is not fixed 15:12:26 <nirik> looking today all the lives that failed failed for deps. 15:12:32 <dgilmore> we have been having lots of weird issues causing compose failures 15:12:33 <nirik> (which doesn't prove anything really) 15:13:07 <dgilmore> nirik: yeah the fcoe bits may have changed or something causing a different code path to be followed 15:13:25 <dgilmore> I guess we need to hassle harold about it 15:13:46 <dgilmore> #topic Open Floor 15:13:51 <nirik> he says it needs to be fixed elsewhere... but sure. 15:13:59 <nirik> oh nice: "ERR anaconda: You have specified that the package 'lv2-fil-plugins' should be installed. This package does not exist. Would you like to ignore this package and continue with installation?" 15:14:07 <nirik> (from todays failed jam-kde) 15:14:14 <dgilmore> reminder I will be on PTO from tomorrow without computer access for the rest of the week 15:14:27 <dgilmore> nirik: thats fun 15:14:41 <dgilmore> nirik: who is he? 15:14:49 <nirik> harold. 15:15:06 <nirik> f25 branching is tomorrow. 15:15:28 <nirik> and I have scheduled 2 outages this week... later today buildsystem and qa, wed for everything else. 15:15:38 <dgilmore> f25 branching is tomorrow 15:15:40 <nirik> pbrobinson: any issues with updating/rebooting arm/ppc/s390 hubs, etc? 15:16:00 <pbrobinson> nirik: nope, I did ppc late last week 15:16:14 <pbrobinson> so it probably doesn't need to be done 15:16:16 <nirik> cool. I would expect things to be smooth... 15:16:30 <nirik> will check if there's anything new since then, but probibly not 15:17:01 <pbrobinson> nirik: I was doing the builders that are on the same host so I did the underlying hypervisor, and the ppc koji/db 15:17:09 <dgilmore> I think mboddu has all of the steps for mass branching okay, he needs to work with pingu on the pkgdb side 15:17:16 <nirik> pbrobinson: cool. Thanks. 15:17:41 <nirik> oh, I have almost all primary builders on f24 now. 15:17:52 <pbrobinson> nirik: I need to rebuild ppc8-0[23] due to them not having a resilient /boot (I forgot to update the kickstart when I'd verified the firmware issue had been fixed) 15:17:54 <nirik> There's one buildhw that needs a new drive first... (Hopefully today) 15:18:15 <pbrobinson> but had issues with getting on the ipmi SoL console, not got back to them 15:18:16 <nirik> and bkernel02 needs doing after I am sure bkernel01 is ok on building/signing 15:18:22 <nirik> ok. 15:18:26 <dgilmore> nirik: okay 15:19:03 <nirik> note that alpha freeze is right after flock 15:19:06 <nirik> 2016-08-09 15:19:41 <dgilmore> nirik: yep 15:20:02 <dgilmore> pbrobinson: how late are you stying up? 15:20:17 <pbrobinson> dgilmore: why? 15:20:38 <dgilmore> We could setup the f24 branched configs in pungi-fedora if you're around for awhile 15:20:43 <dgilmore> so they are ready to go 15:20:56 <nirik> f25 you mean? 15:20:57 <dgilmore> then the cronjob stuff needs to be sorted tomorrow 15:21:01 <dgilmore> yeah f25 15:21:06 <dgilmore> too many releases 15:21:30 <pbrobinson> f25 branched configs... 15:21:36 <dgilmore> for pungi 15:21:53 <dgilmore> or I could send in a PR for them 15:22:00 <dgilmore> that you can review tomorrow 15:22:05 <pbrobinson> yea, I think I'm mostly good with that, was going to do them tomorrow morning my time over coffee 15:22:18 <dgilmore> okay 15:22:34 <dgilmore> the branched one should come from rawhide 15:22:56 <dgilmore> as there was quite a bit of simplification of the config that happened in pungi 15:23:03 <pbrobinson> I mean it's mostly just branching off rawhide and updating both master and f25 with appropriate bits 15:23:14 <dgilmore> yeah 15:23:42 <pbrobinson> so I don't forsee any issues there, I'm mostly up to date with the pungi bits 15:23:56 <dgilmore> pbrobinson: found out with fedora-release and fedora-repos on friday that you can not send in a pr that adds a new branch 15:23:58 <pbrobinson> maybe just atomic that I'd have queries about 15:24:01 * nirik can enable the cron job after branching is done 15:24:06 <dgilmore> the branch has to be made on the master repo 15:24:11 <nirik> unless someone else is 15:24:26 <dgilmore> nirik: either you or pbrobinson can 15:24:31 <pbrobinson> nirik: cron job in ansible is easy too ;-) 15:24:45 <nirik> yeah, it's easy to do... 15:25:26 <dgilmore> does anyone have any questions on branching? 15:25:31 <pbrobinson> dgilmore: is there anything special needed for Atomic, or do we not enable that straight away and leave it to maxamillion for the 2 week stuff? 15:25:49 <dgilmore> pbrobinson: it is enabled from day 1 15:26:02 <dgilmore> pbrobinson: should only need a tweak in the config 15:26:17 <dgilmore> pbrobinson: it will need a new branch and config in the atomic config repo 15:26:25 <pbrobinson> do we have to create empty atomic repos or will pungi deal with that stuff too 15:26:32 <pbrobinson> ah, yep, cool 15:26:35 <pbrobinson> can deal with that 15:26:38 <dgilmore> pungi is supposed to deal with it 15:26:52 <dgilmore> pbrobinson: walters maintains the configs 15:27:02 <dgilmore> I will bug him to get the branch done 15:27:10 <dgilmore> I do not have access to do so afaik 15:27:31 <maxamillion> +1 15:27:32 <pbrobinson> OK 15:28:38 <pbrobinson> easy enough to chase him if it's not there when everything else is ready to go 15:28:41 <dgilmore> pbrobinson: https://pagure.io/fedora-atomic 15:28:46 <dgilmore> only walters has access 15:29:30 <pbrobinson> that's a bit of a SPOF that should likely be fixed :) 15:29:49 <dgilmore> yes 15:30:09 <dgilmore> it is his baby 15:30:57 <dgilmore> anyone have anything else on branching? 15:31:31 <pbrobinson> don't think so 15:31:35 <dgilmore> #info discussion on the steps needed for branching 15:31:46 <pbrobinson> what time will the core bit be kicked off? 15:31:57 <dgilmore> paul frields complained the other week there was not enough use of #info etc 15:32:06 <dgilmore> mboddu: ^ 15:32:43 <mboddu> I am thinking of starting the process probably at 9 15:33:07 <dgilmore> 14:00 UTC 15:33:11 <mboddu> yep 15:33:30 <dgilmore> hopefully the pkgdb side is much faster than last time 15:33:42 <nirik> hopefully so 15:33:44 <dgilmore> f24 branching pkgdb was about 8 or 9 hours 15:33:48 <pbrobinson> mboddu: cool, just let me know of delays/progress etc on #fedora-releng 15:33:56 <dgilmore> but pingu thinks he fixed that 15:34:08 <mboddu> pbrobinson: will do, thanks 15:34:09 * nirik has to run and grab more coffee. 15:34:30 <dgilmore> #info mass branching to start around 14:00 UTC 15:34:58 <dgilmore> I am going to cancel next weeks meeting now 15:35:13 <dgilmore> as at the least I and likely others will be on the way to flock 15:36:10 <pbrobinson> yep 15:38:41 <dgilmore> I have nothing else 15:39:05 <dgilmore> I will close up in 2 minutes unless tehre is something someone wants covered 15:42:47 <dgilmore> #endmeeting