14:38:54 <dgilmore> #startmeeting RELENG (2016-07-25)
14:38:54 <dgilmore> #meetingname releng
14:38:54 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
14:38:54 <zodbot> Meeting started Mon Jul 25 14:38:54 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:38:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:38:54 <zodbot> The meeting name has been set to 'releng_(2016-07-25)'
14:38:54 <zodbot> The meeting name has been set to 'releng'
14:38:54 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:38:57 <dgilmore> #topic init process
14:39:15 * pbrobinson o/
14:39:19 <nirik> morning
14:40:57 <maxamillion> .hello maxamillion
14:40:58 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:41:16 <bowlofeggs> .hello bowlofeggs
14:41:18 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <bowlofeggs@electronsweatshop.com>
14:42:16 <dgilmore> #topic #6420 Hypothetical maximum speed mirroring
14:42:24 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6420
14:42:33 <dgilmore> not sure there is really much to cover here
14:42:45 <dgilmore> we have ssecondary integration to go
14:43:08 <dgilmore> and figure out how to ensure we send fedmsg updates for all pushes
14:43:21 <nirik> yeah.
14:43:39 <nirik> perhaps we can finish that up at a flock workshop... just needs doing
14:46:19 <dgilmore> #info follow up in person at flock over getting https://pagure.io/quick-fedora-mirror support 100% done
14:46:37 <dgilmore> #topic Secondary Architectures updates
14:46:37 <dgilmore> #topic Secondary Architectures update - ppc
14:46:55 <dgilmore> pbrobinson: how is ppc this week?
14:47:05 <pbrobinson> few issues come up recently with some glibc changes, mostly I think we're getting on top of them
14:47:29 <pbrobinson> some build VMs upgraded to F-24, hoping to get the rest done this week
14:48:33 <pbrobinson> an issue with the compose report I need to look into, composes themselves look to be running OK
14:48:56 <pbrobinson> that's about it really
14:49:20 <dgilmore> pbrobinson: is the issue due to the symlink being updated differently?
14:49:42 <pbrobinson> dgilmore: no idea, when would that have changed?
14:49:44 <dgilmore> I had fixed a bug in pungi not dealing with releases only having 24 or Rawhide etc
14:49:57 <dgilmore> there was some code expecting everything to be 24.,0 etc
14:50:11 <dgilmore> pbrobinson: depends when you updated pungi to whats in rawhide
14:50:34 <dgilmore> I do not think that pungi in f24 ever got fixed
14:51:23 <pbrobinson> sounds like it may be related
14:51:37 <dgilmore> http://ppcpkgs.fedoraproject.org/compose/rawhide/
14:52:00 <dgilmore> there is a latest-Fedora- and latest-Fedora-Rawhide there
14:52:21 <dgilmore> so likely
14:52:49 <pbrobinson> dgilmore: so given I never heard about this change what needs to be done to fix it?
14:52:54 <dgilmore> pbrobinson: the version of pungi in rawhide also has a lot of simplification of the config in it
14:53:40 <dgilmore> https://pagure.io/pungi-fedora/c/6ef9598cd461d68656e0083c97fdf242d225b1b3?branch=master
14:56:03 <dgilmore> anything else?
14:56:08 <dgilmore> #topic Secondary Architectures update - s390
14:56:14 <dgilmore> sharkcz: hows s390?
14:56:48 <dgilmore> I guess he is not here?
14:56:58 <dgilmore> #topic Secondary Architectures update - arm
14:57:04 <dgilmore> hows aarch64 pbrobinson
14:57:05 <dgilmore> ?
14:57:18 <pbrobinson> mostly OK, mostly up to date
14:57:41 <dgilmore> cool
14:57:48 <pbrobinson> we have a fixed lorax that is getting the compose further but the one I kicked off this morning failed in other ways
14:58:10 <pbrobinson> so I have to try and find time to debug that tomorrow along with all the other stuff that will be happening this week
14:58:13 <dgilmore> I saw the email from the guy that built a tianocore image for armv7
14:58:23 <pbrobinson> all the builders are now on F-24
14:58:28 <dgilmore> cool
14:58:39 <dgilmore> pbrobinson: want me to have a look at the logs?
14:59:14 <pbrobinson> yea, but it's useless on it's own without all the other supporting stuff like grub2/anaconda support etc and it's a small corner case compared to other devices so I don't see it's of much use
14:59:28 <dgilmore> pbrobinson: that was my thoughts also
14:59:52 <dgilmore> and without hardware that runs it I see much less value in doing the work to make it work
14:59:57 <pbrobinson> dgilmore: would love you to if you've got time, it's complaining of non bootable stuff so I suspect it's still a lorax problem, that's as far as I got
15:00:16 <dgilmore> pbrobinson: will make some time to go over it
15:02:17 <dgilmore> anything else arm wise to cover?
15:02:36 * nirik notes we should look at primary images too... a bunch seem to have fallen out today I think... would be nice to get them all back working...
15:02:40 <pbrobinson> and vendor confirmed what I suspected that moonshot has same firmware issue to mustang, awaiting a fix for both
15:02:58 <dgilmore> awesome
15:03:04 <dgilmore> hopefully the fix comes quickly
15:03:26 <dgilmore> nirik: yeah. lorax seems to be particulary fragile right now
15:03:49 <pbrobinson> they seem to have done a bunch of hack and slash without a lot of testing
15:03:53 <nirik> did the workaround for dnf barfing on unavailable in comps ever go out? anyhow, can look out of meeting
15:04:02 <dgilmore> #info vendor confimred moonshot has same firmware bug as mustang
15:04:13 <dgilmore> nirik: yeah
15:04:30 <nirik> ok.
15:04:47 <dgilmore> nirik: but I suspect that dnf made packages listed in comps strict again
15:05:04 <dgilmore> though I think anaconda explictly sets strict to false
15:05:21 <dgilmore> but we need to get the whole support of arches in comps sorted
15:05:26 <pbrobinson> did the fix for the random failing lives ever land in lorax (I seem to remember hearing the issue was discovered, but maybe I dreamt that)
15:05:40 <dgilmore> pbrobinson: it was an issue in dracut
15:05:45 <dgilmore> and I am not 100% sure
15:06:17 <nirik> I never heard that it got discovered.
15:06:26 <pbrobinson> they made the package support strict in lorax explicitly, it was the cause of issues in aarch64 due to lack of kexec support
15:06:30 <dgilmore> nirik: yeah adamw foun dit
15:06:31 <nirik> workstation today failed because libreoffice is busted.
15:06:32 <dgilmore> found it
15:06:38 <dgilmore> and it was in dracut
15:06:58 <nirik> the weird fsck/umount thing?
15:07:04 <nirik> or are we talking about something else?
15:08:04 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1282981
15:08:19 <dgilmore> nirik: /dev/shm being left open
15:08:19 <nirik> oh yeah, that...
15:08:30 <dgilmore> nirik: which is the fsck/umount
15:08:30 <nirik> thats not the same bug
15:08:52 <dgilmore> dracut when making the initramfs was opening /dev/shm and it was not getting freed up
15:09:05 <dgilmore> resulting in it being unable to be unmounted
15:09:08 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1315541
15:09:57 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1347436 may be related also
15:10:16 <dgilmore> it was due to fcoe being included in the initramfs
15:10:50 <nirik> huh. fun times.
15:10:55 <nirik> anyhow..
15:11:51 <dgilmore> #info looks like the dracut bug causing lives to randomly fail is not yet fixed
15:12:11 <dgilmore> pbrobinson: to my quick glance it is not fixed
15:12:26 <nirik> looking today all the lives that failed failed for deps.
15:12:32 <dgilmore> we have been having lots of weird issues causing compose failures
15:12:33 <nirik> (which doesn't prove anything really)
15:13:07 <dgilmore> nirik: yeah the fcoe bits may have changed or something causing a different code path to be followed
15:13:25 <dgilmore> I guess we need to hassle harold about it
15:13:46 <dgilmore> #topic Open Floor
15:13:51 <nirik> he says it needs to be fixed elsewhere... but sure.
15:13:59 <nirik> oh nice: "ERR anaconda: You have specified that the package 'lv2-fil-plugins' should be installed.  This package does not exist.  Would you like to ignore this package and continue with installation?"
15:14:07 <nirik> (from todays failed jam-kde)
15:14:14 <dgilmore> reminder I will be on PTO from tomorrow without computer access for the rest of the week
15:14:27 <dgilmore> nirik: thats fun
15:14:41 <dgilmore> nirik: who is he?
15:14:49 <nirik> harold.
15:15:06 <nirik> f25 branching is tomorrow.
15:15:28 <nirik> and I have scheduled 2 outages this week... later today buildsystem and qa, wed for everything else.
15:15:38 <dgilmore> f25 branching is tomorrow
15:15:40 <nirik> pbrobinson: any issues with updating/rebooting arm/ppc/s390 hubs, etc?
15:16:00 <pbrobinson> nirik: nope, I did ppc late last week
15:16:14 <pbrobinson> so it probably doesn't need to be done
15:16:16 <nirik> cool. I would expect things to be smooth...
15:16:30 <nirik> will check if there's anything new since then, but probibly not
15:17:01 <pbrobinson> nirik: I was doing the builders that are on the same host so I did the underlying hypervisor, and the ppc koji/db
15:17:09 <dgilmore> I think mboddu has all of the steps for mass branching okay, he needs to work with pingu on the pkgdb side
15:17:16 <nirik> pbrobinson: cool. Thanks.
15:17:41 <nirik> oh, I have almost all primary builders on f24 now.
15:17:52 <pbrobinson> nirik: I need to rebuild ppc8-0[23] due to them not having a resilient /boot (I forgot to update the kickstart when I'd verified the firmware issue had been fixed)
15:17:54 <nirik> There's one buildhw that needs a new drive first... (Hopefully today)
15:18:15 <pbrobinson> but had issues with getting on the ipmi SoL console, not got back to them
15:18:16 <nirik> and bkernel02 needs doing after I am sure bkernel01 is ok on building/signing
15:18:22 <nirik> ok.
15:18:26 <dgilmore> nirik: okay
15:19:03 <nirik> note that alpha freeze is right after flock
15:19:06 <nirik> 2016-08-09
15:19:41 <dgilmore> nirik: yep
15:20:02 <dgilmore> pbrobinson: how late are you stying up?
15:20:17 <pbrobinson> dgilmore: why?
15:20:38 <dgilmore> We could setup the f24 branched configs in pungi-fedora if you're around for awhile
15:20:43 <dgilmore> so they are ready to go
15:20:56 <nirik> f25 you mean?
15:20:57 <dgilmore> then the cronjob stuff needs to be sorted tomorrow
15:21:01 <dgilmore> yeah f25
15:21:06 <dgilmore> too many releases
15:21:30 <pbrobinson> f25 branched configs...
15:21:36 <dgilmore> for pungi
15:21:53 <dgilmore> or I could send in a PR for them
15:22:00 <dgilmore> that you can review tomorrow
15:22:05 <pbrobinson> yea, I think I'm mostly good with that, was going to do them tomorrow morning my time over coffee
15:22:18 <dgilmore> okay
15:22:34 <dgilmore> the branched one should come from rawhide
15:22:56 <dgilmore> as there was quite a bit of simplification of the config that happened in pungi
15:23:03 <pbrobinson> I mean it's mostly just branching off rawhide and updating both master and f25 with appropriate bits
15:23:14 <dgilmore> yeah
15:23:42 <pbrobinson> so I don't forsee any issues there, I'm mostly up to date with the pungi bits
15:23:56 <dgilmore> pbrobinson: found out with fedora-release and fedora-repos on friday that you can not send in a pr that adds a new branch
15:23:58 <pbrobinson> maybe just atomic that I'd have queries about
15:24:01 * nirik can enable the cron job after branching is done
15:24:06 <dgilmore> the branch has to be made on the master repo
15:24:11 <nirik> unless someone else is
15:24:26 <dgilmore> nirik: either you or pbrobinson can
15:24:31 <pbrobinson> nirik: cron job in ansible is easy too ;-)
15:24:45 <nirik> yeah, it's easy to do...
15:25:26 <dgilmore> does anyone have any questions on branching?
15:25:31 <pbrobinson> dgilmore: is there anything special needed for Atomic, or do we not enable that straight away and leave it to maxamillion for the 2 week stuff?
15:25:49 <dgilmore> pbrobinson: it is enabled from day 1
15:26:02 <dgilmore> pbrobinson: should only need a tweak in the config
15:26:17 <dgilmore> pbrobinson: it will need a new branch and config in the atomic config repo
15:26:25 <pbrobinson> do we have to create empty atomic repos or will pungi deal with that stuff too
15:26:32 <pbrobinson> ah, yep, cool
15:26:35 <pbrobinson> can deal with that
15:26:38 <dgilmore> pungi is supposed to deal with it
15:26:52 <dgilmore> pbrobinson: walters maintains the configs
15:27:02 <dgilmore> I will bug him to get the branch done
15:27:10 <dgilmore> I do not have access to do so afaik
15:27:31 <maxamillion> +1
15:27:32 <pbrobinson> OK
15:28:38 <pbrobinson> easy enough to chase him if it's not there when everything else is ready to go
15:28:41 <dgilmore> pbrobinson: https://pagure.io/fedora-atomic
15:28:46 <dgilmore> only walters has access
15:29:30 <pbrobinson> that's a bit of a SPOF that should likely be fixed :)
15:29:49 <dgilmore> yes
15:30:09 <dgilmore> it is his baby
15:30:57 <dgilmore> anyone have anything else on branching?
15:31:31 <pbrobinson> don't think so
15:31:35 <dgilmore> #info discussion on the steps needed for branching
15:31:46 <pbrobinson> what time will the core bit be kicked off?
15:31:57 <dgilmore> paul frields complained the other week there was not enough use of #info etc
15:32:06 <dgilmore> mboddu: ^
15:32:43 <mboddu> I am thinking of starting the process probably at 9
15:33:07 <dgilmore> 14:00 UTC
15:33:11 <mboddu> yep
15:33:30 <dgilmore> hopefully the pkgdb side is much faster than last time
15:33:42 <nirik> hopefully so
15:33:44 <dgilmore> f24 branching pkgdb was about 8 or 9 hours
15:33:48 <pbrobinson> mboddu: cool, just let me know of delays/progress etc on #fedora-releng
15:33:56 <dgilmore> but pingu thinks he fixed that
15:34:08 <mboddu> pbrobinson: will do, thanks
15:34:09 * nirik has to run and grab more coffee.
15:34:30 <dgilmore> #info mass branching to start around 14:00 UTC
15:34:58 <dgilmore> I am going to cancel next weeks meeting now
15:35:13 <dgilmore> as at the least I and likely others will be on the way to flock
15:36:10 <pbrobinson> yep
15:38:41 <dgilmore> I have nothing else
15:39:05 <dgilmore> I will close up in 2 minutes unless tehre is something someone wants covered
15:42:47 <dgilmore> #endmeeting