16:32:54 <dgilmore> #startmeeting RELENG (2016-03-07)
16:32:54 <zodbot> Meeting started Mon Mar  7 16:32:54 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:32:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:32:54 <zodbot> The meeting name has been set to 'releng_(2016-03-07)'
16:32:54 <dgilmore> #meetingname releng
16:32:54 <zodbot> The meeting name has been set to 'releng'
16:32:55 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
16:32:55 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
16:32:57 <dgilmore> #topic init process
16:33:03 <nirik> .hello kevin
16:33:04 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
16:33:20 <maxamillion> .hello maxamillion
16:33:21 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
16:33:25 * pbrobinson waves
16:34:49 <dgilmore> lets start
16:34:53 * nirik goes to grab quick coffee.
16:35:11 <dgilmore> #topic f24 status
16:35:15 <dgilmore> 775143
16:35:24 <dgilmore> uggh
16:36:10 <dgilmore> so
16:36:17 <dgilmore> f24.
16:36:26 <dgilmore> right now the nightlies are coming out
16:36:40 <dgilmore> the email goes to test@ but devel@ is dropping it
16:39:03 <nirik> yeah, I have been digging on that this morning
16:39:12 <nirik> found a traceback... we arent sure why yet
16:39:17 <dgilmore> there is no atomic yet
16:39:41 <dgilmore> cloud images are failing to install 32 bit
16:39:55 <dgilmore> I wonder if we do not just disable trying to build them
16:40:26 * nirik supports disabling any x86 32bit thing. ;)
16:41:05 <nirik> I am pretty sure they didn't want it in f23 even...
16:41:07 <dgilmore> I think we shouldnt disable it
16:41:26 <dgilmore> but we have no way to say try it and do not fail the build if it fails
16:42:04 <maxamillion> dgilmore: so will the two-week atomic release scheduled for tomorrow be affected by this? or is this f24 specific?
16:42:42 <dgilmore> maxamillion: its f24 and rawhide specific
16:42:51 <maxamillion> dgilmore: +1 thanks
16:43:51 <dgilmore> change freeze is tomorrow
16:44:02 <dgilmore> we need to be able to deliever a TC or RC
16:44:02 <maxamillion> fun
16:44:09 <dgilmore> not sure we can right now
16:44:26 <dgilmore> it needs to have all release blocking deliverbles
16:45:15 <maxamillion> dgilmore: is there an idea of what all needs to be done to make that happen or are you still finding issues as you go?
16:45:35 <maxamillion> dgilmore: is this something that others could help with in the near term or would adding new people to the process just slow things down?
16:45:42 <dgilmore> maxamillion: need to figure out whst exactly we need to do for a production compose
16:46:28 <dgilmore> it will need a script written like nightly.sh that does what we need for TC/RC
16:47:14 <dgilmore> we have to make some adjustments for secondary arches also
16:47:45 <maxamillion> rgr
16:47:57 <dgilmore> does anyone have any questions on it?
16:48:18 <maxamillion> is it anything that anyone else can help with or is it just a matter of y'all going through the motions to sort it out?
16:48:51 <dgilmore> people can help by lookinga t things that are failing
16:49:23 <dgilmore> someone could try to read the pungi docs and write a release compose script
16:49:23 <nirik> dgilmore: you think it's too late for me to move spins-kickstarts to pagure.io ?
16:49:32 <dgilmore> nirik: nope
16:49:50 * masta looks in
16:49:51 <nirik> ok, I will see about doing that later today... and updating the urls wherever I can find them
16:50:11 * nirik needs to pick a name... spin-kickstarts is not accurate.
16:50:36 <dgilmore> nirik: kojid.conf and a bunc of places in pungi-fedora
16:50:47 <dgilmore> nirik: also builder iptables rules will need adjusting
16:50:56 <pbrobinson> nirik: ALL_THE_COOL_SHIT ?
16:51:02 <dgilmore> nirik: fedora-kickstarts?
16:51:21 <maxamillion> dgilmore: +1 - I like fedora-kickstarts, we could have a README that explains further what all is in the repo
16:51:25 <dgilmore> the atomic lorax templates should live elsewhere
16:51:26 <nirik> sure.
16:51:49 <nirik> usually I don't like fedora in the name due to trademark, but it's just descriptive here I guess.
16:52:21 <dgilmore> not really different to fedora-release or fedora-repos
16:52:29 <nirik> true
16:53:03 <dgilmore> someone doing a remix and adding things not in fedora would have to rename it
16:53:52 <nirik> right, but they would have to rename those other things too...
16:54:16 <maxamillion> dgilmore: where should people be looking to evaluate the things that are failing?
16:54:17 <nirik> we should be better about syncing generic-release, generic-repos and generic-kickstarts
16:54:55 <dgilmore> maxamillion: depends what it is
16:55:01 <pbrobinson> dgilmore: what advantage do we get my moving the templates elsewhere, why not call it something other than kickstarts and have it all there
16:55:02 <dgilmore> and what its cause is
16:55:40 <pbrobinson> generic-templates ?
16:55:46 <maxamillion> dgilmore: right, but you suggested that's somewhere that people could help and I'm asking for guideance on how people can help ... and if the answer is "well, it depends" then I don't really know how people can help
16:56:10 <maxamillion> (I think I worded that weird, but hopefully you get what I'm saying)
16:56:12 <dgilmore> lorax split the lorax templates out
16:56:18 <dgilmore> I felt they should go there
16:56:44 <nirik> pbrobinson: the non live stuff is in pungi I think... like server dvd ks and such...
16:56:52 <dgilmore> bcl disagreed
16:57:05 <dgilmore> nirik: the install kickstarts are removed
16:57:31 <dgilmore> all that is there now is live, cloud and arm kickstarts
16:57:36 <nirik> yes. I'm saying the only things left in that repo are the live ks's and cloud and arm
16:57:47 <dgilmore> and the atomic installer lorax templates
16:58:33 <dgilmore> maxamillion: well if its broken deps, you file bugs or fix packages
16:58:46 <dgilmore> maxamillion: if its anaconda puking you file a bug
16:59:07 <maxamillion> dgilmore: right, I was asking where to logs are ... where are we supposed to look for these failures?
16:59:10 <nirik> well, I don't care what we call it. I thought fedora-kickstarts was ok, but if it has non ks stuff thats not entirely accurate.
16:59:10 <dgilmore> maxamillion: if its a problem in the compose configs you fix it and send a patch
16:59:30 <dgilmore> maxamillion: depends what it is. if its done in koji or not
16:59:53 <dgilmore> nirik: fedora-compose-configs
17:00:13 <dgilmore> or just compose-configs
17:00:26 <dgilmore> though it does not have all the compose configs
17:00:34 <dgilmore> we could move pungi-fedora into it
17:00:54 <pbrobinson> but I want a purple bike shed!
17:01:01 <nirik> fedora-kickstarts-and-templates-and-a-shed
17:01:09 <dgilmore> and-a-pony
17:01:30 <pbrobinson> just call it the fedora-shed and then we can shove any shit in there including a pony :-D
17:02:54 <nirik> I already made a fedora-kickstarts earlier... but I can abandon it or try and rename it if needed.
17:03:09 <dgilmore> maxamillion: https://kojipkgs.fedoraproject.org/compose/ all composes and logs land under there, I assume people know that
17:03:34 <maxamillion> dgilmore: I didn't know if something had changed with the new tooling or not
17:03:49 <dgilmore> maxamillion: its more open
17:04:05 <maxamillion> dgilmore: +
17:04:06 <maxamillion> +1
17:04:20 <dgilmore> so lets decide a name for the kickstarts etc out of the meeting
17:04:59 <nirik> sure.
17:05:08 <dgilmore> does anyone have anything in regards to f24 status they wish to discuss?
17:05:46 <pbrobinson> ppc and arm should be composing soonish, I have most of the infra in place for arm, should have ppc done tomorrow
17:05:56 <dgilmore> cool
17:06:32 <nirik> just under the wire for freeze. :)
17:06:55 <dgilmore> I have pulled in a patch to pungi that should let us more easily share the pungi configs
17:07:27 <pbrobinson> nirik: secondary arches have generally been slightly more cowboy esq unfortunately, and well you know something about waiting for a long time for a vlan ;)
17:07:33 <dgilmore> just need to adjust the configs to define the deliverables for the secondary arches
17:07:53 <nirik> pbrobinson: yeah, always something. ;)
17:08:06 <pbrobinson> nirik: yep!
17:09:47 <dgilmore> #topic Secondary Architectures updates
17:09:58 <dgilmore> lets do all the secondaries as one
17:10:03 <dgilmore> since sharkcz is not here
17:10:10 <dgilmore> pbrobinson: how is ppc and arm?
17:10:45 <pbrobinson> so arm mass rebuild is pretty much done, about 130 behind, so likely a few stragglers in there
17:11:13 <dgilmore> cool
17:11:20 <pbrobinson> been working over the weekend and today to get stuff ready for pungi4, so now have a builder on it's own channel, cleanly rebuild
17:11:29 <dgilmore> we should probably cover saturdays events here also
17:11:41 <dgilmore> awesome
17:11:44 <pbrobinson> ppc is a little further behind
17:12:03 <pbrobinson> but from mem a little over 2K left to go so it's not too far off
17:12:30 <dgilmore> not great but not terrible
17:12:37 <pbrobinson> I worked with nirik late last week to get stuff ready for the infra changes and I upgraded firmwares on Power8 boxes over the weekend
17:12:55 <dgilmore> will the p8 boxes be faster?
17:12:57 <pbrobinson> yea, for some reason this cycle the shadow mass rebuild has been like treacle
17:13:22 <pbrobinson> dgilmore: the firmware is so I can rebuild the hosts to ppc64le as that's what's supported in el7
17:13:39 <dgilmore> cool
17:13:39 <nirik> and I managed to finally find qemu-kvm-rhev for ppc64le....
17:13:42 <nirik> it was not easy. ;)
17:13:50 <dgilmore> nirik: fun
17:13:51 <pbrobinson> dgilmore: then I'll rebuild the hosts to that and move all the builders and infra as I do over to the new subnet
17:14:08 <dgilmore> I like how we sometimes make what Red Hat ships hard to get
17:14:11 <pbrobinson> as well as new ansible compose boxes and new clean hub
17:14:14 <nirik> hopefully ansible will make the builders rebuild trivial anyhow. ;)
17:14:26 <pbrobinson> all to latest standards and all in ansible
17:14:46 <dgilmore> only taken 5 years?  maybe more
17:14:50 <pbrobinson> nirik: yes, the builders are the easy part, and I tested that previously when I moved them to f23
17:15:05 <pbrobinson> dgilmore: hey! I've not been in this role THAT long!
17:15:31 <dgilmore> pbrobinson: :) I know, but I have been trying to standardise things for a long time
17:15:45 <dgilmore> its slowly getting there
17:15:53 <pbrobinson> dgilmore: yep, and it's taken longer than I would have liked but then even I have to sleep
17:16:08 <dgilmore> indeed we all do
17:16:16 <dgilmore> and all can only do so much
17:16:20 <pbrobinson> but we're almost there... so close I can taste it!
17:16:21 <nirik> we are making progress. ;)
17:16:33 <dgilmore> we are indeed
17:16:52 <pbrobinson> once I've finished ppc up, and we've got alpha out of the way I'll then start to look @ the moonshot
17:16:53 <dgilmore> anything else specific to cover?
17:17:13 <pbrobinson> nope, over all ppc/arm are looking pretty good, no major blockers
17:17:33 <dgilmore> pbrobinson: :) one of the slowest things in teh primary compose today is how long it takes to xz compress the arm disk images
17:17:44 <pbrobinson> as a side note we got a workaround for the armv7 devices not booting so even that will be pushed in time for freeze tomorrow :)
17:18:01 <dgilmore> nice
17:18:09 <pbrobinson> dgilmore: well that's a change for the books!
17:18:22 <dgilmore> so quickly, I still need to file a bug.
17:18:33 <dgilmore> I updated koji on the builders on friday
17:18:37 <pbrobinson> dgilmore: how long does the total pungi4 compose currently take?
17:18:43 <dgilmore> a result of the update was a rpm update
17:18:47 <dgilmore> pbrobinson: 4 hours
17:19:07 <dgilmore> pbrobinson: secondaries will be quicker
17:19:19 <pbrobinson> dgilmore: yes, we produce a LOT less :)
17:20:30 <dgilmore> a result of the rpm update was that the function koji uses to read and record what is in the buildroot failed to return any data
17:20:47 <dgilmore> and we were not recording in the koji db what was in the buildroot
17:21:08 <dgilmore> its data koji-shadow uses to mimic the buildroot
17:21:41 <dgilmore> we still have the rpms in the buildroot log and we know what koji repo was used
17:21:52 <dgilmore> so we have what is needed to reproduce the buildroot
17:21:54 <pbrobinson> so I pushed manually most of those I believe now over the weekend for f24, still need to look at other releases
17:22:47 <dgilmore> so it is all far from ideal
17:22:51 <dgilmore> but not end of world
17:23:13 <nirik> I have another quick thing to mention that happened sat night...
17:23:19 <dgilmore> nirik: sure
17:24:09 <nirik> kojipkgs went unresponsive... I rebooted it, but it's /boot was hosed. I tried various things to get it booting, but no luck, so I just rebuilt it from scratch. So, there's likely a short time when builds failed around then... but it should be back to working fine now.
17:24:30 <dgilmore> nirik: cool. good thing we can do that
17:24:49 <nirik> it was quiicker to just reinstall than fix. ;)
17:25:13 <nirik> I suppose I should file a xfs bug and see if they want the /boot
17:25:21 <pbrobinson> and sat night is probably about the most convenient time for an outage
17:25:30 <pbrobinson> nirik: we support /boot on xfs now?
17:25:46 <pbrobinson> nirik: el7 presumably?
17:25:58 <nirik> yeah, el7 and yes its supposed to be supported.
17:26:00 <dgilmore> pbrobinson: el7 and fedora server on x86_64 use it
17:26:12 <nirik> xfs_repair would oops
17:26:25 <nirik> it was just hosed.
17:26:26 <pbrobinson> dgilmore: I knew they did for root, didn't know they did now for /boot too
17:26:38 <dgilmore> 919851
17:26:47 <dgilmore> 660225
17:26:49 <dgilmore> 359460
17:27:00 <dgilmore> sorry bout that
17:27:17 <pbrobinson> dgilmore: the server install I did on the mustang has ext4 for /boot using the F23 defaults
17:27:54 <maxamillion> dgilmore: having a hell of a day with your yubikey aren't you?
17:28:05 <pbrobinson> LOL
17:29:16 <dgilmore> maxamillion: its new and i hate it already
17:29:23 <dgilmore> I went and pulled it out
17:29:44 <maxamillion> dgilmore: Nano?
17:29:44 <dgilmore> pbrobinson: I think its only on x86_64 where /boot is xfs
17:29:47 <dgilmore> maxamillion: yeah
17:30:04 <dgilmore> anything else
17:30:09 <dgilmore> if not I will close up
17:30:13 <pbrobinson> is that the one RCM sent out? Mine is still in the envelope!
17:30:17 <maxamillion> dgilmore: I love my Nano, but it definitely takes some getting used to
17:30:20 <dgilmore> qa are pinging me for blocker bug review
17:30:22 * pbrobinson is done!
17:30:33 <dgilmore> pbrobinson: yes it is
17:31:13 <dgilmore> lighting the fuse
17:31:23 <dgilmore> 30
17:31:40 <dgilmore> 15
17:31:51 <dgilmore> 5
17:31:58 <dgilmore> #endmeeting