16:04:11 <mboddu> #startmeeting RELENG (2019-07-25)
16:04:11 <zodbot> Meeting started Wed Jul 24 16:04:11 2019 UTC.
16:04:11 <zodbot> This meeting is logged and archived in a public location.
16:04:11 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:04:11 <zodbot> The meeting name has been set to 'releng_(2019-07-25)'
16:04:11 <mboddu> #meetingname releng
16:04:11 <zodbot> The meeting name has been set to 'releng'
16:04:11 <mboddu> #chair nirik tyll sharkcz masta pbrobinson pingou puiterwijk mboddu dustymabe ksinny jednorozec
16:04:11 <zodbot> Current chairs: dustymabe jednorozec ksinny masta mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
16:04:12 <mboddu> #topic init process
16:04:26 <nirik> morning
16:04:37 <mboddu> morning nirik
16:04:38 <mboddu> How goes?
16:04:53 <sharkcz> hi guys
16:04:55 <jednorozec> hey
16:05:08 <nirik> todays better than yesterday was. ;)
16:05:09 <bcotton> .hello2
16:05:10 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
16:06:08 <mboddu> nirik: Good to hear that, I know you were swamped yesterday
16:06:17 <mboddu> Any good news on communishift?
16:06:32 * mboddu waves at sharkcz jednorozec and bcotton
16:06:55 <nirik> yeah, got it reinstalled. ;)
16:07:01 <mboddu> Oh while bcotton is here, I want to suggest a little change to schedule, basically how we word things
16:07:10 <nirik> I thought for sure it couldn't be dns... but then... it was in the end... dns.
16:07:30 <mboddu> Well, glad that its fixed :)
16:07:44 <mboddu> Lets start with open floor
16:08:25 <mboddu> Or, lets have the topic of mass rebuild individually
16:08:40 <mboddu> #topic #8555 Fedora 31 Mass Rebuild
16:08:46 <mboddu> #link https://pagure.io/releng/issue/8555
16:09:20 <mboddu> I started the mass rebuild few min back and so far its going good
16:09:23 <nirik> hopefully it will go smoothly.
16:09:33 <mboddu> Fingers crossed :D
16:09:43 <nirik> keep an eye  out for builders with issues...
16:09:51 <mboddu> Yup
16:10:35 <mboddu> #info Mass rebuild for rpms has started and the mass rebuild for modules will start after the current mass rebuild is done, so that the koji is not overloaded
16:10:35 * nirik would like to go over ppc64le and s390x changes, can do that in another section or whenever.
16:11:11 <mboddu> bcotton: ^^ Can you make the change in the next schedule preparation?
16:11:23 <mboddu> nirik: Yeah, we will have separate topic for it :D
16:11:59 <bcotton> mboddu: yes, let's follow up after the meeting with what specific changes you want. i'm planning on working on future schedules later today anyway
16:12:31 <mboddu> bcotton: Thats called great timing :P . Sure, we can go over it in the afternoon
16:13:04 <mboddu> #topic Faster composes
16:13:14 <mboddu> Rawhide composes are now taking <5 hours
16:13:16 <mboddu> Woot woot woot
16:13:40 <mboddu> nirik: You wanna add you comments on ppc64le and s390x?
16:13:47 <nirik> 4.5 yesterday, but near 5 today.
16:14:05 <nirik> I need to look at stuff and see whats now slowing us down.
16:14:08 <mboddu> Yup, I averaged it out to <5 :D
16:14:12 <nirik> sure.
16:14:16 <nirik> So, on s390x:
16:14:59 <nirik> they setup a kvm virthost for us, and it's running 10 s390x kvm buildvm guests
16:15:12 <bcotton> how long did they used to take?
16:15:13 <nirik> we still have the old 15 buildvm's that are Z/VM
16:15:39 <nirik> they suggested we keep the z/vm ones for now
16:15:42 <mboddu> bcotton: On a good day 8+ hrs
16:16:17 <mboddu> nirik: Any reason why they want us to keep the z/vm?
16:16:46 <mboddu> Esp after knowing kvm has helped us a lot in improving the compose/build time
16:17:07 <nirik> well, they said that z/vm is very sold and well known, kvm could have bugs or issues...
16:17:28 <nirik> but I can ask them in a while about moving that resource over to the kvm side.
16:17:40 <mboddu> Okay
16:18:09 <nirik> On ppc64le:
16:18:31 <nirik> our 2 power9 boxes were being very slow. I am still not sure if it's a bug or just the disks they have are slower causing it.
16:18:51 <nirik> basically any fdatasync's in the guest vm's was taking like 5 seconds to come back
16:19:06 <nirik> so, I moved all the guests to use libvirt's 'unsafe' cache mode.
16:19:33 <nirik> basically this is where the guest does a fdatasync and qermu says 'sure, it's on disk' (lie) and the host then syncs it out as it can.
16:24:30 <mboddu> #info 10 of s390x buildvm's are moved to KVM and 15 are still on Z/VM. Couple of them are added to compose channel and couple for image channel. This improved the compose/build times on s390x
16:24:48 <nirik> grrr.
16:24:58 <nirik> sorry, dog emergency had to step away
16:25:05 <mboddu> nirik: Okay
16:25:36 <nirik> anyhow, I'm not sure that mode will work long term for the builders, but it should at least only fail safely I think.
16:25:43 <nirik> ie, corrupt the guest
16:25:56 <nirik> or crash and cause a build to need resubmitted.
16:26:31 <nirik> anyhow, we will see how they do during the mass rebuild.
16:26:43 <mboddu> #info nirik moved all the ppc64le vm's to use libvirt's 'unsafe' cache mode, this improved the runroot tasks time from couple of hours to few min which inturn improved the compose time.
16:27:07 <mboddu> #ppc64le fix is temporary until we figure out if the issue is with storage or a bug.
16:27:16 <sharkcz> I guess it could cause issues in case of forced power-off or crashes of the VMs, not during regular runtime
16:27:27 <nirik> yeah.
16:27:32 <nirik> and reinstalling them is just a playbook.
16:27:40 <nirik> they have 0 persistent data we need.
16:27:56 <nirik> There was talk about doing this for all the builders in a recent devel list thread.
16:28:12 <nirik> but there was at least one person saying it wasn't very tested for long uptimes
16:28:16 <sharkcz> ack, and the data in flight are the buildroots, so the guests themselves should be quite safe
16:28:35 <sharkcz> but yes, time will show :-)
16:28:44 <nirik> so if it works fine here, we might consider it for other guests as well.
16:30:17 <mboddu> +1
16:30:32 <mboddu> So, moving on
16:30:42 <mboddu> #topic #8513 F32 system-wide change: x86-64 micro-architecture update
16:30:48 <mboddu> #link https://pagure.io/releng/issue/8513
16:31:14 <mboddu> nirik: I wanted to bring this up and see how it will affect infra side
16:31:33 <nirik> unknown, it's still being discussed. The orig proposal is pretty much no go I think.
16:32:45 <nirik> I think runtime detection would be best, then next best would be package level (rpm/dnf know and install the x86_64.avx2 package for you), and then only very last would be making another arch.
16:34:02 <sharkcz> nirik: +1
16:34:04 <mboddu> Okay, then I will leave as is and get back to it once its approved
16:34:13 <mboddu> s/once/if/
16:36:02 * nirik nods
16:37:54 <mboddu> #topic #7445 [RFE][PATCH] make $releasever return "rawhide" on Rawhide
16:38:00 <mboddu> #link https://pagure.io/releng/issue/7445
16:38:18 <mboddu> Some movement is happening on the ticket, its been a while since we talked about it
16:38:53 <nirik> yeah, would be nice to get this done. ;)
16:39:09 * mboddu checking how dnf is now identifying the releasever as rawhide
16:41:50 <mboddu> nirik: Cant find it, do you know?
16:43:35 <nirik> I don't recall off hand. I think there was a patch, but I don't know.
16:43:37 <nirik> would have to dig
16:45:47 <mboddu> nirik: Yeah, github search is not very helpful either - https://github.com/rpm-software-management/dnf/search?q=rawhide&unscoped_q=rawhide
16:46:34 <mboddu> Anyway, we need to look at it and make any changes if necessary
16:46:44 <mboddu> #topic Open Floor
16:46:52 <nirik> one small note:
16:46:52 <mboddu> Anybody has anything to share?
16:46:59 <mboddu> nirik: Go ahead
16:47:13 <nirik> I enabled srpm repo in f31-build... I should go close that ticket.
16:47:34 <nirik> There is still one bug with it... it includes the noarch packages also currently. There's a koji patch that I hope to land to fix that soon.
16:47:50 <mboddu> nirik: Okay, thanks for the update
16:48:02 <mboddu> Oh, I forgot about epel8
16:49:35 <mboddu> #info nirik enabled srpm repo in f31-build and there is one bug with it. He is expecting the koji patch to land to fix it soon.
16:49:41 <nirik> oh yeah.
16:49:49 <mboddu> #topic EPEL8
16:50:36 <mboddu> I have made the changes with koji, pdc, fedpkg and fedscm-admin and processed a couple hundred tickets for epel8 branches
16:51:03 <mboddu> There is still one issue - https://pagure.io/fedpkg/issue/345
16:51:18 <mboddu> I was supposed to talk to lsedlar today, but mass rebuild took my morning
16:52:40 <mboddu> #info EPEL8 is shaping up good and we are expecting a compose sometime next week
16:52:49 <mboddu> nirik: Anything you want to add?
16:53:12 <nirik> not really. What all do we need for composes?
16:53:17 <nirik> just a pungi config?
16:53:40 <mboddu> nirik: Yup and based on if we decide to sync them to mirrors yet or not, we need scripts to do so
16:55:04 <mboddu> Moving on to rawhide gating topic
16:55:09 <mboddu> #topic Rawhide Gating
16:55:46 <mboddu> #info We are expecting to deploy single build rawhide gating tomorrow.
16:55:54 <nirik> well, playground will just need it's sync in the cron job... but epel8 will use new-updates-push I would think.
16:56:37 <mboddu> Are we going with bodhi for epel8, haven't we decided to enable it later?
16:56:53 <nirik> yeah, not sure...
16:56:54 <mboddu> bodhi uses new-updates-push
16:57:16 <nirik> I guess we could just manually populate it first, then enable bodhi
16:57:30 <mboddu> Well, depending on what we decide we have to make things work, either scripts/pungi or bodhi
16:57:34 <nirik> sure.
16:57:44 <nirik> right now actually stuff goes to epel8 tag right?
16:57:50 <mboddu> Yes
16:58:13 <nirik> then I think we just decide when we have enough done and enable bodhi on it... it will automatically add all the epel8 tagged things in.
16:59:00 <smooge> sounds good.. I was going to bring that up in 2 minutes in #fedora-meeting
16:59:11 <mboddu> Sure, that works too, but the reason why we decided to not to use bodhi is to ship the content faster to users and no BRO's initially to make the epel8 builds easier
17:00:52 <nirik> yes...
17:01:22 <nirik> but that was for the initial bringup... if we want to keep doing that we will need I guess a rawhide/playground like compose script for it for now.
17:02:47 <mboddu> nirik: We should do playground nightly composes and if we think we have enough content in epel8, then we can just enable bodhi
17:03:16 <nirik> sure.
17:03:44 <mboddu> Okay, lets close the meeting
17:03:48 <mboddu> Thanks everyone for joining
17:03:52 <mboddu> #endmeeting