16:04:11 #startmeeting RELENG (2019-07-25) 16:04:11 Meeting started Wed Jul 24 16:04:11 2019 UTC. 16:04:11 This meeting is logged and archived in a public location. 16:04:11 The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:04:11 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:04:11 The meeting name has been set to 'releng_(2019-07-25)' 16:04:11 #meetingname releng 16:04:11 The meeting name has been set to 'releng' 16:04:11 #chair nirik tyll sharkcz masta pbrobinson pingou puiterwijk mboddu dustymabe ksinny jednorozec 16:04:11 Current chairs: dustymabe jednorozec ksinny masta mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll 16:04:12 #topic init process 16:04:26 morning 16:04:37 morning nirik 16:04:38 How goes? 16:04:53 hi guys 16:04:55 hey 16:05:08 todays better than yesterday was. ;) 16:05:09 .hello2 16:05:10 bcotton: bcotton 'Ben Cotton' 16:06:08 nirik: Good to hear that, I know you were swamped yesterday 16:06:17 Any good news on communishift? 16:06:32 * mboddu waves at sharkcz jednorozec and bcotton 16:06:55 yeah, got it reinstalled. ;) 16:07:01 Oh while bcotton is here, I want to suggest a little change to schedule, basically how we word things 16:07:10 I thought for sure it couldn't be dns... but then... it was in the end... dns. 16:07:30 Well, glad that its fixed :) 16:07:44 Lets start with open floor 16:08:25 Or, lets have the topic of mass rebuild individually 16:08:40 #topic #8555 Fedora 31 Mass Rebuild 16:08:46 #link https://pagure.io/releng/issue/8555 16:09:20 I started the mass rebuild few min back and so far its going good 16:09:23 hopefully it will go smoothly. 16:09:33 Fingers crossed :D 16:09:43 keep an eye out for builders with issues... 16:09:51 Yup 16:10:35 #info Mass rebuild for rpms has started and the mass rebuild for modules will start after the current mass rebuild is done, so that the koji is not overloaded 16:10:35 * nirik would like to go over ppc64le and s390x changes, can do that in another section or whenever. 16:11:11 bcotton: ^^ Can you make the change in the next schedule preparation? 16:11:23 nirik: Yeah, we will have separate topic for it :D 16:11:59 mboddu: yes, let's follow up after the meeting with what specific changes you want. i'm planning on working on future schedules later today anyway 16:12:31 bcotton: Thats called great timing :P . Sure, we can go over it in the afternoon 16:13:04 #topic Faster composes 16:13:14 Rawhide composes are now taking <5 hours 16:13:16 Woot woot woot 16:13:40 nirik: You wanna add you comments on ppc64le and s390x? 16:13:47 4.5 yesterday, but near 5 today. 16:14:05 I need to look at stuff and see whats now slowing us down. 16:14:08 Yup, I averaged it out to <5 :D 16:14:12 sure. 16:14:16 So, on s390x: 16:14:59 they setup a kvm virthost for us, and it's running 10 s390x kvm buildvm guests 16:15:12 how long did they used to take? 16:15:13 we still have the old 15 buildvm's that are Z/VM 16:15:39 they suggested we keep the z/vm ones for now 16:15:42 bcotton: On a good day 8+ hrs 16:16:17 nirik: Any reason why they want us to keep the z/vm? 16:16:46 Esp after knowing kvm has helped us a lot in improving the compose/build time 16:17:07 well, they said that z/vm is very sold and well known, kvm could have bugs or issues... 16:17:28 but I can ask them in a while about moving that resource over to the kvm side. 16:17:40 Okay 16:18:09 On ppc64le: 16:18:31 our 2 power9 boxes were being very slow. I am still not sure if it's a bug or just the disks they have are slower causing it. 16:18:51 basically any fdatasync's in the guest vm's was taking like 5 seconds to come back 16:19:06 so, I moved all the guests to use libvirt's 'unsafe' cache mode. 16:19:33 basically this is where the guest does a fdatasync and qermu says 'sure, it's on disk' (lie) and the host then syncs it out as it can. 16:24:30 #info 10 of s390x buildvm's are moved to KVM and 15 are still on Z/VM. Couple of them are added to compose channel and couple for image channel. This improved the compose/build times on s390x 16:24:48 grrr. 16:24:58 sorry, dog emergency had to step away 16:25:05 nirik: Okay 16:25:36 anyhow, I'm not sure that mode will work long term for the builders, but it should at least only fail safely I think. 16:25:43 ie, corrupt the guest 16:25:56 or crash and cause a build to need resubmitted. 16:26:31 anyhow, we will see how they do during the mass rebuild. 16:26:43 #info nirik moved all the ppc64le vm's to use libvirt's 'unsafe' cache mode, this improved the runroot tasks time from couple of hours to few min which inturn improved the compose time. 16:27:07 #ppc64le fix is temporary until we figure out if the issue is with storage or a bug. 16:27:16 I guess it could cause issues in case of forced power-off or crashes of the VMs, not during regular runtime 16:27:27 yeah. 16:27:32 and reinstalling them is just a playbook. 16:27:40 they have 0 persistent data we need. 16:27:56 There was talk about doing this for all the builders in a recent devel list thread. 16:28:12 but there was at least one person saying it wasn't very tested for long uptimes 16:28:16 ack, and the data in flight are the buildroots, so the guests themselves should be quite safe 16:28:35 but yes, time will show :-) 16:28:44 so if it works fine here, we might consider it for other guests as well. 16:30:17 +1 16:30:32 So, moving on 16:30:42 #topic #8513 F32 system-wide change: x86-64 micro-architecture update 16:30:48 #link https://pagure.io/releng/issue/8513 16:31:14 nirik: I wanted to bring this up and see how it will affect infra side 16:31:33 unknown, it's still being discussed. The orig proposal is pretty much no go I think. 16:32:45 I think runtime detection would be best, then next best would be package level (rpm/dnf know and install the x86_64.avx2 package for you), and then only very last would be making another arch. 16:34:02 nirik: +1 16:34:04 Okay, then I will leave as is and get back to it once its approved 16:34:13 s/once/if/ 16:36:02 * nirik nods 16:37:54 #topic #7445 [RFE][PATCH] make $releasever return "rawhide" on Rawhide 16:38:00 #link https://pagure.io/releng/issue/7445 16:38:18 Some movement is happening on the ticket, its been a while since we talked about it 16:38:53 yeah, would be nice to get this done. ;) 16:39:09 * mboddu checking how dnf is now identifying the releasever as rawhide 16:41:50 nirik: Cant find it, do you know? 16:43:35 I don't recall off hand. I think there was a patch, but I don't know. 16:43:37 would have to dig 16:45:47 nirik: Yeah, github search is not very helpful either - https://github.com/rpm-software-management/dnf/search?q=rawhide&unscoped_q=rawhide 16:46:34 Anyway, we need to look at it and make any changes if necessary 16:46:44 #topic Open Floor 16:46:52 one small note: 16:46:52 Anybody has anything to share? 16:46:59 nirik: Go ahead 16:47:13 I enabled srpm repo in f31-build... I should go close that ticket. 16:47:34 There is still one bug with it... it includes the noarch packages also currently. There's a koji patch that I hope to land to fix that soon. 16:47:50 nirik: Okay, thanks for the update 16:48:02 Oh, I forgot about epel8 16:49:35 #info nirik enabled srpm repo in f31-build and there is one bug with it. He is expecting the koji patch to land to fix it soon. 16:49:41 oh yeah. 16:49:49 #topic EPEL8 16:50:36 I have made the changes with koji, pdc, fedpkg and fedscm-admin and processed a couple hundred tickets for epel8 branches 16:51:03 There is still one issue - https://pagure.io/fedpkg/issue/345 16:51:18 I was supposed to talk to lsedlar today, but mass rebuild took my morning 16:52:40 #info EPEL8 is shaping up good and we are expecting a compose sometime next week 16:52:49 nirik: Anything you want to add? 16:53:12 not really. What all do we need for composes? 16:53:17 just a pungi config? 16:53:40 nirik: Yup and based on if we decide to sync them to mirrors yet or not, we need scripts to do so 16:55:04 Moving on to rawhide gating topic 16:55:09 #topic Rawhide Gating 16:55:46 #info We are expecting to deploy single build rawhide gating tomorrow. 16:55:54 well, playground will just need it's sync in the cron job... but epel8 will use new-updates-push I would think. 16:56:37 Are we going with bodhi for epel8, haven't we decided to enable it later? 16:56:53 yeah, not sure... 16:56:54 bodhi uses new-updates-push 16:57:16 I guess we could just manually populate it first, then enable bodhi 16:57:30 Well, depending on what we decide we have to make things work, either scripts/pungi or bodhi 16:57:34 sure. 16:57:44 right now actually stuff goes to epel8 tag right? 16:57:50 Yes 16:58:13 then I think we just decide when we have enough done and enable bodhi on it... it will automatically add all the epel8 tagged things in. 16:59:00 sounds good.. I was going to bring that up in 2 minutes in #fedora-meeting 16:59:11 Sure, that works too, but the reason why we decided to not to use bodhi is to ship the content faster to users and no BRO's initially to make the epel8 builds easier 17:00:52 yes... 17:01:22 but that was for the initial bringup... if we want to keep doing that we will need I guess a rawhide/playground like compose script for it for now. 17:02:47 nirik: We should do playground nightly composes and if we think we have enough content in epel8, then we can just enable bodhi 17:03:16 sure. 17:03:44 Okay, lets close the meeting 17:03:48 Thanks everyone for joining 17:03:52 #endmeeting