16:00:44 <mboddu> #startmeeting RELENG (2019-07-18)
16:00:44 <zodbot> Meeting started Wed Jul 17 16:00:44 2019 UTC.
16:00:44 <zodbot> This meeting is logged and archived in a public location.
16:00:44 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:44 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:44 <zodbot> The meeting name has been set to 'releng_(2019-07-18)'
16:00:44 <mboddu> #meetingname releng
16:00:44 <zodbot> The meeting name has been set to 'releng'
16:00:45 <mboddu> #chair nirik tyll sharkcz masta pbrobinson puiterwijk mboddu dustymabe ksinny jednorozec
16:00:45 <zodbot> Current chairs: dustymabe jednorozec ksinny masta mboddu nirik pbrobinson puiterwijk sharkcz tyll
16:00:45 <mboddu> #topic init process
16:00:59 <nirik> morning
16:01:08 * sharkcz is here today
16:01:25 * mboddu waves at nirik and sharkcz
16:01:57 <mboddu> So, lets get started
16:02:03 <mboddu> #topic EPEL8
16:03:32 <mboddu> nirik: I am not sure whether I told you the reason why I asked for fxx and fxx-candidate targets. https://pagure.io/fedscm-admin/pull-request/16, look at Stephan comment on epel8-playground-candidate target
16:04:29 <nirik> uh, looking, but there's a lot of comments there.
16:04:42 <mboddu> So, I talked to Dennis few mins back and he said, they were created for different reasons back in the day and later he merged them back together.
16:05:08 <mboddu> nirik: Sorry, cant give you the link since the PR got rebased later
16:05:19 <mboddu> And the link dies :(
16:05:21 <nirik> ok
16:05:58 <mboddu> When I say merged I mean both of them serve the same purpose now
16:06:23 <nirik> right, ok
16:06:46 <mboddu> But fedpkg uses fxx-candidate target to identify the target if no target is specified while package.cfg has fxx targets listed
16:06:50 <nirik> but fedpkg is still... right.
16:07:22 <nirik> so lets just use both now and if we clean this up later we can fix it everywhere (which will be a big pain, but oh well)
16:07:33 <mboddu> So, should we create epel8-playground target and keep continuing it, or should we ask people to change and start using fxx-candidate
16:07:54 <mboddu> nirik: Okay, you answered before I asked the question :D
16:08:08 <mboddu> I will create the target and update the PR after the meeting
16:08:14 <nirik> ok
16:08:15 <mboddu> And that should be it
16:08:31 <nirik> and we can start branching and building and then composing. ;)
16:08:36 <mboddu> smooge: ^ FYI, we might be ready to start branching in couple of hours
16:09:00 <mboddu> nirik: Right, but we are holding on composing for a while, until we have some builds
16:09:12 <mboddu> I am guessing may be next week
16:09:13 <nirik> sure
16:09:20 <mboddu> Or later, up to smooge
16:09:31 <nirik> we need a number of basethings
16:09:35 <mboddu> Yup
16:10:08 <mboddu> That reminds me, I dont know if you have seen my message couple of days back
16:10:13 * mboddu digs up the logs
16:11:09 <mboddu> [09:22:34] <mboddu> nirik: You remember the buildroots for epel8 and epel8-playground, I think I found something, not sure if its an issue or something we might want to have
16:11:09 <mboddu> [09:22:58] <mboddu> epel8 buildroot = rhel + epel (stable+overrrides)
16:11:09 <mboddu> [09:23:59] <mboddu> epel8-playground = rhel + epel8-playground + epel8(only stable)
16:11:09 <mboddu> [09:24:57] <mboddu> From what I think it might be desired since when they move stuff epel8-playground to epel8, this might be helpful for them
16:11:34 <smooge> .here
16:12:08 <nirik> yeah, we will want that indeed.
16:12:40 <smooge> i saw I needed to add some epel8 points to koji-gc and was wondering when that should go in. it needs keys
16:12:40 <mboddu> Awesome
16:12:46 <smooge> oh sorry
16:13:45 <mboddu> smooge: Yeah, we need update koji configs and some scripts
16:15:26 <mboddu> nirik: We need to build fedpkg and fedscm-admin, I kinda own fedscm-admin and I can build it, but fedpkg we might have to add the patch and build it
16:15:52 <nirik> ok. or can you ask upstream to do a new release? that might be easiest.
16:16:33 <mboddu> nirik: Actually let me send them an email right away, before they go out for the day
16:16:47 * mboddu writes an email, bbiab
16:20:14 <mboddu> Sent the email
16:20:50 <mboddu> Anything else on EPEL8?
16:21:58 <nirik> nope
16:22:05 <mboddu> Okay, moving on
16:22:15 <mboddu> #topic Alternate Architecture Updates
16:22:26 <mboddu> sharkcz: Any updates?
16:22:46 <mboddu> nirik: I noticed that switching s390x to KVM didn't help us much, it took the same amount of time to run the compose
16:22:52 <sharkcz> nope, but thanks to nirik for the new s390x builders, they are very fast even for the images
16:22:54 <mboddu> I didn't look at the image times
16:22:59 <nirik> yes, I was disappointed as well...
16:23:08 <nirik> something is taking longer, I hope to look into it.
16:23:23 <nirik> I think it might be ppc64le
16:23:25 <mboddu> nirik: Yeah, me to :)
16:23:32 <sharkcz> yes, it;'s ppc64le
16:23:48 <sharkcz> s390x container was ~15 minutes
16:23:53 <nirik> I was gonna try and update/reboot them again...it might just be storage tho
16:24:11 <nirik> I guess I could only enable 1/2 of them and see if that helps
16:24:30 <mboddu> enable what?
16:25:09 <nirik> the builders? there are 9 of them on each power9 box... I could only enable 4 or 5 and see if they do things better without contending on the storage.
16:25:22 <mboddu> nirik: Ahh, got it
16:25:43 <nirik> or perhaps the problem is something else. I am not sure.
16:26:00 <sharkcz> nirik: is it p8 or p9 hw?
16:26:12 <nirik> it's the p9's that are slow.
16:26:27 <sharkcz> yes, then it could be the sata storage
16:26:52 <sharkcz> but it could be upgraded IMO
16:27:15 <nirik> yeah. I asked about that... we need more proof that that is the problem
16:27:51 <sharkcz> ack
16:28:20 <nirik> but I can try just 4 enabled per and do a compose and see?
16:28:43 <sharkcz> worth trying, for sure
16:28:48 <mboddu> nirik: ack
16:29:02 <sharkcz> are they used for the images? so with nested virt?
16:29:03 <nirik> but thats only 3 of them in the image channel.
16:29:18 <nirik> yeah, nested virt.
16:30:08 <sharkcz> so maybe I/O inside nested virt ...
16:30:25 <sharkcz> it's getting complex
16:30:28 <nirik> https://koji.fedoraproject.org/koji/taskinfo?taskID=36293914
16:30:33 <mboddu> In recent container base image - https://koji.fedoraproject.org/koji/taskinfo?taskID=36297774
16:30:36 <nirik> 2+ hours for a ostree?
16:30:50 <mboddu> s390x completed in 5 min, ppc64le took almost and hour and failed
16:31:49 <mboddu> s/and/an/
16:32:00 <nirik> so yeah, wonder if the nested virt is not happy somehow
16:32:23 <nirik> host has options kvm_hv nested=1
16:32:36 <sharkcz> the cpu will be proabbly OK, but I/O inside the nested VM can be the problem
16:32:53 <nirik> guests have:
16:32:56 <nirik> <features>
16:32:56 <nirik> <nested-hv state='on'/>
16:32:56 <nirik> </features>
16:33:24 <nirik> so... hum...
16:33:28 <sharkcz> what's the storage setup? LVs or image files?
16:33:48 * mboddu wishes oz throws some timestamps in the logs
16:34:07 <nirik> oh, right I can't do that. I was thinking I could just go back to the p8 ones... but we need the p9 ones or qemu has issues.
16:34:13 <nirik> sharkcz: lv's
16:34:26 <sharkcz> hmm
16:34:43 <nirik> it's got so much space... sadly it's 7200rpm sata.
16:34:51 <nirik> /dev/md2   vg_guests lvm2 a--  <21.83t 20.54t
16:35:00 <nirik> buildvm-ppc64le-01.ppc.fedoraproject.org vg_guests -wi-ao---- 141.60g
16:35:04 <sharkcz> but I guess we can try to replicate the process outside koji on another p9 and will see
16:35:32 <nirik> yeah, that would be great if can...
16:36:17 <sharkcz> we have a boston p9 in brno, will plan for some testing there
16:36:25 <mboddu> sharkcz: Do you have a p9 box?
16:36:29 <sharkcz> or I can test at home :-)
16:36:45 * mboddu is jealous of sharkcz :P
16:36:47 <nirik> so, I am going to update/reboot them, and then let me put some on the second one...
16:36:51 <nirik> nice
16:36:56 <sharkcz> but my disks are even worse, 500G SATAs ;-)
16:36:58 <mboddu> ack
16:37:09 <nirik> worth going to 5.2 kernel?
16:37:21 <sharkcz> works here
16:37:48 <sharkcz> but don't think it brings any perf changes
16:39:17 <nirik> ok
16:39:26 <mboddu> sharkcz: We are using 5.1.16-200.fc29.ppc64le
16:42:12 <nirik> hum, so I have 01/02 in compose and image. Let me move image to different dedicated ones. Perhaps it's doing runroot tasks at the same time as images on those.
16:43:52 <mboddu> #info sharkcz will be doing some experiments to gather some facts on ppc64le slowness and nirik will 4 builders on each p9 box and see if it helps with any storage competition issue.
16:43:54 <nirik> so, I'll update/reboot/shuffle channels around and fire a new compose and we can see from there. ;)
16:44:07 <mboddu> nirik: ack, let me know if you need any help
16:44:23 <mboddu> Moving 01 and 02 out of compose channel?
16:44:30 <mboddu> Or the other way around?
16:44:47 <mboddu> removing 01 and 02 from image and add couple more there?
16:45:01 <nirik> the second.
16:45:09 <mboddu> Awesome
16:45:12 <nirik> so, 01/02 only in compose, 03/04/10/11 in image
16:45:22 <mboddu> Oh 4 for image, nice
16:47:09 <mboddu> #info nirik will remove buildvm-ppc64le-01/02.ppc.fedoraproject.org ppc64le builders from image channel and add 04,10,11 ppc64le builders to image channel which makes 4 in image channel including 03
16:51:15 <nirik> wow... checkout the screenshot here: https://koji.fedoraproject.org/koji/taskinfo?taskID=36284507
16:52:27 <sharkcz> that doesn't look right ...
16:53:24 <nirik> anyhow...
16:54:15 <mboddu> Yeah, I wonder whats causing it
16:54:38 <mboddu> Anyway, anything else?
16:54:49 <smooge> sharkcz, nirik the recommendation I got from an ibm'ers was that the p9 on sata needed to use hardware raid or it bypassed the onboard cache in jbod mode.
16:55:41 <smooge> that hardware raid 'should' be raid6 but even raid0 with 1 drive per raid0 should enable the cache
16:56:08 <nirik> smooge: I find that pretty hard to beleive, but ok
16:56:19 <smooge> i don't know how valid that advise is but it is a controller item
16:56:22 <nirik> linux claims the controller is using cache
16:57:36 <nirik> Write cache: enabled, read cache: enabled, supports DPO and FUA
16:58:02 <sharkcz> on the other hand if JBOD is just a pass-thru, so it won't cache on the controller
16:58:19 <sharkcz> nirik: isn't it just the disk's cache?
16:58:35 <nirik> could be.
16:58:46 <smooge> it may be a multiple definition of using the cache. Again.. that is what I got told as a probable reason for the poor performance
16:58:52 <sharkcz> I would say for higher performance you will need cache from the I/O controller too
16:59:04 <nirik> whats the disk benchmark to use these days?
16:59:32 <sharkcz> no idea :-)
16:59:37 <sharkcz> maybe hdparm still
17:01:43 <sharkcz> the "use the raid function" makes sense to me
17:02:07 <sharkcz> to some degree
17:02:59 <mboddu> Okay, we passed the scheduled time, lets take the remaining discussion to #fedora-releng
17:03:03 <mboddu> Thanks guys for joining
17:03:11 <mboddu> #endmeeting