15:02:46 <nirik> #startmeeting RELENG (2021-09-28)
15:02:46 <zodbot> Meeting started Tue Sep 28 15:02:46 2021 UTC.
15:02:46 <zodbot> This meeting is logged and archived in a public location.
15:02:46 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
15:02:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:46 <zodbot> The meeting name has been set to 'releng_(2021-09-28)'
15:02:48 <nirik> #meetingname releng
15:02:48 <zodbot> The meeting name has been set to 'releng'
15:02:57 <nirik> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
15:02:57 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:03:04 <nirik> #topic init process
15:03:26 <nirik> I am here, but also it's beta release morning, so I reserve the right to drop this meeting in favor of any problems with the release. ;)
15:04:03 * nirik waits to see who is around.
15:04:44 * dustymabe waves
15:04:54 <dustymabe> i've got a few others coming but we're held up in another meeting
15:06:47 <dustymabe> welcome jschinta
15:06:56 <dustymabe> we'll get started here in a few
15:07:04 <jschinta> Hi Dusty (from 46.5.175.*)
15:08:22 <dustymabe> just a few more minutes nirik
15:08:40 <dustymabe> waiting on ravanelli and jlebon to join
15:08:59 <nirik> no worries from my side... but I sure hope mboddu can make it... otherwise it's just me on the releng side. ;)
15:09:21 <mboddu> Sorry, I am late here
15:09:37 <mboddu> I can take over it nirik, if you want me to
15:09:48 <nirik> sure. ;)
15:10:31 <mboddu> Okay, lets wait for the coreos folks to join
15:10:42 <dustymabe> mboddu: if you have another topic you can start with that
15:12:06 <mboddu> Okay, let me check
15:12:13 * mboddu hasn't prepared for this meeting
15:13:28 <mboddu> #topic Tracking eln composes in failed-composes
15:13:49 <mboddu> There was a request to track eln composes as well and I am planning to add it to the compose-tracker
15:14:26 <dustymabe> ok they should be here any minute
15:14:34 <mboddu> Is this okay to track them in one place? I am okay with it, unless someone says otherwise
15:14:51 <jlebon> .hello2
15:14:54 <dustymabe> 👋 jlebon
15:14:54 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
15:14:56 <dustymabe> .hi
15:14:57 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
15:14:58 <nirik> mboddu: I'm fine with it.
15:15:13 <nirik> I think having them in one place is nice... unless eln folks disagree.
15:15:27 <jschinta> .hi (from 46.5.175.*)
15:15:28 <zodbot> jschinta: Sorry, but user 'jschinta' does not exist
15:15:47 <jlebon> hi all! :)
15:15:52 <dustymabe> just pinged renata
15:16:06 <dustymabe> we'll give her a minute and then start
15:16:19 <dustymabe> oops - after the current topic :)
15:16:32 <mboddu> Hello all from the coreos team :)
15:16:40 <dustymabe> 👋👋👋
15:17:09 <mboddu> I think we can get started with the dustymabe's topic
15:17:17 <mboddu> You can take it Dusty :)
15:17:38 <dustymabe> want to switch to #topic s390x and ppc64le strategy for Fedora CoreOS ?
15:17:50 <mboddu> Sure
15:18:26 <dustymabe> I don't think I'm a chair so I can't do it
15:18:36 <mboddu> dustymabe: You are one of the chairs, but anyway
15:18:41 <mboddu> #topic s390x and ppc64le strategy for Fedora CoreOS
15:18:42 <dustymabe> oh then I will :)
15:18:45 <dustymabe> oh good
15:19:18 <dustymabe> ok, so.. the aarch64 pilot where we use a node to SSH into and do builds is working well for aarch64
15:19:41 <dustymabe> jschinta: is from IBM and interested in getting the FCOS pipeline up and running for s390x
15:19:43 <nirik> glad to hear it. :)
15:19:53 <mboddu> Nice
15:19:57 <nirik> sorry that we haven't been able to get something better setup in rdu-cc. ;(
15:20:06 <dustymabe> and ravanelli recently joined our teams and has had a focus on ppc64le in the past
15:20:15 <dustymabe> nirik: no worries, we'll get it sorted out eventually :)
15:20:31 <nirik> yeah, we can be pretty determined. ;)
15:20:47 <dustymabe> so basically, we're looking to try to use a similar strategy for adding those two architectures to our pipeline
15:20:51 * mboddu is a bit worried on s390x, being in different data center, but lets see...
15:21:01 <dustymabe> the one problem we have here is hardware
15:21:31 <dustymabe> jschinta: has access to hardware but it's on the IBM side and not publicly accessible (at least not for what we need, which is bare metal so we can run virt for the builds)
15:21:43 <jschinta> mboddu: Yeah, the Problem we have with s390x is getting a public LPAR (from 46.5.175.*)
15:22:02 <nirik> we do not have any public lpars either. :)
15:22:03 <dustymabe> yes, for s390x specifically
15:22:11 <jschinta> We have LPAR behind the IBM Firewall, but no one is offering a public LPAR (from 46.5.175.*)
15:22:18 <dustymabe> nirik: correct, but how do you do koji builds on s390x ?
15:22:31 <dustymabe> you have one that is at least accessible to fedora infra somehow?
15:22:56 <jschinta> We can get zVMs or KVMs, but since FCOS is using KVM in it's builds, there is a noticable performance impact (from 46.5.175.*)
15:23:01 <mobrien> Its a nation holiday in cz so jednorozec may not be here for the meeting today
15:23:23 <jschinta> I'm not sure how we do koji builds, since i'm not involved there (from 46.5.175.*)
15:23:25 <nirik> yes. currently we have 2 lpars in a machine in boston. We have internal network to/from it from iad2. It's firewalled off to hell and back... but we have enough access to reach the builders there and they can reach our koji hub
15:23:32 <mobrien> stupid matrix
15:23:56 <dustymabe> nirik: could we possibly re-use that same setup?
15:23:57 <nirik> but... we also are likely to be moving soon to another machine...
15:24:40 <nirik> we are currently on a z13 box, and because centos9-stream and eln have moved to z14 specific instructions, we will likely be moving to a z15 in the next few months.
15:25:05 <nirik> dustymabe: perhaps. We had to make some infosec promises there. We definitely 100% cannot make them publicly accessable.
15:25:33 <dustymabe> right, for our needs I think it's sufficient if our pipeline can access them
15:25:35 <nirik> so, where would access need to come from? centos-ci?
15:25:38 <dustymabe> so whereever the pipeline is running
15:25:55 <dustymabe> correct, it runs in centos CI right now, but we have talked with you all about moving it to fedora infra at some point too
15:26:11 <nirik> if the pipeline is in iad2... yeah, I think we can make it work. Centos-ci... I don't think thats going to be possible
15:26:16 <nirik> yeah
15:26:31 <dustymabe> is centos-ci in IAD2 ?
15:26:31 <nirik> so, after that moves we could probibly setup a composer for you...
15:26:34 <nirik> no .
15:26:37 <nirik> it's in RDU
15:26:40 <dustymabe> ahh
15:26:43 <dustymabe> good deal
15:26:50 <nirik> and there's no link between RDU and that boston lab. ;(
15:26:53 <dustymabe> ok so let me restate
15:26:55 <jlebon> is IAD2 where the new openshift instance is?
15:27:08 <nirik> yep. thats our 'main' datacenter.
15:28:12 <dustymabe> ok so jschinta - our plans are to move our FCOS pipeline to fedora infra at some point in the future (away from CentOS CI) and that new datacenter has a network setup that would allow us to access the LPAR that fedora uses
15:28:23 <mboddu> I am trying to remember the issue with transferring ostree objects between BOS and PHX2 data center back in the day. It was taking too long and we disabled it.
15:28:40 <nirik> FYI, our stg and prod openshift 4 clusters are up now... the folks doing that work are just now setting up kubevirt/storage/metrics... so it should be ready for use before too much longer.
15:28:53 <dustymabe> nirik: well that's good news
15:29:03 <jschinta> dustymabe: If we could use that LPAR that would be great (from 46.5.175.*)
15:29:36 <nirik> at that point it would likely be on the new lpar we move to, but I guess we will see when that happens. ;)
15:29:51 <dustymabe> nirik: obviously there are some decent bandwidth needs associated, but I assume we can flesh those out when we can test things
15:30:19 <nirik> sometimes the BW between boston and iad2 isn't great... but it's been enough for koji builds.
15:30:46 <dustymabe> yeah - we'll see what we can do
15:30:53 <jlebon> cool, keep us posted for when we can start testing around there to prepare for migrating!
15:31:07 <nirik> also, I assume you would want a FCOS node for this? do you have images/etc to do that? or would we need to make your a fedora node and bootstrap it into fcos after building things?
15:31:09 <mboddu> But transferring the tiny ostree objects was the problem, but we could try again and see what happens
15:31:12 <dustymabe> ok - jschinta sound good for s390x side? we'll circle back around the first of the year and see where things stand
15:31:56 <nirik> mboddu: huh, I don't recall that issue... I would hope there's not too much transfer from a fcos compose...
15:31:58 <dustymabe> nirik: an FCOS node would be ideal - but if we need to we could try to share with a koji builder (just needs podman and KVM for the most part)
15:32:15 <jschinta> I need to check with my PO if we need something sooner, but i think no one is complaining if there is a possible setup (from 46.5.175.*)
15:32:21 <dustymabe> there's a decent amount of transfer for these FCOS runs because we create disk images :(
15:32:45 <dustymabe> we do compress them, though
15:32:54 <nirik> yeah, we do that in fedora composes too. Although we might want to make sure they don't happen always at the same time.
15:33:03 <dustymabe> fair
15:33:10 <dustymabe> we'll work out the details - as always
15:33:17 <dustymabe> ok, on to ppc64le
15:33:30 <dustymabe> same problem, different arch
15:33:36 <dustymabe> what's our ppc64le story look like
15:33:44 <dustymabe> what does*
15:33:49 <mboddu> That shouldn't be a problem, they are in same data center
15:33:54 <nirik> that looks better, because thats in the same dc. ;)
15:33:55 <nirik> yeah
15:34:11 <dustymabe> are we able to get one dedicate (i.e. installed with FCOS) or do we need to share?
15:34:11 <nirik> so we can probibly get you a vm without too much pain...
15:34:17 <dustymabe> *vm*
15:34:26 <dustymabe> we kind of need bare metal :(
15:34:29 <nirik> yep. I don't think we can dedicate an entire machine to you
15:34:31 <dustymabe> not unless nested virt is just really good
15:34:33 <nirik> why?
15:34:41 <nirik> well fedora uses it for all out composes
15:34:42 <dustymabe> we use kvm to make the disk images
15:35:06 <jschinta> i can only say for s390x, but it takes about 6-8h to run the kola tests in a zVM (from 46.5.175.*)
15:35:16 <nirik> we have 4 power9 boxes... and are using them all heavily for builders
15:35:30 <nirik> jschinta: yeah, kvm should be a lot faster tho right?
15:35:48 <dustymabe> nirik: i'm willing to give it a shot and see how the performance is
15:36:00 <nirik> yeah, lets try at least? :)
15:36:06 <mboddu> IIRC, when we switched fom zVM to kvm, we saved more than 50% time in image building
15:36:18 <nirik> there's a lot of changes coming in power10 down the road too...
15:36:20 <dustymabe> basically we run virt to do the builds and also run about 20-40 VMs per build to do tests
15:36:40 <jschinta> nirik: haven't gotten a kvm to check (from 46.5.175.*)
15:37:19 <dustymabe> ravanelli: have you tried to run FCOS composes using nested virt (i.e. running a compose on a VM?)
15:37:51 <nirik> nested virt seems to work pretty welll on ppc64le...
15:38:03 <ravanelli> dustymabe: yeah. It worked well
15:38:06 <nirik> and also on s390x really (well, kvm, zvm is slow)
15:38:23 <dustymabe> ravanelli: ok, we can try that then :)
15:38:51 <ravanelli> The only problem I'm right now is with RHEL8 as host
15:38:57 <mboddu> We have been nested virt for all our image builds and they seem to be fine
15:38:57 <dustymabe> nirik: is this another thing where we should wait til after the we move FCOS to IAD2 ?
15:38:58 <ravanelli> But with Fedora, that works
15:39:24 <dustymabe> or can we start to use that now (i.e. talking via SSH from RDU to IAD2 hardware)
15:39:46 <nirik> dustymabe: yeah, probibly so? although we mgith have more power9 stuff in rdu before too long
15:40:05 <ravanelli> I joined a little bit late here. But for Power we have a povervms  server now
15:40:19 <ravanelli> If we need more, we can just ask
15:40:26 <nirik> ravanelli: oh? whats the issue with rhel8?
15:40:46 <nirik> rhel8 host that is
15:40:49 <ravanelli> Some qemu tests are failing, but the build went well
15:41:05 <ravanelli> Yeah, rhel8 as host
15:41:06 <dustymabe> ravanelli: some qemu FCOS CI tests?
15:41:06 <nirik> huh, odd. ok. Well, our power boxes are rhel8 host...
15:41:22 <nirik> s390x is fedora
15:41:52 <ravanelli> yeah, fcos tests.
15:42:00 <dustymabe> got ya, we'll figure that out
15:42:04 <ravanelli> nirik: I just opened it yesterday https://bugzilla.redhat.com/show_bug.cgi?id=2008271
15:42:50 <dustymabe> ok ravanelli - there's some things we need to do to get our multi-arch pipeline code cleaned up a little bit
15:42:50 * nirik subscribes
15:42:53 <ravanelli> It is a p8, p9? nirik?
15:43:05 <nirik> power9
15:43:17 <dustymabe> what do you say we circle back in a month and request a ppc64le VM then and we'll start to add it to the pipeline
15:43:21 <ravanelli> interesting
15:43:40 <nirik> dustymabe: sounds good.
15:43:54 <ravanelli> dustymabe: The server I got is basically for fcos
15:44:21 <nirik> ah so you all can start testing/working on it now with that server?
15:44:22 <dustymabe> ravanelli: oh, I didn't realize you got a server
15:45:05 <ravanelli> nirik: yeah, we just need to pass this issue described in the bug
15:45:20 <ravanelli> But it is just a matter of adding the keys there to share access
15:45:23 <dustymabe> ravanelli: is it your server? you could put Fedora on it
15:45:43 <ravanelli> dustymabe: yeah, I requested it for the fcos CI
15:46:10 <dustymabe> interesting. OK. so the question is.. do we need a VM from fedora infra/releng?
15:46:40 <dustymabe> I think the answer is still yes.. from a "official fedora" perspective. It's nice to build our bits from an environment that fedora infra/releng control.
15:46:46 <nirik> IMHO once you move your pipeline you should use local/known machines...
15:46:50 <ravanelli> previously I had 2 vms in a Brazilian university, but the lack of support there is an issue. I needed to update the firmware and wasn't able to.
15:46:56 <ravanelli> So I got this one in IBM cloud
15:47:40 <dustymabe> ravanelli: does that plan still sound good? test with IBM cloud - when we go to prod we'll try to use the fedora infra provided machine?
15:48:05 <ravanelli> dustymabe: anything that works.
15:48:09 <dustymabe> +1
15:48:43 <dustymabe> nirik: mboddu: we'll circle back with you in a month on ppc64le and potential FCOS pipeline move to IAD2
15:48:52 <nirik> alrighty
15:48:55 <ravanelli> Let's test IBM cloud with bandwith and stuff like that.
15:49:02 <ravanelli> So we can see how it goes as well
15:49:02 <dustymabe> +1
15:49:26 <dustymabe> I think that's all from us.. anything else jlebon jschinta ravanelli ?
15:49:34 <ravanelli> nope
15:49:40 <nirik> sounds reasonable.
15:49:41 <mboddu> dustymabe: +1
15:49:51 <jschinta> no (from 46.5.175.*)
15:50:02 <dustymabe> thanks nirik mboddu
15:50:06 <nirik> dustymabe: could you or someone update that infra ticket with what we discussed today? so folks know it's still moving forward/we have a plan? ;)
15:50:20 <dustymabe> yeah - let me see if I can find it
15:50:22 <nirik> (and that we all agree it's the same plan. :)
15:51:07 <nirik> https://pagure.io/fedora-infrastructure/issue/8370
15:51:23 <dustymabe> yep - will update
15:51:25 <dustymabe> thanks all
15:51:30 * dustymabe goes AFK for a bit
15:52:08 <mboddu> Thanks dustymabe and everyone else
15:52:15 <mboddu> #topic Open Floor
15:52:24 <mboddu> #info F35 Beta is out
15:52:31 <nirik> hurray!
15:52:31 <mboddu> Anything else from your end nirik ?
15:52:54 <nirik> nope... don't think so
15:53:56 <mboddu> Okay, then I will close this meeting
15:54:00 <mboddu> Thanks everyone for attending
15:54:03 <mboddu> #endmeeting