15:05:51 <davdunc> #startmeeting fedora_cloud_meeting
15:05:51 <zodbot> Meeting started Thu Feb 17 15:05:51 2022 UTC.
15:05:51 <zodbot> This meeting is logged and archived in a public location.
15:05:51 <zodbot> The chair is davdunc. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
15:05:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:05:51 <zodbot> The meeting name has been set to 'fedora_cloud_meeting'
15:06:00 <davdunc> #topic roll call
15:06:02 * mhayden waves with emoji this time 👋🏻
15:06:08 <davdunc> chair mhayden
15:06:10 <davdunc> :)
15:06:15 <mhayden> 🪑
15:06:16 <davdunc> #chair mhayden
15:06:16 <zodbot> Current chairs: davdunc mhayden
15:06:19 <dustymabe> .hi
15:06:20 <zodbot> dustymabe: Something blew up, please try again
15:06:22 <davdunc> hah.
15:06:24 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
15:06:28 <davdunc> #chair dustymabe
15:06:28 <zodbot> Current chairs: davdunc dustymabe mhayden
15:06:30 * mhayden is worried that dustymabe blew up
15:06:42 <dustymabe> :(
15:06:46 <Eighth_Doctor> .hello ngompa
15:06:46 <davdunc> he probably blew up zodbot
15:06:46 <zodbot> Eighth_Doctor: Something blew up, please try again
15:06:49 <zodbot> Eighth_Doctor: An error has occurred and has been logged. Please contact this bot's administrator for more information.
15:06:53 <davdunc> #chair ngompa
15:06:53 <zodbot> Current chairs: davdunc dustymabe mhayden ngompa
15:07:02 <Eighth_Doctor> 😩
15:07:10 <davdunc> #chair Eighth_Doctor
15:07:10 <zodbot> Current chairs: Eighth_Doctor davdunc dustymabe mhayden ngompa
15:07:12 <davdunc> derp.
15:07:25 <davdunc> whoo hoo.
15:07:43 <davdunc> this is looking good this morning/afternoon/evening
15:08:23 <davdunc> okay.
15:08:26 <davdunc> I think we are all here.
15:08:30 <davdunc> #topic Action items from last meeting
15:09:21 <davdunc> Action item from #345
15:09:23 <davdunc> davdunc to return to the issue 345 in next meeting to secure an author or co-author for blog post for community comment
15:09:33 <davdunc> I didn't get this completed.
15:09:47 <davdunc> but Eighth_Doctor  is this something we can pen together?
15:10:17 <Eighth_Doctor> sure
15:10:34 <davdunc> vacation and stuff  kinda ruined my last week.
15:11:21 <davdunc> I was also need to work with cloud-init folks to find out how to enable timeserver configurations  using cloud-init with ipv6 on Amazon EC2.
15:11:46 <davdunc> I was also need  == I also need
15:12:04 <davdunc> Issue #347
15:12:39 <davdunc> Issue #355 Add AZURE image testing  is another one we have on our plate.
15:12:50 <davdunc> but that's going to be addressed a little later.
15:13:46 <davdunc> Issue #365 Please publish AWS aarch64 AMI for Fedora 35 is on the agenda for now.
15:15:14 <davdunc> anyone wants to discuss any of these actions before we get into the regular agenda?
15:15:53 <mhayden> keep rollin'
15:16:14 <davdunc> okay.
15:16:26 <davdunc> #topic 365 Please publish AWS aarch64 AMI for Fedora 35
15:16:43 <davdunc> #link https://pagure.io/cloud-sig/issue/365
15:17:09 <davdunc> So this is in infra right now, but the big deal is that the euca tools don't support the publication.
15:17:16 <davdunc> fedimg needs to be replaced.
15:17:18 <davdunc> it's time.
15:18:14 <Eighth_Doctor> we've had an issue about it for a while: https://pagure.io/cloud-sig/issue/301
15:18:19 <davdunc> yea.
15:18:29 <davdunc> #link https://pagure.io/cloud-sig/issue/301
15:18:42 <davdunc> so I am ready.
15:18:45 <mhayden> should we evaluate the current projects we have available to us in Fedora to replace fedimg? something with a list of trade-offs?
15:18:57 <davdunc> mhayden: that's a great idea.
15:19:03 <Eighth_Doctor> we've already got a couple of contenders
15:19:22 <davdunc> Eighth_Doctor has done a lot of work on plumbing in kiwi
15:19:23 <mhayden> i figured we already have some tools packaged that could work
15:19:28 <Eighth_Doctor> 1. SUSE Enceladus' mash
15:19:40 <Eighth_Doctor> 2. CoreOS Assembler's "ore"
15:19:59 <mhayden> are we looking for an image builder, uploader, or something that does both jobs well?
15:19:59 <Eighth_Doctor> I'm personally in favor of mash, because being in Python means that extending for more clouds is considerably easier
15:20:12 <davdunc> image builder can't make the images yet.
15:20:14 <mhayden> osbuild-composer / osbuild are in Fedora as well
15:20:19 <Eighth_Doctor> and it also would work well the img-proof cloud image testing tool
15:20:41 <davdunc> yea. the ec2-img-utils will work with it.
15:20:58 <Eighth_Doctor> mhayden: osbuild has one major gap for Fedora: https://github.com/osbuild/osbuild/issues/517
15:21:16 <davdunc> #link https://github.com/osbuild/osbuild/issues/517
15:21:41 <Eighth_Doctor> until that gap is closed, osbuild is not workable for Fedora Cloud
15:21:44 <mhayden> ah okay. it looks like gicmo has already taken a look there
15:21:57 <Eighth_Doctor> there are other issues, but that's the biggest blocker
15:22:28 <Eighth_Doctor> (there's also the issue that there's a broken dnfdb in produced images: https://github.com/osbuild/osbuild/issues/455)
15:22:41 <mhayden> for what it's worth, RHEL has gone with image builder already and i can relay those issues to the team
15:23:12 <Eighth_Doctor> yeah I'm aware of that, I'm also aware that they haven't considered either of those issues as important for osbuild
15:23:19 <dustymabe> my personal opinion is keep building images the same way (for now) and use `ore`
15:23:31 <dustymabe> that's justy because I know the tool
15:23:35 <dustymabe> just*
15:23:44 <dustymabe> not because it is superior
15:23:49 <Eighth_Doctor> if they change their minds, then we can re-evaluate, mhayden
15:24:13 <Eighth_Doctor> dustymabe: I have two problems with "ore": it's in Go and it's vendored into CoreOS Assembler
15:24:26 <davdunc> dustymabe I understand that, but that moves our maintenance to go from python.
15:24:31 <davdunc> yea.
15:24:49 <mhayden> sounds like we all want a little m`ore` from our image build process 🤭
15:24:58 <Eighth_Doctor> lol
15:25:04 * mhayden tried
15:25:24 <Eighth_Doctor> moving image building tools is on the roadmap, mostly because I can't get an imgfac build working to save my life
15:25:27 <davdunc> good effort!
15:25:40 <mhayden> davdunc: 🥇
15:25:49 <Eighth_Doctor> currently, we're looking at kiwi, but it's a problem that frankly I'm putting off for later
15:26:18 <davdunc> yea. I wanted to put it off, but we are running into publication issues now, so it's going to have to change.
15:26:23 <Eighth_Doctor> the image upload and testing stuff is a more severe problem, since it blocks us for Azure too
15:26:40 <davdunc> the imagefac will convert to 2to3 pretty easilly ,but then we need new euca tools
15:26:51 <Eighth_Doctor> img-mash and img-proof are not specifically tied to kiwi
15:26:57 <Eighth_Doctor> so we can do those pieces now
15:27:30 <Eighth_Doctor> I already made changes a while ago to support testing RH-family distros in img-proof
15:27:36 <dustymabe> wasn't there an issue recently where fedimg wasn't working at all?
15:27:46 <davdunc> it's the euca tools dustymabe
15:28:12 <Eighth_Doctor> so we just need img-mash and img-proof packaged in Fedora to start moving things over
15:28:42 <davdunc> but we can just add in the img-mash and test with img-proof with little modification. It fits with os-build tool
15:29:22 <davdunc> sorry.
15:29:25 <davdunc> not os-build
15:29:39 <davdunc> OBS
15:30:26 <davdunc> let's look ore is a good fit, but moves our language support from go to python. That's a learning curve for me.
15:31:01 <davdunc> img-mash isn't in the standard chain of tools for our downstream, but we are pretty deviant from downstream right now.
15:31:26 <davdunc> but speaking of downstream, we need to add ELN into the publication process.
15:31:39 <davdunc> so whatever we do does need to include that.
15:32:04 <davdunc> sgallagh: has been patiently waiting for that for months.
15:32:48 <davdunc> so we might want to land an os-build process in parallel to the img-mash
15:33:04 <davdunc> img-mash will handle our problem fast.
15:33:16 <davdunc> ore will get us consistent with fcos
15:33:30 <davdunc> os-build would be great for downstream combatibility.
15:34:10 <davdunc> I'm partial to img-mash today.
15:34:33 <davdunc> with os-build as a future path, when it supports btrfs more consistently.
15:34:35 <dustymabe> whatever works for me
15:34:48 <davdunc> dustymabe: thanks.
15:34:55 <dustymabe> just telling you what I would do if I were implementing it
15:34:56 <davdunc> Any other strong opinions.
15:35:06 <dustymabe> and we also need to identify this as a huge liability for us
15:35:14 <davdunc> dustymabe: I think you have a great idea.
15:35:20 <dustymabe> if fedimg stops working we could be in danger of missing F36
15:35:25 <davdunc> yea.
15:35:44 <dustymabe> at this point we have way more "friends" that know ore than fedimg
15:35:54 <davdunc> agreed.
15:36:24 <davdunc> but Eighth_Doctor has already created a pipeline with kiwi -> img-mash and I think we have a solid handle there too.
15:37:17 <davdunc> the os-build is something I think that we should all strive to support just for the sake of the community
15:37:23 <Eighth_Doctor> osbuild doesn't do uploads
15:37:36 <Eighth_Doctor> img-mash+img-proof would work with imgfac, kiwi, or osbuild just fine
15:37:43 <davdunc> got it.
15:37:57 <davdunc> i think os-build does do uploads in some cases, doesn't it?
15:38:11 <Eighth_Doctor> the osbuild-composer thing has the upload feature
15:38:12 <davdunc> i know it supports ec2 uploads.
15:38:17 <Eighth_Doctor> not osbuild itself
15:38:20 <davdunc> aha.
15:38:22 <davdunc> gotcha
15:38:30 <Eighth_Doctor> we can't use osbuild-composer no matter what
15:38:39 <Eighth_Doctor> it just doesn't fit with our infra
15:39:03 <Eighth_Doctor> though possibly gicmo might be trying to do weird things to bridge the two
15:39:24 <Eighth_Doctor> (I remain skeptical of the sustainability of it, it reeks of the Atomic Reactor architecture)
15:39:53 <davdunc> so do we know if there is work being done on integration between the fedora infra and os-build or if gicmo might be interested in working with us on that?
15:40:28 <davdunc> I know that Eighth_Doctor has put a lot into making kiwi work for image builds and the img-mash is an easy replacement.
15:40:46 <davdunc> ...img-mash is an easy replacement for fedimg.
15:40:55 <Eighth_Doctor> if gicmo wants to help us by getting the btrfs stuff going, we can look at osbuild more seriously
15:41:06 <Eighth_Doctor> but that's a hard blocker for us
15:41:19 <davdunc> Eighth_Doctor I agree with that wholeheartedly.
15:41:28 <Eighth_Doctor> and I've been working with the koji folks to add support for building images with kiwi for us
15:41:38 <Eighth_Doctor> that is pretty much in place with koji 1.28
15:42:01 <davdunc> Eighth_Doctor I'd like to work on that if you have some time to work with me on Friday.
15:42:16 <Eighth_Doctor> sure
15:42:25 <davdunc> 👍
15:42:37 <Eighth_Doctor> I don't think koji 1.28 is out yet, but we can still do some prep
15:42:41 <davdunc> so I'll add some time to your calendar and we'll get into it.
15:42:46 <Eighth_Doctor> or if you have a way to deploy koji easily, we can play with the feature
15:43:00 <davdunc> i do have an easy way to put the playground together.
15:43:32 <davdunc> oh, speaking of which, dustymabe we have access to C7g instance types in ec2 now.
15:43:54 <davdunc> need to add that to the cloud and fcos testing.
15:44:05 <dustymabe> what's special about those instances?
15:44:12 <davdunc> that's the Graviton 3 instance types.
15:44:29 <davdunc> so new neoverse ARM proc.
15:45:00 <davdunc> #action davdunc to add c7g instance types to AWS testing
15:46:15 <davdunc> okay.
15:46:45 <davdunc> so
15:47:55 <davdunc> #proposed use img-mash to update the image uploader for all public cloud uploads.
15:48:01 <dustymabe> davdunc: +1 I'll see if we can add it to FCOS testing
15:48:21 <dustymabe> but really - can we rescue the m4.large (and friends) instance type first?
15:48:23 <davdunc> +1 that's a great idea dustymabe!
15:48:33 <dustymabe> we're pinned on the kernel in F35 again
15:48:35 <davdunc> dustymabe: yes.
15:48:39 <dustymabe> we really need a fix upstream
15:48:39 <davdunc> ugh!
15:48:57 <dustymabe> so jforbes can pull it and we can unpin our kernel
15:49:02 <davdunc> mhayden: you good with img-mash for now.
15:49:14 <mhayden> let's give it a try!
15:49:17 <davdunc> and yes dustymabe
15:49:20 <davdunc> Eighth_Doctor?
15:49:33 <davdunc> sorry for the roll call on the proposal.
15:49:47 <dustymabe> davdunc: I don't know what else to do other than send an announcement to our users they need to migrate their instances because in the next month we're moving on
15:49:55 <Eighth_Doctor> yeah
15:49:56 <Eighth_Doctor> I'm good with that
15:50:12 <davdunc> #agreed move to img-mash from imgfac for the uploads.
15:50:25 <Eighth_Doctor> uhh
15:50:28 <jforbes> dustymabe: still no resolution from Amazon yet? Rather surprised about that
15:50:30 <Eighth_Doctor> isn't the uploader fedimg?
15:50:35 <davdunc> sorry.
15:50:37 <davdunc> yes.
15:50:38 <Eighth_Doctor> so it'd be moving from fedimg to img-mash
15:50:49 <dustymabe> jforbes: i'm surprised too
15:50:58 <dustymabe> davdunc: would it help if we made noice on social media?
15:51:00 <davdunc> #agreed move to img-mash from fedimg for uploads
15:51:06 <dustymabe> noise - rather
15:51:29 <davdunc> jforbes: I threw it over the wall and then went on vacation.
15:51:46 <davdunc> it's really all about follow through on this side.
15:52:14 <jforbes> Yeah, just that most other distros are also on kernels that are broken there too. I assumed they would prioritize fixing that
15:52:48 <davdunc> It's on my list of things to add to hit harder today and tomorrow.
15:53:16 <dustymabe> jforbes: I assume a revert of that single commit for newer kernels is still out of the question?
15:53:22 <davdunc> The xen team is actively looking into it. I know that's happening. I just don't know if I have missed a request.
15:53:46 <davdunc> okay.
15:53:50 <davdunc> so ...
15:54:01 <dustymabe> sorry davdunc - didn't mean to derail
15:54:15 <jforbes> dustymabe: At this point, it is not advised.  We revert for 5.16, and this gets forgotten about again, until 5.17 is broken... it needs to be fixed upstream
15:54:56 <davdunc> #topic Xen instances not booting with 5.15+ kernels without patch revert
15:55:19 <dustymabe> jforbes: understood
15:55:32 <davdunc> since we are talking about it.
15:55:34 <dustymabe> :)
15:56:00 <dustymabe> sorry davdunc - i know you are just the messenger. I hope it doesn't seem like we are faulting you specifically here
15:56:15 <davdunc> no no!
15:56:37 <dustymabe> I just wish there was something public for the amazon team we could point at and express our feelings
15:56:47 <davdunc> I don't feel like it's me alone on the spot.
15:56:55 <davdunc> but I do need to keep the pressure on.
15:57:08 <dustymabe> i could tweet and tag msw or something?
15:57:29 <davdunc> It's always a possibility.
15:57:30 <dustymabe> we could get people to open support tickets
15:57:32 <davdunc> :)
15:57:54 <davdunc> The support ticket could help.
15:57:55 * dustymabe makes jdoss open a support ticket
15:58:19 <davdunc> hopefully they could attach that to the same issue in the xen queue now.
15:58:34 <davdunc> I'll add a status update to the fcos gh issue.
15:58:40 <dustymabe> thank you
15:59:32 <davdunc> okay.
15:59:41 <davdunc> #topic open floor
15:59:51 <davdunc> oops.
16:00:14 <davdunc> #action davdunc to add status update to fcos ticket on kernel bug for xen instance types.
16:00:33 <davdunc> anything else we need to cover?
16:01:19 <davdunc> #link https://github.com/fedora-infra/fedimg/pull/162 looks like phsmoura looked into the euca-import-volume issue.
16:01:29 * mhayden must scurry away
16:01:33 <davdunc> okay.
16:01:38 <davdunc> thanks for being here mhayden
16:01:52 <davdunc> I think that's all for today.
16:02:09 <davdunc> we have our projects. I'll let you all know where we are in the next meeting.
16:03:23 <davdunc> #endmeeting