2024-07-17 16:31:09 <@gurssing:matrix.org> !startmeeting fedora_coreos_meeting
2024-07-17 16:31:12 <@meetbot:fedora.im> Meeting started at 2024-07-17 16:31:09 UTC
2024-07-17 16:31:12 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2024-07-17 16:31:32 <@gurssing:matrix.org> !topic roll call
2024-07-17 16:31:59 <@siosm:matrix.org> !hi
2024-07-17 16:32:01 <@zodbot:fedora.im> Timothée Ravier (siosm) - he / him / his
2024-07-17 16:32:11 <@hricky:fedora.im> !hi
2024-07-17 16:32:11 <@jlebon:fedora.im> !hi
2024-07-17 16:32:12 <@zodbot:fedora.im> None (jlebon)
2024-07-17 16:32:12 <@zodbot:fedora.im> Hristo Marinov (hricky) - he / him / his
2024-07-17 16:32:23 <@marmijo:fedora.im> !hi
2024-07-17 16:32:24 <@zodbot:fedora.im> Michael Armijo (marmijo)
2024-07-17 16:32:43 <@jbtrystram:matrix.org> !hi
2024-07-17 16:32:46 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his
2024-07-17 16:32:53 <@gurssing:matrix.org> !hi gursewak
2024-07-17 16:32:56 <@zodbot:fedora.im> Gursewak Singh (gursewak)
2024-07-17 16:33:37 <@ydesouza:fedora.im> !hi
2024-07-17 16:33:38 <@zodbot:fedora.im> Yasmin Valim de Souza (ydesouza)
2024-07-17 16:34:14 <@aaradhak:matrix.org> !hi aaradhak
2024-07-17 16:34:16 <@zodbot:fedora.im> Aashish Radhakrishnan (aaradhak)
2024-07-17 16:34:27 <@ravanelli:matrix.org> !hi ravanelli
2024-07-17 16:34:29 <@zodbot:fedora.im> Renata Ravanelli (ravanelli)
2024-07-17 16:34:33 <@gurssing:matrix.org> !topic Action items from last meeting
2024-07-17 16:34:56 <@gurssing:matrix.org> No actions items that I can see from last meeting notes.
2024-07-17 16:35:14 <@gurssing:matrix.org> Moving onto topics for today.
2024-07-17 16:35:21 <@gurssing:matrix.org> !topic Different partition layout after re-provisioning between version 39 and 40
2024-07-17 16:35:27 <@gurssing:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1745
2024-07-17 16:36:37 <@dustymabe:matrix.org> !hi
2024-07-17 16:36:39 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his
2024-07-17 16:37:40 <@gurssing:matrix.org> Jonathan Lebon: Do you want to introduce this issue?
2024-07-17 16:37:57 <@jlebon:fedora.im> sure
2024-07-17 16:40:49 <@jlebon:fedora.im> so in f40, our disk images moved from being partitioned by sgdisk (in cosa) to sfdisk (in osbuild). and while the resulting table is mostly the same, there are some tiny differences that have surfaced to users. this is one of them.
2024-07-17 16:40:49 <@jlebon:fedora.im> basically, the TL;DR there is that the automatic partition growing we do is now growing the last partition to a slightly different size, and so anyone expecting the exact same result (e.g. like in the data persistence case) will notice this
2024-07-17 16:42:08 <@jlebon:fedora.im> all the nitty-gritty details are in the last two comments there, and i initially thought this was worth sending a coreos-status post for it, but i'm thinking it may not yet be
2024-07-17 16:43:04 <@jlebon:fedora.im> the good news is that data loss is not a concern there at least in theory. i've provided steps also to access their data for anyone hitting this
2024-07-17 16:43:34 <@jlebon:fedora.im> but... at first glance it'll certainly _appear_ like data loss
2024-07-17 16:43:58 <@siosm:matrix.org> that's what worries me
2024-07-17 16:44:19 <@siosm:matrix.org> I don't think a coreos-status post would hurt.
2024-07-17 16:44:51 <@siosm:matrix.org> Let's re-phrase that positively: A coreos-status post would probably be nice
2024-07-17 16:45:20 <@jlebon:fedora.im> certainly not against it. it was just not clear how prevalent an issue this really is
2024-07-17 16:45:23 <@siosm:matrix.org> Should we make a test for that?
2024-07-17 16:45:45 <@jlebon:fedora.im> anyone have countme data offhand for how many people are still on < f40?
2024-07-17 16:46:01 <@jlebon:fedora.im> we definitely should have a data persistence test
2024-07-17 16:46:21 <@hricky:fedora.im> When I tested it, there was no data loss on my end.
2024-07-17 16:48:13 <@siosm:matrix.org> It's really a shame that XFS can't be shrunk
2024-07-17 16:48:16 <@jlebon:fedora.im> right. all the data is still there. changing a partition entry in the table doesn't change its contents. and most (all?) filesystems are smart enough to error out early on during mounting when it sees that its container appears truncated
2024-07-17 16:50:44 <@jlebon:fedora.im> anyway, personally lean towards no coreos-status post unless e.g. at least one other person shows up on that tracker issue. but ok with sending one too.
2024-07-17 16:50:56 <@siosm:matrix.org> I don't know what to do. If someone wants to draft a coreos-status post that would be nice. Otherwise there is probably nothing we can really do to fix this
2024-07-17 16:51:56 <@jlebon:fedora.im> filed https://github.com/coreos/fedora-coreos-tracker/issues/1763
2024-07-17 16:52:06 <@jlebon:fedora.im> filed https://github.com/coreos/fedora-coreos-tracker/issues/1763 for the tests part
2024-07-17 16:52:37 <@jlebon:fedora.im> i guess let's move on?
2024-07-17 16:52:45 <@siosm:matrix.org> proposed: we move on and do nothing for now
2024-07-17 16:52:53 <@gurssing:matrix.org> +1
2024-07-17 16:53:00 <@dustymabe:matrix.org> +1
2024-07-17 16:53:04 <@marmijo:fedora.im> +1
2024-07-17 16:53:07 <@aaradhak:matrix.org> +1
2024-07-17 16:53:08 <@dustymabe:matrix.org> thanks for investigating Jonathan Lebon 
2024-07-17 16:53:09 <@ravanelli:matrix.org> +1
2024-07-17 16:53:15 <@jlebon:fedora.im> ack
2024-07-17 16:53:26 <@gurssing:matrix.org> Me move on then.
2024-07-17 16:53:29 <@gurssing:matrix.org> !topic tracker: Fedora 41 changes considerations
2024-07-17 16:53:35 <@gurssing:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1714
2024-07-17 16:54:20 <@mnguyen:fedora.im> !hi
2024-07-17 16:54:21 <@zodbot:fedora.im> Michael Nguyen (mnguyen)
2024-07-17 16:54:58 <@siosm:matrix.org> 109 (https://github.com/coreos/fedora-coreos-tracker/issues/1759) will be deferred to F42 as we found issues with rpm-ostree composes
2024-07-17 16:55:33 <@siosm:matrix.org> And it was too short before the mass rebuild to fix them. We'll retry once F41 is branched
2024-07-17 16:56:36 <@siosm:matrix.org> So we should look at 126 & 127?
2024-07-17 16:56:40 <@jlebon:fedora.im> 126: we already don't ship ifcfg in FCOS. will be interested to see once that change lands in CentOS Stream/RHEL.
2024-07-17 16:56:40 <@jlebon:fedora.im> 127. should be transparent to us
2024-07-17 16:57:30 <@siosm:matrix.org> 213, 214 & 215 should not impact us as well
2024-07-17 16:58:03 <@siosm:matrix.org> Thanks marmijo for running the script and updating the issue!
2024-07-17 16:58:09 <@marmijo:fedora.im> great! seems like it's a pretty short list of impacts to FCOS
2024-07-17 16:58:18 <@marmijo:fedora.im> You're welcome
2024-07-17 16:59:12 <@siosm:matrix.org> 213, 214 & 215 should not impact us as well (packages not included in FCOS)
2024-07-17 16:59:50 <@jlebon:fedora.im> gursewak:  i think we can move on 
2024-07-17 17:00:09 <@siosm:matrix.org> Maybe to https://github.com/coreos/fedora-coreos-tracker/issues/1324? I added it right before the meeting
2024-07-17 17:00:26 <@gurssing:matrix.org> Got it.
2024-07-17 17:00:36 <@gurssing:matrix.org> !topic Platform Request: Hetzner
2024-07-17 17:00:44 <@gurssing:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1324
2024-07-17 17:01:44 <@siosm:matrix.org> I've been looking at "quick wins" for platforms were are close to supporting them but a few bits are missing.
2024-07-17 17:01:48 <@siosm:matrix.org> Hetznner is one of them
2024-07-17 17:02:29 <@siosm:matrix.org> There is support in Afterburn & Ignition, and Flatcar are making images for it. I wrote https://github.com/coreos/fedora-coreos-docs/pull/654 based on their docs
2024-07-17 17:03:07 <@siosm:matrix.org> What remains on the Fedora CoreOS side is to publish an image with the right platform ID.
2024-07-17 17:04:08 <@siosm:matrix.org> The provisioning steps on Hetzner are a bit weird but it's not something we can really fix. The images are however "clean" in that they are installed offline and never booted so there should not be any issue.
2024-07-17 17:05:15 <@siosm:matrix.org> So the question is: Should we build & publish Hetzner images from our pipeline? Are the steps in the docs PR acceptable for our docs standards?
2024-07-17 17:06:01 <@dustymabe:matrix.org> I think I have an opinion on this
2024-07-17 17:06:01 <@jlebon:fedora.im> having read the docs, there are a lot of caveats and tricky bits there that I think makes publishing images for it not slightly but much better UX-wise
2024-07-17 17:06:03 <@siosm:matrix.org> From my perspective, it's better to have some docs with a lot of warnings rather than no docs, and leave out users "in the cold" / looking at other options.
2024-07-17 17:06:49 <@dustymabe:matrix.org> 
2024-07-17 17:06:49 <@dustymabe:matrix.org> Jonathan Lebon mind saying that a different way. it confused me 
2024-07-17 17:06:49 <@dustymabe:matrix.org> > I think makes publishing images for it not slightly but much better UX-wise
2024-07-17 17:06:52 <@jlebon:fedora.im> i like the idea of "restamp and upload yourself" for some platforms, but for this one it really looks way more painful than that
2024-07-17 17:07:12 <@jlebon:fedora.im> dustymabe: does ^ help?
2024-07-17 17:07:20 <@dustymabe:matrix.org> I think so
2024-07-17 17:07:57 <@dustymabe:matrix.org> so here's my piece:
2024-07-17 17:07:57 <@dustymabe:matrix.org> 
2024-07-17 17:07:57 <@dustymabe:matrix.org> I think we should either produce an official image OR make some standard way for people to restamp an image. i.e. `coreos-installer disk-image-customize --platform=hetzner`
2024-07-17 17:08:15 <@jlebon:fedora.im> i should've said "it makes publishing images for it not only _slightly_ better, but _much_ better UX-wise"
2024-07-17 17:09:10 <@dustymabe:matrix.org> for clouds where we foresee wanting to be in their official image list (i.e. the cloud provider takes our image and makes it available OR there is a way for us to upload and publish them) then obviously we should create and produce that image 
2024-07-17 17:09:26 <@siosm:matrix.org> The main benefit here is that it avoids a download / restamp / upload step. Everything happens directly in the cloud and is thus much faster if we have an image ready to use
2024-07-17 17:09:29 <@jlebon:fedora.im> the restamp part definitely would be nice to have better UX, but I don't think that helps a lot in this case AIUI
2024-07-17 17:11:00 <@siosm:matrix.org> yes, that's the issue. It makes it simpler (no need for guestfish, the script, etc.) but that does not remove the download/upload part
2024-07-17 17:11:35 <@dustymabe:matrix.org> if we produce an official image does that remove the download/upload part? 
2024-07-17 17:11:41 <@siosm:matrix.org> I don't know what Hetzner has planned for "custom images"
2024-07-17 17:11:49 <@siosm:matrix.org> yes
2024-07-17 17:12:11 <@dustymabe:matrix.org> how? does their tool accept a URL (i.e. can download directly from our s3 bucket)? 
2024-07-17 17:12:35 <@siosm:matrix.org> you can download a bz2 compressed RAW image from HTTPS
2024-07-17 17:12:41 <@siosm:matrix.org> you can download a bz2 compressed RAW image from an HTTPS URL
2024-07-17 17:12:57 <@siosm:matrix.org> https://github.com/apricote/hcloud-upload-image#usage
2024-07-17 17:13:23 <@siosm:matrix.org> you directly create a snapshot in the hetzner cloud from the image
2024-07-17 17:13:48 <@dustymabe:matrix.org> right, but is that tool downloading it locally and then uploading it to hetzner? or is it telling hetzner to go retrieve it?
2024-07-17 17:13:53 <@dustymabe:matrix.org> sorry for all the questions
2024-07-17 17:14:01 <@siosm:matrix.org> it's telling the server to retrieve it
2024-07-17 17:14:08 <@siosm:matrix.org> (those are good questions)
2024-07-17 17:14:21 <@dustymabe:matrix.org> cool - then you are right :), sorry had to ask because it can be confusing 
2024-07-17 17:14:30 <@jlebon:fedora.im> see also https://github.com/apricote/hcloud-upload-image?tab=readme-ov-file#about
2024-07-17 17:15:25 <@dustymabe:matrix.org> wow that's pretty unfortunate
2024-07-17 17:15:27 <@siosm:matrix.org> > 4. Download the disk image from within the rescue system
2024-07-17 17:15:37 <@dustymabe:matrix.org> you'd think a large cloud would have that all figured out by now 
2024-07-17 17:15:46 <@ravanelli:matrix.org> How hard would be for us to produce an official image?
2024-07-17 17:16:04 <@dustymabe:matrix.org> Renata Ravanelli: not hard at all, we're just deciding if we should
2024-07-17 17:16:29 <@jlebon:fedora.im> travier: so even if we built images, you'd still need the snapshotting dance, right?
2024-07-17 17:16:52 <@siosm:matrix.org> Jonathan Lebon: yes
2024-07-17 17:17:03 <@siosm:matrix.org> but it would be much easier/faster
2024-07-17 17:19:05 <@siosm:matrix.org> (note: we are almost at time)
2024-07-17 17:19:07 <@jbtrystram:matrix.org> I guess now we know why hetzner is cheap :) 
2024-07-17 17:20:03 <@ravanelli:matrix.org> Yeah, looking at the PR/docs in how to get it working, seems a lot for the users to handle/know, using some non official tools. I wonder if we would get a lot of issues/questions coming to us from users trying to get it working and failing.
2024-07-17 17:20:04 <@jlebon:fedora.im> seems to match with https://docs.hetzner.com/robot/dedicated-server/operating-systems/installing-custom-images as well. that's pretty sad UX :(
2024-07-17 17:20:53 <@jbtrystram:matrix.org> Somewhat related : it would be nice to spend the time to look at the platforms requests issues to make a list of all the targets we could address with a `coreos-installer customize-iso --platform` to have an idea of how many birds the stone would reach
2024-07-17 17:21:43 <@jlebon:fedora.im> kinda ambivalent on producing official images in that case. i thought we would've been able to upload the snapshot ourselves. to me, the restamping is not really the most painful part of that process
2024-07-17 17:22:13 <@siosm:matrix.org> I have hetzner & oracle cloud as current "target" for the restamp flow
2024-07-17 17:22:32 <@siosm:matrix.org> maybe scaleway
2024-07-17 17:22:34 <@jlebon:fedora.im> but definitely nice to save a download/upload trip for users. so cool with it too
2024-07-17 17:22:39 <@dustymabe:matrix.org> Jonathan Lebon: the `hcloud-upload-image` tool does it all for you, though?
2024-07-17 17:24:07 <@siosm:matrix.org> If we went the packer (no longer open source) route, we could use an existing image instead and restamp it live.
2024-07-17 17:24:23 <@siosm:matrix.org> If we went with the `packer` option (no longer open source) route, we could use an existing image instead and restamp it live.
2024-07-17 17:24:33 <@siosm:matrix.org> but that's almost more complicated
2024-07-17 17:25:10 <@jlebon:fedora.im> dustymabe: right, but that adds time and having to build/installer a "fourth-party" tool has a higher UX cost IMO 
2024-07-17 17:25:16 <@jlebon:fedora.im> dustymabe: right, but that adds time and having to build/install a "fourth-party" tool has a higher UX cost IMO
2024-07-17 17:25:39 <@siosm:matrix.org> I downloaded the provided release binaries
2024-07-17 17:25:50 <@dustymabe:matrix.org> IMO the tool is just covering a gap in hetzner's UX, which I would expect them to close in the future
2024-07-17 17:25:54 <@siosm:matrix.org> I downloaded the provided release binaries for hcloud-upload-image
2024-07-17 17:26:20 <@jlebon:fedora.im> yup, agree. and not blocking on that. just saying that i think that's the larger UX gap. not the restamping part
2024-07-17 17:26:25 <@dustymabe:matrix.org> if they did that then the user would take our image, provide it to `hcloud` CLI and get an cloud image out of it
2024-07-17 17:26:39 <@dustymabe:matrix.org> yeah, the restamping is the part we control though
2024-07-17 17:27:01 <@dustymabe:matrix.org> "run this guestfish script" isn't a great UX (for the part we control)
2024-07-17 17:27:26 <@dustymabe:matrix.org> so to me either we give them the whole image they can use or provide them a better UX for restamping (i.e. via coreos-installer)
2024-07-17 17:28:00 <@jlebon:fedora.im> or put it in a container: https://github.com/coreos/fedora-coreos-docs/pull/652#discussion_r1679953285
2024-07-17 17:28:10 <@jlebon:fedora.im> or... i guess put it in the coreos-installer container
2024-07-17 17:28:24 <@jlebon:fedora.im> so it's not yet another image to maintain
2024-07-17 17:29:05 <@dustymabe:matrix.org> I don't really understand why coreos-installer can't just pick up this functionality pretty easy
2024-07-17 17:29:35 <@siosm:matrix.org> We are close to at time and there is not urgency for this issue so many we should pick it up next week?
2024-07-17 17:29:36 <@dustymabe:matrix.org> cp disk image to new file, mount raw disk image, edit platform ID, unmount 
2024-07-17 17:29:38 <@gurssing:matrix.org> Reminder that we are almost out of time:)
2024-07-17 17:29:42 <@siosm:matrix.org> We are close to time and there is not urgency for this issue so many we should pick it up next week?
2024-07-17 17:30:18 <@siosm:matrix.org> We are close to time and there is no urgency for this issue so many we should pick it up next week?
2024-07-17 17:30:39 <@siosm:matrix.org> or are there any opposition to publishing an hetzner image?
2024-07-17 17:30:48 <@jlebon:fedora.im> dustymabe: yeah, possibly. would require root.
2024-07-17 17:31:13 <@jlebon:fedora.im> +1 from me
2024-07-17 17:31:16 <@dustymabe:matrix.org> no opposition to publishing a hetzner image here, but... I feel the same way about that Oracle Cloud PR that was just linked to
2024-07-17 17:31:41 <@dustymabe:matrix.org> either publish an official image OR bundle restamping into `coreos-installer`
2024-07-17 17:32:24 <@jbtrystram:matrix.org> +1
2024-07-17 17:32:41 <@jlebon:fedora.im> i would phrase it as "provide a better UX for restamping" instead of assuming coreos-installer
2024-07-17 17:33:00 <@dustymabe:matrix.org> Jonathan Lebon: yes, that's what I meant... I was jumping to a potential solution
2024-07-17 17:33:34 <@jlebon:fedora.im> i feel like just having the script live in the container would be a good compromise
2024-07-17 17:34:06 <@siosm:matrix.org> We're overtime. Let's vote on this next week?
2024-07-17 17:34:24 <@gurssing:matrix.org> I'll keep the meeting tag on this issue then.
2024-07-17 17:34:34 <@gurssing:matrix.org> Skipping open floor and ending the meeting as we are out of time.
2024-07-17 17:34:57 <@siosm:matrix.org> maybe a quick 1 minute open floor?
2024-07-17 17:35:02 <@gurssing:matrix.org> Sure
2024-07-17 17:35:11 <@gurssing:matrix.org> !topic Open Floor
2024-07-17 17:36:21 <@gurssing:matrix.org> !endmeeting