<@gurssing:matrix.org>
16:31:09
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:31:12
Meeting started at 2024-07-17 16:31:09 UTC
<@meetbot:fedora.im>
16:31:12
The Meeting name is 'fedora_coreos_meeting'
<@gurssing:matrix.org>
16:31:32
!topic roll call
<@siosm:matrix.org>
16:31:59
!hi
<@zodbot:fedora.im>
16:32:01
Timothée Ravier (siosm) - he / him / his
<@hricky:fedora.im>
16:32:11
!hi
<@jlebon:fedora.im>
16:32:11
!hi
<@zodbot:fedora.im>
16:32:12
None (jlebon)
<@zodbot:fedora.im>
16:32:12
Hristo Marinov (hricky) - he / him / his
<@marmijo:fedora.im>
16:32:23
!hi
<@zodbot:fedora.im>
16:32:24
Michael Armijo (marmijo)
<@jbtrystram:matrix.org>
16:32:43
!hi
<@zodbot:fedora.im>
16:32:46
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@gurssing:matrix.org>
16:32:53
!hi gursewak
<@zodbot:fedora.im>
16:32:56
Gursewak Singh (gursewak)
<@ydesouza:fedora.im>
16:33:37
!hi
<@zodbot:fedora.im>
16:33:38
Yasmin Valim de Souza (ydesouza)
<@aaradhak:matrix.org>
16:34:14
!hi aaradhak
<@zodbot:fedora.im>
16:34:16
Aashish Radhakrishnan (aaradhak)
<@ravanelli:matrix.org>
16:34:27
!hi ravanelli
<@zodbot:fedora.im>
16:34:29
Renata Ravanelli (ravanelli)
<@gurssing:matrix.org>
16:34:33
!topic Action items from last meeting
<@gurssing:matrix.org>
16:34:56
No actions items that I can see from last meeting notes.
<@gurssing:matrix.org>
16:35:14
Moving onto topics for today.
<@gurssing:matrix.org>
16:35:21
!topic Different partition layout after re-provisioning between version 39 and 40
<@gurssing:matrix.org>
16:35:27
!link https://github.com/coreos/fedora-coreos-tracker/issues/1745
<@dustymabe:matrix.org>
16:36:37
!hi
<@zodbot:fedora.im>
16:36:39
Dusty Mabe (dustymabe) - he / him / his
<@gurssing:matrix.org>
16:37:40
Jonathan Lebon: Do you want to introduce this issue?
<@jlebon:fedora.im>
16:37:57
sure
<@jlebon:fedora.im>
16:40:49
so in f40, our disk images moved from being partitioned by sgdisk (in cosa) to sfdisk (in osbuild). and while the resulting table is mostly the same, there are some tiny differences that have surfaced to users. this is one of them.
<@jlebon:fedora.im>
16:40:49
basically, the TL;DR there is that the automatic partition growing we do is now growing the last partition to a slightly different size, and so anyone expecting the exact same result (e.g. like in the data persistence case) will notice this
<@jlebon:fedora.im>
16:42:08
all the nitty-gritty details are in the last two comments there, and i initially thought this was worth sending a coreos-status post for it, but i'm thinking it may not yet be
<@jlebon:fedora.im>
16:43:04
the good news is that data loss is not a concern there at least in theory. i've provided steps also to access their data for anyone hitting this
<@jlebon:fedora.im>
16:43:34
but... at first glance it'll certainly _appear_ like data loss
<@siosm:matrix.org>
16:43:58
that's what worries me
<@siosm:matrix.org>
16:44:19
I don't think a coreos-status post would hurt.
<@siosm:matrix.org>
16:44:51
Let's re-phrase that positively: A coreos-status post would probably be nice
<@jlebon:fedora.im>
16:45:20
certainly not against it. it was just not clear how prevalent an issue this really is
<@siosm:matrix.org>
16:45:23
Should we make a test for that?
<@jlebon:fedora.im>
16:45:45
anyone have countme data offhand for how many people are still on < f40?
<@jlebon:fedora.im>
16:46:01
we definitely should have a data persistence test
<@hricky:fedora.im>
16:46:21
When I tested it, there was no data loss on my end.
<@siosm:matrix.org>
16:48:13
It's really a shame that XFS can't be shrunk
<@jlebon:fedora.im>
16:48:16
right. all the data is still there. changing a partition entry in the table doesn't change its contents. and most (all?) filesystems are smart enough to error out early on during mounting when it sees that its container appears truncated
<@jlebon:fedora.im>
16:50:44
anyway, personally lean towards no coreos-status post unless e.g. at least one other person shows up on that tracker issue. but ok with sending one too.
<@siosm:matrix.org>
16:50:56
I don't know what to do. If someone wants to draft a coreos-status post that would be nice. Otherwise there is probably nothing we can really do to fix this
<@jlebon:fedora.im>
16:51:56
filed https://github.com/coreos/fedora-coreos-tracker/issues/1763
<@jlebon:fedora.im>
16:52:06
filed https://github.com/coreos/fedora-coreos-tracker/issues/1763 for the tests part
<@jlebon:fedora.im>
16:52:37
i guess let's move on?
<@siosm:matrix.org>
16:52:45
proposed: we move on and do nothing for now
<@gurssing:matrix.org>
16:52:53
+1
<@dustymabe:matrix.org>
16:53:00
+1
<@marmijo:fedora.im>
16:53:04
+1
<@aaradhak:matrix.org>
16:53:07
+1
<@dustymabe:matrix.org>
16:53:08
thanks for investigating Jonathan Lebon
<@ravanelli:matrix.org>
16:53:09
+1
<@jlebon:fedora.im>
16:53:15
ack
<@gurssing:matrix.org>
16:53:26
Me move on then.
<@gurssing:matrix.org>
16:53:29
!topic tracker: Fedora 41 changes considerations
<@gurssing:matrix.org>
16:53:35
!link https://github.com/coreos/fedora-coreos-tracker/issues/1714
<@mnguyen:fedora.im>
16:54:20
!hi
<@zodbot:fedora.im>
16:54:21
Michael Nguyen (mnguyen)
<@siosm:matrix.org>
16:54:58
109 (https://github.com/coreos/fedora-coreos-tracker/issues/1759) will be deferred to F42 as we found issues with rpm-ostree composes
<@siosm:matrix.org>
16:55:33
And it was too short before the mass rebuild to fix them. We'll retry once F41 is branched
<@siosm:matrix.org>
16:56:36
So we should look at 126 & 127?
<@jlebon:fedora.im>
16:56:40
126: we already don't ship ifcfg in FCOS. will be interested to see once that change lands in CentOS Stream/RHEL.
<@jlebon:fedora.im>
16:56:40
127. should be transparent to us
<@siosm:matrix.org>
16:57:30
213, 214 & 215 should not impact us as well
<@siosm:matrix.org>
16:58:03
Thanks marmijo for running the script and updating the issue!
<@marmijo:fedora.im>
16:58:09
great! seems like it's a pretty short list of impacts to FCOS
<@marmijo:fedora.im>
16:58:18
You're welcome
<@siosm:matrix.org>
16:59:12
213, 214 & 215 should not impact us as well (packages not included in FCOS)
<@jlebon:fedora.im>
16:59:50
gursewak: i think we can move on
<@siosm:matrix.org>
17:00:09
Maybe to https://github.com/coreos/fedora-coreos-tracker/issues/1324? I added it right before the meeting
<@gurssing:matrix.org>
17:00:26
Got it.
<@gurssing:matrix.org>
17:00:36
!topic Platform Request: Hetzner
<@gurssing:matrix.org>
17:00:44
!link https://github.com/coreos/fedora-coreos-tracker/issues/1324
<@siosm:matrix.org>
17:01:44
I've been looking at "quick wins" for platforms were are close to supporting them but a few bits are missing.
<@siosm:matrix.org>
17:01:48
Hetznner is one of them
<@siosm:matrix.org>
17:02:29
There is support in Afterburn & Ignition, and Flatcar are making images for it. I wrote https://github.com/coreos/fedora-coreos-docs/pull/654 based on their docs
<@siosm:matrix.org>
17:03:07
What remains on the Fedora CoreOS side is to publish an image with the right platform ID.
<@siosm:matrix.org>
17:04:08
The provisioning steps on Hetzner are a bit weird but it's not something we can really fix. The images are however "clean" in that they are installed offline and never booted so there should not be any issue.
<@siosm:matrix.org>
17:05:15
So the question is: Should we build & publish Hetzner images from our pipeline? Are the steps in the docs PR acceptable for our docs standards?
<@dustymabe:matrix.org>
17:06:01
I think I have an opinion on this
<@jlebon:fedora.im>
17:06:01
having read the docs, there are a lot of caveats and tricky bits there that I think makes publishing images for it not slightly but much better UX-wise
<@siosm:matrix.org>
17:06:03
From my perspective, it's better to have some docs with a lot of warnings rather than no docs, and leave out users "in the cold" / looking at other options.
<@dustymabe:matrix.org>
17:06:49
<@dustymabe:matrix.org>
17:06:49
Jonathan Lebon mind saying that a different way. it confused me
<@dustymabe:matrix.org>
17:06:49
> I think makes publishing images for it not slightly but much better UX-wise
<@jlebon:fedora.im>
17:06:52
i like the idea of "restamp and upload yourself" for some platforms, but for this one it really looks way more painful than that
<@jlebon:fedora.im>
17:07:12
dustymabe: does ^ help?
<@dustymabe:matrix.org>
17:07:20
I think so
<@dustymabe:matrix.org>
17:07:57
so here's my piece:
<@dustymabe:matrix.org>
17:07:57
<@dustymabe:matrix.org>
17:07:57
I think we should either produce an official image OR make some standard way for people to restamp an image. i.e. `coreos-installer disk-image-customize --platform=hetzner`
<@jlebon:fedora.im>
17:08:15
i should've said "it makes publishing images for it not only _slightly_ better, but _much_ better UX-wise"
<@dustymabe:matrix.org>
17:09:10
for clouds where we foresee wanting to be in their official image list (i.e. the cloud provider takes our image and makes it available OR there is a way for us to upload and publish them) then obviously we should create and produce that image
<@siosm:matrix.org>
17:09:26
The main benefit here is that it avoids a download / restamp / upload step. Everything happens directly in the cloud and is thus much faster if we have an image ready to use
<@jlebon:fedora.im>
17:09:29
the restamp part definitely would be nice to have better UX, but I don't think that helps a lot in this case AIUI
<@siosm:matrix.org>
17:11:00
yes, that's the issue. It makes it simpler (no need for guestfish, the script, etc.) but that does not remove the download/upload part
<@dustymabe:matrix.org>
17:11:35
if we produce an official image does that remove the download/upload part?
<@siosm:matrix.org>
17:11:41
I don't know what Hetzner has planned for "custom images"
<@siosm:matrix.org>
17:11:49
yes
<@dustymabe:matrix.org>
17:12:11
how? does their tool accept a URL (i.e. can download directly from our s3 bucket)?
<@siosm:matrix.org>
17:12:35
you can download a bz2 compressed RAW image from HTTPS
<@siosm:matrix.org>
17:12:41
you can download a bz2 compressed RAW image from an HTTPS URL
<@siosm:matrix.org>
17:12:57
https://github.com/apricote/hcloud-upload-image#usage
<@siosm:matrix.org>
17:13:23
you directly create a snapshot in the hetzner cloud from the image
<@dustymabe:matrix.org>
17:13:48
right, but is that tool downloading it locally and then uploading it to hetzner? or is it telling hetzner to go retrieve it?
<@dustymabe:matrix.org>
17:13:53
sorry for all the questions
<@siosm:matrix.org>
17:14:01
it's telling the server to retrieve it
<@siosm:matrix.org>
17:14:08
(those are good questions)
<@dustymabe:matrix.org>
17:14:21
cool - then you are right :), sorry had to ask because it can be confusing
<@jlebon:fedora.im>
17:14:30
see also https://github.com/apricote/hcloud-upload-image?tab=readme-ov-file#about
<@dustymabe:matrix.org>
17:15:25
wow that's pretty unfortunate
<@siosm:matrix.org>
17:15:27
> 4. Download the disk image from within the rescue system
<@dustymabe:matrix.org>
17:15:37
you'd think a large cloud would have that all figured out by now
<@ravanelli:matrix.org>
17:15:46
How hard would be for us to produce an official image?
<@dustymabe:matrix.org>
17:16:04
Renata Ravanelli: not hard at all, we're just deciding if we should
<@jlebon:fedora.im>
17:16:29
travier: so even if we built images, you'd still need the snapshotting dance, right?
<@siosm:matrix.org>
17:16:52
Jonathan Lebon: yes
<@siosm:matrix.org>
17:17:03
but it would be much easier/faster
<@siosm:matrix.org>
17:19:05
(note: we are almost at time)
<@jbtrystram:matrix.org>
17:19:07
I guess now we know why hetzner is cheap :)
<@ravanelli:matrix.org>
17:20:03
Yeah, looking at the PR/docs in how to get it working, seems a lot for the users to handle/know, using some non official tools. I wonder if we would get a lot of issues/questions coming to us from users trying to get it working and failing.
<@jlebon:fedora.im>
17:20:04
seems to match with https://docs.hetzner.com/robot/dedicated-server/operating-systems/installing-custom-images as well. that's pretty sad UX :(
<@jbtrystram:matrix.org>
17:20:53
Somewhat related : it would be nice to spend the time to look at the platforms requests issues to make a list of all the targets we could address with a `coreos-installer customize-iso --platform` to have an idea of how many birds the stone would reach
<@jlebon:fedora.im>
17:21:43
kinda ambivalent on producing official images in that case. i thought we would've been able to upload the snapshot ourselves. to me, the restamping is not really the most painful part of that process
<@siosm:matrix.org>
17:22:13
I have hetzner & oracle cloud as current "target" for the restamp flow
<@siosm:matrix.org>
17:22:32
maybe scaleway
<@jlebon:fedora.im>
17:22:34
but definitely nice to save a download/upload trip for users. so cool with it too
<@dustymabe:matrix.org>
17:22:39
Jonathan Lebon: the `hcloud-upload-image` tool does it all for you, though?
<@siosm:matrix.org>
17:24:07
If we went the packer (no longer open source) route, we could use an existing image instead and restamp it live.
<@siosm:matrix.org>
17:24:23
If we went with the `packer` option (no longer open source) route, we could use an existing image instead and restamp it live.
<@siosm:matrix.org>
17:24:33
but that's almost more complicated
<@jlebon:fedora.im>
17:25:10
dustymabe: right, but that adds time and having to build/installer a "fourth-party" tool has a higher UX cost IMO
<@jlebon:fedora.im>
17:25:16
dustymabe: right, but that adds time and having to build/install a "fourth-party" tool has a higher UX cost IMO
<@siosm:matrix.org>
17:25:39
I downloaded the provided release binaries
<@dustymabe:matrix.org>
17:25:50
IMO the tool is just covering a gap in hetzner's UX, which I would expect them to close in the future
<@siosm:matrix.org>
17:25:54
I downloaded the provided release binaries for hcloud-upload-image
<@jlebon:fedora.im>
17:26:20
yup, agree. and not blocking on that. just saying that i think that's the larger UX gap. not the restamping part
<@dustymabe:matrix.org>
17:26:25
if they did that then the user would take our image, provide it to `hcloud` CLI and get an cloud image out of it
<@dustymabe:matrix.org>
17:26:39
yeah, the restamping is the part we control though
<@dustymabe:matrix.org>
17:27:01
"run this guestfish script" isn't a great UX (for the part we control)
<@dustymabe:matrix.org>
17:27:26
so to me either we give them the whole image they can use or provide them a better UX for restamping (i.e. via coreos-installer)
<@jlebon:fedora.im>
17:28:00
or put it in a container: https://github.com/coreos/fedora-coreos-docs/pull/652#discussion_r1679953285
<@jlebon:fedora.im>
17:28:10
or... i guess put it in the coreos-installer container
<@jlebon:fedora.im>
17:28:24
so it's not yet another image to maintain
<@dustymabe:matrix.org>
17:29:05
I don't really understand why coreos-installer can't just pick up this functionality pretty easy
<@siosm:matrix.org>
17:29:35
We are close to at time and there is not urgency for this issue so many we should pick it up next week?
<@dustymabe:matrix.org>
17:29:36
cp disk image to new file, mount raw disk image, edit platform ID, unmount
<@gurssing:matrix.org>
17:29:38
Reminder that we are almost out of time:)
<@siosm:matrix.org>
17:29:42
We are close to time and there is not urgency for this issue so many we should pick it up next week?
<@siosm:matrix.org>
17:30:18
We are close to time and there is no urgency for this issue so many we should pick it up next week?
<@siosm:matrix.org>
17:30:39
or are there any opposition to publishing an hetzner image?
<@jlebon:fedora.im>
17:30:48
dustymabe: yeah, possibly. would require root.
<@jlebon:fedora.im>
17:31:13
+1 from me
<@dustymabe:matrix.org>
17:31:16
no opposition to publishing a hetzner image here, but... I feel the same way about that Oracle Cloud PR that was just linked to
<@dustymabe:matrix.org>
17:31:41
either publish an official image OR bundle restamping into `coreos-installer`
<@jbtrystram:matrix.org>
17:32:24
+1
<@jlebon:fedora.im>
17:32:41
i would phrase it as "provide a better UX for restamping" instead of assuming coreos-installer
<@dustymabe:matrix.org>
17:33:00
Jonathan Lebon: yes, that's what I meant... I was jumping to a potential solution
<@jlebon:fedora.im>
17:33:34
i feel like just having the script live in the container would be a good compromise
<@siosm:matrix.org>
17:34:06
We're overtime. Let's vote on this next week?
<@gurssing:matrix.org>
17:34:24
I'll keep the meeting tag on this issue then.
<@gurssing:matrix.org>
17:34:34
Skipping open floor and ending the meeting as we are out of time.
<@siosm:matrix.org>
17:34:57
maybe a quick 1 minute open floor?
<@gurssing:matrix.org>
17:35:02
Sure
<@gurssing:matrix.org>
17:35:11
!topic Open Floor
<@gurssing:matrix.org>
17:36:21
!endmeeting