<@jlebon:fedora.im>
16:30:49
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:30:49
Meeting started at 2024-02-07 16:30:49 UTC
<@meetbot:fedora.im>
16:30:49
The Meeting name is 'fedora_coreos_meeting'
<@jlebon:fedora.im>
16:30:54
!topic roll call
<@dustymabe:matrix.org>
16:30:59
!hi
<@zodbot:fedora.im>
16:31:02
Dusty Mabe (dustymabe) - he / him / his
<@jmarrero:matrix.org>
16:32:25
!hi
<@zodbot:fedora.im>
16:32:27
No Fedora Accounts users have the @jmarrero:matrix.org Matrix Account defined
<@jmarrero:matrix.org>
16:32:35
I think I still need to link this thing.
<@spresti:fedora.im>
16:33:16
!hi
<@zodbot:fedora.im>
16:33:17
Steven Presti (spresti)
<@gurssing:matrix.org>
16:33:29
!hi gursewak
<@zodbot:fedora.im>
16:33:30
Gursewak Singh (gursewak)
<@jbtrystram:matrix.org>
16:34:00
!hi jbtrystram
<@zodbot:fedora.im>
16:34:02
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@jlebon:fedora.im>
16:34:14
alrighty, let's start!
<@jlebon:fedora.im>
16:34:25
!topic Action items from last meeting
<@jlebon:fedora.im>
16:34:33
travier open an issue to track hardening our units
<@jlebon:fedora.im>
16:34:52
i don't think i've seen that issue come out
<@jlebon:fedora.im>
16:35:07
oh wait, nvm
!info travier filed https://github.com/coreos/fedora-coreos-tracker/issues/1662
<@jlebon:fedora.im>
16:35:17
!info travier field https://github.com/coreos/fedora-coreos-tracker/issues/1662
<@jlebon:fedora.im>
16:35:59
ok, let's move on to the first
<@jlebon:fedora.im>
16:36:02
ok, let's move on to the first topic
<@jlebon:fedora.im>
16:36:03
!topic Consider zstd for compression of shipped artifacts
<@jlebon:fedora.im>
16:36:09
dustymabe: want to intro this one?
<@spresti:fedora.im>
16:36:42
!link https://github.com/coreos/fedora-coreos-tracker/issues/1660
<@dustymabe:matrix.org>
16:37:30
yep. this came up recently in a conversation with @baude - he was noticing it was taking a long time to decompress the xz applehv image and wondering on how to improve it. We switched to compress/decompress with zstd and I was thoroughly impressed by the performance (at least on his M2 mac)
<@dustymabe:matrix.org>
16:37:43
so I decided to dig in a little bit further and this ticket is the result
<@dustymabe:matrix.org>
16:38:23
one pain point we have (on the infra/releng side) is how long our pipelines take to run. Compression os a big part of that so I was interested to know what some real world comparison would look like
<@dustymabe:matrix.org>
16:38:42
There are some results in the ticket in a table
<@dustymabe:matrix.org>
16:39:39
It looks like level 10 zstd compression may give us a reasonable speedup versus extra storage cost ratio
<@spresti:fedora.im>
16:40:30
So 90% faster but 6% more space?
<@dustymabe:matrix.org>
16:40:33
One other thing I was proposing was to switch over our rawhide stream to use zstd compression as a way to flesh out the investigation a little more: https://github.com/coreos/fedora-coreos-config/pull/2840
<@jlebon:fedora.im>
16:40:33
wait, i'm confused. the impetus for this was to make decompression for users faster, but in testing the difference doesn't seem drastic
<@dustymabe:matrix.org>
16:40:59
which should give us some more data on actual speedup versus size
<@jlebon:fedora.im>
16:41:26
e.g. in https://github.com/coreos/fedora-coreos-tracker/issues/1660#issue-2109301823 we have 13s vs 9s. and the different levels doesn't seem to affect that much
<@jlebon:fedora.im>
16:41:34
how come baude saw a drastic differnce?
<@jlebon:fedora.im>
16:41:36
how come baude saw a drastic difference?
<@jbtrystram:matrix.org>
16:41:59
maybe he have some hardware zstd acceleration ?
<@dustymabe:matrix.org>
16:42:17
Jonathan Lebon: correct. That's on my one system. My hard drives are pretty fast. and it's also when I was doing the compress and decompress back to back (maybe local caching) helped?
<@dustymabe:matrix.org>
16:42:48
jbtrystram: maybe?
<@jlebon:fedora.im>
16:43:00
jbtrystram: possibly. though that's not something we should assume most people have
<@jlebon:fedora.im>
16:43:27
so i think the discussion mostly is now about the pipeline speedup aspect
<@jlebon:fedora.im>
16:43:33
so i think the discussion mostly is now about the pipeline speedup aspect IIUC
<@dustymabe:matrix.org>
16:43:42
I'm interested in others experience/input (i.e. try it out on your system and report performance results)
<@ravanelli:matrix.org>
16:43:46
.hi
<@jlebon:fedora.im>
16:44:16
dustymabe: makes sense. i can give it a try too
<@jbtrystram:matrix.org>
16:44:59
I'll try and report back as well
<@dustymabe:matrix.org>
16:45:04
Jonathan Lebon: that's a big part of it, yes. If we could speed up our pipelines compression by a significant factor it would help us out. The `rawhide` test should at least give us some data
<@dustymabe:matrix.org>
16:45:47
jbtrystram: Jonathan Lebon if you want to play around with the level of compression you can just modify cmd-compress like I'm doing here: https://github.com/coreos/coreos-assembler/pull/3721
<@dustymabe:matrix.org>
16:46:34
one other thing we have to consider in this is compatibility - i.e. is it reasonable to assume that zstd is as ubiquitous as xz now? Also, does coreos-installer support it?
<@dustymabe:matrix.org>
16:49:29
That's mostly it from my side
<@jlebon:fedora.im>
16:50:41
i feel conflicted about this :)
<@dustymabe:matrix.org>
16:52:31
That's fair :) - I think we're still just information gathering
<@jlebon:fedora.im>
16:52:39
i think 15% smaller images is worth spending 30 mins rather than e.g. 5 mins in the pipeline. most of the time, we don't actually care how long pipeline builds take. except during prod builds where we have to retry on e.g. test flakes. it'd be great to have the pipeline go faster, but normally by the time we get to archive step, we're just about to upload to S3 and any failure after that are e.g. cloud tests and upgrade tests which can be retried.
<@jlebon:fedora.im>
16:53:32
from a user's perspective going from 661M to 757M for the qemu image for example is not great optics
<@jlebon:fedora.im>
16:54:11
if decompression was drastically different, then I could see an argument
<@jlebon:fedora.im>
16:54:16
if decompression was drastically different, then I could see an argument for bettering UX
<@dustymabe:matrix.org>
16:54:25
but the uncompressed size is the same? that's just the bandwidth that gets used (which yeah, not great)
<@dustymabe:matrix.org>
16:55:16
right. I'm interested in others experience with the decompression speed. I also need to try it on an image that I didn't just compress on my same system and see if it makes a difference
<@jlebon:fedora.im>
16:56:33
makes sense, yeah. would be good to have more info on that
<@jbtrystram:matrix.org>
16:57:54
looking around, it appears zstd offer faster decompression speed overall (for similar compression ratio), but I don't know enough about our storage costs to weight in :)
<@jbtrystram:matrix.org>
16:58:06
worth noting : since Fedora 33, the filesystem is compressed by default with zstd.
<@dustymabe:matrix.org>
16:58:41
jbtrystram: you mean on BTRFS
<@jlebon:fedora.im>
16:59:56
!info we would like to gather more info on decompression speed and invite people to try out the zstd vs xz paths on their systems and report results.
<@jlebon:fedora.im>
17:00:03
dustymabe: is that accurate? ^
<@jbtrystram:matrix.org>
17:00:18
looks like RPMs where switched to zstd as well in F31 : https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
<@dustymabe:matrix.org>
17:02:06
Jonathan Lebon: yeah. seems accurate. Maybe we can agree to try it out in rawhide to give people real world artifacts to pull and decompress and also flesh out any potential bugs (like coreos-installer support)
<@jlebon:fedora.im>
17:02:30
your over concern about compatibility is a good one. i'm not sure how to evaluate it.
maybe we could check how far back are zstd tools available -- e.g. is it in CentOS 7?
<@jlebon:fedora.im>
17:02:45
(not meant to be answered here)
<@jlebon:fedora.im>
17:03:30
coreos-installer has zstd support for the initramfs at least
<@jlebon:fedora.im>
17:03:57
for `coreos-installer download`, it wouldn't be hard to test locally. you point it at the image using `--image-url ...` and then pass `--decompress`
<@dustymabe:matrix.org>
17:04:18
+1
<@jlebon:fedora.im>
17:04:50
briefly looking at the code, it doesn't look like it
<@jlebon:fedora.im>
17:05:12
https://github.com/coreos/coreos-installer/blob/ed985e44d3ce343d9301ffa00fea7e590547c962/src/download.rs#L114
<@jlebon:fedora.im>
17:06:04
!info this would require adding zstd image decompression support to coreos-installer
<@dustymabe:matrix.org>
17:06:12
it's OK we can fix it if we decide it's useful enough
<@mnguyen:fedora.im>
17:06:42
I vaguely remember two things baude mentioned in our meeting
<@jlebon:fedora.im>
17:06:47
right, let's evaluate first if it's worth it, and then we can figure out the missing pieces
<@mnguyen:fedora.im>
17:07:03
one is xz isn't included by default on mac?
<@music:fedora.im>
17:07:35
the standard `zstd` libraries and tools also support multi-threaded compression, while `xz` libraries don’t
<@dustymabe:matrix.org>
17:08:26
I think we can move on to another topic
<@jlebon:fedora.im>
17:08:45
music: xz supports multithreaded compression, no? we pass `-T<N>` in our pipelines
<@jlebon:fedora.im>
17:08:50
dustymabe: ack, sounds good
<@jlebon:fedora.im>
17:09:06
!topic tracker: Fedora 40 changes considerations
<@jlebon:fedora.im>
17:09:12
!link https://github.com/coreos/fedora-coreos-tracker/issues/1626
<@jlebon:fedora.im>
17:09:55
looks like there's a few more that were added
<@music:fedora.im>
17:10:35
Hmm, you’re right. Added in 2014, apparently, and I just never noticed.
<@jlebon:fedora.im>
17:10:40
!topic Update the system JDK in Fedora from java-17-openjdk to java-21-openjdk.
<@jlebon:fedora.im>
17:10:43
!link https://fedoraproject.org/wiki/Changes/Java21
<@jlebon:fedora.im>
17:10:51
we don't ship java
<@jlebon:fedora.im>
17:11:12
!topic ROCm 6 Release
<@jlebon:fedora.im>
17:11:16
!link https://fedoraproject.org/wiki/Changes/ROCm6Release
<@jlebon:fedora.im>
17:12:11
cool, TIL.
we don't ship this.
<@jlebon:fedora.im>
17:12:22
!topic PyTorch Release
<@jlebon:fedora.im>
17:12:25
!link https://fedoraproject.org/wiki/Changes/PyTorchRelese
<@jlebon:fedora.im>
17:12:58
we don't ship PyTorch
<@jlebon:fedora.im>
17:13:04
and similarly for the last one (iotop)
<@spresti:fedora.im>
17:13:32
Should we info it?
<@jlebon:fedora.im>
17:13:56
!info none of the new changes should affect FCOS directly. we don't ship any of these packages by default.
<@spresti:fedora.im>
17:14:21
Thank you! (sorry was not sure if we should)
<@jlebon:fedora.im>
17:14:43
np! thanks for reminding me. presumably i should've done it for each one
anyway, shall we move on to the last topic?
<@jlebon:fedora.im>
17:15:47
!topic highlights from FCOS 2023
<@jlebon:fedora.im>
17:15:52
no links for this one
<@dustymabe:matrix.org>
17:16:41
yeah. mainly one of our team members wants to do a presentation and include some highlights from FCOS for 2023
<@dustymabe:matrix.org>
17:16:55
I figure maybe we can sift through https://fedoraproject.org/coreos/release-notes?arch=x86_64&stream=stable ?
<@dustymabe:matrix.org>
17:17:05
or source inspiration from somewhere else
<@dustymabe:matrix.org>
17:17:16
I can probably try to pull and look at the countme stats again
<@dustymabe:matrix.org>
17:18:23
I feel like most of what we have been up to is maintenance, new platform support, and "behind the scenes" type stuff like OSBuild and the Container Native stuff
<@dustymabe:matrix.org>
17:18:55
From the release notes here's what jumped out to me:
```
Added Apple Hypervisor support
Support root on iSCSI (non-HBA)
Platform Request: Microsoft Hyper-V
Added support for ppc64le
GCP added aarch64 image
Kubevirt
Included podman quadlet functionality
```
<@jlebon:fedora.im>
17:18:59
yeah definitely. which could be worth highlighting too depending on the crowd
<@dustymabe:matrix.org>
17:19:10
not sure if all of those are worthy or not
<@jlebon:fedora.im>
17:19:42
maybe 7 is too small?
<@jlebon:fedora.im>
17:21:17
(going over the list too)
<@jlebon:fedora.im>
17:22:47
nmstate enablement
Fedora 38 rebase
ISO from RAM work maybe?
<@dustymabe:matrix.org>
17:23:02
WFM - depends on the audience probably
<@jlebon:fedora.im>
17:23:16
i'd add the osbuild work too honestly and let them decide if it makes sense to talk about that
<@dustymabe:matrix.org>
17:23:24
yeah
<@jlebon:fedora.im>
17:24:09
seems like we have a good list :)
<@jlebon:fedora.im>
17:24:29
should move to open floor?
<@jlebon:fedora.im>
17:25:00
!topic Open Floor
<@jlebon:fedora.im>
17:25:28
anything anyone wants to bring up?
<@dustymabe:matrix.org>
17:25:43
nothing from me today other than I might not make meetings for a few weeks later this month/early next month
<@spresti:fedora.im>
17:26:29
Been spending a bit of time working on the fcos action, the action-items are not getting matched right still.
Soon should be fixed https://github.com/coreos/fcos-meeting-action/pull/69
<@dustymabe:matrix.org>
17:26:42
should we remind the next person in the rotation?
<@jbtrystram:matrix.org>
17:27:04
that's me i think
<@spresti:fedora.im>
17:27:28
I hope to have it fixed by then ;'(
<@jlebon:fedora.im>
17:27:45
jbtrystram: consider yourself reminded :)
<@jmarrero:matrix.org>
17:27:49
on https://github.com/coreos/fedora-coreos-tracker/issues/1647 We still think this is related to bootc somehow? I am not sure from the conversation how is this not just a kernel issue.
<@dustymabe:matrix.org>
17:28:59
jmarrero: not bootc, but something ostree/bootc related maybe? most other variants of Fedora aren't reporting issues only us (bootc and coreos)
<@dustymabe:matrix.org>
17:29:26
I'd love for someone to dig in further
<@jmarrero:matrix.org>
17:29:55
OK I will bring it to my meetings with the team and Colin and see what we find.
<@dustymabe:matrix.org>
17:30:28
jmarrero: i reserve the right to be wrong about the problem being bootc/ostree related :)
<@jlebon:fedora.im>
17:30:33
probably not bootc related if we're hitting it
<@dustymabe:matrix.org>
17:30:42
but from the current information that's where it leads
<@jlebon:fedora.im>
17:30:57
this looks like some grub config issue maybe?
<@dustymabe:matrix.org>
17:31:25
Jonathan Lebon: yeah. i'm mostly pointing at "something in our stack"
<@jmarrero:matrix.org>
17:31:25
It's weird because it's only aarch64 right?
<@dustymabe:matrix.org>
17:31:39
correct
<@dustymabe:matrix.org>
17:32:17
IIRC this started happening before we enabled OSBuild so it shouldn't be OSBuild related
<@jmarrero:matrix.org>
17:32:25
I am not sure how different arm and 86 grub configs are.
<@dustymabe:matrix.org>
17:32:30
IIRC this started happening before we enabled OSBuild in rawhide so it shouldn't be OSBuild related
<@jmarrero:matrix.org>
17:32:42
I am not sure how different arm and x86 grub configs are.
<@dustymabe:matrix.org>
17:33:18
jmarrero: maybe we should just start with "can you reproduce the problem" and go from there?
<@jmarrero:matrix.org>
17:33:42
Yeah, I'll dig.
<@jlebon:fedora.im>
17:33:46
Peter talks about changes that were done in qemu to work around this. would be good to dig into that too
<@dustymabe:matrix.org>
17:34:31
yeah I don't really understand that point from him. if that were true wouldn't the other variants be having trouble in their testing?
<@dustymabe:matrix.org>
17:34:34
i.e. openQA?
<@dustymabe:matrix.org>
17:34:42
which I assume uses qemu
<@dustymabe:matrix.org>
17:34:59
anyway we can probably close out the meeting and take this to the CoreOS channel
<@jlebon:fedora.im>
17:35:02
i would assume so. except our grub configs are very unique to us currently
<@jlebon:fedora.im>
17:35:21
so might be a combination of bug in QEMU + specific grub serial config or something
<@jlebon:fedora.im>
17:35:25
anyway, agreed
<@jlebon:fedora.im>
17:35:39
anything else for open floor?
<@jbtrystram:matrix.org>
17:36:07
I have some findings to share about the kdump SSH issue but not sure if this is the right place ?
<@jlebon:fedora.im>
17:36:24
jbtrystram: let's take that to the FCOS channel too :)
<@jlebon:fedora.im>
17:36:49
closing the meeting in 30 seconds if quiet
<@jlebon:fedora.im>
17:37:21
!endmeeting