<@siosm:matrix.org>
16:30:17
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:30:19
Meeting started at 2025-05-14 16:30:17 UTC
<@meetbot:fedora.im>
16:30:19
The Meeting name is 'fedora_coreos_meeting'
<@siosm:matrix.org>
16:30:54
!topic roll call
<@dustymabe:matrix.org>
16:31:26
!hi
<@zodbot:fedora.im>
16:31:28
Dusty Mabe (dustymabe) - he / him / his
<@tlbueno:fedora.im>
16:31:31
!hi
<@zodbot:fedora.im>
16:31:32
Tiago Bueno (tlbueno) - he / him / his
<@marmijo:fedora.im>
16:31:36
!hi
<@zodbot:fedora.im>
16:31:37
Michael Armijo (marmijo)
<@mnguyen:fedora.im>
16:31:53
!hi mnguyen
<@zodbot:fedora.im>
16:31:54
Michael Nguyen (mnguyen)
<@ravanelli:matrix.org>
16:32:00
!hi ravanelli
<@hricky:fedora.im>
16:32:00
!hi
<@zodbot:fedora.im>
16:32:01
Renata Ravanelli (ravanelli)
<@zodbot:fedora.im>
16:32:01
Hristo Marinov (hricky) - he / him / his
<@jlebon:fedora.im>
16:33:02
!hi
<@zodbot:fedora.im>
16:33:03
None (jlebon)
<@jdoss:beeper.com>
16:33:44
!hi
<@zodbot:fedora.im>
16:33:46
Joe Doss (jdoss)
<@jbtrystram:matrix.org>
16:34:54
!hi
<@zodbot:fedora.im>
16:34:56
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@siosm:matrix.org>
16:35:19
!topic Action items from last meeting
<@siosm:matrix.org>
16:35:28
!link https://discussion.fedoraproject.org/t/fedora-coreos-community-meeting-minutes-2025-05-07/152389
<@siosm:matrix.org>
16:35:49
I don't see any action items from the last meeting
<@siosm:matrix.org>
16:35:59
!topic Review Fedora 43 Release Schedule
<@siosm:matrix.org>
16:36:05
!link https://fedorapeople.org/groups/schedule/f-43/f-43-key-tasks.html
<@siosm:matrix.org>
16:36:36
I don't think we have anything specific for this topic right now. We'll keep reviewing the incoming changes
<@aaradhak:matrix.org>
16:36:52
!hi aaradhak
<@zodbot:fedora.im>
16:36:55
Aashish Radhakrishnan (aaradhak)
<@dustymabe:matrix.org>
16:37:36
If we have any changes we want to submit we should try to do so before July
<@siosm:matrix.org>
16:38:16
indeed
<@siosm:matrix.org>
16:38:40
OK, which topic should we start with?
<@dustymabe:matrix.org>
16:39:15
1730 :)
<@siosm:matrix.org>
16:39:58
!topic revisit python discussion
<@siosm:matrix.org>
16:40:02
!link https://github.com/coreos/fedora-coreos-tracker/issues/1730
<@siosm:matrix.org>
16:40:24
Which is back again due to nfs-utils-coreos
<@siosm:matrix.org>
16:40:28
!link https://github.com/coreos/fedora-coreos-tracker/issues/1942
<@siosm:matrix.org>
16:42:04
this nfs-utils package really some work
<@dustymabe:matrix.org>
16:42:09
<@dustymabe:matrix.org>
16:42:09
my general stance is that I'm done fixing python related packaging issues.
<@dustymabe:matrix.org>
16:42:09
my current proposal is to let this happen in rawhide and if someone fixes it before F43, great. If not, F43+ will have python
<@jlebon:fedora.im>
16:42:32
there was renewed interest in reworking the nfs-utils package. did that happen already?
<@dustymabe:matrix.org>
16:42:52
Jonathan Lebon: https://src.fedoraproject.org/rpms/nfs-utils/pull-request/14#comment-262471
<@jlebon:fedora.im>
16:43:28
ok. do we know if the rework fixes this?
<@dustymabe:matrix.org>
16:43:47
nope, but anyone is welcome to investigate
<@siosm:matrix.org>
16:46:14
I'll take a look at this; probably with one of our new team members
<@jlebon:fedora.im>
16:46:37
+ nfs-python-utils (to elminate the python dependency in other packages)
<@jlebon:fedora.im>
16:46:37
+ - Move all of the (optional) programs that require a python interpreter to
<@jlebon:fedora.im>
16:46:37
https://src.fedoraproject.org/fork/smayhew/rpms/nfs-utils/c/8730d56b2763a64a94e1e3ca1cc3527755fcbfb7?branch=new-pkg-split-squash has:
<@jlebon:fedora.im>
16:46:37
<@jlebon:fedora.im>
16:46:43
<@jlebon:fedora.im>
16:46:43
https://src.fedoraproject.org/fork/smayhew/rpms/nfs-utils/c/8730d56b2763a64a94e1e3ca1cc3527755fcbfb7?branch=new-pkg-split-squash has:
<@jlebon:fedora.im>
16:46:43
- Move all of the (optional) programs that require a python interpreter to
<@jlebon:fedora.im>
16:46:43
- nfs-python-utils (to elminate the python dependency in other packages)
<@dustymabe:matrix.org>
16:47:26
> my current proposal is to let this happen in rawhide and if someone fixes it before F43, great. If not, F43+ will have python
<@dustymabe:matrix.org>
16:47:26
I'm happy for a solution to exist, but would still like to propose:
<@dustymabe:matrix.org>
16:47:26
<@dustymabe:matrix.org>
16:47:47
IOW we timebox this effort.
<@jlebon:fedora.im>
16:47:51
I'll note that this is also valuable for rhel-bootc/fedora-bootc minimal image users that just want to add nfs client support
<@siosm:matrix.org>
16:49:16
Let's say, if we haven't fixed this in 2 weeks, we drop the python exclude from Rawhide?
<@siosm:matrix.org>
16:49:24
I don't like it
<@dustymabe:matrix.org>
16:49:58
Can we put that in a proposal?
<@jlebon:fedora.im>
16:50:11
yeah, was going to suggest a similar thing. let's give it some time since it's actively being worked on it seems
<@dustymabe:matrix.org>
16:50:44
to be clear, if python3 gets added to rawhide it doesn't mean we can't remove it later. but we'd want to make sure any "fixing" happens before it hits our stable streams
<@siosm:matrix.org>
16:50:49
it's been in this weird half state for a while now
<@siosm:matrix.org>
16:51:37
Given the version numbers, I expect that we'll have this issue in F42 pretty soon
<@siosm:matrix.org>
16:51:42
https://src.fedoraproject.org/rpms/nfs-utils
<@dustymabe:matrix.org>
16:52:44
travier: this may be more related to this commit: https://src.fedoraproject.org/rpms/nfs-utils/c/02f028903bda100bc18c5cf9ac85eb085e45251c?branch=rawhide
<@dustymabe:matrix.org>
16:52:44
which is only on `rawhide`
<@dustymabe:matrix.org>
16:52:44
<@siosm:matrix.org>
16:53:46
interesting, this is unexpected
<@siosm:matrix.org>
16:53:59
anyway, we'll look at it
<@siosm:matrix.org>
16:55:32
I don't have a good proposed so if you have dusty, go for it
<@siosm:matrix.org>
16:55:53
as soon as we drop the exclude, it's going to be harder to re-add it
<@siosm:matrix.org>
16:56:06
I don't really want to do it
<@dustymabe:matrix.org>
16:57:01
!proposed we'll wait a few weeks to see if a solution to the problem presents itself. At that point we'll drop the exclude for python for rawhide. If the problem still persists once we switch over to F43 for our stable streams then we'll keep python3 in the base in the future. Anytime before F43 it can be removed without issue if a solution to the problem exists.
<@dustymabe:matrix.org>
16:57:45
I only agree if it makes it into stable streams. If it's just rawhide I disagree
<@jlebon:fedora.im>
16:58:08
i think it does in the sense that we then don't notice if something else sneaks in that requires python
<@siosm:matrix.org>
16:58:20
yeah, that's my concern
<@jlebon:fedora.im>
16:58:29
but anyway, +1 to the proposed
<@dustymabe:matrix.org>
16:58:58
It's pretty easy to add a test for that specific case if you are worried about it. `rpm -e python3` and make sure the list in the error message only contains nfs-utils
<@siosm:matrix.org>
17:00:33
My vote will stay at 0. The proposed is reasonable, but I don't think we should include Python in FCOS so I'm not voting for it;
<@marmijo:fedora.im>
17:00:34
Suggestion: "if the problem hasn't been fixed at that point we'll drop the exclude for python for rawhide"
<@marmijo:fedora.im>
17:00:37
but +1 from me
<@siosm:matrix.org>
17:00:44
My vote will stay at 0. The proposed is reasonable, but I don't think we should include Python in FCOS so I'm not voting for it.
<@dustymabe:matrix.org>
17:01:40
marmijo: want to do a full !proposed ? not sure exactly what edit you want me to make
<@marmijo:fedora.im>
17:02:52
Sure, the wording could just be interpreted as: "we'll drop the exclude regardless of whether the issue is fixed". I can rewrite it.
<@marmijo:fedora.im>
17:03:48
!proposed we'll wait a few weeks to see if a solution to the problem presents itself. At that point, if the issue hasn't been resolved, we'll drop the exclude for python for rawhide. If the problem still persists once we switch over to F43 for our stable streams then we'll keep python3 in the base in the future. Anytime before F43 it can be removed without issue if a solution to the problem exists.
<@dustymabe:matrix.org>
17:04:17
+1
<@marmijo:fedora.im>
17:05:01
this vote is in line with a previous decision: https://github.com/coreos/fedora-coreos-tracker/issues/1730#issuecomment-2137959378
<@jlebon:fedora.im>
17:05:16
+1
<@jbtrystram:matrix.org>
17:05:18
if that can't be solved the last resort will be to publish the nfs-utils as a sysext I guess :D
<@jbtrystram:matrix.org>
17:05:29
anyway, +1
<@siosm:matrix.org>
17:05:36
0 :)
<@aaradhak:matrix.org>
17:05:52
+1
<@dustymabe:matrix.org>
17:05:55
2. If not fixed by the time `next` is rebased to F43 - python3 enters production FCOS streams
<@dustymabe:matrix.org>
17:05:55
1. If not fixed by June - python3 enters rawhide
<@dustymabe:matrix.org>
17:05:55
<@dustymabe:matrix.org>
17:05:55
TL;DR our milestones are:
<@siosm:matrix.org>
17:06:48
!agreed we'll wait a few weeks to see if a solution to the problem presents itself. At that point, if the issue hasn't been resolved, we'll drop the exclude for python for rawhide. If the problem still persists once we switch over to F43 for our stable streams then we'll keep python3 in the base in the future. Anytime before F43 it can be removed without issue if a solution to the problem exists.
<@nemric:relativit.fr>
17:07:09
I didn't have the strength to say that :D
<@siosm:matrix.org>
17:07:32
!agreed we'll wait a few weeks to see if a solution to the problem presents itself. At that point, if the issue hasn't been resolved, we'll drop the exclude for python for rawhide. If the problem still persists once we switch over to F43 for our stable streams then we'll keep python3 in the base in the future. Anytime before F43 it can be removed without issue if a solution to the problem exists.
<@siosm:matrix.org>
17:07:46
is zodbot out?
<@siosm:matrix.org>
17:08:25
<del>is zodbot out?</del>
<@marmijo:fedora.im>
17:08:35
I see responses from the bot
<@siosm:matrix.org>
17:08:58
Let's move to the next topic
<@siosm:matrix.org>
17:09:03
!topic Migrate existing systems to OCI updates
<@siosm:matrix.org>
17:09:08
!link https://github.com/coreos/fedora-coreos-tracker/issues/1890
<@siosm:matrix.org>
17:09:30
Dustty/JB who wants to take this one?
<@siosm:matrix.org>
17:09:38
Dusty/JB who wants to take this one?
<@jbtrystram:matrix.org>
17:11:00
I added this one, just to send a reminder that we should trigger the migration of the testing stream to deploy-via-OCI on the next update
<@jbtrystram:matrix.org>
17:11:26
so if there are any concern now is the time to raise it :)
<@jbtrystram:matrix.org>
17:12:12
The migration script have been shipped for a few releases now and is under /usr/libexec/coreos-oci-rebase
<@dustymabe:matrix.org>
17:12:13
hmm. were we going to do `next` first ?
<@jbtrystram:matrix.org>
17:12:33
it does not do much when executing : simply tell zincati to get the next update via OCI.
<@jbtrystram:matrix.org>
17:12:44
yes sorry
<@jbtrystram:matrix.org>
17:12:50
I always mix up next and testing
<@dustymabe:matrix.org>
17:12:57
:)
<@dustymabe:matrix.org>
17:13:02
so lets game this out
<@dustymabe:matrix.org>
17:13:34
I say we have two releases where `next` has the migration script. Then we can consider enabling it for `testing` (which will flow into `stable`).
<@jlebon:fedora.im>
17:13:57
hmm, i thought we had the rollout plan written up somewhere
<@jbtrystram:matrix.org>
17:14:25
dustymabe: you mean the migration script enabled by default
<@jlebon:fedora.im>
17:14:46
i see https://github.com/coreos/fedora-coreos-tracker/issues/1823#issuecomment-2465422672, though that comment is older than i remember the last time talking about this
<@dustymabe:matrix.org>
17:15:01
jbtrystram: yes, basically the migration script activated
<@jbtrystram:matrix.org>
17:15:27
we have a few sentences here : https://hackmd.io/U2_VVU2IRV6J_AkF6o7VUA?edit
<@jbtrystram:matrix.org>
17:16:25
so it was "for next, on third release after deploy-via-container default in bootimages"
<@jbtrystram:matrix.org>
17:16:39
it's been default on bootimages for more than 3 releases I think
<@dustymabe:matrix.org>
17:16:43
:) - we're way past that at this point
<@jlebon:fedora.im>
17:16:48
dustymabe: i'd say 2 releases is the bare minimum for validating
<@dustymabe:matrix.org>
17:17:03
Jonathan Lebon: ok, 3 ?
<@jlebon:fedora.im>
17:17:09
SGTM
<@dustymabe:matrix.org>
17:18:18
TBH it would make our life easier to just roll this out with the F43 update.. but we're also going to stop shipping any OSTree updates with F43 and I don't think it would be good to do that at the same time
<@jlebon:fedora.im>
17:18:25
that was highly optimistic
<@jlebon:fedora.im>
17:18:25
<@jlebon:fedora.im>
17:18:25
> end of march
<@jlebon:fedora.im>
17:18:47
yeah, agree
<@jbtrystram:matrix.org>
17:19:39
+1
<@dustymabe:matrix.org>
17:19:57
May 28 - First next release with migration script
<@dustymabe:matrix.org>
17:19:57
June 25 - 3rd ""
<@dustymabe:matrix.org>
17:19:57
July 9th - First testing release with migration script (rolls into `stable` in two weeks)
<@dustymabe:matrix.org>
17:19:57
Jun 11 - 2nd ""
<@jbtrystram:matrix.org>
17:20:14
we can push back the decomission of the ostree updates but i don't think it's worth waiting until october
<@dustymabe:matrix.org>
17:20:27
OR should we keep it out of `stable` for longer than the normal promotion?
<@jlebon:fedora.im>
17:21:02
dustymabe: yeah, was going to suggest that. this is the pattern we normally follow, but for this particular one, update issues need time to manifest
<@jlebon:fedora.im>
17:22:03
though i acknowledge it's awkward too to have testing and stable differ
<@dustymabe:matrix.org>
17:22:19
Aug 6 - 3rd
<@dustymabe:matrix.org>
17:22:19
July 9th - First testing release with migration script
<@dustymabe:matrix.org>
17:22:19
July 23 - 2nd
<@dustymabe:matrix.org>
17:22:19
Aug 20 - First `stable` release with migration script
<@dustymabe:matrix.org>
17:22:19
May 28 - First next release with migration script
<@dustymabe:matrix.org>
17:22:19
Jun 11 - 2nd ""
<@dustymabe:matrix.org>
17:22:19
June 25 - 3rd ""
<@jlebon:fedora.im>
17:23:10
it's crazy when you think of time in 2 week increments how fast you go down the calendar
<@jbtrystram:matrix.org>
17:23:26
> Aug 20 - First stable release with migration script
<@jbtrystram:matrix.org>
17:23:26
that get quite close to f43 freeze
<@jbtrystram:matrix.org>
17:23:33
that get quite close to f43 freeze
<@jbtrystram:matrix.org>
17:23:33
> Aug 20 - First stable release with migration script
<@jbtrystram:matrix.org>
17:23:33
<@dustymabe:matrix.org>
17:23:39
Jonathan Lebon: and our release cadence is fast by almost all standards
<@dustymabe:matrix.org>
17:24:10
jbtrystram: that shouldn't affect us
<@jlebon:fedora.im>
17:25:22
dustymabe: yeah, agree re. operational complexity
<@jbtrystram:matrix.org>
17:25:24
I know, i am just saying that the timeframe where people will be able to do OCI + ostree to sort out their migration is quite narrower as I had thougt
<@dustymabe:matrix.org>
17:25:34
Jun 11 - 2nd ""
<@dustymabe:matrix.org>
17:25:34
Aug 20 - First `stable` release with migration script enabled
<@dustymabe:matrix.org>
17:25:34
May 28 - First next release with migration script enabled
<@dustymabe:matrix.org>
17:25:34
July 23 - 2nd
<@dustymabe:matrix.org>
17:25:34
July 9th - First testing release with migration script enabled
<@dustymabe:matrix.org>
17:25:34
June 25 - 3rd ""
<@dustymabe:matrix.org>
17:25:34
Aug 6 - 3rd
<@jbtrystram:matrix.org>
17:25:59
a little less than 3 months (if f43 is on schedule)
<@dustymabe:matrix.org>
17:26:13
They'll still have til F43 GA - yeah. 3 months should be plenty IMO
<@dustymabe:matrix.org>
17:26:29
and TBH they should have hit this already if they ever deploy any new nodes (since F42)
<@dustymabe:matrix.org>
17:27:15
I don't have anything else for this topic.. other than we should take this timeline to the ticket
<@jlebon:fedora.im>
17:28:23
let's go to open floor with the few minutes left
<@siosm:matrix.org>
17:29:45
!topic Open Floor
<@siosm:matrix.org>
17:30:05
(I'll update the ticket with the timeline above)
<@dustymabe:matrix.org>
17:30:58
!info 05/2025 edition of FCOS numbers: https://discussion.fedoraproject.org/t/fedora-coreos-numbers-05-2025-edition/153283
<@jlebon:fedora.im>
17:32:19
wow, aarch64 overtaking x86_64
<@jlebon:fedora.im>
17:32:29
did not expect that
<@dustymabe:matrix.org>
17:33:02
Not sure if there is anybody that wants to work on pushing that over the finish line or not
<@dustymabe:matrix.org>
17:33:02
Another topic: since there was an afterburn release we are now unblocked on shipping proxmox: https://github.com/coreos/fedora-coreos-tracker/issues/1652
<@dustymabe:matrix.org>
17:33:02
<@jlebon:fedora.im>
17:33:11
i guess that's mostly podman machine, though?
<@dustymabe:matrix.org>
17:33:24
Yeah. TBH I'm not entirely sure
<@jlebon:fedora.im>
17:33:35
would be nice to have that breakdown
<@jlebon:fedora.im>
17:34:05
but i guess it'd require a separate countme bucket
<@siosm:matrix.org>
17:34:13
yep
<@marmijo:fedora.im>
17:34:49
re: proxmox, i'll work on producing official images, and I can also work on a few items in that list.
<@siosm:matrix.org>
17:34:55
or our users use aarch64 as it's usually cheaper in clouds :)
<@jbtrystram:matrix.org>
17:34:58
half the people are disabling auto-update
<@dustymabe:matrix.org>
17:35:25
jbtrystram: yeah. it's a problem
<@jbtrystram:matrix.org>
17:35:25
it's crazy, we work so hard for those to be reliable :(
<@jlebon:fedora.im>
17:35:35
travier: probably some element of it, though i'm dubious it accounts for the bulk of its growth
<@dustymabe:matrix.org>
17:35:55
though I probably need to dig into the numbers more. For example I wonder for the "transient" nodes, what is the release breakdown?
<@siosm:matrix.org>
17:35:57
agree, it's likely podman desktop growth first
<@jbtrystram:matrix.org>
17:36:04
looking forward to see some RISC-V numbers :)
<@dustymabe:matrix.org>
17:36:36
jbtrystram: I have a RISC-V system up and running - but the repos it has enabled aren't the Fedora ones (where countme=1 is enabled)
<@dustymabe:matrix.org>
17:37:04
jbtrystram: I have a RISC-V system up and running - but the repos it has enabled aren't the main Fedora ones (where countme=1 is enabled)
<@jbtrystram:matrix.org>
17:37:09
Nice
<@nemric:relativit.fr>
17:37:37
what are transient nodes ?
<@dustymabe:matrix.org>
17:38:57
they've been up for less than a week
<@dustymabe:matrix.org>
17:39:15
i.e. either brand new nodes.. or batch processing nodes they will get destroyed after they do their work
<@dustymabe:matrix.org>
17:39:26
I just looked and 20233 transient nodes are F40
<@nemric:relativit.fr>
17:39:32
I mean, does that include live ones ?
<@dustymabe:matrix.org>
17:39:48
likely a single big player that has stabilized on a particular release of F40 based FCOS for batch processing
<@dustymabe:matrix.org>
17:40:22
It includes any nodes that have only been living for 1 week or less :)
<@dustymabe:matrix.org>
17:41:13
but yes.. if you run your workloads from "live" media then you'll likely rarely get to a sys_age of 3 or 4 (you might hit 2)
<@dustymabe:matrix.org>
17:41:49
3 - Six Months (5-24 weeks)
<@dustymabe:matrix.org>
17:41:49
```
<@dustymabe:matrix.org>
17:41:49
client age
<@dustymabe:matrix.org>
17:41:49
1 - First week
<@dustymabe:matrix.org>
17:41:49
2 - First Month (2-4 weeks)
<@dustymabe:matrix.org>
17:41:49
4 - More than Six Months (> 24 weeks)
<@dustymabe:matrix.org>
17:41:49
```
<@nemric:relativit.fr>
17:44:23
yes, I reboot every 2 weeks, and if needed for some k8s/cri-o updates, so I'm surfing between transiant and static
<@nemric:relativit.fr>
17:44:44
yes, I reboot every 2 weeks, and if needed for some k8s/cri-o updates, so I'm surfing between transient and static
<@siosm:matrix.org>
17:45:53
Alright, let's close this meeting
<@siosm:matrix.org>
17:46:02
we're overtime
<@siosm:matrix.org>
17:46:29
!endmeeting