<@dustymabe:matrix.org>
16:30:05
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:30:07
Meeting started at 2024-01-03 16:30:05 UTC
<@meetbot:fedora.im>
16:30:07
The Meeting name is 'fedora_coreos_meeting'
<@dustymabe:matrix.org>
16:30:16
!topic roll call
<@dustymabe:matrix.org>
16:30:25
.hi dustymabe
<@apiaseck:matrix.org>
16:31:18
.hello c4rt0
<@jdoss:beeper.com>
16:31:29
.hello jdoss
<@dustymabe:matrix.org>
16:31:44
!chair apiaseck jdoss
<@jlebon:fedora.im>
16:31:59
.hello jlebon
<@dustymabe:matrix.org>
16:32:21
!chair Jonathan Lebon
<@jbtrystram:matrix.org>
16:33:57
.hello jbtrystram
<@dustymabe:matrix.org>
16:34:10
!chair jbtrystram
<@dustymabe:matrix.org>
16:34:34
welcome all - we'll get started soon. /me pings travier to see if he is around too
<@dustymabe:matrix.org>
16:34:58
also - happy new year to all - we made it
<@dustymabe:matrix.org>
16:35:44
!topic Action items from last meeting
<@jdoss:beeper.com>
16:35:47
\o/
<@dustymabe:matrix.org>
16:35:56
!info there are no action items from the last meeting.
<@dustymabe:matrix.org>
16:36:23
!topic Create a new repo to host Container images needed by Kola tests
<@dustymabe:matrix.org>
16:36:28
!link https://github.com/coreos/fedora-coreos-tracker/issues/1639
<@aaradhak:matrix.org>
16:37:21
.hello aaradhak
<@apiaseck:matrix.org>
16:37:40
Oh - that's a good idea
<@dustymabe:matrix.org>
16:37:42
jbtrystram: want to intro this one?
<@jbtrystram:matrix.org>
16:37:57
Sure :)
<@jbtrystram:matrix.org>
16:39:12
Some kola tests depend on external container images. so far there were only one : a `tang` image used for the LUKS tests. With the iSCSI boot stuff I had to add another one : a `target` service to serve an iscsi target.
<@jbtrystram:matrix.org>
16:39:45
now, the images are owned by the authors of the tests, which are fine, as long as they are around
<@jbtrystram:matrix.org>
16:40:10
I propose to create a new repository under github.com/coreos to host, build and publish those images
<@dustymabe:matrix.org>
16:40:43
I had resisted this model (i.e. individual container images for tests) in the past because it's hard to manage/maintain. I didn't realize we had the tang one slip through
<@jbtrystram:matrix.org>
16:40:48
I created a POC repo here : https://github.com/jbtrystram/kola-container-images
<@dustymabe:matrix.org>
16:40:55
In the past I've recommended we just build the container image on the host and then use it
<@dustymabe:matrix.org>
16:41:20
but I agree it can be nicer to have a container image already in a registry somewhere
<@jlebon:fedora.im>
16:42:23
yeah, if the test is relying on internet access anyway, it seems more efficient to use ready made images
<@jbtrystram:matrix.org>
16:43:01
it could be paired with a dedicated org in quay
<@jlebon:fedora.im>
16:43:45
this makes sense to me overall
hmm, i don't think it's worth another quay org to manage. we could put it under the existing coreos-assembler one
<@jbtrystram:matrix.org>
16:44:29
Jonathan Lebon: i forgot about this one, the coreos-assembler org is a great fit for this
<@dustymabe:matrix.org>
16:44:56
I think part of the reason I wanted to not do this in the past was we didn't have a good way to build multi-arch container images, but we do have a mechanism for that now (we do it for cosa and for the fcos-buildroot containers)
<@dustymabe:matrix.org>
16:45:44
so we'd need to have new jenkins jobs for the new containers OR we'd need to parameterize the job somehow
<@jbtrystram:matrix.org>
16:45:56
dustymabe: I reused coreos/actions-lib/build-container@main
<@dustymabe:matrix.org>
16:46:22
jbtrystram: that's not going to work for multi-arch
<@jlebon:fedora.im>
16:47:07
an alternative btw when evaluating these is to just ship the tools necessary as part of cosa so we can reuse the cosa image itself. i think for the iSCSI stuff it'd probably add too much, but it might be a good fit for the tang one
<@dustymabe:matrix.org>
16:47:13
we need something like https://github.com/coreos/fedora-coreos-pipeline/blob/main/jobs/build-fcos-buildroot.Jenkinsfile
<@dustymabe:matrix.org>
16:47:32
Jonathan Lebon: true.
<@jbtrystram:matrix.org>
16:48:02
dustymabe: maybe i am understanding this wrong (this was my first try on multiarch images) but this seems to work fine : https://quay.io/repository/jbtrystram/tang?tab=tags&tag=latest
<@dustymabe:matrix.org>
16:48:02
COSA is very much a kitchen sink today
<@jlebon:fedora.im>
16:48:12
dustymabe: yeah, probably time to parametrize that job
<@dustymabe:matrix.org>
16:48:30
jbtrystram: how are those images built?
<@jbtrystram:matrix.org>
16:49:06
https://github.com/jbtrystram/kola-container-images/tree/main/.github/workflows
<@jbtrystram:matrix.org>
16:49:36
GH actions, using the `coreos/actions-lib/build-container` action
<@dustymabe:matrix.org>
16:49:48
using emulation or cross compilation or something? I highly doubt github CI has ppc and s390 builders
<@jlebon:fedora.im>
16:49:51
it uses qemu-user-static
<@jlebon:fedora.im>
16:50:07
https://github.com/coreos/actions-lib/blob/6f48d6e84f52dd5d3da0b89510003f21f7313016/build-container/action.yml#L93C33-L93C49
<@jlebon:fedora.im>
16:50:20
so emulation
<@dustymabe:matrix.org>
16:50:42
interesting.. should we rely on that?
<@jlebon:fedora.im>
16:51:25
interestingly, the build times don't look terrible: https://github.com/jbtrystram/kola-container-images/actions
<@dustymabe:matrix.org>
16:52:25
honestly I feel a bit uneasy about it
<@jlebon:fedora.im>
16:52:47
i'm not sure. maybe? i'd say i'd probably want all our multi-arch images to be built the same way. so if we keep build-cosa, i'd rather do it the same way for those too
<@dustymabe:matrix.org>
16:52:58
i.e. this is now load bearing infra and who has insight into the failures.. who maintains it over time?
<@jlebon:fedora.im>
16:53:49
to play devil's advocate, we could probably figure out build failure notifications.
but obviously, that's work vs what we already have set up in the pipeline
<@jbtrystram:matrix.org>
16:54:39
i am happy to make that work with Jenkins to have everything aligned. As said, that's just a POC repo to start the discussion
<@dustymabe:matrix.org>
16:55:07
+1 - ok - so what is the open question we're trying to solve here? IF we should do it, or WHERE we should do it?
<@dustymabe:matrix.org>
16:55:25
can we agree the quay.io/coreos-assembler ORG is probably the right place to push to?
<@jbtrystram:matrix.org>
16:55:51
IF first, then let's vote the WHERE/HOW condition if needed
<@jlebon:fedora.im>
16:56:17
+1 from me
<@dustymabe:matrix.org>
16:56:27
+1
<@apiaseck:matrix.org>
16:56:31
+1
<@dustymabe:matrix.org>
16:56:36
anyone opposed to `IF` ?
<@jbtrystram:matrix.org>
16:56:41
+1
<@jlebon:fedora.im>
16:56:57
that was what I was voting on :)
<@dustymabe:matrix.org>
16:56:59
haven't heard any detractors so far
<@dustymabe:matrix.org>
16:57:28
ok so WHERE seems close to agreed on too.. now HOW
<@jlebon:fedora.im>
16:57:44
one suggestion is to not make this a separate repo but just a folder in the cosa one
<@dustymabe:matrix.org>
16:57:50
I'd argue to keep it consistent and just another jenkins job that uses our multi-arch builders
<@dustymabe:matrix.org>
16:58:32
Jonathan Lebon: define repo.. git repo or quay registry repo ?
<@jlebon:fedora.im>
16:58:42
that also allows atomically adding tests and supporting Dockerfile.
git repo
<@jlebon:fedora.im>
16:59:08
that also allows atomically adding/modifying tests and supporting Dockerfile.
git repo
<@dustymabe:matrix.org>
16:59:31
yeah, maybe that's a good idea, though if the repo is forked from another it might make it harder to keep track/and/or pull in changes
<@dustymabe:matrix.org>
16:59:43
maybe that's TBD pending investigation
<@jlebon:fedora.im>
17:00:29
not sure i follow
<@jbtrystram:matrix.org>
17:01:23
dustymabe: the containerfiles are very simple, and not forked from other repos so I don't think that's an issue. I like jonathan's solution
<@dustymabe:matrix.org>
17:02:28
https://github.com/jbtrystram/targetcli-containers says : `forked from alicefr/targetcli-containers`
<@dustymabe:matrix.org>
17:03:19
anyway let's take this tangent to the FCOS matrix channel after the meeting - i'll add a proposed here now:
<@dustymabe:matrix.org>
17:03:28
!proposed we will build manifest listed container images for specific tests using Jenkins jobs and our multi-arch builders (like we build cosa and fcos-buildroot today) and store them in the quay.io/coreos-assembler org in quay.
<@jlebon:fedora.im>
17:03:31
gotcha, i follow now
yeah, sounds good 👍️
<@dustymabe:matrix.org>
17:03:35
votes?
<@apiaseck:matrix.org>
17:03:37
!agreed
<@dustymabe:matrix.org>
17:04:28
!undo
<@dustymabe:matrix.org>
17:04:54
anyone else want to vote on the proposed?
<@jbtrystram:matrix.org>
17:05:50
should the proposal mentions where we store the containerfiles ?
<@dustymabe:matrix.org>
17:06:08
!agreed we will build manifest listed container images for specific tests using Jenkins jobs and our multi-arch builders (like we build cosa and fcos-buildroot today) and store them in the quay.io/coreos-assembler org in quay.
<@dustymabe:matrix.org>
17:06:23
jbtrystram: I think that's TBD - use your best judgement :)
<@dustymabe:matrix.org>
17:06:42
we can discuss further in the FCOS channel after the meeting
<@jbtrystram:matrix.org>
17:07:06
ok
<@dustymabe:matrix.org>
17:07:09
!topic aarch64 failing to upgrade with /boot filesystem full
<@dustymabe:matrix.org>
17:07:14
!link https://github.com/coreos/fedora-coreos-tracker/issues/1637
<@dustymabe:matrix.org>
17:07:56
so unfortunately we had to cancel our last set of releases for 2023 because we encountered some upgrade test failures on aarch64
<@dustymabe:matrix.org>
17:08:19
it appears the autopruning code may have some corner case that isn't handled
<@dustymabe:matrix.org>
17:08:49
there was a PR added to OSTree to enhance logging in this area: https://github.com/ostreedev/ostree/pull/3123
<@jlebon:fedora.im>
17:08:55
planning to look at this today
are releases blocked on this currently?
<@dustymabe:matrix.org>
17:09:03
Jonathan Lebon: yes :(
<@dustymabe:matrix.org>
17:09:19
it was hard to progress without understanding the problem more completely
<@jlebon:fedora.im>
17:09:20
ouch ok
<@dustymabe:matrix.org>
17:09:40
we didn't want to ship out further updates that could exacerbate the problem
<@jlebon:fedora.im>
17:10:03
it's good that we can reproduce it manually at least
<@dustymabe:matrix.org>
17:10:15
and the week the problem showed itself I was technically away and not able to dedicate appropriate eyes on the investigation
<@dustymabe:matrix.org>
17:10:38
yes, it's rather easy to reproduce - let me know if you need help
<@dustymabe:matrix.org>
17:10:51
the currently built `next` in the compose repo should trigger the problem
<@dustymabe:matrix.org>
17:11:03
the currently built `next` in the compose OSTree repo should trigger the problem
<@dustymabe:matrix.org>
17:11:25
or `testing` too I think
<@jlebon:fedora.im>
17:11:55
ack, thanks
<@dustymabe:matrix.org>
17:12:01
or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path
<@dustymabe:matrix.org>
17:12:11
or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path).
<@dustymabe:matrix.org>
17:12:49
yeah, grab a debug pod on the aarch64 builder and see if you can repro.. if you can't I'll jump in to help. Thanks so much Jonathan Lebon
<@jlebon:fedora.im>
17:13:01
yeah, this might just need a slight tweak in the autoprune code. it's a bit sad that it has become load-bearing now on some arches
<@dustymabe:matrix.org>
17:13:11
!info we are debugging the issue in https://github.com/coreos/fedora-coreos-tracker/issues/1637 and will try to get out releases as soon as we understand the problem more fully
<@dustymabe:matrix.org>
17:13:39
Jonathan Lebon: at least we know it does work when tests pass, though :)
<@dustymabe:matrix.org>
17:14:22
I would like to accelerate us changing our /boot partition strategy at some point in the future, but $priorities just make it hard
<@dustymabe:matrix.org>
17:14:38
!topic tracker: Fedora 40 changes considerations
<@dustymabe:matrix.org>
17:14:44
!link https://github.com/coreos/fedora-coreos-tracker/issues/1626
<@dustymabe:matrix.org>
17:15:06
ok there are a few new changes for us to consider
<@dustymabe:matrix.org>
17:15:21
!info 121. Changes/Linker Error On Security Issues
<@dustymabe:matrix.org>
17:15:29
!link https://fedoraproject.org/wiki/Changes/Linker_Error_On_Security_Issues
<@jlebon:fedora.im>
17:16:18
should be transparent to us
<@dustymabe:matrix.org>
17:16:30
seems transparent to us as failures would happen during rpm builds?
<@jlebon:fedora.im>
17:16:37
well, to FCOS at least.
yeah
<@dustymabe:matrix.org>
17:17:00
!info 121. Changes/Linker Error On Security Issues should be transparent to us as failures would happen during rpm builds before reaching FCOS builds.
<@dustymabe:matrix.org>
17:17:10
!info 122 389_Directory_Server_3.0.0
<@dustymabe:matrix.org>
17:17:16
!link https://fedoraproject.org/wiki/Changes/389_Directory_Server_3.0.0
<@jlebon:fedora.im>
17:17:43
i think we can safely skip this one
<@jlebon:fedora.im>
17:17:54
i think we can safely skip this one; we don't ship that package
<@dustymabe:matrix.org>
17:18:01
Jonathan Lebon: some text for the `!info` ?
<@jlebon:fedora.im>
17:18:25
we don't ship 389-ds-base
<@jlebon:fedora.im>
17:18:42
!info we don't ship 389-ds-base
<@dustymabe:matrix.org>
17:18:50
!info 122 389_Directory_Server_3.0.0 - Nothing to do, we don't ship 389-ds-base.
<@dustymabe:matrix.org>
17:19:00
!info 123. DNF: Do not download filelists by default
<@dustymabe:matrix.org>
17:19:08
!link https://fedoraproject.org/wiki/Changes/DNFConditionalFilelists
<@dustymabe:matrix.org>
17:19:55
Do we know if there are any implications for `rpm-ostree` here and how it handles yum repos?
<@jlebon:fedora.im>
17:20:23
rpm-ostree already doesn't download filelists by default IIRC
<@dustymabe:matrix.org>
17:21:01
Jonathan Lebon: i.e. you can't just `rpm-ostree install /usr/bin/foo` ?
<@jlebon:fedora.im>
17:21:31
you can. `/usr/bin` is in the primary metadata
<@jlebon:fedora.im>
17:21:59
i'll have to double-check. i think we do know to download it if you request e.g. /usr/share/...
<@jlebon:fedora.im>
17:22:23
but that seems independent of this change since i think the change is happening at the dnf layer and not libdnf
<@dustymabe:matrix.org>
17:22:32
ok but in general you think we shouldn't have any issues because of this? or should we assign an action item/open an issue?
<@jlebon:fedora.im>
17:23:20
i think so, but i think it's probably worth verifying current rpm-ostree client-side behaviour wrt filelists
<@dustymabe:matrix.org>
17:24:21
```core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/
31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck
8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck
bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck
repomd.xml```
<@dustymabe:matrix.org>
17:24:38
Like this
````core@apu2:~$
31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck
8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck
bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck
repomd.xml```
````
<@dustymabe:matrix.org>
17:24:51
Like this
```
core@apu2:~$
31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck
8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck
bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck
repomd.xml
```
<@dustymabe:matrix.org>
17:25:02
Like this
```
core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/
31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck
8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck
bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck
repomd.xml
```
<@jlebon:fedora.im>
17:25:34
hmm, confusingly it looks like some paths do skip it and some don't
<@jlebon:fedora.im>
17:25:40
that might just be an oversight
<@dustymabe:matrix.org>
17:26:10
ok let's open an issue to investigate further..
<@jlebon:fedora.im>
17:26:16
i'd peel that off as a separate rpm-ostree issue
<@jlebon:fedora.im>
17:26:17
yeah
<@dustymabe:matrix.org>
17:26:39
!info 123. DNF: Do not download filelists by default - We'll open an issue and investigate this further
<@dustymabe:matrix.org>
17:27:18
!info 213. Wget2 as wget
!link https://fedoraproject.org/wiki/Changes/Wget2asWget
<@dustymabe:matrix.org>
17:27:34
!info 213. Wget2 as wget
<@dustymabe:matrix.org>
17:27:39
!link https://fedoraproject.org/wiki/Changes/Wget2asWget
<@dustymabe:matrix.org>
17:27:55
!info 213. Wget2 as wget - Nothing to do, we don't ship wget.
<@dustymabe:matrix.org>
17:28:01
^^ agree?
<@jlebon:fedora.im>
17:28:11
👍️
<@dustymabe:matrix.org>
17:28:34
!topic Open Floor
<@dustymabe:matrix.org>
17:28:44
anyone with anything for open floor?
<@dustymabe:matrix.org>
17:28:55
jdoss: Adam B ?
<@jdoss:beeper.com>
17:29:25
I got nothing this week. Happy new year folks. Lets make FCOS even more awesome in 2024!
<@dustymabe:matrix.org>
17:29:28
it's worth noting I ran into this myself this past week: https://github.com/coreos/fedora-coreos-tracker/issues/1641
<@ajbouh:fedora.im>
17:29:43
i've been trying to find and fix bugs associated with adding additional files to an FCOS build
<@jdoss:beeper.com>
17:30:15
I guess I am still fighting ipxe booting RPi CM 4s with FCOS but that is a swamp of pain and suffering right now.
<@ajbouh:fedora.im>
17:30:16
would love to know the right way to navigate that sort of thing, since it feels like i'm swimming against the current with everything i try
<@jdoss:beeper.com>
17:30:37
I just use layering for everything now.
<@ajbouh:fedora.im>
17:31:01
yeah, i want it baked in for the live image and to work offline, so hard mode x2
<@jdoss:beeper.com>
17:31:28
I like your speed Adam. 🙂
<@dustymabe:matrix.org>
17:31:41
Adam B: yeah. I'm not going to lie, `cosa` is very much built for building coreos and deviating from that too much might yield some pain
<@ajbouh:fedora.im>
17:31:43
the rootfs squash image inside the initrd is currently the blocker for me, since cpio has a 4GB max for file entries
<@ajbouh:fedora.im>
17:32:10
"some" :)
<@ajbouh:fedora.im>
17:32:25
honestly cosa has been pretty solid
<@dustymabe:matrix.org>
17:32:48
Adam B: one option may be to write the ISO to partition one of a disk image (i.e. USB key) and write other files to a filesystem on partition 2
<@dustymabe:matrix.org>
17:33:24
then after the install pick those files up and copy them into the right places
<@ajbouh:fedora.im>
17:33:29
yes, i've realized this might be my only option...
<@ajbouh:fedora.im>
17:33:39
still trying to figure out how this impacts updates
<@ajbouh:fedora.im>
17:34:03
and also how to safely modify the disk image without breaking its bootability
<@dustymabe:matrix.org>
17:34:58
you can also do https://matrix.to/#/!YWqcsiUQiCaqimYdQT:fedoraproject.org/$h-bXNIpf0JvGsss02bHdwjpjK4Mb-vPWGYBPajyIQ1c?via=fedoraproject.org&via=fedora.im&via=matrix.org but just write a the raw disk image into partition 2 and install that instead of what is baked in the ISO
<@dustymabe:matrix.org>
17:35:13
since it sounds like you are building the raw disk metal image without issue
<@ajbouh:fedora.im>
17:35:51
yes, raw disk image builds alright
<@ajbouh:fedora.im>
17:35:55
slowly, but alright
<@dustymabe:matrix.org>
17:36:01
Adam B: let's keep chatting over in the FCOS channel after the meeting
<@dustymabe:matrix.org>
17:36:13
anyone with anything else for open floor? if no I'll close out the meeting soon
<@dustymabe:matrix.org>
17:37:17
!endmeeting