<@nimbinatus:matrix.org>
15:00:32
!startmeeting Fedora_bootc_Initiative
<@meetbot:fedora.im>
15:00:36
Meeting started at 2025-07-15 15:00:32 UTC
<@meetbot:fedora.im>
15:00:36
The Meeting name is 'Fedora_bootc_Initiative'
<@nimbinatus:matrix.org>
15:01:16
!topic roll call
<@nimbinatus:matrix.org>
15:01:27
!hi
<@zodbot:fedora.im>
15:01:30
Laura Santamaria (nimbinatus) - she / her / hers
<@jcpunk:fedora.im>
15:01:34
!hi
<@zodbot:fedora.im>
15:01:35
Pat Riehecky (jcpunk) - he / him / his
<@rsturla:fedora.im>
15:01:52
!hi
<@zodbot:fedora.im>
15:01:53
None (rsturla)
<@hricky:fedora.im>
15:02:00
!hi
<@zodbot:fedora.im>
15:02:01
Hristo Marinov (hricky) - he / him / his
<@nimbinatus:matrix.org>
15:02:57
While we're waiting for folks to trickle in, anyone have any topics of interest for the agenda?
<@jcpunk:fedora.im>
15:03:40
tacitly, transient etc and some method to preserve host ssh keys (so I don't constantly get "host key changed ID")
<@nimbinatus:matrix.org>
15:04:49
I'll give roll call a couple more minutes and folks can keep suggesting more topics, and then I'll kick off the agenda with that topic. Thanks!
<@jcpunk:fedora.im>
15:05:17
(initial chat up at ostree - https://github.com/ostreedev/ostree/issues/3402)
<@jeckersb:fedora.im>
15:05:42
!hi
<@zodbot:fedora.im>
15:05:43
John Eckersberg (jeckersb)
<@dustymabe:matrix.org>
15:06:21
!hi
<@zodbot:fedora.im>
15:06:22
Dusty Mabe (dustymabe) - he / him / his
<@nimbinatus:matrix.org>
15:08:16
All right, let's kick off!
<@jmarrero:matrix.org>
15:08:32
!hi
<@zodbot:fedora.im>
15:08:33
Joseph Marrero (jmarrero)
<@nimbinatus:matrix.org>
15:08:38
!topic transient `/etc` and preserving host ssh keys
<@nimbinatus:matrix.org>
15:08:46
!link https://github.com/ostreedev/ostree/issues/3402
<@nimbinatus:matrix.org>
15:09:05
Pat Riehecky: want to kick off the chat?
<@jcpunk:fedora.im>
15:09:17
Sure
<@jcpunk:fedora.im>
15:09:53
The problem with a truly transient system is hosts have some local state that it really helps to preserve - ssh host keys, /etc/machiene-id, etc
<@jcpunk:fedora.im>
15:10:45
When these change it can cause some frustration, users need to fiddle with `known_hosts` or systemlogs no longer track to the same host
<@jcpunk:fedora.im>
15:11:03
But, preserving things in /etc is directly at odds with making it transient
<@jcpunk:fedora.im>
15:11:45
So, the real challenge is : what should a rational workflow look like to keep transient /etc doing its job but also somehow actually restore local system state?
<@jcpunk:fedora.im>
15:12:53
My initial thoughts involve some sort of composefs layer of `/var/etc` that sits _under_ `/usr/etc` so the any file in the image automatically takes precidence. But that begins embedding my workflow into the tooling
<@jcpunk:fedora.im>
15:13:25
I know there is a chat "somewhere" about podman style secrets, but honestly I haven't had time to find/follow it
<@jcpunk:fedora.im>
15:14:34
That is more or less my introductory thoughts
<@jmarrero:matrix.org>
15:16:43
I think the issue linked has the solution from Colin but there is no direct example you can copy and paste.
<@jmarrero:matrix.org>
15:18:36
I can take this to the issue try to come up with an example that is clearer and add it to the docs. Or you have a problem with the answer proposed there?
<@jcpunk:fedora.im>
15:18:51
I'm unsure what all I should target as my `Before` and so forth
<@jcpunk:fedora.im>
15:19:20
A detailed example would be sufficient, the boot sequence is a bit opaque to me
<@jcpunk:fedora.im>
15:20:25
I tried a few tricks and usually ended up either before /etc got setup correctly or after ssh noticed the keys were missing
<@nimbinatus:matrix.org>
15:20:32
!action Write up a detailed example for the docs based on Colin's answer in the issue with a clarification on the boot sequence (could be a great issue opened with a `good first issue` tag for new contributors)
<@dustymabe:matrix.org>
15:21:00
jmarrero: I think this might be a special case of dynamic overlays though
<@dustymabe:matrix.org>
15:22:10
i.e. I think in this case Pat Riehecky wants to persist something generated on first boot into subsequent boots.
<@dustymabe:matrix.org>
15:22:10
<@dustymabe:matrix.org>
15:22:10
so for dynamic overlays on the first boot they would have some service or something that took those pieces and then created the config (i.e. however we decide dynamic overlays should be defined) to be applied on subsequent boots
<@dustymabe:matrix.org>
15:22:44
note he stated that in this particular case (ssh host keys) they can't be generated elsewhere and pulled from the network
<@dustymabe:matrix.org>
15:23:04
<@dustymabe:matrix.org>
15:23:04
i.e. I think in this case Pat Riehecky wants to persist something generated on first boot into subsequent boots.
<@dustymabe:matrix.org>
15:23:04
so for dynamic overlays, on the first boot they would have some service or something that took those pieces and then created the config (i.e. however we decide dynamic overlays should be defined) to be applied on subsequent boots
<@jmarrero:matrix.org>
15:23:55
But you could generate them on first boot, copy them to another place before shutdown and pull them back on the second boot no?
<@dustymabe:matrix.org>
15:25:03
Exactly, yes. Perhaps we should just add that as a noted sub-workflow of dynamic overlays that maybe we'd like to make easier.
<@jmarrero:matrix.org>
15:25:37
Yeah I think that makes sense.
<@dustymabe:matrix.org>
15:26:34
Pat Riehecky: I'm not sure of the entire boot sequence right now with transient etc/ but you may be able to workaround things for now with a carefully crafted systemd-tmpfiles entry that (on subsequent boots) copies in your host keys if that exist under /var/etc/
<@dustymabe:matrix.org>
15:26:50
Pat Riehecky: I'm not sure of the entire boot sequence right now with transient etc/ but you may be able to workaround things for now with a carefully crafted systemd-tmpfiles entry that (on subsequent boots) copies in your host keys if they exist under /var/etc/
<@dustymabe:matrix.org>
15:27:49
OR just configure sshd to look in more places for the host keys (if that's possible)
<@jcpunk:fedora.im>
15:28:57
I'd rather something that copies stuff in so I can rely on `restorecon` and avoid as many local changes as possible
<@jcpunk:fedora.im>
15:31:33
I've only got a handful of files to persist, so manually listing them for systemd-tmpfiles.d is probably workable but i'll need to think about the initial copy over...
<@nimbinatus:matrix.org>
15:37:51
All right, so it sounds like we have two doc updates: subworkflows of dynamic overlays (plus examples) and the server-based example from Colin's reply at https://github.com/ostreedev/ostree/issues/3402#issuecomment-2779606668 ?
<@jcpunk:fedora.im>
15:38:40
ssh server key rotation might be a fun edge case to think through as well. I'd swear ssh server added a way to do that automatically but my searches come up with nothing
<@jmarrero:matrix.org>
15:41:14
Pat Riehecky: can you add any additional details about your use case and possible edge cases to the issue? I will try to come up with a couple of examples at it seems like documenting transient etc use-cases and examples would be helpful.
<@dustymabe:matrix.org>
15:41:27
<@dustymabe:matrix.org>
15:41:27
> ssh server key rotation
<@dustymabe:matrix.org>
15:41:27
go nuclear and re-deploy from scratch :)
<@jcpunk:fedora.im>
15:44:59
My three biggest issues are : ssh server keys (my users are often students with limited technical skill and not always strong technical English), kerberos keytab (/etc/krb5.keytab is our solution to a lot of trusted auth problems as our edge devices don't typically have TPM), /etc/machine-id (We send the logs to a central source, but the host name is often less useful than the machine-id for determining if the same physical SOC is problematic as it moves around the lab area)
<@jcpunk:fedora.im>
15:45:41
Industry wise, I work at an international science lab (fnal.gov) so a large number of my remote users have limited overlap with my work hours
<@jmarrero:matrix.org>
15:49:42
I think we are officially overtime? Is there any other topics nimbinatus ?
<@nimbinatus:matrix.org>
15:50:18
I've got us down until top of the hour, but I certainly can close the meeting if folks would like. We don't have any other topics on the agenda that I'm aware of.
<@nimbinatus:matrix.org>
15:51:09
Anyone opposed to closing out? I'll close at 5 before the top of the hour if not
<@jmarrero:matrix.org>
15:52:42
I been following this old invite, maybe we need to update it to an hr.
<@jmarrero:matrix.org>
15:52:44
https://calendar.fedoraproject.org/SIGs/2025/1/27/#m10982
<@nimbinatus:matrix.org>
15:53:14
Ah, interesting. Yes, I've had it on my calendar as a full hour. I'll check in with Jason on how to update that
<@nimbinatus:matrix.org>
15:55:37
Thanks for coming, folks! Chat next week if not before
<@nimbinatus:matrix.org>
15:55:45
!endmeeting