16:31:10 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:10 <zodbot> Meeting started Wed Aug 16 16:31:10 2023 UTC.
16:31:10 <zodbot> This meeting is logged and archived in a public location.
16:31:10 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:31:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:10 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:15 <dustymabe> #topic roll call
16:31:27 <travier> .hello siosm
16:31:33 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:31:37 <gursewak> .hi
16:31:41 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:31:48 <ravanelli> .hi
16:31:52 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:32:23 <dustymabe> #chair travier gursewak ravanelli
16:32:23 <zodbot> Current chairs: dustymabe gursewak ravanelli travier
16:33:38 <spresti> .hi
16:33:40 <zodbot> spresti: spresti 'Steven Presti' <spresti@redhat.com>
16:33:48 <dustymabe> #chair spresti
16:33:48 <zodbot> Current chairs: dustymabe gursewak ravanelli spresti travier
16:35:04 <jmarrero> .hi
16:35:05 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:35:29 <marmijo> .hello marmijo
16:35:30 <zodbot> marmijo: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:36:21 <dustymabe> #chair jmarrero marmijo
16:36:21 <zodbot> Current chairs: dustymabe gursewak jmarrero marmijo ravanelli spresti travier
16:36:32 <dustymabe> #topic Action items from last meeting
16:36:46 <dustymabe> IIUC the last meeting was canceled
16:37:23 <dustymabe> well - it was started but then not enough people were here
16:37:36 <dustymabe> so we'll move on
16:37:57 <dustymabe> #topic New Package Request: AWS amazon-ssm-agent for AWS EC2 AMI
16:38:03 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1537
16:38:44 <dustymabe> travier: you already had a reply in the ticket
16:39:20 <dustymabe> I also linked to https://github.com/coreos/fedora-coreos-tracker/issues/95 where we had discussions about not including cloud agents in the base images
16:39:39 <travier> dustymabe: there was one action item from last meeting
16:39:45 <travier> I re-actionned it
16:39:47 <dustymabe> travier: oh?
16:39:59 <dustymabe> ahh I see it now
16:40:03 <aaradhak85> .hello aaradhak85
16:40:04 <zodbot> aaradhak85: Sorry, but user 'aaradhak85' does not exist
16:40:06 <dustymabe> yes that action was completed
16:40:11 <travier> ok good
16:40:12 <dustymabe> #chair aaradhak85
16:40:12 <zodbot> Current chairs: aaradhak85 dustymabe gursewak jmarrero marmijo ravanelli spresti travier
16:42:01 <travier> Overall I don't like adding agents that do more than a simple health check.
16:42:16 <spresti> So from my reading of the issue, we really should not include the package. It feels like its opening a box of worms
16:42:17 <travier> like a "ping: I'm up" check
16:42:38 <dustymabe> travier: agreed
16:43:05 <dustymabe> I do wish we had an answer for people like this. even if it was in the FAQ
16:43:27 <dustymabe> i.e. I think I'd be happy linking from the FAQ to a community supported method for getting the agent installed
16:43:35 <travier> As it's an external package, layering is a fine option if they want to take the risk
16:43:45 <dustymabe> travier: agree
16:43:51 <travier> we don't have to make all our users take the risk
16:44:41 <spresti> +1
16:45:16 <dustymabe> #proposed We will not include the amazon-ssm-agent in Fedora CoreOS. We have generally not shipped cloud agents in the past and the package is only available from third party repos. If users want to layer this package they have that option.
16:45:24 <travier> +1
16:45:32 <spresti> +1
16:46:27 <jmarrero> +1
16:46:41 <dustymabe> #agreed We will not include the amazon-ssm-agent in Fedora CoreOS. We have generally not shipped cloud agents in the past and the package is only available from third party repos. If users want to layer this package they have that option.
16:46:51 <dustymabe> (I'll count myself as +1 there)
16:47:07 <gursewak> +1 to that as well.
16:47:20 <dustymabe> gursewak: 👍
16:47:29 <dustymabe> #topic low memory systems with large disks fail to provision
16:47:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1535
16:47:43 * dustymabe gives a moment to read the ticket
16:49:47 <dustymabe> TL;DR people can't really boot small memory systems any longer with a large disk
16:49:58 <dustymabe> unless they set up a separate mount for /var
16:50:02 <spresti> So why do we reprovision root at 200gb? i was under the impression it was a larger amount where the issue was more prominant?
16:50:12 <dustymabe> spresti: we do it at 100g+
16:51:08 <dustymabe> yeah the performance regressions get worse the larger it gets, but we were operating off of the recommendations from the XFS team and they basically recommended anything above a certain agcount should get repaved (which turned out to be right around 100G)
16:53:11 <spresti> Ah ok, that makes sense. I like the option 1 you recommended seems like an all rounder
16:53:38 <travier> I'm OK with disableing the reprovision if memory is low, so option 1
16:53:41 <spresti> I wonder though should we add a part 2 into that fix, where we also bail if ram is  < 3gb and the hdd is 12432143?
16:54:12 <spresti> ignore my numbers on hdd size, just trying to indicate too large
16:54:44 <ravanelli> if we go for 1, should we add some message to let the users know about the issues with large disks?
16:55:34 <travier> Usually the use case for low memory systems is Raspberry boards or small cloud instance. Those don't usually have large disks
16:55:36 <dustymabe> spresti: that's the behavior today IIUC. if HDD size is large then autosave-xfs/reprovisioning will be requested. when the reprovisioning code starts up it checks memory and if it's too low it bails out.
16:56:07 <dustymabe> travier: think backup server
16:56:34 <dustymabe> memory requirements: low
16:56:38 <dustymabe> disk requirements: high
16:56:39 <travier> would it be the root partition that is large and not a set of RAID disks?
16:56:49 <spresti> I think there is a diffrence. Today we would bail at 100g but instead we bail when it becomes too long I thought we encountered timeouts in the past
16:56:59 <dustymabe> travier: good point. it depends on how you architected it
16:57:14 <travier> but yes, there are always cases
16:57:46 <dustymabe> spresti: so in that case we'd have to find another threshold in which we'd decide to hard fail
16:57:55 <spresti> ie if we have 3gb of ram we repro at 100g but if we dont we try and ride with the old implemnetation and warn about slowness, and then finaly fail at some threshold.
16:58:36 <travier> If you have more than 100GB, I'd say that you can afford the /var split
16:58:46 <dustymabe> travier: indeed
16:58:56 <dustymabe> I think the hard part is getting the user to understand the problem
16:59:06 <spresti> Ah thats fair travier
16:59:11 <dustymabe> all they know is that their instance no longer gets provisioned
16:59:23 <dustymabe> i.e. we implemented autosave-xfs without them ever requesting it
16:59:41 <dustymabe> so their system that provisioned fine 6 months ago, now fails
16:59:51 <travier> agree it's not create that it's failing the boot where it was working before
17:00:42 <spresti> Sorry for waffling so much I think I am all in on option 1. (ignore my idea about another threshold)
17:00:49 <dustymabe> spresti: :)
17:01:03 <travier> which is why I like the "don't do anything if we can't, but warn using CLHM"
17:01:15 <dustymabe> so in option 1. we decide that on low memory systems we won't request autosave-xfs
17:01:21 <travier> We can warn about memory and /var split
17:01:33 <ravanelli> +1
17:01:37 <dustymabe> and then later (an independent process) drop down a CLHM helper
17:02:15 <dustymabe> travier: i'm thinking the CLHM helper can be implemented completely independent of the "memory checking" - WDYT?
17:02:25 <travier> agree
17:02:36 <dustymabe> i.e. it's useful in all cases
17:02:41 <travier> it was like that before
17:02:49 <travier> so the CLHM addition is a "feature"
17:03:05 <dustymabe> right.. now what it will cause (probably) is existing systems to start "warning" to their owners
17:03:14 <dustymabe> but raising awareness of this problem is probably a good thing
17:03:24 <travier> how long have we shipped the transpose xfs workaround in releases so far?
17:03:33 <travier> since how long*
17:03:43 <dustymabe> I think it landed in like june(ish)
17:04:20 <travier> not that long ago so that should be fine
17:04:38 <dustymabe> yeah
17:05:26 <dustymabe> #proposed We will attempt to make the autosave-xfs code not request a reprovision if the memory in the system is too low. We will also develop a CLHM helper script that checks agcount in the root filesystem (if it is XFS) and give the user a proper message if it's too low.
17:05:43 <ravanelli> +1
17:06:06 <spresti> +1
17:06:08 <aaradhak85> +1
17:06:10 <marmijo> +1
17:06:18 <mnguyen> +1
17:06:20 <jmarrero> +1
17:06:32 <travier> +1
17:06:37 <gursewak> +1
17:06:48 <dustymabe> #agreed We will attempt to make the autosave-xfs code not request a reprovision if the memory in the system is too low. We will also develop a CLHM helper script that checks agcount in the root filesystem (if it is XFS) and give the user a proper message if it's too low.
17:07:16 <dustymabe> #topic Low memory machines fails to intialize/boot fcos
17:07:20 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1540
17:07:32 <dustymabe> If this issue seems like the last one, I assure you they are different :)
17:07:48 <travier> :)
17:07:57 <dustymabe> in this case someone is trying to boot a 512M AWS instance and it can't even load the initramfs
17:08:23 <dustymabe> I guess the open question I have is.. do we care?
17:08:37 <dustymabe> also do we have any docs anywhere (I couldn't find any) that specifies minimum requirements
17:09:06 <travier> https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/welcome/Hardware_Overview/#hardware_overview-specs
17:09:36 <travier> I don't think we have FCOS specific requirements
17:10:01 <spresti> wow thats like a pi zero
17:10:26 <dustymabe> travier: nice - I didn't find that when I searched before
17:10:49 <dustymabe> honestly I think that bit of documentation is enough
17:10:54 <travier> We could add a small note in https://docs.fedoraproject.org/en-US/fedora-coreos/platforms/ maybe?
17:11:15 <dustymabe> TBH I'm completely surprised 512M was working at all before
17:11:44 <dustymabe> thought it's nice to know we were fitting in that footprint OK and still able to update (though I guarantee no packages were layered)
17:11:52 <dustymabe> up until recently
17:12:20 <dustymabe> #info we have documentation for minimum system requirements at https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/welcome/Hardware_Overview/#hardware_overview-specs
17:12:36 <dustymabe> I don't think we need a #agreed for this one. Do you?
17:13:42 <jmarrero> I think that makes sense.
17:13:56 <travier> I think we could go as low as 1CPU 1GB RAM 15GB disk but that's so small you would not run that many containers on it
17:14:23 <dustymabe> yeah, in this case the user was using the host as a bastion (so just ssh running)
17:14:50 <travier> or we default to the Fedora ones and users are free to try below, but we're not obligated to make sure it works
17:15:30 <dustymabe> #topic tracker: Fedora 39 changes considerations
17:15:35 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1491
17:15:51 <dustymabe> ok there are a few we haven't covered yet
17:16:05 <dustymabe> subtopic 126. Deprecating libuser and removing passwd package from Fedora
17:16:12 <dustymabe> #link https://fedoraproject.org/wiki/Changes/LibuserDeprecation
17:16:28 <dustymabe> travier: jmarrero: I was going to lean on you here to tell me if rpm-ostree was OK with this change
17:16:48 <travier> I don't think that should impact us
17:17:11 <dustymabe> travier: any other supported context I should add to the NOTE about it?
17:17:17 <dustymabe> supporting*
17:18:33 <travier> rpm-ostree and ostree do not link to it
17:18:51 <travier> as far as I can see
17:18:52 <dustymabe> #info This should not impact us. rpm-ostree and ostree do not link to libuser.
17:19:00 <dustymabe> travier: that's good enough for me
17:19:13 <dustymabe> subtopic 128. LLVM 17
17:19:20 <dustymabe> #link https://fedoraproject.org/wiki/Changes/LLVM-17
17:19:54 <dustymabe> #info This should not affect us or should be transparent.
17:20:16 <dustymabe> subtopic 129. Migrate NetworkManager ifcfg profiles to keyfile
17:20:25 <dustymabe> #link https://fedoraproject.org/wiki/Changes/MigrateIfcfgToKeyfile
17:20:42 <dustymabe> #info This should not affect us. Fedora CoreOS never supported ifcfg style network configuration files.
17:20:43 <travier> we've already done that haven't we?
17:20:50 <travier> +1
17:20:59 <dustymabe> travier: yep. We never supported the old style from the beginning (we actively removed the plugin)
17:21:11 <dustymabe> subtopic 130. IBus 1.5.29
17:21:18 <dustymabe> #link https://fedoraproject.org/wiki/Changes/IBus_1.5.29
17:21:30 <travier> does not impact us
17:21:56 <dustymabe> #info Nothing to do. We don't ship IBUS.
17:22:11 <dustymabe> subtopic 131. Use Noto fonts for Indic (Indian language) scripts
17:22:17 <dustymabe> #link https://fedoraproject.org/wiki/Changes/Indic_Noto_fonts
17:22:19 <travier> nothing to do
17:22:27 <dustymabe> #info We don't ship fonts.
17:22:40 <dustymabe> subtopic 132. Anaconda WebUI for Fedora Workstation by default
17:22:49 <travier> we don't ship anaconda
17:22:50 <dustymabe> #link https://fedoraproject.org/wiki/Changes/AnacondaWebUIforFedoraWorkstation
17:22:58 <dustymabe> it's interesting to me that this is a system wide change
17:23:22 <dustymabe> #info Nothing for us to do. We don't ship Anaconda and this change is targetted at Workstation.
17:23:35 <dustymabe> subtopic 133. Build JDKs once, repack everywhere
17:23:41 <dustymabe> #link https://fedoraproject.org/wiki/Changes/BuildJdkOncePackEverywhere
17:23:54 <dustymabe> #info Nothing to do. We don't ship Java.
17:24:06 <dustymabe> subtopic 134. GNU Toolchain Update (gcc 13.2, binutils 2.40, glibc 2.38, gdb 13.2)
17:24:12 <dustymabe> #link https://fedoraproject.org/wiki/Changes/GNUToolchainF39
17:24:39 <dustymabe> #info This would should be transparent to us. We'll pick up packages as they get rebuilt and actively test them in our `rawhide` stream.
17:25:01 <dustymabe> ok now moving on to the self contained changes
17:25:12 <dustymabe> someone please stop me if there is anything to bring up!
17:25:23 <dustymabe> subtopic 220. Clean Systemd-boot installs
17:25:30 <dustymabe> #link https://fedoraproject.org/wiki/Changes/cleanup_systemd_install
17:26:17 <travier> does not impact us
17:26:30 <dustymabe> This one is interesting though
17:26:51 * dustymabe often wonders if eventually GRUB should be replaced
17:27:07 <dustymabe> #info Nothing for us to do. We will continue to use GRUB for now.
17:27:22 <dustymabe> subtopic 221. No custom Qt theming for Fedora Workstation
17:27:29 <dustymabe> #link https://fedoraproject.org/wiki/Changes/NoCustomQtThemingForWorkstation
17:27:40 <dustymabe> #info Nothing for us to do. This one is Workstation specific.
17:27:50 <dustymabe> subtopic 222. Retire Modularity
17:27:56 <dustymabe> #link https://fedoraproject.org/wiki/Changes/RetireModularity
17:28:20 <dustymabe> Well I can't say I didn't see this one coming ever since modularity was introduced :)
17:28:55 <dustymabe> We've already done some work to remove the modular repos from F39+ so that should be all.
17:29:12 <dustymabe> though I think maybe a little more work to make sure updating systems don't have issues
17:29:37 <dustymabe> this was/is the issue for that https://github.com/coreos/fedora-coreos-tracker/issues/1513
17:30:03 <dustymabe> #info We've already done some work in https://github.com/coreos/fedora-coreos-tracker/issues/1513 to remove modular repos from F39+.
17:30:24 <dustymabe> subtopic 223. Sericea and Sway Spin Xorg-less
17:30:32 <dustymabe> #link https://fedoraproject.org/wiki/Changes/sericea-xorgless
17:30:45 <dustymabe> #info Nothing for us to do. This is Sericea and Sway spin specific.
17:30:58 <dustymabe> subtopic 224. Further reduce Fedora-specific build flags in non-RPM Python extensions
17:31:04 <dustymabe> #link https://fedoraproject.org/wiki/Changes/Python_Extension_Flags_Reduction
17:31:24 <dustymabe> #info Nothing for us to do. We don't ship python.
17:31:35 <dustymabe> subtopic 225. Passkey authentication for centrally managed users
17:31:45 <dustymabe> #link https://fedoraproject.org/wiki/Changes/Passkey_authentication_centrally_managed_users
17:32:12 <dustymabe> hmm. I don't know if anyone has tried to register an FCOS system with AD, but I guess it would work
17:32:36 <dustymabe> #info No work for us to do here. If there are AD managed FCOS systems I imagine the change could be picked up by them.
17:32:44 <dustymabe> Not sure if that ^^ is completely accurate, but let me know
17:32:55 <dustymabe> subtopic 226. ibus-anthy 1.5.15
17:33:03 <dustymabe> #link https://fedoraproject.org/wiki/Changes/ibus-anthy_1.5.15
17:33:12 <dustymabe> #info Nothing for us to do. We don't ship ibus.
17:33:22 <dustymabe> subtopic 227. Enable auto-updates by default in Fedora Kinoite
17:33:29 <dustymabe> #link https://fedoraproject.org/wiki/Changes/KDEKinoiteAutoUpdateByDefault
17:33:31 <travier> nothing to do :)
17:33:42 <dustymabe> #info Nothing for us to do. This is kinoite specific.
17:33:49 <dustymabe> #topic open floor
17:34:04 <travier> thanks for going through this
17:34:06 <dustymabe> travier: though I do wonder - you are enabling auto updates on kinoite but not silverblue?
17:34:20 <travier> it's "already" enabled on Silverblue :)
17:34:25 <dustymabe> oh, nice
17:34:30 <dustymabe> didn't know that
17:34:42 <travier> This is GUI specific. Silverblue does it in GNOME Software, Kinoite in Discover
17:34:45 <dustymabe> but I do run SB - so it's weird that I didn't know that
17:34:50 <dustymabe> ahh ok that's why
17:34:56 <dustymabe> I run i3wm and not Gnome
17:35:04 <dustymabe> anyone with any topics for open floor?
17:35:24 <dustymabe> we do need someone to start working on https://github.com/coreos/fedora-coreos-tracker/issues/1502 soon - beta is right around the corner
17:35:43 <travier> indeed
17:35:51 <dustymabe> oh and I guess we could use a volunteer for the work we discussed earlier about https://github.com/coreos/fedora-coreos-tracker/issues/1535
17:36:21 <dustymabe> (sorry for letting the meeting run long)
17:36:53 <travier> well, we skipped a week
17:37:03 <dustymabe> i'll end the meeting soon unless new discussion starts
17:37:36 <dustymabe> #endmeeting