16:29:14 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:14 <zodbot> Meeting started Wed Jun 28 16:29:14 2023 UTC.
16:29:14 <zodbot> This meeting is logged and archived in a public location.
16:29:14 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:14 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:19 <dustymabe> #topic roll call
16:29:43 <ravanelli> .hi
16:29:44 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:29:49 <jdoss> .hi
16:29:50 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:30:02 <gshomo> .hi
16:30:03 <zodbot> gshomo: gshomo 'greg shomo' <gshomo@gmail.com>
16:30:04 <quentin9696[m]> .hi
16:30:07 <zodbot> quentin9696[m]: Sorry, but user 'quentin9696 [m]' does not exist
16:30:59 <dustymabe> #chair ravanelli jdoss gshomo quentin9696[m]
16:30:59 <zodbot> Current chairs: dustymabe gshomo jdoss quentin9696[m] ravanelli
16:31:33 <marmijo[m]> .hello marmijo
16:31:34 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:32:08 <dustymabe> #chair marmijo[m] Guidon
16:32:08 <zodbot> Current chairs: Guidon dustymabe gshomo jdoss marmijo[m] quentin9696[m] ravanelli
16:32:25 <apiaseck> .hello c4rt0
16:32:26 <zodbot> apiaseck: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:32:38 <dustymabe> #chair apiaseck
16:32:38 <zodbot> Current chairs: Guidon apiaseck dustymabe gshomo jdoss marmijo[m] quentin9696[m] ravanelli
16:33:23 <dustymabe> ok let's get started
16:33:28 <dustymabe> #topic Action items from last meeting
16:33:35 <dustymabe> #info there were no action items from last meeting
16:33:57 <dustymabe> but I do know that last time we did report that travier opened https://github.com/coreos/fedora-coreos-tracker/issues/1512
16:34:08 <travier> .hello siosm
16:34:09 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:34:13 <dustymabe> ravanelli: I know you were working on that with him.. were you able to get a change proposal submitted?
16:34:41 <mnguyen_> .hello mnguyen
16:34:42 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:34:47 <dustymabe> #chair travier mnguyen_
16:34:47 <zodbot> Current chairs: Guidon apiaseck dustymabe gshomo jdoss marmijo[m] mnguyen_ quentin9696[m] ravanelli travier
16:34:55 <ravanelli> dustymabe: I created https://fedoraproject.org/wiki/Changes/EnableFwupdRefreshByDefault
16:35:24 <ravanelli> I would appreciate some reviews/inputs on it
16:35:26 <dustymabe> Nice!
16:35:30 <ravanelli> So we can move it to page complete
16:35:54 <travier> ravanelli: Can you paste the link to that page in the tracking issue?
16:36:00 <dustymabe> #info please review https://fedoraproject.org/wiki/Changes/EnableFwupdRefreshByDefault and provide feedback for travier and ravanelli
16:36:16 <travier> We'll have to share it with the server & iot groups after that
16:36:41 <ravanelli> issue: https://github.com/coreos/fedora-coreos-tracker/issues/1512
16:36:56 <dustymabe> nice work to you both
16:37:02 <dustymabe> i'll try to give my feedback
16:37:20 <ravanelli> 👍
16:37:26 <dustymabe> #topic F39 Change: No default fedora-repos-modular [Consideration]
16:37:26 <travier> yes, I meant: could you paste the link to the wiki in the github issue?
16:37:40 <ravanelli> travier: ooh, yes
16:37:43 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1513
16:37:47 <travier> 👍
16:38:09 <spresti> .hi
16:38:10 <zodbot> spresti: spresti 'Steven Presti' <spresti@redhat.com>
16:38:23 <dustymabe> We discussed the current $topic a bit last week (summarized in https://github.com/coreos/fedora-coreos-tracker/issues/1513#issuecomment-1601239985)
16:38:55 <dustymabe> any thoughts on the path forward for us on the two options presented in that comment?
16:39:05 <dustymabe> A) try to detect when someone has layered packages and add in fedora-repos-modular as a layered package itself
16:39:08 <travier> I'd would lean towards communication only
16:39:12 <dustymabe> B) send communications and give users commands to use to convert the package to a layered package before the change lands
16:39:16 <travier> so B
16:40:06 <spresti> The only problem with B (IMO) is it is kind of a silent fail right?
16:40:18 <dustymabe> spresti: indeed
16:40:21 <travier> It should be easy to fix (?) and we can't always guarentee that arbitrary layered packages will always work on updates
16:40:35 <dustymabe> travier: correct
16:41:02 <dustymabe> I think the only semi-common case of layering a modular package may be cri-o
16:41:23 <travier> I think this is part of the things that we can not guarentee. We can not guarentee updates will always work once you layer packages
16:41:45 <spresti> Could we create a unit that detects this? and logs warnings (saying that repo x is no longer included) then roll back to fix their update path?
16:42:37 <dustymabe> spresti: not sure I understand - the machine won't need rolling back (IIUC) because it failed the update and it's just happy sitting there on the old version
16:43:23 <spresti> Yeah I used bad verbage "rollback" I mean default the setup aka option A
16:43:51 <spresti> I think I am leaning more to A (from my perspective)
16:43:58 <spresti> Sorry for the confusion.
16:44:32 <dustymabe> yeah, option A is an option. The question is mostly whether it's worth the effort here (i.e. developing a solution, testing the solution, making sure it keeps working over time)
16:45:02 <dustymabe> and also trying to make it only apply the workaround if actually needed (i.e. modular packages exist)
16:45:12 <dustymabe> modular layered packages*
16:45:53 <dustymabe> I think I'm leaning towards B here too.
16:46:21 <dustymabe> but would be willing to revisit our solution if beta testing (i.e. when `next` switches to F39) yields bad results or new information
16:46:33 <spresti> So at what point is a non-updated server a breaking problem?
16:47:12 <dustymabe> define "breaking problem"?
16:48:23 <spresti> Breaking being not able to update after applying change
16:48:44 <dustymabe> I think there is some nuance here..
16:49:06 <dustymabe> in this particular case the core OSTree doesn't have any problem updating
16:49:24 <dustymabe> it's IF you applied layered packages AND they were modular packages coming from modular repos
16:49:47 <dustymabe> when you layer packages you assume some risk (i.e. we don't test that upgrade path in FCOS CI)
16:50:22 <dustymabe> so I don't see it as a big deal breaker that some machines could stop to update.. after all there are multiple ways people could ID the problem and resolve it
16:50:26 <spresti> ^ thats what I needed thank you; yeah B makes most sense.
16:50:36 <dustymabe> 1 - watch coreos-status email announcements (where we will tell them about this)
16:51:02 <dustymabe> 2 - run `next` and preview what's coming. If your `next` nodes stop updating then you should try to figure out why
16:52:09 <mnguyen_> +1 to B
16:52:22 <spresti> B++
16:53:00 <dustymabe> #proposed We will communicate the removal of modular repos from Fedora CoreOS in F39 and give users steps to resolve the problems on the systems. If users are running `next` or `testing` streams they will also notice updates not coming in before there `stable` systems stop receiving updates.
16:53:59 <dustymabe> votes?
16:54:02 <apiaseck> +1
16:54:06 <ravanelli> dustymabe: Would mind explaning how users would solve it?
16:54:28 <ravanelli> would it be before trying to update?
16:54:48 <travier> +1
16:55:17 <travier> uninstalling the pacakge and re-installing the non-modular version should do it
16:55:21 <quentin9696[m]> +1
16:55:30 <travier> agree that we should give an example with the announcemment
16:55:35 <dustymabe> ravanelli: assuming the modular package that the user was layering is still a modular package, then I think they could just resolve the problem by re-layering in the `fedora-repos-modular` package (i.e. if it is currently a base package then it will be tracked as an inactive base package, but the request will exist).
16:55:56 <travier> or indeed re-adding the repo works
16:56:21 <travier> apparently modules will be going away in Fedora at some point given the lack of traction
16:56:31 <ravanelli> Thanks dustymabe and travier
16:56:35 <ravanelli> +1 so
16:56:35 <dustymabe> ravanelli: but yes, there will be a way for the user to run a command on their current system (non-updated) to put their machine in a position to auto-update again
16:56:56 <dustymabe> #agreed We will communicate the removal of modular repos from Fedora CoreOS in F39 and give users steps to resolve the problems on the systems. If users are running `next` or `testing` streams they will also notice updates not coming in before there `stable` systems stop receiving updates.
16:57:03 <dustymabe> yep.. I think examples will be useful
16:57:09 <dustymabe> ok moving on to the next ticket
16:57:17 <dustymabe> #topic Adding an LVM devices file by default
16:57:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1517
16:57:41 * dustymabe will give some time here for people to read the description
16:58:46 <dustymabe> TL;DR both Host (CoreOS) and VM Guest (could be any linux) are trying to control/access block devices with LVM signatures created by the guest.
16:59:49 <jdoss> This seems like a good addition. I am using FCOS with qemu to run VMs. It makes the a great VM host node OS. Anything that makes running VMs easier is a +1 from me. I wish qemu could be included too so I didn't have to layer it, but I know that is a bigger discussion.
17:00:11 <dustymabe> jdoss: +1
17:01:09 <jdoss> It doesn't help that qemu pulls the kitchen sink in with it.
17:01:38 <dustymabe> ha - but that's part of what makes qemu great.. it is a swiss army knife
17:01:52 <jdoss> yeah for sure
17:02:22 <dustymabe> The approach listed in the ticket is to:
17:02:29 <dustymabe> 1) create an empty devices file for new installs
17:02:38 <dustymabe> 2) migrate existing systems to use and lvmdevices file
17:03:17 <travier> looks good to me
17:03:29 <dustymabe> this goal is that 2) is a one time migration and 1) is something we do for new installs for now, but hopefully have lvm2 RPM own that piece in the future (I opened an RFE at https://bugzilla.redhat.com/show_bug.cgi?id=2217510)
17:03:35 <travier> not sure we should provide the transition script but we can try
17:04:24 <dustymabe> any glaring problems with this proposal (before I do #proposed)?
17:05:15 <dustymabe> actually - a good question here is when we should roll this out? immediately? do we need to give some warnings to users?
17:05:44 <dustymabe> the migration script should behave such that existing LVM devices attached to a system (i.e. if someone was using a block device with LVM on it for /var/) should continue to work
17:06:14 <dustymabe> what wouldn't work is if say someone did a new install and then attached an existing block device to the newly installed system with a previously created LV
17:07:02 <travier> I don't think that's something we can reasonably support
17:07:04 <dustymabe> but they could be easily imported with a single command
17:07:45 <travier> for PXE / ephemeral setups, it would mean that we would need to keep the migration script forever
17:08:00 <travier> (maybe?)
17:08:22 <dustymabe> well.. it would just mean the user would have to change their provisioning to include a file or an import step
17:08:26 <travier> 🤔
17:09:01 <travier> or they could remove the file via Ignition?
17:09:05 <dustymabe> so they could write the lvmdevices file via Ignition with the contents they want or run an import command (provided by LVM) to do the same
17:09:14 <dustymabe> travier: correct.. they could remove the file too
17:09:41 <dustymabe> to get back to the current behavior
17:09:50 <travier> hum, can we actually remove files with Ignition?
17:09:57 <dustymabe> I believe we can
17:10:16 <dustymabe> spresti: might recall off the top. I'd need to look up docs
17:10:20 <travier> https://github.com/coreos/ignition/issues/739
17:10:45 <dustymabe> ahh - ok then :)
17:11:03 <spresti> Sorry not of the top of my head
17:11:06 <travier> so this needs a tmpfiles.d entry or a script but it's doable
17:11:12 <dustymabe> right
17:12:10 <dustymabe> travier: so given that limitation, what do you still think of 1) and 2) ?
17:12:41 <travier> I think we should do 1
17:12:52 <travier> 2 is more complex, not sure if we should do it
17:13:20 <travier> but if we end up doing it for RHCOS, then might as well have it in FCOS
17:13:40 <dustymabe> well. we kind of have to do 2 (at least a one time migration) if we do 1. because otherwise people's upgrading systems stop working
17:14:10 <travier> we kind of don't support LVM in ignition but fair
17:14:12 <dustymabe> or I see.. 1) only applies to new installs
17:14:42 <travier> yes, only for new installations, but that's maybe harder to do
17:15:24 <dustymabe> we could definitely do it - it just wouldn't be using a file (there would have to be a unit with some logic)
17:16:42 <travier> those kind of automated migations are really hard to do as we don't know what users have put in place
17:16:53 <travier> what if there is already a file there?
17:17:02 <dustymabe> maybe we can go with this proposal for now and if we find any new information during testing we can bring it back to the ticket?
17:17:08 <dustymabe> #proposed we will ship an empty lvmdevices file for new installs and also add a migration script for existing systems that will populate an lvmdevices file with appropriate content so that existing systems using LVM will continue to work.
17:17:22 <travier> +1
17:17:27 <spresti> +1
17:18:25 <dustymabe> any other votes?
17:18:38 <apiaseck> I need time to digest it
17:18:52 <travier> 🥪
17:18:57 <apiaseck> +1
17:19:08 <ravanelli> +1
17:19:15 <marmijo[m]> +1
17:19:23 <dustymabe> #agreed we will ship an empty lvmdevices file for new installs and also add a migration script for existing systems that will populate an lvmdevices file with appropriate content so that existing systems using LVM will continue to work.
17:19:46 <dustymabe> Of course, if new information comes out or if we find a better way to achieve the goal. We'll bring that information to the ticket and pivot.
17:19:59 <dustymabe> #topic tracker: Fedora 39 changes considerations
17:20:04 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1491
17:20:09 <travier> 👍
17:20:23 <dustymabe> ok - I updated the description this morning with new changes that have come in
17:20:35 <dustymabe> subtopic 117. Retire AWS CLI version 1 package awscli
17:21:44 <dustymabe> I think maybe the only thing that this implies for us is that we need to make sure the new package and not the old package is getting installed in COSA...
17:21:50 <dustymabe> and... I just checked
17:21:58 <dustymabe> [coreos-assembler]$ rpm -qa | grep aws
17:22:00 <dustymabe> python3-awscrt-0.16.19-1.fc38.x86_64
17:22:02 <dustymabe> awscli2-2.11.18-1.fc38.noarch
17:22:07 <dustymabe> so we already have `awscli2`
17:22:33 <dustymabe> so no action for us?
17:23:08 <quentin9696[m]> quixk question about taht
17:23:09 <spresti> Well that turned out to be easy.
17:23:31 <travier> 👍
17:23:32 <dustymabe> quentin9696[m]: go for it
17:23:32 <quentin9696[m]> why FCOS don't include by default awscli on AWS AMI ?
17:23:54 <dustymabe> quentin9696[m]: maybe let's cover that in open floor
17:24:06 <quentin9696[m]> dustymabe: sure
17:24:08 <dustymabe> subtopic 118. No fedora-repos-modular in default installation
17:24:26 <dustymabe> ok this is the topic we discussed earlier - covered by https://github.com/coreos/fedora-coreos-tracker/issues/1513
17:24:39 <dustymabe> subtopic 119. LIBFFI 34 static trampolines
17:25:18 <dustymabe> I think this one should be transparent to us. No changes for us to implement since we just consume packages.
17:25:31 <dustymabe> 👍/👎 ?
17:26:06 <spresti> If its just consumed then +1
17:26:14 <travier> do we ship that?
17:26:27 <travier> we do
17:26:45 <dustymabe> subtopic 120. Flatpaks without Modules
17:26:55 <dustymabe> we dont ship flatpak, nothing for us to do.
17:27:05 <dustymabe> subtopic 121. Increase vm.max_map_count value
17:27:08 <travier> 119: should be transparent
17:27:51 <dustymabe> increasing vm.max_map_count should be transparent to us. I don't know of a reason this would cause problems
17:28:09 <travier> +1
17:28:11 <dustymabe> subtopic 122. Make Toolbx a release-blocking deliverable and have release-blocking test criteria
17:29:06 <dustymabe> We ship the toolbox software but not the toolbox image (gets pulled from registry). Overall this shouldn't affect us (other than making sure the toolbox container images are in good shape before release).
17:29:30 <dustymabe> subtopic 217. Aspell Deprecation
17:29:42 <dustymabe> I'm pretty sure we don't ship this.
17:29:53 <dustymabe> subtopic 218. Automatic Cloud Reboot On Updates
17:30:06 <dustymabe> This has to do with the Fedora Cloud image and cloud-init specifically. Doesn't apply to us.
17:30:15 <dustymabe> subtopic 219. Vagrant 2.3
17:30:19 <dustymabe> We don't ship vagrant
17:30:27 <dustymabe> :)
17:30:45 <dustymabe> Anything on those that we should go over before moving to open floor?
17:30:53 <spresti> Wow awesome thank you for running through them so fast
17:31:10 <dustymabe> spresti: we were running out of time and the last few were easy :)
17:31:14 <dustymabe> #topic open floor
17:31:17 <dustymabe> quentin9696[m]: you had something?
17:31:55 <quentin9696[m]> yes, can we add by default AWS CLI on AWS AMI provided by FCOS ?
17:31:58 <travier> LGTM for the other changes
17:32:05 <dustymabe> FYI: I won't be around next week for the meeting. I know there will be a bunch of other people out too for Holiday. Someone could volunteer to run the meeting OR we could cancel the next one.
17:32:34 <travier> we don't include any cloud tools in the image (all images have the same packages)
17:32:54 <dustymabe> quentin9696[m]: we ship the same FCOS image everywhere, so that would mean we'd need to ship the AWS CLI packages and gcloud packages and Azure client packages everywhere
17:33:01 <dustymabe> the list goes on
17:33:01 <travier> I should be able to run the meeting
17:33:08 <dustymabe> travier++
17:33:14 <spresti> thank you travier
17:33:21 <gshomo> thank you, everyone !
17:33:41 <dustymabe> quentin9696[m]: so it gets rather large if you try to include them all.. running them from a container however - pretty easy to do
17:34:02 <dustymabe> you could even do it in toolbox if you like (just install the packages you want/need)
17:34:13 <travier> (gcloud is very big from memory)
17:34:27 <quentin9696[m]> dustymabe: oh ok, in my mind you shipped modified version to adapt to the cloud provider. That's ok for me, it was just curious about that
17:34:54 <dustymabe> quentin9696[m]: the only real modifications we make to images is to ship a different platform ID in the kernel arguments for the image
17:34:56 <quentin9696[m]> dustymabe: actually we run a container with AWS CLI
17:35:14 <dustymabe> all the software inside the image can then "differ" in behavior based on that platform ID kernel argument
17:35:25 <dustymabe> so it's all the same bits inside, they just may behave slightly different
17:35:49 <dustymabe> any other topics for open floor?
17:36:01 <Guidon> For aws-cli: just so you know the issue: running a container is very heavy (RAM/CPU). It’s really a pain to manager.
17:36:13 <Guidon> Especially on low-RAM machines
17:36:16 <quentin9696[m]> thanks for the explanations
17:37:05 <dustymabe> Guidon: it depends on what that container is doing :) after all a container is mostly just a linux process
17:37:46 <dustymabe> maybe grabbing the container image and laying it out on the filesystem is a bit compute/network intensive, but once the setup happens and the process is running I wouldn't expect to have a different footprint than running the process natively
17:38:32 * dustymabe will close the meeting now
17:38:38 <dustymabe> #endmeeting