16:30:19 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:19 <zodbot> Meeting started Wed Aug  2 16:30:19 2023 UTC.
16:30:19 <zodbot> This meeting is logged and archived in a public location.
16:30:19 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:19 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:23 <dustymabe> #topic roll call
16:30:25 <dustymabe> .hi
16:30:26 <zodbot> dustymabe: Something blew up, please try again
16:30:28 <spresti> .hi
16:30:29 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:32 <zodbot> spresti: Something blew up, please try again
16:30:34 <zodbot> spresti: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:43 <ravanell_> .hi
16:30:44 <zodbot> ravanell_: Something blew up, please try again
16:30:47 <zodbot> ravanell_: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:49 <jmarrero> .hi
16:30:50 <zodbot> jmarrero: Something blew up, please try again
16:30:53 <zodbot> jmarrero: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:15 <JasonBrooks[m]> .hello jasonbrooks
16:31:16 <zodbot> JasonBrooks[m]: Something blew up, please try again
16:31:20 <zodbot> JasonBrooks[m]: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:25 * jmarrero hides from the explosions
16:31:25 <dustymabe> zodbot must be at flock today
16:31:36 <JasonBrooks[m]> 🙂
16:31:49 <spresti> lol
16:31:57 <ravanell_> lol
16:32:25 <dustymabe> #chair spresti ravanell_ jmarrero JasonBrooks[m]
16:32:25 <zodbot> Current chairs: JasonBrooks[m] dustymabe jmarrero ravanell_ spresti
16:32:42 <marmijo[m]> .hello marmijo
16:32:43 <zodbot> marmijo[m]: Something blew up, please try again
16:32:46 <zodbot> marmijo[m]: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:33:40 <dustymabe> #chair marmijo[m]
16:33:40 <zodbot> Current chairs: JasonBrooks[m] dustymabe jmarrero marmijo[m] ravanell_ spresti
16:33:46 <dustymabe> ok let's get started :)
16:33:59 <dustymabe> #topic Action items from last meeting
16:34:34 <dustymabe> we've had a few meetings where attendance and topics were sparse (I know I've been out a few Wednesdays in July)
16:34:42 <dustymabe> so..
16:34:50 <dustymabe> #info There were no action items from last meeting
16:35:28 <dustymabe> moving on to meeting tickets (reminder, please add the meeting label to tickets you'd like to discuss)
16:35:32 <dustymabe> #topic kola reprovision tests are failing on ppc64le
16:35:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1489
16:36:31 <dustymabe> For this one we have some pretty basic provisioning tests failing on ppc64le
16:36:42 <dustymabe> which is an architecture we just added to our production streams
16:37:07 <dustymabe> we first saw the issue in rawhide and reported a bug against the kernel in Fedora
16:37:42 <dustymabe> now it is in Fedora 38 since the 6.4 kernel has landed there
16:38:18 <dustymabe> marmijo[m]: and I did a kernel git bisect yesterday and found the offending commit: https://github.com/coreos/fedora-coreos-tracker/issues/1489#issuecomment-1661427107
16:38:54 <dustymabe> however, there is currently no fix. We need to decide if we want to pin the kernel on 6.3 for our production streams OR decide to ship the 6.4+ kernel with this regression
16:39:57 <jmarrero> Can we pin only for ppc64le?
16:40:37 <dustymabe> jmarrero: good question
16:40:44 <dustymabe> in theory we can pin only for ppc64le
16:41:00 <dustymabe> but I don't think we've ever tried it
16:41:52 <dustymabe> it's typically nice to know that the software is the same across the board, but this could be a good excuse to flex that muscle
16:42:10 <dustymabe> should we explore that as an option?
16:42:15 <marmijo[m]> That might be a good option to explore so that our new ppc64le builds are stable for our users.
16:43:02 <spresti> I think it makes sense to do so. I am worried how often we would revisit this pin ? eventually there will be a breaking point?
16:43:03 <ravanell_> dustymabe: I think we already did it in the past, pin only for some arch the kernel
16:43:12 * ravanell_ checking
16:43:39 <dustymabe> ravanell_: yeah I would be interested to know if you can find an example (for fcos). from my memory I don'
16:43:42 <dustymabe> don't recall ever doing it
16:43:58 <jmarrero> I would imagine we would lift the pin once it's fixed upstream or if there is a CVE on the pinned version?
16:44:24 <dustymabe> jmarrero: correct
16:44:44 <dustymabe> ok so plan A would be to only pin for ppc64le
16:45:02 <dustymabe> but let's for a moment assume that we can't do that for some reason
16:45:27 <dustymabe> if we can't pin for only one arch - should we pin for all, or ship the update and tell ppc64le users to use old boot media
16:45:37 <dustymabe> IIUC the problem is related to first boot so existing systems should be fine
16:46:41 <spresti> I think pining for all honestly.
16:46:44 <marmijo[m]> Does that mean that updates wouldnt see the regression, only new installs?
16:47:04 <jmarrero> Or ppc64le install base is tiny no?
16:47:09 <jmarrero> Our*
16:48:02 <dustymabe> marmijo[m]: I think so. I think only new installs are going to be doing "root reprovisioning" which are the tests that are failing
16:48:30 <jmarrero> I think we could pin for all, and unpin if there is a reason to. But I would be fine with with telling the ppc64le to use older boot media too.
16:48:38 <dustymabe> but in general it seems like there is some fundamental issues with zram block devices on that architecture so there may be real world applications that hit problems too
16:49:15 <dustymabe> jmarrero: yes, I think the number of users here is ...
16:49:22 <dustymabe> almost no one other than us
16:49:26 <dustymabe> but maybe
16:50:04 <dustymabe> i think the countme stats recently were like 10 ppc64le nodes
16:50:37 <dustymabe> so we'll still explore plan A (pin only on ppc64le)
16:51:04 <dustymabe> but in the event plan A doesn't pan out - let's just ship the new kernel?
16:51:15 <dustymabe> i'm really on the fence about it
16:51:16 <marmijo[m]> I think that’s the ideal plan for now. I’d also vote for just informing our users if it’s really that low.
16:51:40 <dustymabe> yeah, though informing the users is kind of not ideal either
16:51:58 <dustymabe> sending a coreos-status post isn't great if it won't affect 99% of users
16:52:39 <dustymabe> spresti: I heard you mention some support for full pin
16:53:26 <spresti> Yeah, I am worried about the deviation between arch's.
16:53:49 <spresti> Since we dont know the turn around on a fix, a pin on one arch scares me a little
16:54:00 <spresti> But the install base being small, maybe thats ok.
16:54:45 <dustymabe> I think I'm fully in the "pin on one arch if we can"
16:54:47 <dustymabe> camp
16:55:01 <dustymabe> i think the open question for me is if we can't pin on one arch
16:55:18 <dustymabe> maybe we just have a vote and see where people are at
16:55:34 <dustymabe> 1. pin for all if we can't pin for one
16:55:51 <dustymabe> 2. let the update land for all and instruct ppc64le users on how to workaround issues
16:56:41 <spresti> 1
16:56:45 <ravanell_> 1
16:56:54 <dustymabe> for me i'm thinking maybe 2. - or maybe we go with 1. for now and then 2. next round or if any CVEs come in
16:57:18 <dustymabe> marmijo[m]: jmarrero: ?
16:57:40 <marmijo[m]> dustymabe: I'm in that same camp. 1 for now but 2 if we need to unpin the other arches
16:57:41 <jmarrero> 1 for now, then 2 if we need to
16:58:26 <dustymabe> ok cool
16:58:32 <dustymabe> let me type up a proposed
16:59:41 <dustymabe> #proposed Since our install base is low we will attempt to pin the kernel for just ppc64le. If that is not an option then we will pin for all for next week's releases and then unpin for all the next cycle or if an important CVE comes in.
17:00:27 <marmijo[m]> +1
17:00:30 <jmarrero> +1
17:00:32 <spresti> +1
17:00:39 <JasonBrooks[m]> +1
17:01:07 <dustymabe> #agreed Since our install base is low we will attempt to pin the kernel for just ppc64le. If that is not an option then we will pin for all for next week's releases and then unpin for all the next cycle or if an important CVE comes in.
17:01:24 <dustymabe> anyone want to volunteer to explore pinning just for ppc64le?
17:01:40 <marmijo[m]> I can take a look at it
17:02:26 <dustymabe> #action marmijo[m] will explore trying to pin the kernel just on one architecture to see if we think it will work for the next round of releases.
17:02:29 <dustymabe> thanks marmijo[m]
17:02:41 <dustymabe> #topic Platform Request - Apple native hypervisor
17:02:41 <marmijo[m]> dustymabe: thank you!
17:02:46 <dustymabe> #link  https://github.com/coreos/fedora-coreos-tracker/issues/1533
17:03:08 <dustymabe> this is a request for a new platform/artifact from baude and the podman team
17:04:55 * dustymabe was going to see if baude was around to answer any questions we may have
17:05:12 <dustymabe> I like how there was no response to "What is the official name of the platform?"
17:06:08 <marmijo[m]> Apple is weird with their naming conventions sometimes. I did a little digging and I think it's literally just called "Hypervisor". I could be wrong though
17:06:24 <dustymabe> fun
17:06:33 <spresti> I would be excited to see this. Since it was just added today, maybe we can ping him and have him attend next meeting for questions?
17:06:53 <dustymabe> he was around earlier
17:07:00 <jmarrero> https://developer.apple.com/documentation/hypervisor
17:07:15 <dustymabe> spresti: i know him and bgilbert did some exploration on the topic in the past - have you seen PRs related to this work?
17:07:23 <JasonBrooks[m]> "The Hypervisor framework"
17:08:04 <spresti> @dustymabe no I have not; I have only seen the ones regarding windows hyperv
17:09:18 <marmijo[m]> We could probably refer to it as "Apple Hypervisor".
17:09:27 <dustymabe> yeah
17:09:40 <dustymabe> I think I've heard baude refer to it as `applehv` before
17:10:27 <dustymabe> It's interesting that it says there is no console provided by default
17:10:41 <dustymabe> i.e. no serial console and no VGA console?
17:10:45 <dustymabe> that would be interesting
17:10:56 <jmarrero> I think applehv is x86_64 only but that is only what reddit says :D
17:11:08 <dustymabe> :)
17:11:16 <dustymabe> ok we'll ask more questions in the ticket
17:11:29 <marmijo[m]> I own some apple silicon so I would be excited to see this as well
17:11:31 <jmarrero> There is a demo I saw a month back from wwdc on how to start and instance programatically
17:11:31 <dustymabe> i'll ask about the console and the official name
17:12:12 <dustymabe> anyone have any objects to us eventually adding this platform if enablement work is done?
17:12:25 <jmarrero> I think: https://developer.apple.com/videos/play/wwdc2022/10002/
17:12:36 <jmarrero> no objections from me.
17:12:43 <JasonBrooks[m]> Seems like an important one to add
17:13:15 <spresti> Nope no objections; looking forward to it.
17:13:26 <marmijo[m]> no objections here
17:13:47 <dustymabe> #proposed There are still some details to work out but adding this platform seems like a good idea to enable FCOS use cases on apple hardware, including podman machine and podman desktop.
17:14:00 <jmarrero> +1
17:14:05 <marmijo[m]> +1
17:14:06 <spresti> +1
17:14:07 <ravanell_> +1
17:14:20 <gursewak> =1
17:14:31 <dustymabe> #agreed There are still some details to work out but adding this platform seems like a good idea to enable FCOS use cases on apple hardware, including podman machine and podman desktop.
17:14:55 <dustymabe> #topic tracker: Fedora 39 changes considerations
17:14:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1491
17:15:24 <dustymabe> ok we have some new changes to discuss
17:15:37 <dustymabe> subtopic 123. Improve Default Font Handling with default-fonts metapackages
17:15:46 <dustymabe> #link https://fedoraproject.org/wiki/Changes/ImproveDefaultFontHandling
17:16:49 <dustymabe> #info DWM: This should not affect us we don't ship fonts packages.
17:17:12 <dustymabe> subtopic 124. Build Fedora Workstation live ISO with Image Builder
17:17:30 <dustymabe> I wonder why this is marked as a systemwide change
17:18:04 <dustymabe> #info DWM: This is for Fedora Workstation and shouldn't affect Fedora CoreOS.
17:18:43 <dustymabe> subtopic 125. Golang 1.21
17:18:55 <dustymabe> #link https://fedoraproject.org/wiki/Changes/golang1.21
17:19:16 <dustymabe> anybody anticipate problems with golang 1.21?
17:20:02 <dustymabe> at least when this lands in rawhide our packages in the rawhide stream will test it
17:20:22 <dustymabe> spresti: I think you are closer to more of the "go" projects - any problems you can think of?
17:21:16 <spresti> With the go projects I have in mind, I don't have any worries from this.
17:21:49 <spresti> Honestly I expect it to be fairly simple (I hope I am not wrong)
17:22:05 <dustymabe> #info DWM: we don't anticipate any fallout from this but our installed FCOS packages will at least test this when it lands in rawhide and thus in our rawhide stream.
17:22:11 <dustymabe> spresti: famous last words :)
17:22:24 <dustymabe> subtopic 126. Deprecating libuser and removing passwd package from Fedora
17:22:32 <dustymabe> #link https://fedoraproject.org/wiki/Changes/LibuserDeprecation
17:24:44 <dustymabe> hmm I don't actually know if we need to do anything for this
17:25:01 <dustymabe> it seems like it's just a dependency of a few packages and those deps are going to be dropped
17:25:19 <dustymabe> but I don't know if there is anything we do on the ostree side that needs consideration ehre
17:25:44 <dustymabe> maybe i'll postpone this discussion until walters or travier can weigh in.. that is unless you think you know the answer jmarrero?
17:26:05 <jmarrero> I have no idea :D
17:26:23 <dustymabe> :)
17:26:28 <dustymabe> ok i'll move on to the next one then
17:26:44 <dustymabe> subtopic 127. Allow Removal of tzdata
17:26:56 <dustymabe> #link https://fedoraproject.org/wiki/Changes/AllowRemovalOfTzdata
17:28:28 <dustymabe> I think for this one we'll need to make sure tzdata continues to be installed
17:28:33 <jmarrero> I could see this affecting people expecting something other than UTC.
17:29:15 <dustymabe> I think the problem is that we have existing systems and people could already be using the tzdata data
17:29:31 <jmarrero> agreed
17:29:33 <dustymabe> so we kind of need to keep it installed
17:29:47 <dustymabe> this should be as simple as just naming the package in the manifests
17:29:52 <dustymabe> i'll open a separate ticket for it
17:30:21 <dustymabe> #info DWM: we think we need to keep the tzdata package around since existing installs could be leveraging that data.
17:30:44 <dustymabe> ok we're running out of time so I'll leave the rest of the changes for next time
17:30:47 <dustymabe> #topic open floor
17:30:54 <dustymabe> anyone with anything for open floor today?
17:32:18 <dustymabe> lively bunch
17:32:29 <dustymabe> thanks JasonBrooks[m] for coming - we always enjoy having you
17:32:35 <dustymabe> feel free to say more
17:32:49 <JasonBrooks[m]> o/
17:33:05 <spresti> Thanks for running it!
17:33:09 <dustymabe> #endmeeting