16:28:50 <dustymabe> #startmeeting fedora_coreos_meeting
16:28:50 <zodbot> Meeting started Wed Nov 24 16:28:50 2021 UTC.
16:28:50 <zodbot> This meeting is logged and archived in a public location.
16:28:50 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:28:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:28:50 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:04 <dustymabe> #topic roll call
16:29:13 <dustymabe> .hellomynameis dustymabe
16:29:14 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:29:52 <nemric> hi
16:29:59 <copperi[m]> :hello copperi
16:30:06 <copperi[m]> .hello copperi
16:30:08 <zodbot> copperi[m]: copperi 'Jan Kuparinen' <copper_fin@hotmail.com>
16:30:11 <dustymabe> we'll see who shows up today considering a good chunk of people are probably AFK this week
16:30:17 <dustymabe> #chair nemric copperi[m]
16:30:17 <zodbot> Current chairs: copperi[m] dustymabe nemric
16:30:19 <dustymabe> welcome :)
16:31:40 <nemric> :)
16:32:25 <dustymabe> it's looking pretty slim
16:32:56 <jlebon> .hello2
16:32:57 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:00 <walters> .hello2
16:33:01 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:33:06 <travier[m]> .hi siosm
16:33:06 <zodbot> travier[m]: Sorry, but user 'travier [m]' does not exist
16:33:12 <dustymabe> sorry travier[m] :)
16:33:14 <travier[m]> .hello siosm
16:33:15 <zodbot> travier[m]: siosm 'Timothée Ravier' <travier@redhat.com>
16:33:19 <dustymabe> #chair jlebon walters travier[m]
16:33:19 <zodbot> Current chairs: copperi[m] dustymabe jlebon nemric travier[m] walters
16:33:21 <travier[m]> :)
16:33:33 <mnguyen> .hello mnguyen
16:33:34 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:33:50 <copperi[m]> did we have a split ?
16:33:50 <dustymabe> #chair mnguyen
16:33:50 <zodbot> Current chairs: copperi[m] dustymabe jlebon mnguyen nemric travier[m] walters
16:34:10 <dustymabe> copperi[m]: not sure - I didn't see a bunch of people leave
16:34:16 <jlebon> i think it's just the thanksgiving week factor
16:34:20 <dustymabe> yup
16:34:32 <dustymabe> means there will be a lot of time for open floor
16:34:37 <dustymabe> #chair davdunc
16:34:37 <zodbot> Current chairs: copperi[m] davdunc dustymabe jlebon mnguyen nemric travier[m] walters
16:34:42 * dustymabe forces davdunc into the meeting :)
16:34:53 <davdunc> ha.
16:34:58 <dustymabe> ok let's get started
16:35:00 <davdunc> .hi
16:35:01 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:35:24 <dustymabe> #topic Action items from last meeting
16:35:54 <dustymabe> There were no official action items from last meeting.. but we did decide to ask more questions about OSBS in Fedora...
16:36:14 <dustymabe> for this:
16:36:18 <dustymabe> #topic Make publicly accessible coreos-assembler builds for architectures != x86_64
16:36:29 <dustymabe> #link https://github.com/coreos/coreos-assembler/issues/2470
16:36:49 <dustymabe> I shot an email over to the Fedora team and got back the following info
16:36:58 <dustymabe> Fedora's OSBS is x86_64 and aarch64 only and there are no plans to add additional architectures.
16:36:59 <dustymabe> The current policy requires building from RPMs and does not allow building from source.
16:37:19 <jlebon> dustymabe: thanks for asking!
16:37:40 <dustymabe> I added a proposal to the ticket that we eliminate Fedora's OSBS from consideration
16:37:48 <ravanelli> .hi
16:37:49 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:37:55 <dustymabe> #chair ravanelli
16:37:55 <zodbot> Current chairs: copperi[m] davdunc dustymabe jlebon mnguyen nemric ravanelli travier[m] walters
16:38:22 <jlebon> i think there's still some value in pushing to quay.io from the multi-arch builders we have at least
16:38:28 <dustymabe> so given this I think our only real options are to:
16:38:47 <dustymabe> well the only option of where to build IMHO is our multi-arch builders
16:39:00 <dustymabe> then the question comes up of "do we push them to quay?"
16:39:22 <dustymabe> considering that the same builder that built the image is the consumer of that image
16:39:51 <dustymabe> either way we need to implement some sort of crude build triggering for the image builds (i.e. tags and such)
16:40:23 <dustymabe> it might be nice to push them to quay for historical reasons (i.e. keeping history of old tags) and such
16:40:25 <jlebon> dustymabe: pushing them to quay will benefit developers,
16:40:35 <jlebon> so i'd say it's worth it
16:40:45 <dustymabe> jlebon: ehh, i'm mixed on the positives of that
16:40:54 <jlebon> not just local, but e.g. if they want to bring up a pod in a multi-arch cluster
16:41:04 <jlebon> it's a pain to require them to build and push somewhere first
16:41:09 <dustymabe> yeah
16:41:11 <dustymabe> ok i'm sold
16:41:13 <ravanelli> having the multi-arch cosa images in quay would help a lot in the brew improvements as well. We need to add the cosa.tar in brew. But it is not add in s3.
16:41:34 <ravanelli> I would need to rebuild it from the commit
16:42:03 <ravanelli> Since the internal clusters don't keep the image there for a long time
16:42:11 <jlebon> ravanelli: a tarball of... the whole container? yuck
16:42:17 <dustymabe> ok so we need to determine what all is needed
16:42:18 <ravanelli> yeah =/
16:42:32 <dustymabe> i'm thinking a single container reference in quay with a manifest list for all arches
16:42:41 <walters> hmm, do we need to add all of cosa to brew?  Ultimately I think if we have lockfiles for cosa all we'd need is the git commit there to reproduce
16:42:45 <jlebon> dustymabe: +1
16:43:13 <jlebon> we don't even need lockfiles technically, just to store the pkglist that was actually used
16:43:15 <ravanelli> walters: Looks we need it, as the oscontainer
16:43:40 <dustymabe> and the other question is, what do we do for x86_64? currently we run those builds directly in jenkins/openshift. I would propose we add an x86_64 builder and build/push that container the exact same way as the others
16:44:02 <walters> ravanelli:  the ostree container image is separate from cosa, right?
16:44:06 <dustymabe> or maybe not - I guess i'm open to options there
16:44:38 <dustymabe> anywho we're getting in the weeds - let me make a high level proposal
16:44:42 <walters> well I think a core tension here is we will need a CI flow that does not necessarily run on all architectures
16:44:46 <jlebon> dustymabe: i guess it depends on whether we want the :latest to represent the exact same git commit on all arches
16:44:53 <ravanelli> walters: it is, but we do need to upload it to brew
16:45:03 <walters> ravanelli: right, agree
16:45:21 <jlebon> if we take full ownership, we can ensure that, but not sure it's worth it
16:45:55 <dustymabe> #proposed We will use our existing multi-arch-builders to build COSA for all architectures and also find a way to push them to quay using manifest lists so that a single tag in quay can be pulled for any architecture we support.
16:46:43 <dustymabe> this also means we stop doing any builds in quay - so we'd be pushing all containers (even the x86_64 one)
16:47:01 * dustymabe wishes quay would just support the arches we want :)
16:47:13 <jlebon> oh? is it not possible to just add to existing tags?
16:47:27 <jlebon> i haven't worked with manifest lists before
16:47:29 <dustymabe> 👍 / 👎 ?
16:47:32 <walters> no, a manifest list is a separate thing from an archful manifest
16:47:48 <copperi[m]> 👍️
16:48:02 <dustymabe> jlebon: you might be able to, but I assume it was something where it would be easier if we pushed them all at the same time
16:48:19 <jlebon> +1, though should reword given walters' comment
16:48:32 <dustymabe> i.e. if quay tags:latest but it takes us an hour to collect the alt arch builds and push them - there is a period of time where that tag is borked for those arches
16:48:58 <jlebon> dustymabe: ack agreed
16:49:05 <dustymabe> #proposed We will use our existing multi-arch-builders to build COSA for all architectures and also find a way to push them to quay using archful manifests so that a single tag in quay can be pulled for any architecture we support.
16:49:19 <jlebon> +1
16:49:30 <dustymabe> TIL there's a difference between manifest lists and archful manifests (clearly I have a lot to learn here)
16:49:44 <travier[m]> +1
16:49:44 <ravanelli> dustymabe: +1
16:50:04 <davdunc> +1
16:50:05 <copperi[m]> +1
16:50:24 <jlebon> maybe we could keep the quay builder for x86_64, but have it push to e.g. :staging. then our builders rebuild whenever :staging is updated, matching the same git commit, and then push everything together to :latest
16:50:24 <dustymabe> #agreed We will use our existing multi-arch-builders to build COSA for all architectures and also find a way to push them to quay using archful manifests so that a single tag in quay can be pulled for any architecture we support.
16:50:27 <walters> rough +1 to proposed, there are some details here
16:50:49 <dustymabe> walters: indeed - always new information that pops up during the investigation/implementation
16:50:55 <dustymabe> this is just the general direction
16:51:49 <dustymabe> ok I don't see jaimelm today so I'll skip his topic
16:51:49 <travier[m]> +1 to jlebon 's idea too
16:52:00 <dustymabe> and the other one is one we'll discuss after the beginning of the year
16:52:33 <dustymabe> #topic https://github.com/coreos/fedora-coreos-tracker/issues/1004
16:52:37 <dustymabe> #undo
16:52:37 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7fce52351940>
16:52:49 <dustymabe> #topic m6i instances fail to boot - kernel crashlooping
16:52:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1004
16:53:03 <dustymabe> davdunc: any updates here?
16:53:33 <davdunc> so.. i  brought it up with the Product team to ensure that it's on the list of issues.
16:53:45 <dustymabe> i'd hate for AL2022 to not boot on m6i instances
16:53:46 <dustymabe> :)
16:53:59 <davdunc> need that 5.14 kernel first. :)
16:54:07 <davdunc> but they are aware of the regression.
16:54:10 <dustymabe> davdunc: #shipit
16:54:30 <davdunc> :
16:54:32 <davdunc> :)
16:54:38 <dustymabe> davdunc: when we get a fix in we'll add a test to our suite to cover that instance type (at lest for basic checks)
16:55:07 <dustymabe> davdunc: is there even an upstream bug report or mail list thread on the topic - if so let's add it to the issue
16:55:12 <davdunc> yea. this is ice lake specific, so it's going to affect several of the instance types.
16:55:36 <davdunc> I put the internal ticket in the bug for tracking.
16:55:45 <davdunc> I don't have an open issue that can be tracked.
16:56:24 * dustymabe thinks maybe we should have a one-off test we run once a week that runs a basic test on each instance type
16:56:24 <davdunc> but it's in the ec2 kernel team issues and I have them working on the pstate implementation.
16:56:30 <dustymabe> i think jlebon has thought of this before
16:56:45 <dustymabe> ok cool - thanks davdunc for the info
16:56:53 <dustymabe> #topic open floor
16:56:54 <davdunc> it's the same for the Fedora kernel all around and FreeBSD.
16:57:05 <dustymabe> anyone have anything for open floor today?
16:57:08 <travier[m]> that might be a good idea, just to boot testing on all instances before the release
16:57:24 <dustymabe> nemric: copperi[m]: thanks for joining the meetings :) - let us know if there's anything you want to bring up (today or in the future)
16:57:26 <travier[m]> or rawhide
16:57:55 <dustymabe> travier[m]: yeah, i'm thinking we want to catch it before release day - that's why I was thinking once a week might be nice
16:57:58 <walters> #link https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/KQKCFXG4CTT6IGCGWAXIX5SLHFBFCEAI/
16:58:25 <davdunc> travier[m]: at least a random sample from all the architectures.
16:59:01 <nemric> ok dusty, I'll create an issue on tracker, I'll need a /etc/systemd/user unit, that's for an ssh-agent.service
16:59:10 <dustymabe> #info all streams of FCOS have now been rebased to Fedora Linux 35 content
16:59:30 <davdunc> yay!
16:59:46 <dustymabe> nemric: yeah I think it would be nice if Ignition supported user units. I don't think it does currently
16:59:56 <dustymabe> I've had to create the files manually
17:00:03 <nemric> it doesn't
17:01:03 <travier[m]> davdunc: +1. Any input about which one we should try would be great :)
17:01:14 <jlebon> dustymabe: so now we can start tracking f36 changes :)
17:01:20 <dustymabe> jlebon: indeed
17:01:22 <jlebon> it never ends
17:01:28 <dustymabe> https://fedoraproject.org/wiki/Releases/36/ChangeSet
17:01:33 <davdunc> travier[m]: I'll put together a list.
17:01:44 <dustymabe> I have an open ticket already but I need to sync over the list so we can start discussing them during the meetings
17:02:04 <jlebon> dustymabe: nice
17:02:13 <dustymabe> here's the ticket to follow: https://github.com/coreos/fedora-coreos-tracker/issues/918
17:02:43 <dustymabe> any other topics for open floor? I have a generic discussion question
17:03:12 <dustymabe> How do we convince more people to try Fedora CoreOS? I think we've built something really useful and want more people to try it
17:03:41 <dustymabe> a devel@ list campaign to ask people if they've tried it or to try it would help
17:04:03 <dustymabe> i've also thought it would be cool to convince the Linux Unplugged folks (jupiter broadcasting) to do a Fedora CoreOS challenge
17:04:26 <dustymabe> though they often like to try to use ZFS in their trials and that's not going to be fun with FCOS
17:04:45 <jlebon> hehe
17:04:54 <jlebon> a community manager would be helpful for this
17:05:14 * dustymabe puts on his community manager hat
17:05:36 <dustymabe> i basically do many jobs - and all of them poorly
17:05:41 <dustymabe> :)
17:05:55 <copperi[m]> nice hat though
17:06:02 <jlebon> dustymabe: i disagree :)
17:06:03 <dustymabe> jlebon: agreed
17:06:11 <dustymabe> oops - was typing that before you sent yours
17:06:30 <jlebon> (re. the poorly part, to be clear)
17:06:52 <dustymabe> anywho - maybe in your free time noodle on some ways to get more people to try it and give us feedback
17:07:12 <dustymabe> we've definitely still got some rough edges (like selinux modifications and such), but I think things are getting better
17:07:19 <jlebon> i feel like the kubernetes work recently might help a lot there
17:07:28 <davdunc> yea. I disagree on the _poorly_ part too. dustymabe your imposter's syndrome is showing. :)
17:07:52 <dustymabe> once we get package layering native support in ignition/butane and selinux stuff working better and maybe something like quadlet i think it'll be really slick
17:08:23 <travier[m]> +1, dustymabe , you're doing great
17:08:48 <jlebon> we've definitely come a long way
17:08:56 <travier[m]> something like quadlet would make creating configs much easier for standalone nodes
17:09:03 <dustymabe> indeed
17:09:20 <travier[m]> right now it's not great to copy paste the output of podman generate
17:09:41 <dustymabe> anywho - i'm clearing out some tech debt in my personal stuff at home and then I'm probably going to try to do a better job of project management/community management/vision for FCOS
17:10:05 <travier[m]> definitely, we're in a place where things work well and we're looking at user experience improvements
17:10:36 <dustymabe> it's also nice that our releases are on a pretty steady heartbeat (every two weeks)
17:10:40 <davdunc> dustymabe: looking forward to the discussions and helping you with write-ups!
17:10:55 <dustymabe> and we've done a good job of not breaking users so far (I think)
17:11:18 <dustymabe> ok, i'll stop rambling - any other topics? or i'll close out in 2ish minutes
17:13:04 <davdunc> dustymabe++
17:13:04 <zodbot> davdunc: Karma for dustymabe changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:13:43 <dustymabe> #endmeeting