16:31:01 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:01 <zodbot> Meeting started Wed Aug 18 16:31:01 2021 UTC.
16:31:01 <zodbot> This meeting is logged and archived in a public location.
16:31:01 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:05 <dustymabe> #topic roll call
16:31:08 <dustymabe> .hi
16:31:09 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:32 <travier> .hello
16:31:32 <zodbot> travier: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:31:35 <travier> .hello siosm
16:31:38 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:06 <dustymabe> #chair travier
16:32:06 <zodbot> Current chairs: dustymabe travier
16:32:33 <jaimelm> .hello2
16:32:34 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:32:52 <jlebon> .hello2
16:32:53 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:02 <dustymabe> #chair jaimelm jlebon
16:33:03 <zodbot> Current chairs: dustymabe jaimelm jlebon travier
16:33:42 <dustymabe> #chair bgilbert
16:33:42 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon travier
16:34:14 <copperi> .hello
16:34:15 <zodbot> copperi: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:34:29 <dustymabe> copperi: try `.hi`
16:34:36 <copperi> .hi
16:34:37 <zodbot> copperi: copperi 'Jan Kuparinen' <copper_fin@hotmail.com>
16:35:37 <dustymabe> #chair copperi
16:35:37 <zodbot> Current chairs: bgilbert copperi dustymabe jaimelm jlebon travier
16:35:39 <dustymabe> #topic Action items from last meeting
16:35:49 <dustymabe> There were surpisingly no action items from last meeting
16:35:54 <bgilbert> .hi
16:35:55 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:36:09 <dustymabe> #topic Differing behavior for aarch64 vs x86_64 disk images
16:36:15 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/855
16:36:37 <dustymabe> it took me way too long to give an update in the ticket with a summary of the discussion last week (and I probably missed some things)
16:37:13 <dustymabe> jlebon: I think we were interested in your take on the "new" problem and proposed solutions
16:37:17 <dustymabe> sorry to put the spotlight on you :)
16:38:06 <jlebon> dustymabe: i'll have to read up and circle back :)
16:38:25 <dustymabe> no worries, that wasn't fair of me
16:38:41 <dustymabe> bgilbert: anything to state before we take a detour to another issue ?
16:38:46 <bgilbert> nope, let's move on for now
16:38:53 <dustymabe> #topic NetworkManager: consider defaulting to EUI-64 for IPv6 SLAAC (at least on OpenStack)
16:38:57 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/907
16:39:22 <lucab> .hi
16:39:23 <dustymabe> I took an action two meetings ago to find out how fedora cloud base handles this issue
16:39:23 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:39:26 <dustymabe> #chair lucab
16:39:26 <zodbot> Current chairs: bgilbert copperi dustymabe jaimelm jlebon lucab travier
16:39:43 <dustymabe> I updated the issue with that information in https://github.com/coreos/fedora-coreos-tracker/issues/907#issuecomment-895455009
16:40:16 <dustymabe> more or less cloud base has cloud-init which writes out an ifcfg file
16:41:06 <dustymabe> the ifconfig doesn't mention anything about addr-gen-mode but because the file exists and the way NM handles configuration read in from files, it ends up effectively using ipv6.addr-gen-mode=eui64
16:41:34 <jaimelm> This came in to OKD yesterday... https://github.com/openshift/okd/issues/817
16:42:00 <dustymabe> if I remove the ifcfg file and reboot the machine (i.e. to get the default config from NM just like is done in FCOS by default) then Fedora Cloud Base shows the same issue
16:42:04 <dustymabe> jaimelm: yep
16:42:57 <dustymabe> Also some chatter in https://bugzilla.redhat.com/show_bug.cgi?id=1743161#c14 and https://github.com/coreos/fedora-coreos-tracker/issues/907#issuecomment-899717858
16:43:02 <dustymabe> same guy I think
16:43:26 <dustymabe> ok, now that we know how cloud base deals with it (mostly by getting lucky) how do we want to proceed?
16:43:30 <jaimelm> Ahhh, yeah, looks like it
16:43:44 <dustymabe> my proposal is short medium and long term
16:44:22 <dustymabe> 1. short term, we try to figure out some what to get this working on openstack images without requiring changes in NM
16:45:17 <dustymabe> 2. medium term, NM implements a default config setting for this value that we can take advantage of with a dropin
16:45:50 <dustymabe> 3. long term, we try to get Fedora to change to it as the default for all server like variants (i.e. stable-privacy makes most sense in a workstation/laptop scenario)
16:46:13 <jaimelm> What are the chances of 3?
16:46:14 <travier> dustymabe: +1 for that plan
16:46:29 <dustymabe> the 1. could also be modified to not be openstack specific, but apply globally to FCOS
16:46:42 <dustymabe> jaimelm: not sure, haven't tried that path before
16:46:42 <jaimelm> ^^
16:46:50 <travier> +1 for all FCOS too
16:46:53 <jaimelm> yeah
16:46:58 <jaimelm> general is better here
16:47:23 <lucab> the short term fix is probably to tell users to setup their own connection profile via Ignition instead of using the default one.
16:47:57 <jaimelm> Sounds like a good plan. It would probably be good to start 3 coinciding with 1 to get the ball rolling, given it's an unkown.
16:48:51 <dustymabe> jaimelm: +1
16:48:57 <dustymabe> lucab: yeah, there might be other options
16:49:08 <dustymabe> would have to explore a bit
16:49:11 <jlebon> hmm, in the propagate path where we copy keyfiles from the initrd to the real root, we could inject it then
16:49:37 <jlebon> but that doesn't cover all cases, notably openstack via configdrive
16:50:20 <dustymabe> yeah
16:50:41 <dustymabe> by default right now we don't write any keyfiles
16:50:51 <dustymabe> and NM generates them on the fly
16:50:54 <jlebon> lucab: +1  i'm not sure it's worth trying to have a super sophisticated short-term hack for this
16:51:10 <dustymabe> these generated NM keyfiles have ipv6.addr-gen-mode=stable-privacy and thats the crux of the problem here
16:51:12 <jaimelm> lucab: that would be good if an example was provided in the docs so people know how to roll their own via ignition.
16:51:32 <lucab> the fact that NM generates it on the fly is a long-term friction I fear, it would be good if we could move out of that
16:51:33 <dustymabe> ok let me try with a proposal
16:51:48 <dustymabe> #proposal We will work with the NetworkManager team to get in place a configuration setting for a default value for ipv6.addr-gen-mode and apply that to all of FCOS when it's ready. In the shorter term we may try to find some other way to set it without requiring the NetworkManager feature to be implemented. We'll also reach out to FESCO and try to convince the rest of Fedora that
16:51:50 <dustymabe> `stable-privacy` makes most sense in a workstation/laptop setting and we should apply eui64 as the default for all server like variants.
16:52:32 <jaimelm> yes.
16:52:32 <jlebon> +1
16:52:34 <dustymabe> hmm is there a limit to the character count on a single line in IRC?
16:52:46 <lucab> also, is https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/962 going to break cloud-init too?
16:52:48 <walters> +1
16:53:18 <jaimelm> Not by the standard I don't think. What client are you using?
16:53:29 <dustymabe> weechat
16:53:50 <dustymabe> any more votes?
16:54:01 <dustymabe> lucab: i'll probably have to look at that separately
16:54:26 <lucab> dustymabe: nevermind, I think I misread the code there
16:54:45 <lucab> +1
16:55:09 <dustymabe> bgilbert: any thoughts?
16:55:48 <bgilbert> nope, sgtm
16:56:13 <dustymabe> ok i'll split this out so maybe we don't have the text wrapping issue
16:56:15 <dustymabe> #agreed We will work with the NetworkManager team to get in place a configuration setting for a default value for ipv6.addr-gen-mode and apply that to all of FCOS when it's ready.
16:56:16 <copperi> dustymabe: irc has 510 character limit / message
16:56:17 <dustymabe> #agreed In the shorter term we may try to find some other way to set it without requiring the NetworkManager feature to be implemented.
16:56:19 <dustymabe> #agreed We'll also reach out to FESCO and try to convince the rest of Fedora that `stable-privacy` makes most sense in a workstation/laptop setting and we should apply eui64 as the default for all server like variants.
16:56:23 <dustymabe> copperi: good to know
16:56:56 <dustymabe> jlebon: ready yet, or should we go to the f35 changes ticket?
16:57:19 <jlebon> dustymabe: f35 changes
16:57:25 <dustymabe> #topic tracker: Fedora 35 changes considerations
16:57:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/856
16:57:50 <dustymabe> ok we started this last week
16:58:03 <dustymabe> we'll pick it up again today
16:58:27 <dustymabe> all the issues in the description that have a ❌ have not been considered yet
16:58:48 <dustymabe> we need to go through them and decide if we can skip, or if they need more investigation, in which case we'll split out a ticket for that investigation
16:58:56 <dustymabe> we'll start with
16:59:06 <dustymabe> item; 1.16 x TRIAGE GNU Toolchain update
16:59:08 <jaimelm> copperi: wow. that seems small. but you're right, I see now in the RFC
16:59:22 <dustymabe> skip? or investigate?
16:59:43 <dustymabe> skip means we think with high confidence this won't affect FCOS
17:00:35 <dustymabe> so this is just a point release update
17:00:46 <dustymabe> gcc 11 was already in f34
17:01:09 <dustymabe> so it will be 11.1 or 11.2
17:01:33 <jlebon> i think just the last three X of the 1.* items probably need more investigation
17:01:50 <dustymabe> jlebon: what about the current one :)
17:02:07 <jlebon> just those, so everything else skip IMO :)
17:02:33 <dustymabe> #info we think we can skip GNU Toolchain update
17:03:02 <dustymabe> anybody else want to agree/disagree with jlebon?
17:03:05 <jlebon> an rhbz related to the third-party one showed up already againt rpm-ostree
17:03:26 <jlebon> so walters has already dug into that one
17:03:46 <walters> 🕳
17:04:21 <lucab> dustymabe: skimming the list, I'd agree that nothing else stands up
17:04:26 <dustymabe> #info Golang 1.17: we don't anticipate any issues skipping
17:04:28 <dustymabe> #info IBus 1.5.25: we don't anticipate any issues skipping
17:04:30 <dustymabe> #info LLVM 13: we don't anticipate any issues skipping
17:04:36 <dustymabe> what about 1.21 x TRIAGE Memory Constraints macros for RPM
17:04:54 <dustymabe> anything in there? any interaction with rpm-ostree compose we need to consider?
17:05:00 <jaimelm> skip
17:05:26 <jlebon> for authselect, there's a lot of history there, but today we don't ship neither it nor the compat, so we can probably skip that one too
17:05:45 <walters> compose has nothing to do with builds, so no
17:05:47 <jlebon> i think it's just related to RPM building
17:06:26 <dustymabe> #info Memory Constraints macros for RPM: we don't anticipate any issues skipping
17:06:30 <dustymabe> #info Python Packaging Guidelines overhaul: we don't anticipate any issues skipping
17:06:40 <dustymabe> #info Remove authselect-compat package: today we don't ship authselect or the compat, skipping
17:06:41 <jlebon> for the yescrypt one, we should sanity-check our docs use that already, which i think it does
17:06:58 <dustymabe> ok hold one sec
17:07:05 <dustymabe> 13:01:33        jlebon | i think just the last three X of the 1.* items probably need more investigation
17:07:13 <dustymabe> so 1.32 1.33 1.34 ?
17:07:49 <bgilbert> jlebon: the docs already use yescrypt
17:07:50 <jlebon> i mean 1.29 1.33 1.34
17:07:53 <dustymabe> ahh ok
17:08:02 <dustymabe> oh :)
17:08:06 <dustymabe> #undo
17:08:06 <zodbot> Removing item from minutes: INFO by dustymabe at 17:06:40 : Remove authselect-compat package: today we don't ship authselect or the compat, skipping
17:08:27 <dustymabe> jlebon: in other words.. we're probably OK, but let's make sure about authselect ?
17:08:57 <jlebon> bgilbert: indeed, looks like it. nice
17:09:10 <dustymabe> #action dustymabe will open a ticket and look for volunteers to investigate
17:09:13 <dustymabe> #undo
17:09:13 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:09:10 : dustymabe will open a ticket and look for volunteers to investigate
17:09:26 <dustymabe> #action dustymabe will open a ticket and look for volunteers to investigate "Third-party Software Mechanism"
17:09:45 <dustymabe> walters: seems like a captive audience already for ^^
17:10:19 <jlebon> dustymabe: re. authselect, sure yeah. just e.g. making sure it's not getting pulled in now or something due to spec file changes
17:10:20 <walters> yeah I think it's on track to be fixed but I will watch and assist
17:10:47 <dustymabe> #action dustymabe will open a ticket and look for volunteers to investigate "Remove authselect-compat package"
17:10:58 <jlebon> i can take that one
17:11:05 <dustymabe> item: 1.34 x TRIAGE Use yescrypt as default hashing method for shadow passwords
17:11:19 <dustymabe> ok skip or no based on new information from bgilbert ?
17:11:27 <jlebon> skip
17:11:30 <bgilbert> +1
17:11:32 <walters> may make sense to add a kola test for it
17:11:41 <bgilbert> walters: for what?
17:11:44 <walters> yescrypt
17:11:52 <bgilbert> that it exists?
17:12:01 <walters> yes ;)
17:12:39 <dustymabe> i could see a kola test that tests password based authentication (i.e. what we put in our docs) as useful
17:12:45 <jaimelm> ^^
17:13:06 <jlebon> IOW, provision with a yescrypt hash, and sanity-check it works?
17:13:18 <dustymabe> yeah, though that wouldn't actually test this change
17:13:36 <dustymabe> i imagine this change is more.. when someone runs `passwd` use yescrypt
17:13:58 <jlebon> so automating a `useradd foo && echo "bar" | passwd foo` or something
17:14:10 <dustymabe> which I don't really think we need to do
17:14:17 <jlebon> that's kinda an anti-pattern for us :)
17:14:42 <dustymabe> yeah.. but a test that automates what we have in our docs, would be useful IMO, but not required for this
17:14:51 <dustymabe> #info Use yescrypt as default hashing method for shadow passwords: we don't anticipate any issues skipping
17:15:04 <bgilbert> yeah, looks like we don't actually test pw right now
17:15:21 <dustymabe> bgilbert: want to open an issue for that and find a volunteer?
17:15:25 <dustymabe> :)
17:15:29 <jlebon> i think it'd make sense to add a non-exclusive test which provisions the VM with a yescrypt hash and tests it
17:15:38 <bgilbert> dustymabe: I'll split the difference and open an issue :-)
17:15:57 <dustymabe> 👍
17:16:03 <dustymabe> ok - time check
17:16:41 <dustymabe> we'll do self contained changes next week
17:16:54 <dustymabe> #topic Differing behavior for aarch64 vs x86_64 disk images
17:16:58 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/855
17:17:28 <dustymabe> only reason I'm pushing this topic is because we're getting close to shipping aarch64 images
17:17:55 <dustymabe> and there are probably implications for RHCOS which we need to consider
17:18:13 <walters> hmmm....what are you thinking of wrt RHCOS?
17:18:24 <walters> we quashed the devmapper variance
17:18:36 <travier> (I haven't reached out to arm folks yet. Will try to do that asap)
17:18:45 <walters> (we're actually already shipipng aarch64 RHCOS)
17:18:58 <bgilbert> walters: building, or shipping?
17:19:02 <walters> ((as preview)) (((also man I sure overuse parentheticals)))
17:19:12 <dustymabe> walters: I overuse them and ""
17:19:14 <jaimelm> walters is a LISP machine
17:19:29 <jlebon> is the argument now that if we don't do something, then as it stands Butane-generated raid1 setups will be using different partition numbers as vanilla systems?
17:19:39 <bgilbert> jlebon: yes
17:20:00 <jlebon> and this is purely cosmetic, right?
17:20:03 <bgilbert> which defeats a design goal
17:20:15 <dustymabe> there were other arguments for symmetry (i.e. why the ticket was opened to begin with)
17:20:47 <bgilbert> jlebon: well, I'm sure there's some script somewhere that assumes partition 4
17:20:51 <walters> in theory we can change this after aarch64 goes stable, right?
17:20:59 <bgilbert> but in general the Butane template is supposed to match the disk image
17:21:30 <dustymabe> walters: yes, but would prefer to change it before to reduce mismatch in the wild
17:21:37 <jlebon> bgilbert: makes sense
17:21:42 <bgilbert> walters: depends on how we fix it, to some degree
17:21:48 <walters> agree it's clearly better to fix it before the first stable release
17:21:55 <dustymabe> correct - to restate the options
17:22:05 <walters> I just don't have a good sense of the potential fallout of changing it later
17:22:20 <dustymabe> 1. break existing RAID template overrides (probably needs a Butane spec bump?)
17:22:22 <dustymabe> 2. change the Ignition matching behavior (needs Ignition spec 4 AFAICT)
17:22:24 <dustymabe> 3. ship empty partitions on aarch64/ppc64le so we don't skip any partition numbers
17:22:26 <dustymabe> 4. live with the inconsistency and update kola tests
17:22:43 <jlebon> is 1. saying we would hardcode partition numbers in the template?
17:22:52 <bgilbert> jlebon: yes
17:23:32 <jlebon> ahh ok, but if we did have the ignition RFE, then it could not have to do that, right?
17:23:59 <bgilbert> if we want to skip partition numbers, we'd still have to hardcode them in the Butane template
17:24:04 <bgilbert> but doing so wouldn't affect user configs
17:24:42 <jlebon> ack, i think i follow
17:25:19 <bgilbert> I'm leaning 3.  the hack is ugly but it avoids spec bumps
17:25:25 <jlebon> long-term, i think what we should strive for is have users never need to write partition numbers
17:25:31 <dustymabe> 3. also has other benefits https://github.com/coreos/fedora-coreos-tracker/issues/855#issuecomment-858026743
17:25:35 <bgilbert> jlebon: agreed
17:25:39 <jaimelm> jlebon++
17:25:39 <zodbot> jaimelm: Karma for jlebon changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:25:53 <jlebon> but under the hood, we really have to do so, and in that respect it makes sense to ship empty partitions if it makes *our* lives easier
17:26:12 <dustymabe> 🎉 🎂 🌞
17:26:44 <travier> +1 for 3
17:27:05 <jaimelm> 3 seems the most reasonable. Avoids spec bumps and shifts the issues away fomr the users.
17:27:16 <walters> agreed
17:27:57 <dustymabe> #proposed we will ship empty partitions to help gain symmetry between the x86_64 aarch64 and ppc64le disk images in order to workaround/fix other bugs and help mitigate user confusion
17:28:10 <jaimelm> yes.
17:28:18 <dustymabe> bgilbert: wording check ^^
17:28:22 <bgilbert> dustymabe: let's add wording that they won't necessarily match the corresponding partition sizes on x86_64
17:29:00 <dustymabe> bgilbert: yes. that's a subtopic I mentioned in https://github.com/coreos/fedora-coreos-tracker/issues/855#issuecomment-867131221
17:29:07 <dustymabe> should we discuss that now ?
17:29:13 <jaimelm> 1 minutes
17:29:17 <dustymabe> :)
17:29:21 <jaimelm> s/minutes/minute
17:29:35 <dustymabe> basically is there an argument for matching them or an argument for not matching them (sizes)
17:29:51 <dustymabe> ok you said necessarily
17:30:11 <dustymabe> #proposed we will ship empty partitions to help gain symmetry between the x86_64 aarch64 and ppc64le disk images in order to workaround/fix other bugs and help mitigate user confusion. The partition sizes won't necessarily be the same on all arches.
17:30:13 <bgilbert> they're not matched now and I wasn't planning to harmonize them
17:30:14 <dustymabe> better ?
17:30:28 <jlebon> +1
17:30:30 <bgilbert> +1
17:30:41 <travier> +1
17:30:47 <dustymabe> +1
17:30:49 <copperi> +1
17:30:59 <jaimelm> yes.
17:31:07 <bgilbert> and for the record:
17:31:09 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/855#issuecomment-896387343
17:31:32 <dustymabe> #agreed we will ship empty partitions to help gain symmetry between the x86_64 aarch64 and ppc64le disk images in order to workaround/fix other bugs and help mitigate user confusion. The partition sizes won't necessarily be the same on all arches.
17:31:48 <dustymabe> bgilbert: anything in particular in that link you wanted to highlight?
17:32:42 <bgilbert> "other bugs" isn't super-specific so I wanted to have it next to a problem description in the minutes
17:32:43 <bgilbert> that's all
17:32:48 <dustymabe> +1
17:33:07 <dustymabe> bgilbert: did you want to execute on that outcome (and re-enable the raid tests)?
17:33:12 <dustymabe> if not we'll try to find someone else
17:33:22 <bgilbert> yeah, makes sense
17:33:25 <dustymabe> #topic open floor
17:33:41 <bgilbert> #action bgilbert to add empty partitions, update Butane, re-enable tests
17:33:42 <dustymabe> #info the multi-arch-pipeline is outputting aarch64 artifacts now
17:34:13 <dustymabe> i'm working through an issue with the aarch64 AMI images https://github.com/coreos/fedora-coreos-tracker/issues/920
17:34:14 <jlebon> woohoo! nice work on this dustymabe!
17:34:14 * jaimelm moonwalks across the open floor.
17:34:23 <jaimelm> I think we're good.
17:34:30 <dustymabe> jlebon: do you mind updating the unofficial builds browser?
17:34:34 <jaimelm> Only 5 minutes over :)
17:34:48 <jlebon> dustymabe: yup, makes sense
17:34:48 <dustymabe> yep - anyone with anything else for open floor?
17:35:03 <walters> dustymabe: mind adding a comment to https://github.com/coreos/fedora-coreos-tracker/issues/13 ?
17:35:03 * dustymabe will have to see if we can update the official page too
17:35:19 <dustymabe> walters: yep will do soon!
17:35:40 * dustymabe closes meeting in 30 s unless conversation continues
17:36:00 <jlebon> dustymabe: let's chat after re. prod/official
17:36:03 <copperi> Thanks dustymabe
17:36:16 <dustymabe> #endmeeting