16:30:46 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:46 <zodbot> Meeting started Wed Dec  2 16:30:46 2020 UTC.
16:30:46 <zodbot> This meeting is logged and archived in a public location.
16:30:46 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:46 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:50 <dustymabe> #topic roll call
16:30:58 <cyberpear> .hello2
16:30:58 <dustymabe> .hello2
16:30:58 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:01 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:17 <jbrooks> .hello2 jasonbrooks
16:31:17 <zodbot> jbrooks: Sorry, but you don't exist
16:31:21 <bgilbert> .hello2
16:31:22 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:39 <jlebon> .hello2
16:31:40 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:59 <jbrooks> .hello jasonbrooks
16:32:00 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:34:34 <travier> .hello siosmù
16:34:35 <travier> .hello siosm
16:34:35 <zodbot> travier: Sorry, but you don't exist
16:34:38 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:34:50 <dustymabe> #chair cyberpear jbrooks bgilbert jlebon jbrooks travier
16:34:50 <zodbot> Current chairs: bgilbert cyberpear dustymabe jbrooks jlebon travier
16:34:55 <dustymabe> red_beard: around today?
16:35:24 <lucab> .hello2
16:35:25 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:36:02 <dustymabe> #chair lucab
16:36:02 <zodbot> Current chairs: bgilbert cyberpear dustymabe jbrooks jlebon lucab travier
16:36:19 <dustymabe> #topic Action items from last meeting
16:36:30 <dustymabe> action items are from red_beard
16:36:34 <dustymabe> he doesn't seem to be around
16:36:41 <dustymabe> #action red_beard to investigate forklifting FCOS artifacts into OVH to see if they work
16:36:48 <dustymabe> #action red_beard to open a ticket to strategize on FCOS RH summit presence
16:37:54 <dustymabe> #topic Move rpmdb path from /usr/share/rpm to /usr/lib/sysimage/rpm
16:38:10 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/639
16:38:21 <dustymabe> i wanted to circle back on this issue and see where it stands
16:38:44 <dustymabe> If we're going to do anything for Fedora 34 we need to make progress on it
16:38:59 <jlebon> i think we're still looking for someone to drive the change at the Fedora level
16:39:31 <dustymabe> jlebon: maybe King_InuYasha can do it for other variants and you can drive it for OSTree variants
16:39:52 <dustymabe> looks like he was half volunteering in https://github.com/coreos/fedora-coreos-tracker/issues/639#issuecomment-701618881
16:40:49 <jlebon> yeah, definitely happy to help with the OSTree variants
16:41:17 <jlebon> i can chat with King_InuYasha and see if we're comfortable leading this together
16:41:53 <dustymabe> #action jlebon to talk with King_InuYasha to see if they can be co-captains for the proposed change to move the rpmdb path (#639)
16:41:59 <dustymabe> thanks!
16:42:21 <jlebon> +1
16:42:24 <dustymabe> #topic default hostname now is `fedora`, used to be `localhost`
16:42:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/649
16:43:06 <dustymabe> ok, we've discussed this one before, but mostly in the context of GCP and long hostnames not getting truncated right
16:43:25 <dustymabe> OKD users have been hitting new issues. One of them reached out to me this week and I found more information
16:43:42 <dustymabe> basically setting the hostname via Reverse DNS doesn't work now
16:44:13 <dustymabe> see https://github.com/coreos/fedora-coreos-tracker/issues/649#issuecomment-736104003
16:45:29 <dustymabe> I've added comments to the bug luca opened to try to add more ammo for "we should revert the fallback hostname change" argument
16:45:50 <dustymabe> https://bugzilla.redhat.com/show_bug.cgi?id=1892235#c9
16:46:08 <dustymabe> at this point I'm wondering if we should do something on the FCOS side
16:46:42 <dustymabe> i.e. a service that set's the hostname to localhost right after systemd sets it to `fedora` or something
16:47:21 <lucab> the change comes from deep inside the RPM, with a build-time flag, I don't think we can things FCOS-only at that level
16:47:37 <lucab> *can do
16:47:55 <dustymabe> right, we can't change the fact that it gets set without re-spinning our own rpm, but we could unset it right after it gets set
16:49:19 <dustymabe> I'm going to try to set up some time with zbyszek to try to figure out a way forward
16:49:57 <cyberpear> does debian set their default to `debian`? -- i.e., was this a good change in the first place? should it be reverse for F34?
16:50:27 <dustymabe> cyberpear: I think that depends on who you ask :)
16:50:37 <jlebon> dustymabe, lucab: hmm, would it work to do this from a generator? it gets set during manager bringup, right?
16:50:39 <dustymabe> i know which way I would vote
16:50:47 <jlebon> as a short-term hack
16:51:15 <dustymabe> jlebon: probably a generator or something that runs very early in the initrd
16:51:31 <dustymabe> if you look at dmesg output you can see when it gets set to `fedora`
16:51:49 <lucab> jlebon: I think so
16:51:51 <jlebon> ok, yup it does get set during manager bringup
16:52:22 <dustymabe> so i'm thinking 1. talk with zbyszek first 2. depending on that conversation we can implement a short term hack
16:52:57 <dustymabe> either way, we've got `stable` set to rebase to 33 this week unless we make an executive decision not to
16:53:25 <dustymabe> so this issue will hit users unless we decide it's big enough that we want to hold off
16:54:07 <jlebon> which platforms rely on reverse lookup?
16:54:47 <dustymabe> the one that the OKD users were hitting were VMWare
16:54:59 <dustymabe> but I think the answer is most likely "custom platform"
16:55:28 <dustymabe> i.e. not one of the big clouds, but more a custom datacenter where DHCP and DNS may be managed by separate entities, etc..
16:56:31 <jlebon> leaning towards not holding the rebase for this
16:56:32 <dustymabe> the OKD users were the first to report an issue here
16:56:39 <dustymabe> lorbus might have more details
16:56:59 <lorbus> \me looks
16:57:31 <dustymabe> yeah, if we don't hold the rebase we can include information in a coreos-status post
16:57:41 <dustymabe> the problem is, what is the intervention?
16:58:01 <dustymabe> ok, for all of those systems that just upgraded, log in to each one of them and echo foobar > /etc/hostname ?
16:58:12 <lucab> bare-metal has the other good share of users with hostname-via-rDNS
16:58:34 <dustymabe> lucab: right.. "custom datacenter" :)
16:58:38 <jlebon> wait, wouldn't this affect firstboot only?
16:58:50 <dustymabe> jlebon: no
16:59:05 <dustymabe> NM doesn't set the hostname statically
16:59:22 <lorbus> there is a whole host of changes wrt resolved and DNS that are posing problems at the moment: Michael Catanzaro might also be a good person to talk to, he wrote up some options here: https://github.com/openshift/okd/issues/401#issuecomment-737337603
16:59:34 <jlebon> hmm ok, that makes it worse than i thought
16:59:51 <lorbus> (that is not the hostname issue specifically, but it's all related I think)
17:00:02 <dustymabe> i didn't pay attention before, but my weechat FCOS box (the one I'm typing on right now) used to get hostname via rDNS and it was set (f32 time frame), now it is `fedora`.
17:02:07 <dustymabe> anybody else with thoughts on what we should do here?
17:02:50 <dustymabe> part of me says "it's fine" because `next` and `testing` have been churning along fine
17:03:09 <dustymabe> part of me says we'd really not want to break people and require them to log in to boxes to fix things
17:03:10 <lorbus> * there is a whole host of changes wrt resolved and DNS that are posing problems at the moment: Michael Catanzaro might also be a good person to talk to, he wrote up some options here: https://github.com/openshift/okd/issues/401#issuecomment-737312124
17:03:31 <dustymabe> lorbus: got it, I'll have to read through that thread in a bit
17:03:50 <jlebon> can we just try to chat with the systemd folks today or tomorrow before releasing stable?
17:04:20 <lorbus> I think I posted a link to the wrong comment, it's a few comments above that one: https://github.com/openshift/okd/issues/401#issuecomment-737312124
17:04:38 <dustymabe> jlebon: yeah I think so
17:05:20 <dustymabe> worst case I say that we just skip the stable release this cycle, get the fix in testing and then pick back up next cycle
17:05:27 <lorbus> talking to the sd folks seems like the best option to me too :)
17:06:16 <dustymabe> one final piece of info.. there was a recent change in NM which *should* raise the priority of rDNS in it's scheme
17:06:46 <dustymabe> I was testing that this morning but found a bug which should be fixed by https://paste.centos.org/view/60847978
17:07:12 <dustymabe> so there are more options on the table, but long term
17:07:43 <dustymabe> #action dustymabe and jlebon to talk to the systemd team to see if we can come to agreement on a path forward for the fallback hostname behavior introduced in systemd in fedora 33
17:08:08 <dustymabe> #topic 2020-12-02: gather status update for Fedora Council
17:08:18 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/688
17:09:26 <dustymabe> help us write down some significant pieces of work we've done recently
17:09:28 <dustymabe> https://hackmd.io/iOmPW-0fSmu2NZ1OvgJylQ?edit
17:10:21 <lucab> dustymabe: do you have a link at hand to the previous one?
17:10:27 <dustymabe> lucab: let me find it
17:11:05 <dustymabe> here it is https://github.com/coreos/fedora-coreos-tracker/issues/650
17:11:55 <dustymabe> bgilbert skunkerk- - do you want to fill in the details about the recent Ignition release?
17:13:33 <bgilbert> done, just the one user-facing item
17:14:43 <dustymabe> bgilbert: you did some other work around playing with the partition scheme, has that landed?
17:15:08 <bgilbert> ah, good point.  yes ithas
17:15:09 <bgilbert> *it has
17:15:45 <dustymabe> so does that mean this `testing` release will now have a new partition layout?
17:15:55 <dustymabe> same partitions, just re-ordered
17:16:20 <bgilbert> right
17:16:34 <bgilbert> and the 4Kn image will have an empty BIOS-BOOT partition
17:16:49 <dustymabe> oh nice, we resolved that
17:17:15 <dustymabe> ok while we wait for people typing in the hackmd
17:17:42 <dustymabe> is it worth bringing up the twitter @coreos discussion - we punted again last week because red_beard wasn't here. punt again this week or should I #topic it?
17:18:57 <dustymabe> lucab: was there an afterburn release this cycle?
17:19:21 <jlebon> dustymabe: given that it's not urgent, let's just punt again?
17:19:27 <dustymabe> jlebon: WFM
17:19:27 <bgilbert> jlebon: +1
17:19:34 <jbrooks> +1
17:19:43 <dustymabe> #topic open floor
17:20:14 <dustymabe> ok.. when we do put out the new `stable` on f33, should we do a coreos-status post ?
17:20:32 * dustymabe checks to see what we did last time
17:21:24 <bgilbert> my usual answer is: not inherently, but if there's known breakage or user action required, yes
17:21:29 <dustymabe> ok we did put out a `Fedora CoreOS rebasing to Fedora 32: known issues; upcoming test day` last time
17:22:21 <jlebon> bgilbert's approach makes sense to me
17:22:26 <dustymabe> though last time we did put it out earlier, when we switched to `testing`
17:22:40 <dustymabe> s/to//.
17:23:19 <dustymabe> ok I'll try to gather up any "known issues" and see if it's worth the email
17:23:38 <dustymabe> depending on the outcome of our rDNS investigation that would definitely be in there
17:23:53 <dustymabe> any other topics for open floor?
17:23:54 <jlebon> dustymabe: +1
17:25:20 <dustymabe> anybody with any brainpower to help figure out why this cpuacct change happened? https://github.com/coreos/fedora-coreos-tracker/issues/682
17:25:52 <dustymabe> i've been leaning way too hard on lucab for stuff like this in the past :)
17:26:18 <jlebon> i looked at that issue report and was like "not today"
17:26:26 <dustymabe> I hear that
17:27:39 <dustymabe> well that's all I had for today
17:28:17 <dustymabe> #endmeeting