16:30:50 <jlebon> #startmeeting fedora_coreos_meeting
16:30:50 <zodbot> Meeting started Wed Nov 25 16:30:50 2020 UTC.
16:30:50 <zodbot> This meeting is logged and archived in a public location.
16:30:50 <zodbot> The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:50 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:55 <cyberpear> .hello2
16:30:56 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:30:57 <jlebon> #topic roll call
16:30:59 <bgilbert> .hello2
16:31:00 <travier_> .hello siosmù
16:31:00 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:01 <travier_> .hello siosm
16:31:03 <zodbot> travier_: Sorry, but you don't exist
16:31:06 <zodbot> travier_: siosm 'Timothée Ravier' <travier@redhat.com>
16:31:10 <dustman> .hello dustymabe
16:31:11 <zodbot> dustman: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:26 <jlebon> #chair cyberpear bgilbert travier_ dustman
16:31:26 <zodbot> Current chairs: bgilbert cyberpear dustman jlebon travier_
16:31:30 <jbrooks> .hello jasonbrooks
16:31:31 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:37 <jlebon> dustman: haha that nick
16:31:41 <jlebon> #chair jbrooks
16:31:41 <zodbot> Current chairs: bgilbert cyberpear dustman jbrooks jlebon travier_
16:32:03 <dustman> :)  - I'm in freenode webchat
16:32:14 <jlebon> nice
16:32:15 <lucab> .hello2
16:32:19 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:21 <jlebon> #chair lucab
16:32:21 <zodbot> Current chairs: bgilbert cyberpear dustman jbrooks jlebon lucab travier_
16:32:30 <jlebon> cool, larger turnout than i expected
16:32:38 <jlebon> let's give it 30 more seconds
16:33:12 <jlebon> ok, let's start!
16:33:15 <jlebon> #topic Action items from last meeting
16:33:22 <jlebon> - red_beard to investigate forklifting FCOS artifacts into OVH to see if they work
16:33:29 <jlebon> - red_beard to open a ticket to strategize on FCOS RH summit presence
16:33:54 <jlebon> red_beard: are you around or is that your ghost? :)
16:35:04 <jlebon> AFAIK, i haven't seen a ticket about the RH summit. and not sure about the other one.
16:35:16 <jlebon> will just re-action both for when red_beard is around
16:35:30 <jlebon> #action red_beard to investigate forklifting FCOS artifacts into OVH to see if they work
16:35:39 <jlebon> #action red_beard to open a ticket to strategize on FCOS RH summit presence
16:35:53 <jlebon> anyone have something to say about this before moving on?
16:36:15 <jlebon> ok, moving on
16:36:17 <travier_> In discussion with OVHcloud folks
16:36:21 <travier_> nothing new for now
16:36:30 <jlebon> ack, thanks travier_
16:36:48 <jlebon> #topic Actually move iptables to the nft backend
16:36:52 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/676
16:37:09 <jlebon> dustman: did you want to introduce that one?
16:37:33 <dustman> :)  - more or less we thought we were switching to use the nft backend when the rest of fedora did
16:37:48 <dustman> but the implementation of the change (via alternatives) didn't work for OSTree based systems
16:38:07 <dustman> we just recently caught it and now need to take steps to get ourselves back in line with Fedora
16:38:19 <dustman> @jlebon - do you want to talk about the options?
16:38:33 <jlebon> sure
16:38:54 <jlebon> so in Fedora, the change came in in f32, and it applied only to new installs
16:38:59 <jlebon> so we'd like to match that for FCOS
16:39:38 <jlebon> the way it works is that /usr/bin/iptables is a symlink to /etc/alternatives/... which is a symlink to either /usr/bin/iptables-legacy or /usr/bin/iptables-nft
16:39:53 <jlebon> right now, the symlinks point to legacy
16:40:06 <jlebon> we'd like to change them to nft, but *only* on new installs
16:40:12 * dustman notes that we'll make the change for iptables and the other few tools that were switched to the nft backend
16:40:38 <jlebon> doing it for new installs though is complicated by the fact that the symlinks are managed by OSTree
16:41:01 <jlebon> so if we change them in the OSTree commit, all nodes will get the change
16:41:47 <jlebon> i do think though it'd be more proper to have it in the commit, and have upgrading nodes carry the diff to stay on legacy
16:42:01 <jlebon> the proposal i posted in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732514979 is to do it in two steps
16:42:44 <jlebon> first, we mark down whether a node was already (manually) changed to use the nft backend
16:43:22 <jlebon> then in a second commit where we actually change the ostree commit, we ship a dracut systemd unit which conditionally reverts back to legacy
16:43:38 <jlebon> the conditions are:
16:43:54 <jlebon> - if it's a first boot, we do nothing (because it's a new node)
16:44:09 <jlebon> - if the node was already on nft, we do nothing
16:44:29 <jlebon> - if the symlinks somehow are already modified from the new defaults, we do nothing
16:44:38 <jlebon> - otherwise, we revert to legacy
16:45:21 <jlebon> the reason for the first barrier is that otherwise there's no way to tell whether a node is now on nft because of the new defaults, or because they had manually already changed it
16:45:45 <dustman> @jlebon - would looking at the rollback deployment tell us?
16:46:25 <travier_> How dangerous would it be to just switch to the nft backend?
16:46:31 <jlebon> dustman: i answered in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-733809643 -- i think it would mostly work, but it's not as obviously correct
16:46:46 <jlebon> travier_: yeah, that's a good question
16:47:11 <jlebon> if we look at OKD at least, it seems like they're already adapted to work with both backends
16:47:51 <jlebon> the second proposal i had was to just do the switch and tell users they can touch a stamp file if they want to stay on legacy (https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968)
16:48:39 <jlebon> i don't know enough about the other apps out there we care about that interact with iptables to know how safe that is
16:49:12 <jlebon> i'm assuming vanilla k8s would be fine too based on OKD
16:49:27 <dustman> yeah, I'm not sure TBH
16:49:57 <dustman> bgilbert, lucab: any thoughts on the best approach here?
16:50:40 <bgilbert> not particularly
16:51:07 <jlebon> how about: sanity-check that docker, podman, kube, and OKD are fine with nft
16:51:21 <jlebon> and if so, just do the more aggressive approach?
16:51:24 <lucab> I'm not too much against the idea of yolo-ing and switching everything to nft, after announcing it
16:51:49 <dustman> considering F32 switched and we haven't heard massive complaining I'd say at least docker and podman are good
16:52:00 <jlebon> right, i'd be surprised as well
16:52:03 <dustman> RHEL is already on NFT, so OKD is covered
16:52:14 <dustman> not sure abotu pure kube
16:52:27 <cyberpear> "once alternatives becomes OSTree-compatible" -- is there a bug tracking this?
16:52:51 <dustman> @cyberpear yeah, it's mentioned in the bug from this topic
16:53:04 <dustman> we need to make efforts there with upstream so that it can work
16:53:06 <lucab> cyberpear: https://github.com/fedora-sysv/chkconfig/issues/27
16:53:06 <travier_> https://github.com/fedora-sysv/chkconfig/issues/27
16:53:06 <jlebon> dustman: but that said, it's worth checking with them because there might be a state-related issues (i.e. if a node used to be on legacy and migrated)
16:53:56 <dustman> ok so I have some thoughts on this..
16:54:01 <dustman> lots of different ways we could go
16:54:17 <cyberpear> thanks for the link!
16:54:18 <travier_> https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables/
16:54:26 <dustman> we could do what jlebon suggested with two barriers (though man does that seem like a lot for a small change)
16:54:32 <dustman> we could yolo
16:54:56 <dustman> we could just stay where we are for now and publish a FAQ entry for people who want to use NFT
16:54:57 <travier_> https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall
16:54:59 <jlebon> hmm, i wonder how iptables-save/restore works across a migration... probably doesn't
16:55:22 <dustman> while we work with upstream for alternatives to work (hopefully f34 we'll be able to use that mechanism)
16:55:28 <travier_> Having both nftables & iptables rules at the same time on a system is not a good idea either apparently
16:56:00 <lucab> dustman: systemd-alternatives?
16:56:12 <dustman> the other OSTree variants are going to need something, too. so maybe the wait til f34 and get alternatives to work right, would be the right approach
16:56:28 <dustman> lucab: you forgot the d.. systemd-alternativesd
16:57:20 <jlebon> worth pointing out that dusty noticed this while browsing twitter -- AFAIK no user has reported this as an issue
16:58:09 <jlebon> so... given that it doesn't seem pressing, waiting it out until alternatives is fixed is not a bad idea
16:58:27 <travier_> A doc page for switching to NFT while we push for the proper alternatives fix seems like a good idea to me
16:58:37 <dustman> +1
16:59:04 <jlebon> to be clear though, it wouldn't change the decisions we have to face between auto-migrating folks or not
16:59:05 <cyberpear> +1 for docs of the issue, wait for proper alternatives support in f34 (is someone working on that?)
16:59:12 <dustman> I think if we go with this we'll need to take the initiative to drive the change to happen in alternatives
16:59:49 <dustman> jlebon, meaning we'd still need two barrier releases, etc?
16:59:50 <jlebon> because i think eventually we should move to nft by default
17:00:22 <jlebon> dustman: right, yeah
17:00:25 <dustman> cyberpear: no one is currently working on that. We'll have to prioritize it.
17:00:27 <cyberpear> do we have a convenient way for folks to see which changes they've missed out on?
17:00:35 <jlebon> having a fixed alternatives just makes the UX for choosing way better
17:00:37 <lucab> dustman: one barrier to switch everything from alternatives-now to alternatives-future, and then another one of nft specifically I guess
17:00:55 <cyberpear> i.e., "which changes are defaults in fresh installs, but since mine started earlier, I didn't get them?"
17:01:20 <dustman> lucab: yeah
17:01:59 <jlebon> ok, let me transform this into a proposal
17:02:47 <jlebon> #proposed we will stay with the legacy backend for now while we pursue a fixed alternatives for OSTree systems. meanwhile, we will document how users can move to the nft backend.
17:03:35 <jlebon> ack/nack? anything missing?
17:03:37 <dustman> +1
17:03:50 <lucab> ack
17:04:15 <jlebon> one more would be nice :)
17:04:28 <jbrooks> +1
17:04:44 <jlebon> #agreed we will stay with the legacy backend for now while we pursue a fixed alternatives for OSTree systems. meanwhile, we will document how users can move to the nft backend.
17:04:51 <jlebon> ok, let's move on
17:05:02 <jlebon> #topic twitter @coreos handle
17:05:29 <jlebon> hmm, need to find the ML link
17:05:43 <dustman> we did decide to wait because red_beard wanted to discuss it further this week
17:05:52 <dustman> though he's not here I don't think :(
17:05:57 <jlebon> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/I67KV5PUTTLJJL7NEYJGLECSQMKVN24F/
17:06:21 <jlebon> did we want to discuss anything from that thread today, or just wait until next week when red_beard is possibly back as well?
17:06:26 <jbrooks> Right, he definitely wanted to discuss it
17:07:04 <jlebon> ack ok, WFM to defer
17:07:27 <jlebon> dustman: ack?
17:07:29 <jbrooks> I don't have anything to add beyond what I wrote in the thread
17:07:33 <dustman> I'm good with defer, unless jbrooks wanted to discuss any logistics related to the twitter poll
17:07:59 <jbrooks> Next step would be deciding if we want to run it
17:08:12 <travier_> cyberpear i.e., "which changes are defaults in fresh installs, but since mine started earlier, I didn't get them?" > I don't think we have anything for that and I'm worrying about that too
17:08:15 <jbrooks> I need to actually talk to the person w/ the keys to post it, if we opt for that
17:08:24 <cyberpear> if we make the account private for the poll, folks can't retweet, limiting the poll's reach, IIUC
17:08:48 <jbrooks> Yes, but it'll still reach followers
17:09:06 <dustman> yeah, are we worried about random people skewing the results?
17:09:14 <bgilbert> it seems odd to me that we'd ask the community what to do with a marketing feed
17:09:18 <jbrooks> We don't have to make it private, but I'm most interested in knowing what followers think
17:09:20 <lucab> one thing I didn't ask is, why isn't RH (as the company owning it) keeping that account in the same vein as coreos-inc did, for all things k8s and for commercial purposes?
17:09:48 <jbrooks> Giving it to the upstream is SOP
17:09:55 <jbrooks> for Red Hat
17:10:18 <dustman> i.e. Red Hat is trying to do "the right thing"?
17:10:32 <jbrooks> @RedHatStorage vs @Ceph
17:10:33 <jbrooks> etc
17:10:54 <cyberpear> add "please only answer if you actually follow the account, and maybe have an option that says "I don't follow @coreos" for folks to see the results
17:11:39 <jbrooks> I don't see the value in collecting data from non-followers
17:11:55 <cyberpear> hence the "please don't answer if you don't follow"
17:12:02 <dustman> so.. in general. I think as a group we're still up in the air on if we want to make this move or not, but I think the information from the poll would be useful
17:12:14 <cyberpear> usually when I see something on twitter, it's only when someone I follow has retweeted it, even if I follow the original account
17:12:29 <dustman> obviously we'll defer some discussion til next week when red_beard is back
17:12:29 <cyberpear> (then again, I follow over 900 accounts)
17:12:40 <dustman> but i'd be +1 to running the poll in the meantime
17:13:15 <dustman> and letting jbrooks conduct it as he sees fit
17:14:44 <dustman> anybody super opposed to that?
17:15:04 <bgilbert> I guess I'm moderately opposed to the poll but I won't -1 it
17:15:51 <dustman> so -0.5 :)
17:16:03 <bgilbert> sure :-)
17:16:15 <jbrooks> We can talk more about it, it's no rush
17:16:30 <jlebon> need a poll to ask if they want a poll
17:16:34 <jbrooks> heh
17:16:44 <bgilbert> wheeee
17:17:08 <jlebon> ok to move on?
17:17:10 <bgilbert> +1
17:17:33 <jlebon> there's one more thing i wanted to talk about if travier_ is still around
17:17:40 <travier_> yep
17:17:48 <jlebon> #topic  manifests/ignition-and-ostree: Remove systemd-repart
17:17:54 <jlebon> #link https://github.com/coreos/fedora-coreos-config/pull/744
17:18:13 <jlebon> travier_: want to briefly introduce what your change is about?
17:18:19 <travier_> Sure
17:19:18 <travier_> Since F33 (I think, have not checked yet), we also ship systemd-repart enabled by default in the initrd and the final root which is a way to change to disk partitions
17:19:27 <travier_> repartition the disk
17:20:02 <travier_> This does nothing right now as we don't provide any config files but user could do that
17:20:34 <travier_> As Ignition is currently the preferred way to do that, this PRs suggest removing systemd-repart from fcos
17:21:00 <jlebon> thanks travier_
17:21:10 <travier_> we have three options: 1. keep things as is, 2. remove systemd-repart from the manifest, 3. split into another rpm
17:21:22 <travier_> s/from/in the manifest/
17:21:27 <dustman> So `systemd-repart` will do nothing by default?
17:21:28 <travier_> 2 is the PR
17:21:37 <jlebon> 4 mask systemd-repart ?
17:21:49 <travier_> 4 won't work for the initramfs I think
17:22:03 <travier_> it does nothing but it is enabled
17:22:25 <jlebon> honestly, i don't think we need to worry about users trying to use repart in the initrd
17:22:52 <dustman> jlebon, right. they'd also need to use a client side generated initrd?
17:23:06 <dustman> in order to add a config file to try to use repart
17:23:08 <jlebon> oh, or does repart in the initrd look into the /etc of /sysroot?
17:23:23 <jlebon> dustman: right, that's my thinking
17:23:23 <travier_> This is not critical but when I took a look at the logs I was wondering why this would be included when we don't use it and probably don't recommend using it
17:23:56 <jlebon> definitely agreed, we should steer users away from it
17:24:13 <jlebon> i think what we were discussing in the PR was how we should do that
17:24:34 <lucab> jlebon: that's possible but quite unlikely, just a few snowflakes source config from /sysroot
17:24:46 <jlebon> IMO, remove-from-packages is a very blunt tool
17:25:31 <lucab> yeah, we learned that it is nasty
17:25:38 <jlebon> because it mutates the package itself behind its back, and it also makes `rpm -ql` lie
17:26:02 <travier_> indeed
17:26:03 <dustman> I think I prefer trying to split the package if there is a legit case for not wanting repart, but I understand that's more work. In the very least masking it would be my preferred method (assuming that is an option).
17:26:21 <lucab> 4 in the short term and 3 in the long run, perhaps?
17:26:56 <dustman> i'm cool with masking even in the long term, but we need a FAQ entry I think (like we're doing for dnsmasq)
17:27:00 <jlebon> my argument for masking was that we already do it for dnsmasq, and it's good to be consistent
17:27:11 <lucab> (assuming RPM maintainers are happy with the split, otherwise only masking)
17:27:13 <cyberpear> can't we just change the fcos preset for the service?
17:27:34 <dustman> i don't even know if that is enough
17:27:35 <jlebon> cyberpear: it's statically enabled. i'm not sure if presets work for those (i have a vague memory it doesn't)
17:27:42 <cyberpear> but yes, RPM split is fine, too; just probablymore work
17:27:45 <dustman> users already have to drop in config files to use it
17:27:47 <cyberpear> fair enough
17:29:01 <jlebon> i also like that masking as a systemd primitive matches well the message we're trying to convey
17:29:32 <jlebon> it's an active action, whereas nuking the unit is an "omission".  i don't know how well that resonates with others, heh
17:29:37 <dustman> travier_ is masking as either a short or long term solution good enough for you?
17:30:47 <travier_> We can go with that. I don't think this will work for the initrd but should not be a problem. I don't know how we would do that though. Directly in the manifest?
17:31:15 <jlebon> travier_: i can help you with that :)
17:31:22 <dustman> yeah, just search for where I did it for dnsmas
17:31:26 <dustman> dnsmasq :)
17:31:39 <jlebon> or that, yeah :)
17:31:40 <travier_> I'll try to work on splitting the RPM upstream
17:32:08 <jlebon> whoops, looks like we're past time
17:32:11 <dustman> travier_ just talk the maintainer of the fedora rpm, he's very receptive
17:32:28 <jlebon> ok, let's move on to open floor quickly
17:32:41 <jlebon> #topic Open Floor
17:32:51 <jlebon> anyone have anything they want to bring up?
17:33:28 <dustman> if anyone knows systemd-resolved well and has some time to maybe work with lorbus or vadim it would be nice to get to the bottom of this issue: https://github.com/coreos/fedora-coreos-tracker/issues/679
17:33:49 <dustman> i really don't think our migration service is the cause of troubles there, but I could be wrong
17:34:14 <jlebon> +1 thanks Dusty
17:34:45 <jlebon> ok, closing the meeting in 49 seconds
17:35:36 <jlebon> #endmeeting