16:30:50 #startmeeting fedora_coreos_meeting 16:30:50 Meeting started Wed Nov 25 16:30:50 2020 UTC. 16:30:50 This meeting is logged and archived in a public location. 16:30:50 The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:50 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:50 The meeting name has been set to 'fedora_coreos_meeting' 16:30:55 .hello2 16:30:56 cyberpear: cyberpear 'James Cassell' 16:30:57 #topic roll call 16:30:59 .hello2 16:31:00 .hello siosmù 16:31:00 bgilbert: bgilbert 'Benjamin Gilbert' 16:31:01 .hello siosm 16:31:03 travier_: Sorry, but you don't exist 16:31:06 travier_: siosm 'Timothée Ravier' 16:31:10 .hello dustymabe 16:31:11 dustman: dustymabe 'Dusty Mabe' 16:31:26 #chair cyberpear bgilbert travier_ dustman 16:31:26 Current chairs: bgilbert cyberpear dustman jlebon travier_ 16:31:30 .hello jasonbrooks 16:31:31 jbrooks: jasonbrooks 'Jason Brooks' 16:31:37 dustman: haha that nick 16:31:41 #chair jbrooks 16:31:41 Current chairs: bgilbert cyberpear dustman jbrooks jlebon travier_ 16:32:03 :) - I'm in freenode webchat 16:32:14 nice 16:32:15 .hello2 16:32:19 lucab: lucab 'Luca Bruno' 16:32:21 #chair lucab 16:32:21 Current chairs: bgilbert cyberpear dustman jbrooks jlebon lucab travier_ 16:32:30 cool, larger turnout than i expected 16:32:38 let's give it 30 more seconds 16:33:12 ok, let's start! 16:33:15 #topic Action items from last meeting 16:33:22 - red_beard to investigate forklifting FCOS artifacts into OVH to see if they work 16:33:29 - red_beard to open a ticket to strategize on FCOS RH summit presence 16:33:54 red_beard: are you around or is that your ghost? :) 16:35:04 AFAIK, i haven't seen a ticket about the RH summit. and not sure about the other one. 16:35:16 will just re-action both for when red_beard is around 16:35:30 #action red_beard to investigate forklifting FCOS artifacts into OVH to see if they work 16:35:39 #action red_beard to open a ticket to strategize on FCOS RH summit presence 16:35:53 anyone have something to say about this before moving on? 16:36:15 ok, moving on 16:36:17 In discussion with OVHcloud folks 16:36:21 nothing new for now 16:36:30 ack, thanks travier_ 16:36:48 #topic Actually move iptables to the nft backend 16:36:52 #link https://github.com/coreos/fedora-coreos-tracker/issues/676 16:37:09 dustman: did you want to introduce that one? 16:37:33 :) - more or less we thought we were switching to use the nft backend when the rest of fedora did 16:37:48 but the implementation of the change (via alternatives) didn't work for OSTree based systems 16:38:07 we just recently caught it and now need to take steps to get ourselves back in line with Fedora 16:38:19 @jlebon - do you want to talk about the options? 16:38:33 sure 16:38:54 so in Fedora, the change came in in f32, and it applied only to new installs 16:38:59 so we'd like to match that for FCOS 16:39:38 the way it works is that /usr/bin/iptables is a symlink to /etc/alternatives/... which is a symlink to either /usr/bin/iptables-legacy or /usr/bin/iptables-nft 16:39:53 right now, the symlinks point to legacy 16:40:06 we'd like to change them to nft, but *only* on new installs 16:40:12 * dustman notes that we'll make the change for iptables and the other few tools that were switched to the nft backend 16:40:38 doing it for new installs though is complicated by the fact that the symlinks are managed by OSTree 16:41:01 so if we change them in the OSTree commit, all nodes will get the change 16:41:47 i do think though it'd be more proper to have it in the commit, and have upgrading nodes carry the diff to stay on legacy 16:42:01 the proposal i posted in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732514979 is to do it in two steps 16:42:44 first, we mark down whether a node was already (manually) changed to use the nft backend 16:43:22 then in a second commit where we actually change the ostree commit, we ship a dracut systemd unit which conditionally reverts back to legacy 16:43:38 the conditions are: 16:43:54 - if it's a first boot, we do nothing (because it's a new node) 16:44:09 - if the node was already on nft, we do nothing 16:44:29 - if the symlinks somehow are already modified from the new defaults, we do nothing 16:44:38 - otherwise, we revert to legacy 16:45:21 the reason for the first barrier is that otherwise there's no way to tell whether a node is now on nft because of the new defaults, or because they had manually already changed it 16:45:45 @jlebon - would looking at the rollback deployment tell us? 16:46:25 How dangerous would it be to just switch to the nft backend? 16:46:31 dustman: i answered in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-733809643 -- i think it would mostly work, but it's not as obviously correct 16:46:46 travier_: yeah, that's a good question 16:47:11 if we look at OKD at least, it seems like they're already adapted to work with both backends 16:47:51 the second proposal i had was to just do the switch and tell users they can touch a stamp file if they want to stay on legacy (https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968) 16:48:39 i don't know enough about the other apps out there we care about that interact with iptables to know how safe that is 16:49:12 i'm assuming vanilla k8s would be fine too based on OKD 16:49:27 yeah, I'm not sure TBH 16:49:57 bgilbert, lucab: any thoughts on the best approach here? 16:50:40 not particularly 16:51:07 how about: sanity-check that docker, podman, kube, and OKD are fine with nft 16:51:21 and if so, just do the more aggressive approach? 16:51:24 I'm not too much against the idea of yolo-ing and switching everything to nft, after announcing it 16:51:49 considering F32 switched and we haven't heard massive complaining I'd say at least docker and podman are good 16:52:00 right, i'd be surprised as well 16:52:03 RHEL is already on NFT, so OKD is covered 16:52:14 not sure abotu pure kube 16:52:27 "once alternatives becomes OSTree-compatible" -- is there a bug tracking this? 16:52:51 @cyberpear yeah, it's mentioned in the bug from this topic 16:53:04 we need to make efforts there with upstream so that it can work 16:53:06 cyberpear: https://github.com/fedora-sysv/chkconfig/issues/27 16:53:06 https://github.com/fedora-sysv/chkconfig/issues/27 16:53:06 dustman: but that said, it's worth checking with them because there might be a state-related issues (i.e. if a node used to be on legacy and migrated) 16:53:56 ok so I have some thoughts on this.. 16:54:01 lots of different ways we could go 16:54:17 thanks for the link! 16:54:18 https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables/ 16:54:26 we could do what jlebon suggested with two barriers (though man does that seem like a lot for a small change) 16:54:32 we could yolo 16:54:56 we could just stay where we are for now and publish a FAQ entry for people who want to use NFT 16:54:57 https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall 16:54:59 hmm, i wonder how iptables-save/restore works across a migration... probably doesn't 16:55:22 while we work with upstream for alternatives to work (hopefully f34 we'll be able to use that mechanism) 16:55:28 Having both nftables & iptables rules at the same time on a system is not a good idea either apparently 16:56:00 dustman: systemd-alternatives? 16:56:12 the other OSTree variants are going to need something, too. so maybe the wait til f34 and get alternatives to work right, would be the right approach 16:56:28 lucab: you forgot the d.. systemd-alternativesd 16:57:20 worth pointing out that dusty noticed this while browsing twitter -- AFAIK no user has reported this as an issue 16:58:09 so... given that it doesn't seem pressing, waiting it out until alternatives is fixed is not a bad idea 16:58:27 A doc page for switching to NFT while we push for the proper alternatives fix seems like a good idea to me 16:58:37 +1 16:59:04 to be clear though, it wouldn't change the decisions we have to face between auto-migrating folks or not 16:59:05 +1 for docs of the issue, wait for proper alternatives support in f34 (is someone working on that?) 16:59:12 I think if we go with this we'll need to take the initiative to drive the change to happen in alternatives 16:59:49 jlebon, meaning we'd still need two barrier releases, etc? 16:59:50 because i think eventually we should move to nft by default 17:00:22 dustman: right, yeah 17:00:25 cyberpear: no one is currently working on that. We'll have to prioritize it. 17:00:27 do we have a convenient way for folks to see which changes they've missed out on? 17:00:35 having a fixed alternatives just makes the UX for choosing way better 17:00:37 dustman: one barrier to switch everything from alternatives-now to alternatives-future, and then another one of nft specifically I guess 17:00:55 i.e., "which changes are defaults in fresh installs, but since mine started earlier, I didn't get them?" 17:01:20 lucab: yeah 17:01:59 ok, let me transform this into a proposal 17:02:47 #proposed we will stay with the legacy backend for now while we pursue a fixed alternatives for OSTree systems. meanwhile, we will document how users can move to the nft backend. 17:03:35 ack/nack? anything missing? 17:03:37 +1 17:03:50 ack 17:04:15 one more would be nice :) 17:04:28 +1 17:04:44 #agreed we will stay with the legacy backend for now while we pursue a fixed alternatives for OSTree systems. meanwhile, we will document how users can move to the nft backend. 17:04:51 ok, let's move on 17:05:02 #topic twitter @coreos handle 17:05:29 hmm, need to find the ML link 17:05:43 we did decide to wait because red_beard wanted to discuss it further this week 17:05:52 though he's not here I don't think :( 17:05:57 #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/I67KV5PUTTLJJL7NEYJGLECSQMKVN24F/ 17:06:21 did we want to discuss anything from that thread today, or just wait until next week when red_beard is possibly back as well? 17:06:26 Right, he definitely wanted to discuss it 17:07:04 ack ok, WFM to defer 17:07:27 dustman: ack? 17:07:29 I don't have anything to add beyond what I wrote in the thread 17:07:33 I'm good with defer, unless jbrooks wanted to discuss any logistics related to the twitter poll 17:07:59 Next step would be deciding if we want to run it 17:08:12 cyberpear i.e., "which changes are defaults in fresh installs, but since mine started earlier, I didn't get them?" > I don't think we have anything for that and I'm worrying about that too 17:08:15 I need to actually talk to the person w/ the keys to post it, if we opt for that 17:08:24 if we make the account private for the poll, folks can't retweet, limiting the poll's reach, IIUC 17:08:48 Yes, but it'll still reach followers 17:09:06 yeah, are we worried about random people skewing the results? 17:09:14 it seems odd to me that we'd ask the community what to do with a marketing feed 17:09:18 We don't have to make it private, but I'm most interested in knowing what followers think 17:09:20 one thing I didn't ask is, why isn't RH (as the company owning it) keeping that account in the same vein as coreos-inc did, for all things k8s and for commercial purposes? 17:09:48 Giving it to the upstream is SOP 17:09:55 for Red Hat 17:10:18 i.e. Red Hat is trying to do "the right thing"? 17:10:32 @RedHatStorage vs @Ceph 17:10:33 etc 17:10:54 add "please only answer if you actually follow the account, and maybe have an option that says "I don't follow @coreos" for folks to see the results 17:11:39 I don't see the value in collecting data from non-followers 17:11:55 hence the "please don't answer if you don't follow" 17:12:02 so.. in general. I think as a group we're still up in the air on if we want to make this move or not, but I think the information from the poll would be useful 17:12:14 usually when I see something on twitter, it's only when someone I follow has retweeted it, even if I follow the original account 17:12:29 obviously we'll defer some discussion til next week when red_beard is back 17:12:29 (then again, I follow over 900 accounts) 17:12:40 but i'd be +1 to running the poll in the meantime 17:13:15 and letting jbrooks conduct it as he sees fit 17:14:44 anybody super opposed to that? 17:15:04 I guess I'm moderately opposed to the poll but I won't -1 it 17:15:51 so -0.5 :) 17:16:03 sure :-) 17:16:15 We can talk more about it, it's no rush 17:16:30 need a poll to ask if they want a poll 17:16:34 heh 17:16:44 wheeee 17:17:08 ok to move on? 17:17:10 +1 17:17:33 there's one more thing i wanted to talk about if travier_ is still around 17:17:40 yep 17:17:48 #topic manifests/ignition-and-ostree: Remove systemd-repart 17:17:54 #link https://github.com/coreos/fedora-coreos-config/pull/744 17:18:13 travier_: want to briefly introduce what your change is about? 17:18:19 Sure 17:19:18 Since F33 (I think, have not checked yet), we also ship systemd-repart enabled by default in the initrd and the final root which is a way to change to disk partitions 17:19:27 repartition the disk 17:20:02 This does nothing right now as we don't provide any config files but user could do that 17:20:34 As Ignition is currently the preferred way to do that, this PRs suggest removing systemd-repart from fcos 17:21:00 thanks travier_ 17:21:10 we have three options: 1. keep things as is, 2. remove systemd-repart from the manifest, 3. split into another rpm 17:21:22 s/from/in the manifest/ 17:21:27 So `systemd-repart` will do nothing by default? 17:21:28 2 is the PR 17:21:37 4 mask systemd-repart ? 17:21:49 4 won't work for the initramfs I think 17:22:03 it does nothing but it is enabled 17:22:25 honestly, i don't think we need to worry about users trying to use repart in the initrd 17:22:52 jlebon, right. they'd also need to use a client side generated initrd? 17:23:06 in order to add a config file to try to use repart 17:23:08 oh, or does repart in the initrd look into the /etc of /sysroot? 17:23:23 dustman: right, that's my thinking 17:23:23 This is not critical but when I took a look at the logs I was wondering why this would be included when we don't use it and probably don't recommend using it 17:23:56 definitely agreed, we should steer users away from it 17:24:13 i think what we were discussing in the PR was how we should do that 17:24:34 jlebon: that's possible but quite unlikely, just a few snowflakes source config from /sysroot 17:24:46 IMO, remove-from-packages is a very blunt tool 17:25:31 yeah, we learned that it is nasty 17:25:38 because it mutates the package itself behind its back, and it also makes `rpm -ql` lie 17:26:02 indeed 17:26:03 I think I prefer trying to split the package if there is a legit case for not wanting repart, but I understand that's more work. In the very least masking it would be my preferred method (assuming that is an option). 17:26:21 4 in the short term and 3 in the long run, perhaps? 17:26:56 i'm cool with masking even in the long term, but we need a FAQ entry I think (like we're doing for dnsmasq) 17:27:00 my argument for masking was that we already do it for dnsmasq, and it's good to be consistent 17:27:11 (assuming RPM maintainers are happy with the split, otherwise only masking) 17:27:13 can't we just change the fcos preset for the service? 17:27:34 i don't even know if that is enough 17:27:35 cyberpear: it's statically enabled. i'm not sure if presets work for those (i have a vague memory it doesn't) 17:27:42 but yes, RPM split is fine, too; just probablymore work 17:27:45 users already have to drop in config files to use it 17:27:47 fair enough 17:29:01 i also like that masking as a systemd primitive matches well the message we're trying to convey 17:29:32 it's an active action, whereas nuking the unit is an "omission". i don't know how well that resonates with others, heh 17:29:37 travier_ is masking as either a short or long term solution good enough for you? 17:30:47 We can go with that. I don't think this will work for the initrd but should not be a problem. I don't know how we would do that though. Directly in the manifest? 17:31:15 travier_: i can help you with that :) 17:31:22 yeah, just search for where I did it for dnsmas 17:31:26 dnsmasq :) 17:31:39 or that, yeah :) 17:31:40 I'll try to work on splitting the RPM upstream 17:32:08 whoops, looks like we're past time 17:32:11 travier_ just talk the maintainer of the fedora rpm, he's very receptive 17:32:28 ok, let's move on to open floor quickly 17:32:41 #topic Open Floor 17:32:51 anyone have anything they want to bring up? 17:33:28 if anyone knows systemd-resolved well and has some time to maybe work with lorbus or vadim it would be nice to get to the bottom of this issue: https://github.com/coreos/fedora-coreos-tracker/issues/679 17:33:49 i really don't think our migration service is the cause of troubles there, but I could be wrong 17:34:14 +1 thanks Dusty 17:34:45 ok, closing the meeting in 49 seconds 17:35:36 #endmeeting