16:28:39 <dustymabe> #startmeeting fedora_coreos_meeting 16:28:39 <zodbot> Meeting started Wed Apr 26 16:28:39 2023 UTC. 16:28:39 <zodbot> This meeting is logged and archived in a public location. 16:28:39 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:28:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:28:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:28:55 <dustymabe> #topic roll call 16:28:58 <dustymabe> .hi 16:28:59 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:29:07 <c4rt0> .hello c4rt0 16:29:08 <zodbot> c4rt0: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com> 16:29:10 <fifofonix> ./hi 16:29:26 <fifofonix> .hi 16:29:27 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com> 16:29:37 <jmarrero> .hi 16:29:38 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com> 16:29:43 <jlebon> .hello2 16:29:44 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:30:15 <dustymabe> #chair fifofonix jmarrero jlebon 16:30:16 <zodbot> Current chairs: dustymabe fifofonix jlebon jmarrero 16:30:41 <bgilbert> .hi 16:30:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:30:50 <rishabhsaini> hi 16:32:00 <dustymabe> #chair rishabhsaini bgilbert 16:32:00 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero rishabhsaini 16:32:10 <dustymabe> will get started here in a minute 16:32:44 <travier> .hello siosm 16:32:45 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com> 16:32:46 <marmijo[m]> .hello marmijo 16:32:48 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com> 16:33:56 <dustymabe> #topic Action items from last meeting 16:34:03 <dustymabe> #chair travier marmijo[m] 16:34:03 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero marmijo[m] rishabhsaini travier 16:34:15 <dustymabe> * bgilbert to write up a comment in the bug 16:34:17 <dustymabe> :) 16:34:39 <ravanelli> .hi 16:34:42 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com> 16:34:43 <dustymabe> #info bgilbert made a beautiful comment to explain things in https://github.com/coreos/fedora-coreos-tracker/issues/1474#issuecomment-1517289046 16:34:48 <dustymabe> thanks bgilbert 16:34:49 <bgilbert> <3 16:34:56 <dustymabe> #chair ravanelli 16:34:56 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero marmijo[m] ravanelli rishabhsaini travier 16:35:13 <dustymabe> that was all for action items.. 16:35:18 <travier> ❤️ 16:35:51 <dustymabe> full disclosure - most of us in the FCOS team are in weeklong meetings this week so I'll only focus on meeting tickets that are time sensitive 16:36:02 <bgilbert> +1 16:36:12 <dustymabe> of which I think this one is: 16:36:20 <dustymabe> #topic CIFS Mounts Failing On Next 38.20230322.1.0 due to SELinux Issue (keyutils) 16:36:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1447 16:36:46 <dustymabe> It's good to see fifofonix here (thank you for opening this issue) 16:36:55 <fifofonix> yw 16:37:46 <dustymabe> as I understand it this is a prolem that prevents CIFS filesystem shares access (blocked by selinux denials) 16:37:53 <dustymabe> s/prolem/problem 16:38:26 <bgilbert> but not all CIFS shares, right? https://github.com/coreos/fedora-coreos-tracker/issues/1447#issuecomment-1520500744 16:38:27 <fifofonix> yes, but i think it is actually restricted to mounting DFS-based shared 16:38:29 <dustymabe> fifofonix: do I understand correctly that it's anyone using CIFS shares on FCOS that would be affected, or does the configuration need to be specific? 16:38:42 <dustymabe> ahh, OK 16:38:43 <bgilbert> how common is the affected configuration likely to be? 16:38:53 * dustymabe is not a CIFS expert 16:39:54 <dustymabe> fifofonix: would you know the answer to ^^ 16:40:01 <fifofonix> DFS is a popular MSFT thing that notionally provides HA mounts. 16:40:15 <dustymabe> HA == high availability ? 16:40:35 <bgilbert> does that require special server-side config, e.g. multiple backend servers? 16:40:44 <fifofonix> Yes. No expert here either. Concept is that there is a single name that will resolve to active backend. 16:41:03 <dustymabe> aside: /me wonders if we should add a samba test (I think Amazon has a service we could hook into for this) 16:41:26 <fifofonix> When we say CIFS we really mean SMB. 16:41:33 <dustymabe> fifofonix: correct, me too 16:41:40 <jlebon> dustymabe: +1 16:41:49 <fifofonix> I think CIFS predates SMB but you still notionally mount as CIFS. 16:42:09 <dustymabe> fifofonix: oh, I thought it was the other way around, but that was just me assuming things 16:42:24 <fifofonix> I think one interesting part is how widely is this used, and the second is what are available workarounds. 16:42:37 <bgilbert> so a) it's not just SMB but HA SMB b) there's a workaround (disable HA) c) there's currently no proper fix 16:42:56 <dustymabe> right. I think c) is the big one that's a concern for me 16:43:04 <bgilbert> (I think CIFS is the name for a particular point-in-time snapshot of the protocol that was documented) 16:43:12 <fifofonix> In my set up I am able to workaround by specifically addressing the primary share server (obviously no good in event of failover). 16:43:18 <dustymabe> if we decided to hold the promotion to stable it would be nice to know how long (i.e. not have it be unbounded) 16:44:03 <jmarrero> This is also the same issue right? https://discussion.fedoraproject.org/t/fc38-mount-cifs-mount-smb-mount-smb3-error-126/81271 16:44:21 * dustymabe clicky 16:44:30 <jlebon> bgilbert: for b), is that something the client can decide? 16:44:46 <fifofonix> That does look the same. 16:45:01 <jlebon> i.e. is the HA configured client side or server side? 16:45:23 <jmarrero> so we got more people hitting this 16:45:32 <dustymabe> so several people/groups are starting to hit this 16:45:33 <bgilbert> jlebon: that was my understanding from fifofonix's GH comment 16:45:37 <dustymabe> yeah 16:45:47 <fifofonix> this is broader than fcos too obvs 16:45:59 <bgilbert> fifofonix, could you confirm that the workaround is client-side? 16:46:17 <dustymabe> jmarrero: I'll try to link that discussion thread to the BZs where we are tracking this already 16:46:45 <dustymabe> bgilbert: +1 - that's good information to find out if fifofonix has it 16:46:59 <fifofonix> so, i can modify the mounts on all my hosts to point to a primary server rather than a DFS name and then move forward. 16:47:22 <dustymabe> so workarounds include: 16:47:34 <dustymabe> 1. some client configuration change in CIFs configuration 16:47:45 <dustymabe> 2. set SELinux to permissive 16:47:59 <dustymabe> 3. hold back updates on your server/fleet 16:48:06 <bgilbert> on balance, I'd be inclined not to delay the promotion to stable 16:48:10 <dustymabe> i guess 3 isn't a workaround per-se 16:48:13 <bgilbert> with or without a coreos-status post 16:48:18 <jlebon> 4. it should work to customize the policy too I think 16:48:26 <dustymabe> jlebon: i.e. server side? 16:48:31 <jlebon> dustymabe: client-side 16:48:32 <fifofonix> with all of the above it requires an alert to the community because action is required. 16:48:35 <dustymabe> oh no, you mean selinux policy 16:48:41 <jlebon> right, yeah 16:48:52 <dustymabe> could we ship a generic client policy update? 16:49:08 <jlebon> dustymabe: can you expand? 16:49:24 <dustymabe> i.e. bake a policy with exceptions into the image we ship 16:49:32 <dustymabe> or is it so specific 16:49:41 <dustymabe> i.e. to the mountpoints or something 16:49:58 <jlebon> dustymabe: ahhh gotcha. we could modify our policy, yes. 16:50:13 <jlebon> but that's circumventing evaluation by selinux folks 16:50:20 <dustymabe> of course, yes 16:50:27 <fifofonix> (i was trying to get a .cil file to apply to do this - but don't know how to gen one from the audit2allow output in BZ) 16:50:55 <dustymabe> jlebon: but it would be better than say if we were to decide to set selinux to permissive for all (which we wouldn't do) 16:51:15 <jlebon> fifofonix: yeah, i think it'd require building the module in a container maybe and then `semodule -i` it on the hosts 16:51:38 <jlebon> dustymabe: definitely 16:52:05 <dustymabe> that is something that would require some investigation as I don't think we've ever done that specifically before 16:52:32 <dustymabe> so 4. is "it should work to customize the policy too I think, client side" 16:52:49 <dustymabe> and 5. would be for us to ship a customized policy in our images 16:53:15 <dustymabe> ok at least we have the options listed out 16:53:51 <dustymabe> ideas on path forward? it is interesting to note that so far we've only got one report of this from our users (thanks fifofonix) 16:53:56 <jlebon> fifofonix: how (un)acceptable is option 1 in your opinion? 16:54:09 <dustymabe> only 4 people subscribed to https://github.com/coreos/fedora-coreos-tracker/issues/1447 16:54:26 <travier> another workaround would be to set the one problematic domain into permissive 16:54:35 <dustymabe> but maybe they are subscribed to the issue and just passively watching 16:54:42 <travier> not the entire node, just one domain 16:54:55 <bgilbert> dustymabe: 4 participating. subscribed count isn't shown AFAIK 16:55:11 <dustymabe> bgilbert: ok yeah, there is a distinction there 16:55:26 <jlebon> travier: is that something you can easily at runtime without a policy update? 16:55:31 <jlebon> easily do* 16:55:40 <travier> yes, if we had semanage but we don't 16:55:45 <fifofonix> i think (i) is acceptable to me but i'm aware of the issue and its extend and have had time to warm to it. 16:55:47 <jlebon> :) 16:55:54 <travier> I'm looking into how that works 16:56:17 <dustymabe> travier: 1 was client configuration on the CIFS side, not selinux 16:56:26 <fifofonix> but, for someone not running next/test ... i think they need some notice so they hold updates / can test changes / etc. 16:56:42 <dustymabe> fifofonix: I think we'd need to send a coreos-status email - so hopefully they would see that 16:56:51 <dustymabe> and then evaluate their options 16:56:58 <travier> dustymabe: it's to replace your 2 option 16:56:59 <fifofonix> +1 16:57:18 <dustymabe> travier: ahh, so then you are talking about 4. I think 16:57:39 <travier> well, we don't have to do it. people impacted by it can do it 16:57:42 <travier> it's a workaround 16:57:45 <dustymabe> correct 16:58:01 <dustymabe> ok thoughts on coming to a conclusion/decision? 16:58:12 <bgilbert> #proposed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled. 16:58:12 <fifofonix> back to 1. when you say client CIFS config I'm interpreting as systemd mount config change on all vms. 16:58:36 <fifofonix> when is f38 stable? 16:58:39 <dustymabe> fifofonix: right. whatever steps you took to change the config client side to stop having the problem 16:58:40 <bgilbert> fifofonix: yeah 16:58:43 <jlebon> fifofonix: next week 16:58:52 <bgilbert> F38 is stable already, but the FCOS stable rebase is next week 16:58:57 <dustymabe> right, next week was the original scheduled date 16:59:20 <fifofonix> ok well deserves 'IMPORTANT' flagging. 16:59:24 <bgilbert> though if the SELinux folks get the info they need and produce a fix by then, we could possibly fast-track 16:59:35 <dustymabe> +1 to proposed 16:59:40 <jlebon> we'd want to send the email out this week 16:59:49 <jlebon> ack to proposed 16:59:50 <dustymabe> bgilbert: agree - if we get a fix soon then we could re-evaluate 16:59:53 <fifofonix> yes, as much notice as possible. 17:00:01 <jlebon> should we reflect that in the proposal? 17:00:03 <jmarrero> +1 to proposed 17:00:06 <bgilbert> fifofonix: are you able to help with a reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2182643 ? 17:00:06 <marmijo[m]> +1 to proposed 17:00:09 <travier> +1 to proposed 17:00:17 <ravanelli> +1 17:00:18 <jlebon> (the "might reevaludate if a fix becomes available" part) 17:00:29 <bgilbert> yeah, let me edit 17:00:31 <dustymabe> jlebon: SGTM 17:00:50 <fifofonix> bgilbert: the biggest component of a reproducer seems to be having a DFS share (i'm not aware that any work). 17:01:10 <bgilbert> fifofonix: yeah, ideally they could get a step-by-step for that 17:02:10 <bgilbert> #proposed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled. If a fix becomes available, we'll evaluate it for fast-tracking into stable. 17:02:19 <dustymabe> +1 17:02:23 <bgilbert> fifofonix: I think you're probably in the best position to help them 17:02:26 <jlebon> aye to proposal 17:02:47 <dustymabe> we'll also need fifofonix help with example workaround steps (if he's willing) 17:02:48 <fifofonix> bgilbert: unfortunately, i don't know how to create one of those. happy to run more detailed logging on my infra. 17:02:51 <bgilbert> if that's actively being worked on, we may want to delay the coreos-status email a bit, but I opted not to put that in the proposed 17:03:10 <dustymabe> "actively being worked on" -> the bug? 17:03:14 <bgilbert> yeah, the bug 17:03:35 <dustymabe> I haven't seen any activity this week. I emailed a few people earlier today, but I think it was already late in the day for them 17:03:43 <bgilbert> fifofonix: hmm, okay. I don't think anyone in this meeting knows how to set that up then 17:03:49 <dustymabe> since more and more people are seeing it in Fedora I think we might be able to get it prioritized, though 17:03:54 <bgilbert> fair 17:04:12 <bgilbert> #agreed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled. If a fix becomes available, we'll evaluate it for fast-tracking into stable. 17:04:28 <dustymabe> should we target a day to try to send that email ^^ 17:04:37 <bgilbert> not later than Friday I assume 17:04:43 <jlebon> bgilbert: +1 17:04:44 <dustymabe> i.e. maybe give it one more day to see if anything pops up in the BZ and then send it 17:04:49 <bgilbert> dustymabe: +1 17:05:00 <dustymabe> ok so try to send thursday, latest Friday 17:05:07 <fifofonix> not hopeful on BZ. has been super quiet. 17:05:13 <bgilbert> yeah 17:05:26 <bgilbert> volunteers to write the post? 17:05:40 <jlebon> Ondrej asked for a reproducer last week, but sounds like that's hard to do 17:06:02 <dustymabe> yeah, maybe we could find a hosted service that shows the problem (but that would be work itself) 17:06:10 <dustymabe> bgilbert: i can try if no one else wants to 17:06:20 <bgilbert> dustymabe: +1 17:06:27 <jlebon> dustymabe: i can help you too 17:06:41 <dustymabe> #action dustymabe to write coreos-status post about CIFS issue for f38 rebase heading to stable 17:07:18 <dustymabe> I'll lean on you fifofonix for some client side workaround steps if you don't mind 17:07:37 <dustymabe> #topic open floor 17:07:38 <fifofonix> dustymabe: will do my best. 17:07:47 <dustymabe> anyone with anything for open floor today? 17:08:02 <dustymabe> OR are there any more time sensitive topics that we should discuss? 17:09:34 <dustymabe> ok I'll close it out 17:09:37 <bgilbert> +1 17:09:40 <dustymabe> #endmeeting