16:28:39 <dustymabe> #startmeeting fedora_coreos_meeting
16:28:39 <zodbot> Meeting started Wed Apr 26 16:28:39 2023 UTC.
16:28:39 <zodbot> This meeting is logged and archived in a public location.
16:28:39 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:28:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:28:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:28:55 <dustymabe> #topic roll call
16:28:58 <dustymabe> .hi
16:28:59 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:29:07 <c4rt0> .hello c4rt0
16:29:08 <zodbot> c4rt0: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:29:10 <fifofonix> ./hi
16:29:26 <fifofonix> .hi
16:29:27 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:29:37 <jmarrero> .hi
16:29:38 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:29:43 <jlebon> .hello2
16:29:44 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:15 <dustymabe> #chair fifofonix jmarrero jlebon
16:30:16 <zodbot> Current chairs: dustymabe fifofonix jlebon jmarrero
16:30:41 <bgilbert> .hi
16:30:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:50 <rishabhsaini> hi
16:32:00 <dustymabe> #chair rishabhsaini bgilbert
16:32:00 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero rishabhsaini
16:32:10 <dustymabe> will get started here in a minute
16:32:44 <travier> .hello siosm
16:32:45 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:46 <marmijo[m]> .hello marmijo
16:32:48 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:33:56 <dustymabe> #topic Action items from last meeting
16:34:03 <dustymabe> #chair travier marmijo[m]
16:34:03 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero marmijo[m] rishabhsaini travier
16:34:15 <dustymabe> * bgilbert to write up a comment in the bug
16:34:17 <dustymabe> :)
16:34:39 <ravanelli> .hi
16:34:42 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:34:43 <dustymabe> #info bgilbert made a beautiful comment to explain things in https://github.com/coreos/fedora-coreos-tracker/issues/1474#issuecomment-1517289046
16:34:48 <dustymabe> thanks bgilbert
16:34:49 <bgilbert> <3
16:34:56 <dustymabe> #chair ravanelli
16:34:56 <zodbot> Current chairs: bgilbert dustymabe fifofonix jlebon jmarrero marmijo[m] ravanelli rishabhsaini travier
16:35:13 <dustymabe> that was all for action items..
16:35:18 <travier> ❤️
16:35:51 <dustymabe> full disclosure - most of us in the FCOS team are in weeklong meetings this week so I'll only focus on meeting tickets that are time sensitive
16:36:02 <bgilbert> +1
16:36:12 <dustymabe> of which I think this one is:
16:36:20 <dustymabe> #topic CIFS Mounts Failing On Next 38.20230322.1.0 due to SELinux Issue (keyutils)
16:36:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1447
16:36:46 <dustymabe> It's good to see fifofonix here (thank you for opening this issue)
16:36:55 <fifofonix> yw
16:37:46 <dustymabe> as I understand it this is a prolem that prevents CIFS filesystem shares access (blocked by selinux denials)
16:37:53 <dustymabe> s/prolem/problem
16:38:26 <bgilbert> but not all CIFS shares, right? https://github.com/coreos/fedora-coreos-tracker/issues/1447#issuecomment-1520500744
16:38:27 <fifofonix> yes, but i think it is actually restricted to mounting DFS-based shared
16:38:29 <dustymabe> fifofonix: do I understand correctly that it's anyone using CIFS shares on FCOS that would be affected, or does the configuration need to be specific?
16:38:42 <dustymabe> ahh, OK
16:38:43 <bgilbert> how common is the affected configuration likely to be?
16:38:53 * dustymabe is not a CIFS expert
16:39:54 <dustymabe> fifofonix: would you know the answer to ^^
16:40:01 <fifofonix> DFS is a popular MSFT thing that notionally provides HA mounts.
16:40:15 <dustymabe> HA == high availability ?
16:40:35 <bgilbert> does that require special server-side config, e.g. multiple backend servers?
16:40:44 <fifofonix> Yes. No expert here either. Concept is that there is a single name that will resolve to active backend.
16:41:03 <dustymabe> aside: /me wonders if we should add a samba test (I think Amazon has a service we could hook into for this)
16:41:26 <fifofonix> When we say CIFS we really mean SMB.
16:41:33 <dustymabe> fifofonix: correct, me too
16:41:40 <jlebon> dustymabe: +1
16:41:49 <fifofonix> I think CIFS predates SMB but you still notionally mount as CIFS.
16:42:09 <dustymabe> fifofonix: oh, I thought it was the other way around, but that was just me assuming things
16:42:24 <fifofonix> I think one interesting part is how widely is this used, and the second is what are available workarounds.
16:42:37 <bgilbert> so a) it's not just SMB but HA SMB b) there's a workaround (disable HA) c) there's currently no proper fix
16:42:56 <dustymabe> right. I think c) is the big one that's a concern for me
16:43:04 <bgilbert> (I think CIFS is the name for a particular point-in-time snapshot of the protocol that was documented)
16:43:12 <fifofonix> In my set up I am able to workaround by specifically addressing the primary share server (obviously no good in event of failover).
16:43:18 <dustymabe> if we decided to hold the promotion to stable it would be nice to know how long (i.e. not have it be unbounded)
16:44:03 <jmarrero> This is also the same issue right? https://discussion.fedoraproject.org/t/fc38-mount-cifs-mount-smb-mount-smb3-error-126/81271
16:44:21 * dustymabe clicky
16:44:30 <jlebon> bgilbert: for b), is that something the client can decide?
16:44:46 <fifofonix> That does look the same.
16:45:01 <jlebon> i.e. is the HA configured client side or server side?
16:45:23 <jmarrero> so we got more people hitting this
16:45:32 <dustymabe> so several people/groups are starting to hit this
16:45:33 <bgilbert> jlebon: that was my understanding from fifofonix's GH comment
16:45:37 <dustymabe> yeah
16:45:47 <fifofonix> this is broader than fcos too obvs
16:45:59 <bgilbert> fifofonix, could you confirm that the workaround is client-side?
16:46:17 <dustymabe> jmarrero: I'll try to link that discussion thread to the BZs where we are tracking this already
16:46:45 <dustymabe> bgilbert: +1 - that's good information to find out if fifofonix has it
16:46:59 <fifofonix> so, i can modify the mounts on all my hosts to point to a primary server rather than a DFS name and then move forward.
16:47:22 <dustymabe> so workarounds include:
16:47:34 <dustymabe> 1. some client configuration change in CIFs configuration
16:47:45 <dustymabe> 2. set SELinux to permissive
16:47:59 <dustymabe> 3. hold back updates on your server/fleet
16:48:06 <bgilbert> on balance, I'd be inclined not to delay the promotion to stable
16:48:10 <dustymabe> i guess 3 isn't a workaround per-se
16:48:13 <bgilbert> with or without a coreos-status post
16:48:18 <jlebon> 4. it should work to customize the policy too I think
16:48:26 <dustymabe> jlebon: i.e. server side?
16:48:31 <jlebon> dustymabe: client-side
16:48:32 <fifofonix> with all of the above it requires an alert to the community because action is required.
16:48:35 <dustymabe> oh no, you mean selinux policy
16:48:41 <jlebon> right, yeah
16:48:52 <dustymabe> could we ship a generic client policy update?
16:49:08 <jlebon> dustymabe: can you expand?
16:49:24 <dustymabe> i.e. bake a policy with exceptions into the image we ship
16:49:32 <dustymabe> or is it so specific
16:49:41 <dustymabe> i.e. to the mountpoints or something
16:49:58 <jlebon> dustymabe: ahhh gotcha. we could modify our policy, yes.
16:50:13 <jlebon> but that's circumventing evaluation by selinux folks
16:50:20 <dustymabe> of course, yes
16:50:27 <fifofonix> (i was trying to get a .cil file to apply to do this - but don't know how to gen one from the audit2allow output in BZ)
16:50:55 <dustymabe> jlebon: but it would be better than say if we were to decide to set selinux to permissive for all (which we wouldn't do)
16:51:15 <jlebon> fifofonix: yeah, i think it'd require building the module in a container maybe and then `semodule -i` it on the hosts
16:51:38 <jlebon> dustymabe: definitely
16:52:05 <dustymabe> that is something that would require some investigation as I don't think we've ever done that specifically before
16:52:32 <dustymabe> so 4. is "it should work to customize the policy too I think, client side"
16:52:49 <dustymabe> and 5. would be for us to ship a customized policy in our images
16:53:15 <dustymabe> ok at least we have the options listed out
16:53:51 <dustymabe> ideas on path forward? it is interesting to note that so far we've only got one report of this from our users (thanks fifofonix)
16:53:56 <jlebon> fifofonix: how (un)acceptable is option 1 in your opinion?
16:54:09 <dustymabe> only 4 people subscribed to https://github.com/coreos/fedora-coreos-tracker/issues/1447
16:54:26 <travier> another workaround would be to set the one problematic domain into permissive
16:54:35 <dustymabe> but maybe they are subscribed to the issue and just passively watching
16:54:42 <travier> not the entire node, just one domain
16:54:55 <bgilbert> dustymabe: 4 participating.  subscribed count isn't shown AFAIK
16:55:11 <dustymabe> bgilbert: ok yeah, there is a distinction there
16:55:26 <jlebon> travier: is that something you can easily at runtime without a policy update?
16:55:31 <jlebon> easily do*
16:55:40 <travier> yes, if we had semanage but we don't
16:55:45 <fifofonix> i think (i) is acceptable to me but i'm aware of the issue and its extend and have had time to warm to it.
16:55:47 <jlebon> :)
16:55:54 <travier> I'm looking into how that works
16:56:17 <dustymabe> travier: 1 was client configuration on the CIFS side, not selinux
16:56:26 <fifofonix> but, for someone not running next/test ... i think they need some notice so they hold updates / can test changes / etc.
16:56:42 <dustymabe> fifofonix: I think we'd need to send a coreos-status email - so hopefully they would see that
16:56:51 <dustymabe> and then evaluate their options
16:56:58 <travier> dustymabe: it's to replace your 2 option
16:56:59 <fifofonix> +1
16:57:18 <dustymabe> travier: ahh, so then you are talking about 4. I think
16:57:39 <travier> well, we don't have to do it. people impacted by it can do it
16:57:42 <travier> it's a workaround
16:57:45 <dustymabe> correct
16:58:01 <dustymabe> ok thoughts on coming to a conclusion/decision?
16:58:12 <bgilbert> #proposed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled.
16:58:12 <fifofonix> back to 1. when you say client CIFS config I'm interpreting as systemd mount config change on all vms.
16:58:36 <fifofonix> when is f38 stable?
16:58:39 <dustymabe> fifofonix: right. whatever steps you took to change the config client side to stop having the problem
16:58:40 <bgilbert> fifofonix: yeah
16:58:43 <jlebon> fifofonix: next week
16:58:52 <bgilbert> F38 is stable already, but the FCOS stable rebase is next week
16:58:57 <dustymabe> right, next week was the original scheduled date
16:59:20 <fifofonix> ok well deserves 'IMPORTANT' flagging.
16:59:24 <bgilbert> though if the SELinux folks get the info they need and produce a fix by then, we could possibly fast-track
16:59:35 <dustymabe> +1 to proposed
16:59:40 <jlebon> we'd want to send the email out this week
16:59:49 <jlebon> ack to proposed
16:59:50 <dustymabe> bgilbert: agree - if we get a fix soon then we could re-evaluate
16:59:53 <fifofonix> yes, as much notice as possible.
17:00:01 <jlebon> should we reflect that in the proposal?
17:00:03 <jmarrero> +1 to proposed
17:00:06 <bgilbert> fifofonix: are you able to help with a reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2182643 ?
17:00:06 <marmijo[m]> +1 to proposed
17:00:09 <travier> +1 to proposed
17:00:17 <ravanelli> +1
17:00:18 <jlebon> (the "might reevaludate if a fix becomes available" part)
17:00:29 <bgilbert> yeah, let me edit
17:00:31 <dustymabe> jlebon: SGTM
17:00:50 <fifofonix> bgilbert: the biggest component of a reproducer seems to be having a DFS share (i'm not aware that any work).
17:01:10 <bgilbert> fifofonix: yeah, ideally they could get a step-by-step for that
17:02:10 <bgilbert> #proposed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled.  If a fix becomes available, we'll evaluate it for fast-tracking into stable.
17:02:19 <dustymabe> +1
17:02:23 <bgilbert> fifofonix: I think you're probably in the best position to help them
17:02:26 <jlebon> aye to proposal
17:02:47 <dustymabe> we'll also need fifofonix help with example workaround steps (if he's willing)
17:02:48 <fifofonix> bgilbert: unfortunately, i don't know how to create one of those. happy to run more detailed logging on my infra.
17:02:51 <bgilbert> if that's actively being worked on, we may want to delay the coreos-status email a bit, but I opted not to put that in the proposed
17:03:10 <dustymabe> "actively being worked on" -> the bug?
17:03:14 <bgilbert> yeah, the bug
17:03:35 <dustymabe> I haven't seen any activity this week. I emailed a few people earlier today, but I think it was already late in the day for them
17:03:43 <bgilbert> fifofonix: hmm, okay.  I don't think anyone in this meeting knows how to set that up then
17:03:49 <dustymabe> since more and more people are seeing it in Fedora I think we might be able to get it prioritized, though
17:03:54 <bgilbert> fair
17:04:12 <bgilbert> #agreed We'll send a coreos-status email with a recommended workaround, and ship F38 to stable as scheduled.  If a fix becomes available, we'll evaluate it for fast-tracking into stable.
17:04:28 <dustymabe> should we target a day to try to send that email ^^
17:04:37 <bgilbert> not later than Friday I assume
17:04:43 <jlebon> bgilbert: +1
17:04:44 <dustymabe> i.e. maybe give it one more day to see if anything pops up in the BZ and then send it
17:04:49 <bgilbert> dustymabe: +1
17:05:00 <dustymabe> ok so try to send thursday, latest Friday
17:05:07 <fifofonix> not hopeful on BZ. has been super quiet.
17:05:13 <bgilbert> yeah
17:05:26 <bgilbert> volunteers to write the post?
17:05:40 <jlebon> Ondrej asked for a reproducer last week, but sounds like that's hard to do
17:06:02 <dustymabe> yeah, maybe we could find a hosted service that shows the problem (but that would be work itself)
17:06:10 <dustymabe> bgilbert: i can try if no one else wants to
17:06:20 <bgilbert> dustymabe: +1
17:06:27 <jlebon> dustymabe: i can help you too
17:06:41 <dustymabe> #action dustymabe to write coreos-status post about CIFS issue for f38 rebase heading to stable
17:07:18 <dustymabe> I'll lean on you fifofonix for some client side workaround steps if you don't mind
17:07:37 <dustymabe> #topic open floor
17:07:38 <fifofonix> dustymabe: will do my best.
17:07:47 <dustymabe> anyone with anything for open floor today?
17:08:02 <dustymabe> OR are there any more time sensitive topics that we should discuss?
17:09:34 <dustymabe> ok I'll close it out
17:09:37 <bgilbert> +1
17:09:40 <dustymabe> #endmeeting