16:29:48 #startmeeting fedora_coreos_meeting 16:29:48 Meeting started Wed May 24 16:29:48 2023 UTC. 16:29:48 This meeting is logged and archived in a public location. 16:29:48 The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:29:48 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:48 The meeting name has been set to 'fedora_coreos_meeting' 16:29:54 #topic roll call 16:29:56 .hi 16:29:57 dustymabe: dustymabe 'Dusty Mabe' 16:30:34 .hi 16:30:35 jdoss: jdoss 'Joe Doss' 16:30:36 .hi 16:30:38 ravanelli: ravanelli 'Renata Ravanelli' 16:30:43 .hello siosm 16:30:44 travier: siosm 'Timothée Ravier' 16:31:01 .hi all 16:31:02 quentin9696[m]: Sorry, but user 'quentin9696 [m]' does not exist 16:31:22 On the agenda (but rather low priority), we have https://github.com/coreos/fedora-coreos-docs/pull/538 16:31:56 .hi 16:31:57 bgilbert: bgilbert 'Benjamin Gilbert' 16:32:30 .hello marmijo 16:32:31 marmijo[m]: marmijo 'Michael Armijo' 16:32:43 .hello2 16:32:44 jlebon: jlebon 'None' 16:32:49 .hello jasonbrooks 16:32:50 JasonBrooks[m]: jasonbrooks 'Jason Brooks' 16:32:55 Not to clutter up that issue with random stuff travier but I love the look of https://github.com/squidfunk/mkdocs-material for doc sites. 16:33:42 #chair jdoss ravanelli travier bgilbert marmijo[m] jlebon JasonBrooks[m] 16:33:42 Current chairs: JasonBrooks[m] bgilbert dustymabe jdoss jlebon marmijo[m] ravanelli travier 16:33:47 did I miss anyone? 16:33:56 #chair quentin9696[m] 16:33:56 Current chairs: JasonBrooks[m] bgilbert dustymabe jdoss jlebon marmijo[m] quentin9696[m] ravanelli travier 16:33:57 jdoss: looks really nice indeed 16:35:18 #topic Action items from last meeting 16:35:24 * jlebon to open issue for investigation dnf5 libdnf for rpm-ostree 16:35:31 * dustymabe to open a ticket to make sure RPMs we control have SPDX Licenses 16:35:35 #info there was no need for jlebon to open a ticket for rpm-ostree dnf5 investigation because one already existed at https://github.com/coreos/rpm-ostree/issues/2139 16:35:42 #info dustymabe opened a ticket to track us updating our RPMs to be SPDX compatible in https://github.com/coreos/fedora-coreos-tracker/issues/1497 16:35:57 #info there was already a libdnf tracker issue at https://github.com/coreos/rpm-ostree/issues/2139 16:36:32 for that SPDX issue I tagged in jmarrero and spresti[m] to see if they were interested in taking part of it. 16:37:25 #chair apiaseck 16:37:25 Current chairs: JasonBrooks[m] apiaseck bgilbert dustymabe jdoss jlebon marmijo[m] quentin9696[m] ravanelli travier 16:37:32 .hello c4rt0 16:37:33 apiaseck: c4rt0 'Adam Piasecki' 16:37:41 🎉 16:37:54 .hi 16:37:55 jmarrero: jmarrero 'Joseph Marrero' 16:37:59 #chair jmarrero 16:37:59 Current chairs: JasonBrooks[m] apiaseck bgilbert dustymabe jdoss jlebon jmarrero marmijo[m] quentin9696[m] ravanelli travier 16:38:13 dustymabe: I missed it that tag, I will take ostree and rpm-ostree. 16:38:20 jmarrero++ 16:38:41 * dustymabe moves on to meeting tickets soon (thanks travier for the reminder about the docs issue) 16:38:53 #topic Enable libostree automatic early pruning 16:38:58 #link https://github.com/coreos/fedora-coreos-tracker/issues/1495 16:39:09 * dustymabe hands the floor to jlebon 16:40:23 #chair fifofonix 16:40:23 Current chairs: JasonBrooks[m] apiaseck bgilbert dustymabe fifofonix jdoss jlebon jmarrero marmijo[m] quentin9696[m] ravanelli travier 16:40:57 so one issue we've been hitting is /boot being so small that it can break updates on some arches 16:41:22 we hit this most recently on aarch64 (and then we had to temporarily remove Qualcomm device tree blobs) 16:41:25 .hello mnguyen 16:41:26 mnguyen: mnguyen 'Michael Nguyen' 16:41:33 #chair mnguyen 16:41:33 Current chairs: JasonBrooks[m] apiaseck bgilbert dustymabe fifofonix jdoss jlebon jmarrero marmijo[m] mnguyen quentin9696[m] ravanelli travier 16:42:02 libostree now has support for essentially deleting the previous kernel and initrd first before copying in the new ones 16:42:12 it knows to do this only if needed 16:42:29 jlebon++ 16:43:18 this would've allowed upgrades on aarch64 to keep working without any intervention. it would also allow ppc64le to work with the current /boot size 16:43:39 the feature is gated by default since it's still new and touches upgrade code 16:43:51 so we'd have to decide whether we want to opt into it 16:44:10 and if so, on which streams & arches 16:44:17 to me it's not really a question of if, but when/how 16:44:32 ideally eventually we can have it turned on by default upstream 16:44:54 so FCOS is a great opportunity to exercise it 16:45:01 Given that we don't have other short term options, I'd vote to enable it as needed for selected arches for now. Once we have had more soak time, we can move to everything by default 16:46:11 is there any alternative? 16:46:34 yeah. for aarch64 maybe a short time in `next` first would be useful (though I hate to re-enable `next-devel` just for this) 16:46:44 for ppc64le, we could technically not stabilize it until we fix the boot issue 16:46:55 the boot space issue* 16:46:59 for ppc64le, since there was no prior ppc64le we could just go ahead and enable it everywhere 16:47:19 jlebon: right we could block ppc64le releasing on that, but we've waited a long time already :( 16:47:21 it's kinda awkward to rely on this right out of the gates 16:47:39 jlebon: I think I'd agree, but also don't think demand for ppc64le is high 16:48:08 one of my motivations for releasing ppc64le is so that we can stop manually updating our ppc64le build server (selfish reason, I know) 16:48:34 yeah, not strongly opposed 16:49:28 for aarch64, if we want to stop doing the qualcomm hack, we don't really have a choice if we want to keep supporting existing nodes 16:50:03 for aarch64 I think the only question in my mind is do we do it for only next for now, or go ahead straight to `testing` (in which case it hits stable in two weeks) 16:51:00 from the libostree side, how much bake time do you feel this needs? what sort of testing would increase confidence? 16:51:46 (though I guess technically we could enable it only for `testing` and not include the change in the `stable` promotions) 16:52:37 bgilbert: i'm not sure. i think if it's in `next` for at least e.g. 3 months and no one reports anything, i'd be satisfied 16:53:19 I'd decrease it to 6 weeks (that's 3 cycles) 16:53:38 (i'm talking about the upstream default here) 16:53:43 oh 16:54:20 IoW we only put the experimental change in `next` and then change the upstream default rather than promoting the config change 16:54:59 I was more asking how invasive the changes are, for the purpose of FCOS trusting them 16:55:26 * dustymabe notes the new extended upgrade tests should help us build confidence here 16:56:19 dustymabe: that's one path. but i'm comfortable having FCOS roll it out to stable before upstream flips it 16:56:30 jlebon: +1 16:56:43 on the upstream side, there's lots of projects using ostree that use it in many different ways from FCOS 16:57:18 how about for aarch64 we put it in next for say 3 cycles (6 weeks) then promote to testing 16:57:56 for ppc64le we just put it everywhere after some testing (or maybe we only release ppc64le on `testing`/`next` initially 16:58:41 that SGTM re. aarch64 16:59:08 well, there's the qualcomm change that needs to go with it 16:59:17 right 16:59:25 it's not too hard to pair those? 16:59:39 with our arch-includes 16:59:39 we could do one cycle with the feature, then undo the qualcomm change, and then bake it for 2 or 3 cycles 16:59:59 i would just add the qualcomm change back in at the same time 17:00:18 obviously we'd have to do some basic testing on this up front to make sure it works well enough in theory 17:00:50 i'm ok with that 17:00:57 cool. 17:01:08 should we re-visit ppc64le? 17:02:01 let's enable it once we have that? 17:02:03 we can publish it to next at the same time? 17:02:22 jlebon: only `next`? 17:02:29 travier +1 17:03:22 #proposed We'll enable this for aarch64 in next for 3 cycles (6 weeks) then promote to testing. We'll enable it for ppc64le right away for all streams. 17:03:24 dustymabe: basically have roll out ppc64le as we opt into it 17:03:50 (moving to a proposal as I don't see concerns raised) 17:03:51 it gets messy otherwise keeping track of the conditionals 17:03:58 jlebon: I think I'm OK with that, but not sure of the implications for say things like the website 17:04:24 travier: thanks for the proposal - I have one concern, which I'll bring up here in just a moment 17:05:10 dustymabe: good point. i'm not sure either. it'd be nice if it handled that scenario without breaking 17:05:38 yeah. maybe we can investigate or give ourselves the freedom to decide later 17:05:38 just seems odd to fast-track it to stable for one arch but not another 17:05:54 what are the website implications? 17:05:58 jlebon: yeah, but for this arch we have no existing users to consider 17:06:21 bgilbert: not sure if the website can handle an architecture not existing for a given release 17:06:25 dustymabe: yeah, it's more of a principle thing :) we're saying it's not ready and it's ready at the same time 17:06:46 dustymabe: a given stream, you mean? that isn't related to this issue, though. any new arch would have that problem 17:07:11 bgilbert: correct.. I think when we enabled s390x we just did it for all of them at the same time 17:07:17 and that was different website code 17:07:18 that seems suboptimal in any event 17:07:20 right 17:07:36 ok let me try to bring it back 17:07:37 jlebon: AFAICT this is an implementation detail? there's no inconsistent messaging if there's no messaging at all 17:08:29 dustymabe: what's the concern that you aluded to earlier? 17:08:39 so we've at least decided we can do this on aarch64 in `next`. enabling `next-devel` just for this is a little heavyweight.. since no one has complained about dropping the quallcommm stuff, should we just let our "6 week" testing cycle play out when we switch next to f39? 17:08:59 bgilbert: yeah, it's more about wanting to be consistent ourselves across arches 17:09:00 heavyweight how? 17:09:23 bgilbert: heavyweight in that the pipeline now runs for both testing-devel and next-devel for every new change 17:09:37 jlebon: we already have weird arch divergences, I don't see it as a big deal 17:09:44 right now next-devel is disabled because they are in lockstep 17:09:52 there's even precedent: the AWS serial console hack we used when launching aarch64 17:10:08 I think it's fine to wait for F39 17:10:14 dustymabe: the purpose of next is to test changes like this, though? 17:10:22 I was going to suggest that as well 17:10:25 bgilbert: it's not, it's just not ideal and there's no rush to get ppc64le out 17:10:28 bgilbert: indeed. I'm just considering the cost 17:10:56 including looking at flakes and what not 17:11:28 if next-devel is identical except for upgrades, we could ignore flakes except for upgrades 17:11:59 i'm also OK with enabling it now (especially if there is other experimentation we want to do) 17:12:13 i think we had talked about maybe enabling oomd or zram there recently 17:12:36 good either way. i kinda like the idea of doing it with the f39 cycle 17:12:37 what I'm getting at is that extra process has its own cost 17:13:17 so if it's easier to just initiate the process so we can move on, there's some merit to that 17:13:17 if we did that, we could just make it cross-arch 17:13:32 but I don't have a strong opinion 17:13:43 forget all the arch stuff and just turn it on in rawhide, then branched, and eventually next, etc... 17:14:39 #proposed We'll enable ostree autopruning for aarch64 in next for at least 3 cycles (6 weeks) then promote to testing. We'll enable it for ppc64le right away for the streams that we decide to release ppc64le for. 17:15:00 note I don't say when we'll enable it in #next :) - we can take that offline 17:15:16 +1 17:15:28 also punted on exactly when to release ppc64le on what streams 17:16:11 +1, sure 17:16:21 * travier notes that we're almost out of time 17:16:28 travier: indeed 17:16:39 votes then #agreed - then next topic 17:16:55 +1 17:17:03 +1 17:17:04 +1 17:17:06 sorry I should have tried to get to a resolution faster on this one 17:17:14 #agreed We'll enable ostree autopruning for aarch64 in next for at least 3 cycles (6 weeks) then promote to testing. We'll enable it for ppc64le right away for the streams that we decide to release ppc64le for. 17:17:23 #topic docs re-organize sections 17:17:26 #link https://github.com/coreos/fedora-coreos-docs/pull/538 17:17:45 travier, you have the floor 17:17:52 dustymabe: nice, you remembered 17:18:01 jlebon: well, I was reminded 17:18:10 at the beginning of the meeting 17:18:12 This idea for this PR is to reduce the number of nested sections that a user commonly has to open when they open the docs 17:18:16 ahh heh 17:18:41 This is inspired by a discussion that we had about RHCOS advanced isntallation docs being heavily nested 17:19:09 in this case, it's likely that most users will directly open the "System Configuration" section every time they open the docs 17:19:34 Thus I'm trying to find a better layout to organize things 17:19:39 thanks for bringing this up. I definitely like the idea to nest under sections that are broader categories 17:19:52 +1 17:20:00 I also think we should split some existing long pages, which is easier to do when we have more categories 17:20:17 (end of presentation) 17:20:24 I think what you have is a good start 17:20:32 I do appreciate the thoughts of making the docs more organized. I always thought the meat of the docs are crammed into one section. 17:20:38 Yeah, also in my opinion it adds to readability 17:20:47 it does kind of suck that System Configuration is kind of a catch all 17:20:52 https://docs.fedoraproject.org/en-US/fedora-coreos/storage/ > The storage page is very long for example and has "unrelated" content 17:20:55 we could maybe split the difference by calling the sections e.g. "Configuring Storage", "Configuring Networking", "Other Configuration" 17:21:01 and yeah, +1 to splitting the storage page 17:21:12 I was thinking a cookbook section of full examples would be helpful too. 17:21:19 "Configuring Storage", "Configuring Networking", "Other Configuration" > +1 17:21:26 jdoss: isn't that mostly what the docs already do? 17:21:50 Well, kinda like a full example of setting up RAID 1 with layering 17:21:52 bgilbert: maybe he's looking for something more comprehensive, kind of like the tutorials? 17:22:22 bgilbert: I like those section names 17:22:24 travier: yeah s/System Configuration/Other Configuration 17:22:42 and then if we ever find a "theme" in the future we can just make a new section 17:22:50 i.e. "User/Group Configuration" 17:23:05 jdoss: hmm. I guess I don't see as much value to combining things together into examples that probably don't match the exact needs of any user 17:23:11 dustymabe: +1 17:23:18 the kernel tuning, kernel args, and kdump pages could be under "Kernel Configuration" 17:23:27 ^^ was typing that up 17:23:40 jlebon: +1 17:23:50 jlebon: +1 17:23:50 bgilbert: I always find a full example help show the bigger picture of configuring common setups super helpful. 17:23:55 now that we have a proper search functionality even if we put things in a section that not everyone would be lead to it could still be OK 17:24:09 +1 17:24:20 should the configuration pages be nested under a single "Configuration" toplevel? 17:24:49 jlebon: maybe.. but maybe only if you could default to the nest being expanded? 17:25:06 otherwise it may become too "clicky" 17:25:16 "how many clicks does it take to get to the center!" 17:25:27 #badjoke - I'll see myself out 17:25:29 .chair dustymabe 17:25:29 dustymabe is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them. 17:25:43 I'd lean no. feels like the extra taxonomy doesn't really add clarity 17:25:58 I am here for the mabejokes 17:26:00 maybe right now. i might be more strongly if we add more toplevels like these 17:26:05 s/be/feel/ 17:26:12 bgilbert: the only thing I think it would help is then we don't have to add the Configuration word to every subsection 17:26:46 so now it's Configuration -> Networking -> Setting a Hostname 17:26:55 we're not constrained from restructuring again later 17:27:03 indeed (though not sure if the URLs change) 17:27:06 What's if we just don't add the "Configuring" word at all? 17:27:26 We're not changing the URL of the pages, just the place on the sidebar 17:27:27 maybe ^^ 17:27:31 travier: +1 17:27:32 "Networking" / "Storage" / "Other"? 17:27:42 "Other" is a bit odd 17:27:48 Other would definition still need Configuration 17:27:49 Misc? 17:27:53 definitely* 17:28:02 not sure why my fingers typed defnition 17:28:20 but "Other" makes sense if it's under "Configuration" :) 17:28:38 * jlebon stops pushing his nesting agenda 17:28:41 travier: +1 17:28:48 jlebon: 17:29:21 #proposed We like the nested topics idea from travier and have given him a lot of feedback on possible options. We trust he'll make informed decisions on the path forward but aren't giving strict guidance. 17:29:22 It YAMLs all the way down. 17:29:49 +1 17:29:53 +1 to sentiment, not wild about the wording for meeting minutes 17:29:54 I don't feel like everything needs to be decided here (hence the proposed) 17:30:01 bgilbert: cut the last sentence? 17:30:05 that's fair. +1 17:30:24 I think "Other" adds to confusion... +1 to everything else on the subject 17:30:31 dustymabe: sure, wfm 17:30:34 +1 17:30:45 ok I'll cut that out when moving to #agreed 17:31:14 +1 17:31:15 #agreed We like the nested topics idea from travier and have given him a lot of feedback on possible options. 17:31:28 any opposed to the edit - speak now or forever... 17:31:34 Thanks, we'll update the PR 17:31:38 I'll* 17:31:39 travier++ 17:31:41 dustymabe: Karma for siosm changed to 1 (for the release cycle f38): https://badges.fedoraproject.org/tags/cookie/any 17:31:53 #topic open floor 17:32:00 anyone with anything for open floor? 17:32:13 we have two FAS groups 17:32:14 I have something 17:32:17 lots of good discussion today from many people. Sorry the first topic went a bit long! 17:32:21 https://accounts.fedoraproject.org/group/coreos/ and https://accounts.fedoraproject.org/group/sysadmin-coreos/ 17:32:36 I gather the second one grants access to FCOS build infra? does anyone know what the first one does? 17:32:54 update the documentation to mention on the wireguard page there is an issue with Pre/Post actions on wireguard config with F38 17:33:05 I'll ask in a bigger group outside of Dusty's DMs. Does anyone have any good provider for testing FCOS on VMware? 17:33:29 jdoss: I use Workstation locally, but that doesn't really help with ESXi testing 17:33:40 quentin9696[m]: is there an open issue for the problem? 17:33:53 bgilbert: yes 17:34:24 https://github.com/coreos/fedora-coreos-tracker/issues/1487 17:34:29 re: FAS groups I don't think the first one does anything and can't remember what the second one does 17:35:20 bgilbert: workstation works on Linux right? 17:35:25 unfortunately the openshift-apps appowners isn't able to use a FAS group - it's just a static list of usernames in the playbook 17:35:26 since it's a known issue and don't seems to be solved quickly, maybe mention it to new users 17:35:39 quentin9696[m]: we don't normally mention open issues in the docs. is there a particular reason it'd be good to do so here? 17:35:51 jdoss: Hum, vaguely remember AWS as some VMware thingy: https://aws.amazon.com/fr/vmware/ 17:35:55 jdoss: yup 17:35:56 bgilbert: ^ 17:36:20 quentin9696[m]: we have other known issues. that's what the issue tracker is for :-) 17:36:32 quentin9696[m]: I'm OK with adding a note to the docs about the open issue. we typically don't, but I can understand how it would be frustrating to follow docs and have something not work 17:36:44 travier: yeah I signed up but no access on my AWS account as of today. 17:37:00 i do wish the issue had gained more traction in the BZ 17:37:06 oh there he is 17:37:10 dustymabe, bgilbert: sysadmin-coreos gives access to the batcave at least it looks like 17:37:16 I'm okayish with the docs note, but it adds maintenance burden and doesn't really relieve the need to search the issue tracker when encountering problems 17:37:23 jdoss: want to work with the selinux maintainers on the BZ for wireguard/selinux? 17:37:26 jdoss: we are starting to use VMC on AWS and working through permissioning to deploy via terraform provider 17:37:41 jlebon: haven't been there in a while. how are the snacks these days? 17:37:44 jlebon++ 17:37:56 if you follow the doc, it use Pre/Post actions so it will not works 17:38:11 while wg quick "works", it's not the best way to setup a wireguar connection on FCOS 17:38:13 dustymabe: I have limited cycles right now so I'm inclined to let them hopefully sort it out. 17:38:35 I am also terrible with SELinux policy 17:38:39 bgilbert: hehe 17:39:20 travier: what do you recommend? I use wgquick on my FCOS jump boxes and it works fine. 17:39:45 You can use NetworkManager native support 17:40:07 I'd love to see some examples if you have any to share. 17:40:38 quentin9696[m]: so I think the concern is over the maintenance burden or then remembering to go clean up the note when the issue is fixed.. would you want to open a PR to the docs and then followup to remove it when the issue is fixed? 17:40:47 https://github.com/coreos/fedora-coreos-docs/issues/286 17:40:57 dustymabe: +1 17:41:14 I've had some nftables issues with the WG interface not being online before the rules get applied and having WG start as part of network-manger's target would be nice. 17:41:24 dustymabe: will do that during the week, yes 17:41:35 cool 17:41:44 wow - this open floor is great! 17:41:52 quentin9696[m]: you could actually file a draft PR with the revert 17:41:59 for tracking 17:42:06 did everyone get what they need? maybe we can move ongoing discussions to #fedora-coreos ? 17:42:15 yeah, will follow up on the FAS groups in #fedora-coreos 17:42:16 dustymabe: it would be cool to do a nerd happyhour to geek out over using FCOS 17:42:38 bgilbert: good idea 17:42:40 an occasional social could be cool to have 17:43:04 sign me up 17:43:13 +1 17:43:17 +1 17:43:19 #info fifofonix and I are thinking of doing a "How do you Fedora CoreOS?" session at the F38 release party next week. 17:43:32 I'd anything to argue with bgilbert over video chat... I am all for it. 17:43:40 haha 17:43:41 lol 17:43:42 .fire jdoss 17:43:42 adamw fires jdoss 17:43:45 ! 17:43:52 arguing with me in IRC isn't good enough? :-P 17:43:57 thou shalt not argue with bgilbert 17:43:59 Not enough. I need more. 17:44:10 :) 17:44:14 :-) 17:44:20 thanks all for the discussion 17:44:24 #endmeeting