16:30:06 #startmeeting fedora_coreos_meeting 16:30:06 Meeting started Wed Jan 26 16:30:06 2022 UTC. 16:30:06 This meeting is logged and archived in a public location. 16:30:06 The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:30:06 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:06 The meeting name has been set to 'fedora_coreos_meeting' 16:30:11 #topic roll call 16:30:16 .hello siosm 16:30:17 travier: siosm 'Timothée Ravier' 16:30:49 #chair siosm 16:30:49 Current chairs: siosm travier 16:31:01 #undo 16:31:01 Removing item from minutes: 16:31:07 #topic roll call 16:31:12 .hi 16:31:13 davdunc: davdunc 'David Duncan' 16:31:25 .hi 16:31:26 saqali: saqali 'Saqib Ali' 16:31:29 .hello2 16:31:30 .hi 16:31:30 jlebon: jlebon 'None' 16:31:31 #chair davdunc 16:31:31 Current chairs: davdunc siosm travier 16:31:33 lorbus: lorbus 'Christian Glombek' 16:31:35 #chair jlebon 16:31:35 Current chairs: davdunc jlebon siosm travier 16:31:36 .hi 16:31:37 dustymabe: dustymabe 'Dusty Mabe' 16:31:38 #chair lorbus 16:31:38 Current chairs: davdunc jlebon lorbus siosm travier 16:31:40 #chair dustymabe 16:31:40 Current chairs: davdunc dustymabe jlebon lorbus siosm travier 16:31:47 #chair saqali 16:31:47 Current chairs: davdunc dustymabe jlebon lorbus saqali siosm travier 16:33:09 .hello2 16:33:10 jaimelm: jaimelm 'Jaime Magiera' 16:33:39 .hi 16:33:40 bgilbert: bgilbert 'Benjamin Gilbert' 16:34:06 #chair jaimelm bgilbert 16:34:06 Current chairs: bgilbert davdunc dustymabe jaimelm jlebon lorbus saqali siosm travier 16:34:45 Alright, let's start 16:34:50 #topic Action items from last meeting 16:35:00 No action from last time as far as I can see 16:35:16 .hi 16:35:17 lucab: lucab 'Luca BRUNO' 16:35:20 Any topic we should discuss first? 16:35:23 #chair lucab 16:35:23 Current chairs: bgilbert davdunc dustymabe jaimelm jlebon lorbus lucab saqali siosm travier 16:35:31 .hi 16:35:31 aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist 16:36:09 .hello miabbott 16:36:10 miabbott: miabbott 'Micah Abbott' 16:36:12 .hi 16:36:13 aaradhak: aaradhak 'Aashish Radhakrishnan' 16:36:18 #chair miabbott aaradhak 16:36:18 Current chairs: aaradhak bgilbert davdunc dustymabe jaimelm jlebon lorbus lucab miabbott saqali siosm travier 16:36:24 #topic Support GRUB bootloader password 16:36:36 #link https://github.com/coreos/fedora-coreos-tracker/issues/134 16:36:54 i tagged this for the meeting to see if anyone wanted to pick it up 16:37:11 looks like miabbott is indicating some desire for it downstream 16:37:37 has been asking for it and our RHCOS PM is getting pressure to make it available 16:38:07 miabbott: thanks for the context 16:38:22 miabbott: said like that, we should tack a RH engineer to work on it :) 16:38:29 task* 16:38:41 .hi 16:38:43 ravanell_: Sorry, but user 'ravanell_' does not exist 16:38:57 That's mostly it for this topic. We can move on. Just wanted to raise awareness. 16:39:04 ok 16:39:10 travier: yeah, it may just come down to that 16:39:24 #topic Project Updates for Community Outreach and Testing for .ign/.bu Changes 16:39:31 #link https://github.com/coreos/fedora-coreos-tracker/issues/799 16:39:40 jaimelm: Do you have updates here? 16:40:11 #chair ravanell_ 16:40:11 Current chairs: aaradhak bgilbert davdunc dustymabe jaimelm jlebon lorbus lucab miabbott ravanell_ saqali siosm travier 16:40:41 Not no. Been out of the loop since before Christmas. 16:41:30 I don't think we've had feedback that the change was confusing. Was this also about editor support? 16:42:04 travier: I think so 16:42:19 yes 16:42:33 I generated notes but need to pr them 16:42:49 jaimelm: mind updating the ticket with the more focused scope? 16:43:04 yeah 16:43:31 Maybe we should focus on a few selected popular editors (vim, nano, emacs, vscode) and call it done? 16:44:01 I did those four a few others. I do think we should limit it and/or crowdsource the rest. 16:44:06 +1 16:44:44 great! 16:45:31 #topic During major version rebases, align FCOS testing stream with GA date 16:45:39 #link https://github.com/coreos/fedora-coreos-tracker/issues/1024 16:45:50 I think we are ready now to talk about this one 16:46:52 The general idea is to try to have a testing release with the Fedora new release content on GA date 16:47:25 correct 16:47:35 I don't think we've had that much trouble with the F35 rebase given the preparation work looking at the changes for 35 16:47:51 yes - and also we have a rawhide stream now 16:47:53 yeah, preparation is key 16:47:55 As long as that prep process continues, seems like a good plan' 16:48:38 Any objections? 16:48:48 of course this is our guidance/goal, if we don't meet it because we're shielding users from something that popped up, it's not a bad thing 16:49:05 beta should be pretty frozen even a week before so `next` is already a good representation of what GA will be. so in that sense, GA is about promoting `next` to `testing` 16:49:27 now, if there are exceptions and respins, we'll have to evaluate how those affect the plan 16:49:35 true, though sometimes there are some last minute additions 16:49:42 but always happen the week before 16:49:47 so we could respin `next` 16:50:00 to match what will be delivered in `testing` on that next tuesday 16:50:04 yeah. and in general, i think those should be low risk by design 16:50:39 one thing to keep in mind I guess, is that `next` won't always be a 1:1 comparison for `next` 16:50:44 sorry for `testing` 16:50:58 we may be experimenting in `next` and so might not be a perfect canary 16:51:03 but usually that is the case 16:51:13 +1 for this change - want to do a #proposed and we can vote? 16:51:16 indeed 16:52:09 #proposed For the Fedora 36 rebase, we will try to align the testing release including the new content with the F36 GA date. 16:52:18 ack 16:52:28 +1 16:52:37 +1 16:52:46 ack 16:53:41 +1 16:53:50 would say "aim to align" instead of "try to align" to be more optimistic, but cool as is :) 16:53:59 #agreed For the Fedora 36 rebase, we will aim to align the testing release including the new content with the F36 GA date. 16:54:13 #topic tracker: Fedora 36 changes considerations 16:54:20 #link https://github.com/coreos/fedora-coreos-tracker/issues/918 16:54:56 This is the tracker issue for Fedora 36 changes. Help looking at those to prepare for the rebase is appreciated! 16:55:24 Feel free to pick up a change and look at it to make sure it does not have negative impact on FCOS 16:56:07 yeah - I'll re run the script to update the list too. We'll try to discuss these during our video meeting next week! First meeting of the month! 16:56:38 Oh right. 16:56:57 some of those have been deferred from F35, so we are did the work for them (e.g. openssl3) 16:57:10 s/are/already/ 16:57:40 lucab_: indeed 16:58:10 I'll go through and mark it with a green check 16:58:18 if there is nothing I'll move to the last topic 16:58:21 that one does link to the previous open (now closed) issue 16:58:26 nothing else* 16:58:40 the "Remove nscd" does as well 16:58:45 travier: +1 16:59:08 #topic Consider proposing Fedora policy prohibiting scriptlets from writing to /var 16:59:11 #link https://github.com/coreos/fedora-coreos-tracker/issues/1067 16:59:44 Does anyone wants to introduce this one? 16:59:48 want* 17:00:04 I can take it 17:01:00 on multiple occasions, we've had breakage when new package versions update their scriptlets to write to /var 17:01:19 the most recent one was authselect, which wanted to perform a config migration and use /var to track the state of that 17:01:50 in rpm-ostree-land, that's a non-starter, because scriptlets are run server-side, where there's no node-specific /var 17:02:18 so we end up going to the package maintainer and asking them to please not do that. and often they're surprised, because Fedora packaging policy and other docs don't mention this constraint. 17:02:44 bgilbert: yes. It puts us in a tough position. 17:03:04 we have tools like the `rawhide` stream and baking the upcoming Fedora major in `next`, which helps us catch these things most of the time 17:03:26 though of course it can also happen with regular package updates in stable Fedora releases 17:03:47 Maybe creating a doc would help best ? 17:04:12 so the ticket proposal is: if we think that packages should generally not write to /var (with an exception for a particular subdir not further discussed here), should we advocate for changing Fedora packaging policy to reflect that? 17:04:38 i expect some pushback, but i think we should try this 17:04:41 Looking down the road, are there any reasons the Packaging Committee might not like the idea? Seems worth pondering. 17:04:57 that would a) provide better docs/notice to packagers, and b) allow automation in Fedora to start checking for this 17:05:05 I think it's helpful going into a proposal knowing how it might get shot down 17:05:31 i suspect a *lot* of packages would be affected, so it might have to be in phases 17:05:53 bgilbert: I agree I think we should try it. jaimelm we have to have strong justification to convince people 17:06:01 I'm skeptical that we could get a policy ban in, because of fundamental tools like 'alternatives' (or similar). I'd aim for a strong recommend. 17:06:10 That probably won't be a hard pull thing but more of a let's get to it objective 17:06:29 also it'd be good to have concrete example code for what packages should do instead 17:06:35 ^^ 17:06:47 maybe more than one 17:06:55 pick a couple use cases 17:07:03 +1 17:07:19 👍️ 17:07:21 most /var things should just use tmpfiles, which i think is already in the guidelines 17:07:40 the big one is actual node-specific logic like migrations, which packagers would have to relearn 17:07:44 to e.g. use a systemd service 17:08:21 * dustymabe also notes that containers don't have systemd typically, so that has to be considered 17:08:44 usually container don't have state in /var either 17:08:52 true 17:09:00 yeah, I guess containers typically don't `dnf update` 17:09:24 I don't see what could block that. This will improve the state of things for IoT, Silverblue/Kinoite as well 17:10:20 not hearing much oposition.. #proposed? 17:10:25 Something that would be great would be to have dnf progressively enforce that by running the scriptlets via systemd-run with RO mounts but that's a bigger change 17:10:26 yeah 17:10:32 dustymabe: I have seen quite a few containers update in build state 17:10:44 I'm not sure if it is unrelated or a pre-req, but we there are also packages shipping content in /var (e.g. vagrant) 17:10:52 s/we// 17:11:16 maybe first stage would be a CI run in src.fp.o and bodhi which warns if if detects /var writes 17:11:31 lucab_: but they aren't trying to touch it in scriptlets? 17:11:57 #proposed We will submit a Fedora packaging policy change to the FPC to strongly recommend that scriplet do not write content in /var 17:12:00 dustymabe: I didn't check, hopefully not 17:12:00 lucab_: those should be handled by rpm-ostree 17:12:24 but yeah still, ideally they use tmpfiles so we don't have to convert for them 17:12:34 I'd suggest s/to the FPC// s/submit/pursue/ 17:12:43 we can hash out process details as we go 17:12:47 travier: should we shoot for "deny" and settle for "strongly recommend" later? 17:12:56 ok, let's not drag that in then 17:13:00 I think the main value here is to get a sense of the goal 17:13:05 +1 17:14:08 #proposed We will pursue a Fedora packaging policy change to strongly recommend that scriplet do not write content in /var 17:14:34 we can add the two levels (recommend / deny) into the proposal 17:14:35 +1 17:14:37 +1 17:14:38 ack 17:14:39 ack 17:14:40 ack 17:14:41 +1 17:14:43 +1 17:14:50 #agreed We will pursue a Fedora packaging policy change to strongly recommend that scriplet do not write content in /var 17:14:55 if we have time maybe we should discuss lua too 17:15:35 that would be good but maybe we should make two separated discussion as this one will get much more pushback 17:15:43 oh yeah 17:15:50 it would be a different proposal 17:15:58 i mean "if we have time" in the meeting today 17:16:23 is the "agreed" an action item for bgilbert? or somebody else? 17:16:35 bgilbert I would assume 17:16:38 I'm not planning to work on it, no :-) 17:16:42 oh 17:16:43 ha 17:16:49 The Lua would probably end-up being a "if you use Lua for scriplets, warn rpm-ostree first!" 17:17:09 well, the idea of getting 1-3 examples prepped seems a good place to start. 17:17:15 can we distribute the love on that? 17:17:17 I can add it to my list of things I never get to 17:17:20 I can work on Change request and docs PR 17:17:29 travier: ❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️ 17:17:34 travier++ 17:17:42 travier++ 17:17:53 (this gets me badges :D) 17:18:02 dustymabe: haha. sadly, i have a list like that too 17:18:03 a badger! 17:18:27 let's hit lua real quick.. mind if I set the topic? 17:18:34 go ahead 17:18:43 #topic lua: either support it or change policy 17:18:45 (but let's keep some time for OD too) 17:18:54 basically we've been in the middle for a long time 17:19:09 we try to get people to change their scriptlets OR 17:19:21 we add an exception and hardcode an alternative in bash (that gets stale) 17:19:39 we either need to do the work to fully support it 17:19:50 OR pursue a policy to disallow it in Fedora 17:20:11 doing the work would require librpm providing an API for it 17:20:13 The middle ground approach just causes needless work for us periodicially 17:20:30 if we had that, we could add support for it in rpm-ostree 17:21:01 jlebon: is that something librpm is willing to provide (i.e. just needs someone to work on it) or are they opposed? 17:21:14 #link https://github.com/coreos/rpm-ostree/issues/749 17:21:19 was the change in rpm not enough to make it work in rpm-ostree? 17:21:20 more of the latter 17:21:30 so they are opposed 17:21:33 https://github.com/rpm-software-management/rpm/issues/1273 17:21:48 https://github.com/rpm-software-management/rpm/issues/1273#issuecomment-996519896 17:21:54 I don't think we will ever be able to get the "ban lua" change accepted. It's too useful for some packages in Fedora 17:22:23 https://github.com/rpm-software-management/rpm/pull/1867 ? 17:22:47 that's not the same thing 17:23:04 Oh, ok 17:23:05 lua scriptlets have context that they rely on 17:23:31 ok so the way I see it is: 17:23:40 1. to implement Lua we need something from librpm 17:23:51 2. librpm isn't willing to provide that 17:23:59 3. we must ask for a policy change to Fedora as a result 17:24:12 the policy change should stir the discussion and possibly librpm will come around 17:24:26 [x] doubt 17:24:26 or maybe an alternative solution will be proposed that is better 17:24:46 Usually the Lua script are there for state management or things that don't apply to rpm-ostree based systems 17:24:57 I see some discussion in the rpm-ostree ticket that there might be alternative (hackier) approaches? 17:25:14 like synthesizing a package with just the lua script and handing it off to rpm 17:25:50 So I think we ask for Lua scriptlet to only do things that can not be done via Bash or that are only for state management that do not apply to rpm-ostree. 17:26:11 we should* 17:26:19 travier: that still sounds like a policy change? 17:26:22 yes 17:26:42 bgilbert: not sure how foolproof that is though. we'd need to dig more into what exactly is the context they run in, and how it'd differ if it actually ran in a separate RPM 17:27:21 right 17:27:34 * dustymabe guesses we could fork librpm :( 17:27:57 anybody with strong opiniions about how we should proceed 17:28:01 that isn't the status quo 17:28:15 jlebon: silly though, could we detect the lua case in rpm-ostree and re-route those specific cases to a dedicated sandboxed transaction on its own? 17:28:51 sorry travier, I ate OD 17:29:23 lucab_: it's worth investigating 17:29:40 alright, let's quickly move to OD 17:29:48 #topic Open Floor 17:30:10 (Will end in 5 minutes) 17:30:37 jlebon: ah, I see Colin already proposed a similarly crazy idea https://github.com/coreos/rpm-ostree/issues/749#issuecomment-644751946 17:30:40 #info we're going to try to ship some ad-hoc releases on the `testing`/`next` stream with some updates for CVEs 17:31:03 #info - next week's community meeting will be a video meeting (first meeting of the month) 17:31:22 we'll discuss the f36 accepted changes 17:31:35 travier: regarding the podman v4 discussion, where did we land on that 17:31:45 do we need to try to re-engage with the podman team on that? 17:32:09 I answered in the thread here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/4JRMF5IWBYZIOWG2UB2YVGXBWAQ36NMZ/ 17:32:15 I have no answer so far 17:32:52 You might have to point them at the list - not sure how much they read fedora devel (I admit I fail a lot of times to read it) 17:33:15 I pointed internally where I asked the question initially 17:33:33 and on the initial issue in FCOS 17:34:03 so let's game this out a bit.. if they don't make any changes to their plans - what do we do? we already said we would ship it in `next` 17:34:17 https://github.com/coreos/fedora-coreos-tracker/issues/1070 17:34:47 Well, we kind of need packages to come from somewhere... 17:35:10 i mean they are going to provide a package to us with podman v4 17:35:27 but they didn't agree upfront to make it co-installable or anything 17:35:39 the idea was just they'd build it and we'd ship it in next IIUC 17:35:43 if we could figure out a path to helping them override the RPM within their constraints, that would give more options wrt the part they care about (podman-machine) 17:36:06 that was discussed as an option, but we didn't agree to that last week 17:36:19 https://github.com/coreos/fedora-coreos-tracker/issues/1070#issuecomment-1016700798 17:36:30 right, we said we'd pursue it, timeframe unspecified 17:36:37 We suggested a lot of things and we received minimal feedback 17:37:21 i think the feedback on the "override" approach is that it causes a long period of delay on startup in which the user has no indication of progress 17:37:35 I don't think they care about running in next per se, so if we're having second thoughts about that... 17:37:35 We agreed to ship v4 in next. Not to remove v3, etc. 17:37:38 dustymabe: right, that's the hurdle 17:38:05 travier: hmm, I feel like that was implied considering they can't both be installed at the same time 17:38:23 We agreed to make podman v4 in next, not to not do the work to ship both too 17:38:25 * 17:38:25 d'ho I'm so late ! 17:39:05 I would really like to get an answer to that question: nothing points out that we can't install both at the time 17:39:09 i think the delayed start concern could be addressed 17:39:11 and choose on first boot 17:39:19 jlebon: in what way? 17:39:20 speaking of v4, from the release notes it also contains data/state migration, so once in it is not safe to rollback 17:40:25 lucab_: yeah I think that is right 17:40:42 dustymabe: `rpm-ostree install --cache-only $URLS_TO_RPMS -a`. this short-circuits metadata fetching and applies it live 17:40:59 ok, to be clear when we did the vote last week, I don't think it was implied that they would need to do the work to make podman v3 and v4 co-installable 17:41:16 dustymabe: agreed 17:41:25 jlebon: it would be an override, not an install, correct? 17:41:34 dustymabe: ahh right, but same idea :) 17:41:37 No, but it did not imply that this would not be a good option 17:41:38 override supports cache-only too 17:42:02 it doesn't support -A, but it's equivalent to calling `rpm-ostree ex apply-live` afterwards 17:42:41 the heavyweight part is fetching metadata and libsolv processing it, which we'd skip 17:42:56 (We are well over) 17:43:01 AIUI, we can satisfy their immediate needs by giving them a one-off OS build so they can meet their schedule, followed by a fast override command 17:43:04 I guess we need to carry this into more discussions with ourselves and the podman team 17:43:18 and then can navigate the FCOS migration to v4 as we wish 17:43:49 if they don't even want to wait for rpm-ostree to create a deployment, they could even `rpm-ostree usroverlay && rpm -Uvh $RPM_URLS` and then in the background run `rpm-ostree override replace` 17:44:07 Note than any override option that we choose will have to be CI tested too 17:44:20 And won't work offline 17:44:57 or in a private network setup 17:45:17 hmm, how do they fetch the FCOS image? or is there a knob to provide your own local image? 17:45:20 which again points us at shipping both in the image 17:45:41 ...we could ship the RPM in the image :-( 17:46:10 https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html 17:46:23 i understand wanting to be conservative, but shipping just v4 seems like an OK option to me in `next` 17:46:30 i guess I've stated that in the ticket already 17:46:32 travier: ack, thanks 17:46:54 assuming it passes our CI 17:47:54 i don't think it's right to go back on what we decided last week. so short-term I think we should add it to next 17:48:01 end meeting, pick up discussion in a more focused gathering tomorrow? 17:48:03 the elephant in the room is that people will have some podman (v3 or v4) installed from brew, and if the daemon does not support both then some breakage is going happen by design 17:48:16 lucab_: +1 17:48:44 It's not going back from what we decided last week to ship both 17:48:51 back on* 17:49:04 Ending at :50 17:49:09 jlebon: I think it's okay to revisit now that we've had more time to understand the implications, as long as we give them _some_ solution meeting their needs 17:49:11 in a minute 17:49:12 travier: yes that is correct, but we'd have to do work to make shipping both possible 17:49:19 and we can't make them do that 17:49:25 travier: that requires more work from them than was implied when that decision took place 17:49:26 we also have a responsibility to our userbase 17:49:45 i don't disagree we should keep exploring options 17:49:55 I don't really understand how that could that much work to copy paste a spec file and tweak som filenames 17:50:20 #endmeeting