16:30:00 #startmeeting fedora_coreos_meeting 16:30:00 Meeting started Wed Jun 12 16:30:00 2019 UTC. 16:30:00 This meeting is logged and archived in a public location. 16:30:00 The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:00 The meeting name has been set to 'fedora_coreos_meeting' 16:30:05 #topic roll call 16:30:07 .hello2 16:30:10 bgilbert: bgilbert 'Benjamin Gilbert' 16:30:10 .hello2 16:30:13 slowrie: slowrie 'Stephen Lowrie' 16:30:21 .hello2 16:30:21 lorbus: lorbus 'Christian Glombek' 16:30:23 .hello2 16:30:24 miabbott: miabbott 'Micah Abbott' 16:30:27 .hello2 16:30:29 dustymabe: dustymabe 'Dusty Mabe' 16:30:41 .hello2 16:30:42 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:32:36 .hello lucab 16:32:36 kaeso[m]: lucab 'Luca Bruno' 16:33:59 .hello sinnykumari 16:34:00 ksinny: sinnykumari 'Sinny Kumari' 16:34:09 #chair slowrie lorbus miabbott dustymabe ajeddeloh kaeso[m] ksinny 16:34:09 Current chairs: ajeddeloh bgilbert dustymabe kaeso[m] ksinny lorbus miabbott slowrie 16:34:14 #topic Action items from last meeting 16:34:17 None! 16:34:35 #topic Stream metadata generator 16:34:40 #link https://github.com/coreos/fedora-coreos-tracker/issues/193 16:34:44 .hello2 16:34:45 jlebon: jlebon 'None' 16:34:52 ksinny: go ahead 16:34:55 #chair jlebon 16:34:55 Current chairs: ajeddeloh bgilbert dustymabe jlebon kaeso[m] ksinny lorbus miabbott slowrie 16:35:01 ok 16:35:31 some context on stream metadata generator 16:35:57 stream metadata generator is a tool which gives stream metadata by taking input release index, release metadata and override file. We will get release index and release metadata from fcos artifacts location for corresponding streams. 16:36:19 We need to decide upon few things 16:36:30 1. where should we keep override file? Should override file for all the streams be at same place or keep in separate branch? 16:37:10 I am thinking maybe we can make a separate project under coreos/ in github and keep it there? 16:37:32 something like stream-oveeride 16:37:41 err, stream-override 16:38:15 jlebon and I discussed this at one point, and I wish I remember what we said 16:38:18 ideally we'd namespace it - coreos/fedora-coreos-stream-override 16:38:25 .hello mnguyen 16:38:27 mnguyen_: mnguyen 'Michael Nguyen' 16:38:31 #chair mnguyen_ 16:38:31 Current chairs: ajeddeloh bgilbert dustymabe jlebon kaeso[m] ksinny lorbus miabbott mnguyen_ slowrie 16:38:42 wfm, I am bad at naming things :) 16:38:58 there's the override file as input, and there's the stream metadata as output. both should have history. 16:39:06 or `fedora-coreos-stream`, and we also use it to keep history? 16:39:07 in principle we could commit both to the same branch 16:39:13 bgilbert: :) 16:39:51 or, like, master branch with all the override files, and then generated branches with the output stream metadata 16:39:52 I am in favor of having both stream and oveeride at same place 16:40:12 ksinny, how are you envisioning that the tool would be run? manually invoked job in CI? 16:40:24 i.e., does the tool itself need to know about Git? 16:40:36 or can a CI job handle the commit & push 16:41:00 bgilbert: I am writing as a separate tool in Go. We can run tie up later in Jenkins or any other way 16:41:27 at least for lockfiles, i was thinking of a separate CI job as well to handle the commit & push 16:41:42 could maybe share some code there 16:42:24 AIUI nothing programmatic should be reading stream metadata from the Git repo 16:42:45 but rather from the uploaded copy 16:43:01 and similarly, nothing should be reading the override file except for the builder, which will look where it's pointed 16:43:17 so I think we can do whatever's convenient, without much risk? 16:43:25 bgilbert: I think we should run it when release.json or releaseindex.json or override file gets updated 16:44:33 ksinny: the point at which an FCOS release has occurred is exactly the moment where stream metadata is updated 16:44:38 so I'd rather keep it manual at first 16:44:56 can always automate 16:45:10 bgilbert: sure, we can keep it manual in the begining 16:45:12 "manual" as in manually invoked 16:45:13 +1 16:45:26 (as in, probably still running in CI) 16:45:48 later on, by looking at timestamp change in json files should do that, isn't it? 16:45:53 yeah, in CI 16:46:11 yeah, could do 16:47:19 that's all from me, I had one more question which I forgot. Might ask in Open Floor if I remember 16:47:40 anything to add on? 16:47:47 jlebon: you haven't stated an opinion about branch structure 16:47:49 any thoughts? 16:48:24 bgilbert: master for overrides and others for historical sound good to me to start 16:48:25 i think i'd prefer to have a flat structure (single branch) 16:48:26 (I'd probably bikeshed the repo name to `fedora-coreos-streams`) 16:49:05 dustymabe: then there wouldn't be atomic updates, no? 16:49:25 dustymabe: there'd be commit pairs: one manual one to update overrides file, then an auto one with the generated consequences 16:49:35 with a push in the middle 16:50:14 works technically, but feels a bit odd 16:50:22 yeah, it'd make git history noisier 16:50:41 bgilbert: sounds like something we could massage - i.e. open PR against repo. bot detects PR and updates it with processed output, then merges 16:50:47 hmm, yeah 16:50:55 to make things more obvious, we could rename `master` to `overrides` 16:50:59 separate branch for stream history and soverride sounds cleaner 16:51:04 and have no master branch at all 16:51:18 well, thinking about dustymabe's point 16:51:31 we need to be able to test override changes outside of prod 16:51:43 possibly that means running the tool locally 16:51:49 i honestly don't know which one is better, jlebon what I was thinking about was the structure of say https://pagure.io/fedora-comps (single branch) vs something like fedora-kickstarts (f29, f30, master) 16:51:57 but it sure would be nice if we could PR a change and have CI show us the result before merging 16:53:35 /shrug but yeah, ultimately I don't have a strong opinion 16:54:12 and the stream output from that bot would also be what we'd upload afterwards? 16:54:50 this essentially changes the definition of the repo from "historical" to "canonical" 16:54:55 bleh 16:55:05 jlebon: yeah, should go to the fcos artifacts directory 16:55:23 (which isn't a bad thing, just calling it out) 16:55:34 ksinny: his point is that the auto-updated PR _defines_ the new stream metadata 16:55:41 rather than having the history just _record_ what was updated 16:55:55 (as a side effect) 16:56:15 yeah, good point. proposal retracted. 16:56:18 i like the idea i think 16:56:22 heh 16:56:25 ot it 16:56:39 got* 16:56:44 jlebon: that implies that auto-generated updates would also go through PR though 16:56:53 i.e. for new releases, the bot would PR and e.g. we'd manually merge 16:57:31 which also seems fine, but is a different workflow 16:57:40 the generator is then not responsible for stream metadata upload 16:57:45 but that happens as a consequence of a PR merge 16:58:38 we should probably move on soon, but, any preferences between those models? 16:59:04 could it be PR-based for the non-mechanical ones, and direct pushes for mechanical? 16:59:14 yes, but seems confusing? 16:59:17 anyway, yeah we can discuss upstream 16:59:20 upload happens at different times 16:59:46 ksinny: want to take an action to summarize in ticket? 16:59:59 bgilbert: ack 17:00:13 I mean yes 17:00:23 #action ksinny to summarize discussion in ticket 17:00:27 and we can discuss further there 17:00:43 +1 17:00:44 +1 17:00:56 #topic Release notes 17:00:58 #link https://github.com/coreos/fedora-coreos-tracker/issues/194 17:01:10 this is something we haven't really discussed at all 17:01:23 presumably we will want release notes 17:01:45 in CL, we never really found a way to do that properly 17:01:52 the issue is that there are multiple versions in flight, and then they promote 17:02:18 so if you get a new stable release, you now care what changes happened in the previous testing cycle 17:02:38 but (if you're not running testing) you don't care about the current testing cycle 17:03:14 any opinions on how this might work? how did release notes work for AH? 17:04:15 (also, unlike CL, we're inheriting a lot of changes. there's a lot of content hidden inside a single line, "- Rebase onto Fedora 31" 17:04:16 ) 17:04:50 jlebon, dustymabe, ksinny? 17:05:16 Given that we don't share the same branching structure as CL, I wonder if release notes based on each branch would work. 17:05:20 In AH we have Two Week automated sent my releng to intended MLs. Also, we send a follow-up email with more details like AMI ID and important changes 17:05:49 Like with CL you constantly hop branches, which is why our notes are basically unsuable, but that 17:05:53 We don't store them anywhere other than sending it to MLs 17:05:53 s no longer the case 17:06:01 bgilbert: for FAH we only send release notes for the stable releases 17:06:11 It might be nice to build something where we can essentially show the changes that have happened to the build as it's progressed to it's current point (using CL version terms essentially I want to see 1845.7's release notes and it shows me all notes from 1845.0 to the current point) 17:06:11 ajeddeloh: isn't it? 17:06:15 and the release notes are mostly automated information 17:06:31 if something significant changes we'll add a paragraph about the change 17:06:31 hmm, have a FAH example handy? 17:06:47 bgilbert: it's not pretty, but it's something :) 17:06:49 * dustymabe finds link 17:06:58 #link https://coreos.com/releases/ 17:06:59 on the CL side 17:07:09 bgilbert: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2019-June/msg00000.html 17:07:31 and follow-up email https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2019-June/msg00002.html 17:07:32 ksinny: right 17:07:39 these are from latest release 17:07:43 does Fedora have a "web changelog" system for RPMs? 17:07:56 we send two emails - planned to send them as one but we never got there 17:08:11 kaeso[m]: see https://github.com/coreos/fedora-coreos-tracker/issues/194#issuecomment-501368280 17:08:14 kaeso[m]: i.e. given input set of rpms and output set show me the changelog ? 17:08:20 okay, so always just a package version delta? 17:08:24 there's never any "fixed xyz"? 17:08:26 bgilbert: not always 17:08:27 one sec 17:08:48 bgilbert: unless I'm missing something we're merging changes into branches, not creating new ones constantly, right? 17:09:21 jlebon: cool, like that, yes 17:09:21 ajeddeloh: in Git, yes 17:09:34 ajeddeloh: but that doesn't really affect the release note UX 17:09:49 kaeso[m]: here is what `rpm-ostree db diff --changelogs` gives me right now on my host: https://paste.fedoraproject.org/paste/07cZ6jArdJWHVLS7FQ0l9A 17:09:50 ajeddeloh: you still have to care about the previous testing release 17:10:01 dustymabe: I was thinking of permalinks for changelog for invidual packages, between "old" and "new" versions 17:10:20 jlebon: nice 17:10:21 #link https://lists.fedoraproject.org/archives/list/rel-eng@lists.fedoraproject.org/message/5WBUM4XTZA7FZQZ3SVDH5GMWRH6472H2/ 17:10:35 ^^ example where I added notes about security update 17:10:56 jlebon: that's really nice 17:11:38 CL's release notes had a mix of things 17:11:45 bgilbert: ah, I was just thinking that within a branch we can look at commits and get a set of changes that is actually what the user expects, independent of how we present that 17:11:47 we used to make more effort to try to summarize relevant changes (i.e. podman version xyz with new features abc) 17:11:59 ajeddeloh: hmm, maybe 17:12:05 my initial question was for something URL-linkable, but a static "db diff" that we publish on each release would work too 17:12:12 but since we're not focusing on AH any longer we've stopped that mostly 17:12:15 in CL: there are package updates (for major packages; we hide minor ones) where we just link to the upstream changelogs 17:12:37 dustymabe: right 17:12:40 if there was a change in the Gentoo packaging but not a new upstream release, that'd generally get lost 17:12:55 and then there are high-level issues of which we're directly aware: CVEs and bug fixes 17:13:17 so that's already sort of a broken system: issues are anywhere from surfaced to completely buried depending on where/how they were fixed 17:13:43 (e.g. if we fixed a problem in Ignition upstream it was buried in the Ignition update) 17:14:27 for FCOS the db diff stuff seems nice for package changes, better than CL 17:14:32 from CL release notes, I like the quick highlighting of very few selected components (kernel, systemd, ignition) 17:14:35 i think it'd be really cool to make it mostly automatically generated, with a tiny header "intro" section for specific things we want to manually call out, e.g. "This release fixes zombie-melted-down-sheep-doom." 17:14:40 but if we fix cosa or fcos-config... 17:15:10 jlebon: +1 17:15:21 jlebon: +1 17:15:24 kaeso[m]: I agree 17:15:35 can we automatically pull out all CVEs? 17:15:40 do we need to regex the rpm changelogs? 17:15:45 if it's attached to the errata, yes 17:16:07 it shows up as a structured entry in the updateinfo xml 17:16:11 nice 17:16:31 well, the errata does, for CVE ids... we do specifically grep for those right now :| 17:16:47 kaeso[m]: I like that too 17:17:18 kaeso[m]: do you think it'd be sufficient for the automated generation to surface particular packages? 17:17:56 kaeso[m]: i.e., without us manually writing notes for those pkgs 17:18:17 I would guess so 17:18:55 okay, so in summary, automatically generated CVE list, automatically generated pkg diff, section for manually listing fixes as appropriate 17:19:28 yes. My typical usage is "quick check that table when somebody wants to use feature $foo which is only in latest version of $bar" 17:20:06 for a select few pkgs, wonder if we could automate linking to upstream release notes 17:20:16 for the github ones it shouldn't be too hard 17:20:27 for the kernel, not sure 17:20:28 jlebon: +1 17:20:39 I do think we should have a historical archive of relnotes 17:20:43 is it important to mail them to a list? 17:21:05 want to keep the noise level down 17:21:33 we could have a separate list for releases 17:21:59 Having something like the coreos website where you can select specific versions would be nice, if we want to also mail them to a list that's fine by me 17:22:34 I was thinking RSS/Atom but those are rumored to be deceased 17:22:52 ~~tooling assumptions~~ 17:22:58 we could tweet :-) 17:23:56 as long as we have something that can be permalinked (ML archives, fixed URL, etc) I'm fine 17:24:05 +1 17:24:20 #action bgilbert to summarize discussion in ticket 17:24:26 anything else on this topic? 17:24:43 yeah no strong opinion. whatever we decide on though, we shouldn't require an account at $WEBSITE to get notifications 17:24:48 +1 17:25:27 #topic Open Floor 17:26:20 #info streams and stream tooling progress: https://github.com/coreos/fedora-coreos-config/issues/100 17:26:43 dustymabe: heh thanks, was about to bring that up :) 17:27:03 would appreciate input on that ticket, esp. if you think there's a cleaner way forward 17:27:11 #info signing project proposal was created for fedora infra: https://github.com/coreos/fedora-coreos-tracker/issues/187#issuecomment-499999387 17:27:25 ^^ looks like we have to wait on them to come back on that 17:27:30 dustymabe++ 17:27:41 i've been told they are a bit overloaded with work right now 17:28:01 #info updates(.stg)coreos.fedoraproject.org progress https://pagure.io/fedora-infrastructure/issue/7825#comment-574361 17:29:04 okay, closing out in 30s 17:29:34 #endmeeting