15:59:41 #startmeeting RELENG (2019-04-25) 15:59:41 Meeting started Wed Apr 24 15:59:41 2019 UTC. 15:59:41 This meeting is logged and archived in a public location. 15:59:41 The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:41 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:59:41 The meeting name has been set to 'releng_(2019-04-25)' 15:59:41 #meetingname releng 15:59:41 The meeting name has been set to 'releng' 15:59:41 #chair nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu dustymabe ksinny jednorozec 15:59:41 Current chairs: dustymabe jednorozec ksinny masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll 15:59:41 #topic init process 15:59:53 morning 16:00:42 nirik: Morning, how goes? 16:01:05 * ksinny_ waves 16:01:17 the usual. ;) 16:01:31 My internet(on mobile network) is very slow, it may get disconnected 16:01:45 * mboddu is really sorry for ksinny_ 16:01:53 So, lets get to that ticket first 16:02:02 no problem at all :) 16:02:10 #topic #8294 coreos-pool and coreos-release koji tags request for FCOS 16:02:12 .hello2 16:02:13 bcotton: bcotton 'Ben Cotton' 16:02:17 #link https://pagure.io/releng/issue/8294 16:02:20 * mboddu waves at bcotton 16:02:26 morning bcotton. 16:02:38 How's the thought leader business? ;) 16:02:47 thriving ;-) 16:03:06 ksinny_: So, what exactly are these tags for? And whats the difference from coreos-continuous tag? 16:04:03 And do they need to follow a release, for ex, should the tags be inherited from any of the fedora base tags (fxx) or any other tags 16:04:32 mboddu: packages tagged to coreos-pool tag is from where FCOS will be built for developmenet and production refs 16:05:14 coreos-release tag will be used to tag builds which will make to FCOS releases(includes production refs) 16:05:14 so this is for packages that override fedora packages that are needed by coreos for prod? 16:05:31 This is to make sure that garbage policy are accordingly 16:05:32 or this is just marking the existing fedora packages ? 16:05:57 I think its just marking the existing fedora packages, iiuc 16:06:07 mboddu: Don't think they need to be inherited from any other tags 16:06:50 we will create a dist-repo with all packages tagged to coreos-pool and will specify specific nvrs while running composes 16:08:09 hum, so those likely could be large? if you tag in every fedora package you need? 16:08:10 mboddu: coreos-continuous tag is mainly for ttsing new features which is not yet released for packages like rpm-ostree, ignition 16:09:37 ksinny_: How many packages are we talking here? Just a ball park number? 16:10:03 nirik: yeah, this is mostly marking the existing packages. In some exception cases where some backport is needed then it may be the case to build a package directly to coreos-pool but for that I think it will first need to cerate repsective branch in dist-git 16:10:13 And also, how many composes per day? 16:11:29 nirik: yeah, it will be a large repo since we will be using a single tag for all pcakges we want to include in FCOS build 16:11:44 so, the fedora ga and fedora-updates tagged packages should not ever get gc'ed... I wonder if you could only tag in those packages that are not in those tags and use all the repos? or you need it to be seperate? 16:11:46 ksinny_: To build a package directly to coreos-pool you dont need a separate dist-git branch, you can use a build target, but if you need to apply your own patches and then build that package with the patch then you might need a separate dist-git branch 16:13:05 mboddu: package list are mostly what we have at https://github.com/coreos/fedora-coreos-config/blob/master/fedora-coreos-base.yaml inlcuding all dependencies 16:14:27 I guess it all sounds fine, but I am concerned with signing time... if we have to resign a pile of packages all the time it's going to take up a lot of robosign cycles... 16:14:37 nirik: It can be gc'ed as per the gc policy we will have (don't think we had discussion about it yet) 16:15:14 nirik: Yeah, that was my next question, ksinny_ : do they need to be signed with a different key? 16:16:19 * masta looks in 16:16:33 so, the fedora ga and fedora-updates tagged packages should not ever get gc'ed... I wonder if you could only tag in those packages that are not in those tags and use all the repos? or you need it to be seperate? 16:16:50 That means, we need to have a pending area to get them signed and robosign config changes and stuff 16:16:55 I think we can take this question on ticket to ge better answer 16:16:59 get* 16:17:14 yeah, lets add some questions and discuss more in ticket? 16:18:04 mboddu: We are fine with using Fedora key 16:18:42 I had a related question in ticket "can we have dist-repo with pakcages having multiple keys"? 16:18:43 #info There are more questions that needs to be answered and all the questions will be asked in the ticket 16:19:15 ksinny_: fedora keys are based on a release though 16:19:41 ksinny_: Yes, you can 16:20:28 huh, how does it do that? multiple copies? 16:21:03 mboddu: yeah, but we will have packages in coreos-pool from multiple Fedora releases (since we are handling next(F31) and stable(F30) 16:21:14 nirik: Not sure how its handled internally though, probably a question to koji folks 16:22:15 ksinny_: And how are you going to do major releases? 16:23:42 ^ reason being, if you are using builds from different releases, is it going to be continuous release or you cut major releases? 16:24:14 mboddu: Suppose Fedora major release hapened (eg. F30) , we will take those packages set and it will bake in into testing-devel and once it looks good we will do testing and then stable release 16:25:13 Release will keep happening every two week with updated content from major release +updates-testing content(after getting baked in for some time in testing and then to stable) 16:26:44 mboddu: https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.md is a good document to get some answers 16:26:48 Simply put, if FCOS 1 (is the release name if we call it) it is based on F30 content, no F31 content and FCOS 2 will be based on F31 content 16:26:57 Is my understanding right? 16:27:35 FCOS new release is going to have multiple streams which is a bit different than Atomic Host, so it can be confusing in beginning 16:28:14 mboddu: yes, FCOS stable stream release will have content only from F30 16:28:40 So, when F31 goes stable, then FCOS stable will have both F30 and F31 content? 16:28:52 Or just F31 content? 16:29:01 But we have a simltanious release with next stream(which tracks upcoming release i.e. F31). This is to let user see what is coming in next release 16:29:47 mboddu: when F31 goes stable, FCOS stable stream release will be F31 based. next stream release will be on F32 (once branching takes place) 16:30:06 Okay 16:30:55 This is a bit similar to how container linux does release. They have separate channels alpha, beta and stable 16:32:23 nirik: So, ^ if we keep using the fedora keys, then we have to change the configs everytime when we branch and a release goes stable 16:32:35 Which is not bad 16:32:41 considering the discusion, does the request in ticket looks fine? 16:32:46 or... perhaps we could version the coreos ones? 16:33:01 just always make them when we make fedora tags... 16:33:18 ie, coreos-release-30, coreos-release-31, coreos-pool-30, etc? 16:33:30 or does that not work because they might have a mix? 16:34:11 nirik: ^ yes, which is why I started the discussion with "follow a release" 16:36:08 Having coreos tags based on Fedora release may work but I might be missing something here, let's add this to the ticket 16:36:20 I'm sure we can figure something that works... just need to try and do soemthing so it aligns best for both of us 16:36:31 agree 16:36:42 nirik: +1 16:37:03 we could just also sign those with fedora keys, but I don't know if another key is a requirement/desired 16:38:32 nirik: As far as I am aware we don't need any other key 16:38:56 nirik: I guess we can make use of fedora release keys somehow 16:39:29 nirik: the great majority of packages we ship will be regular Fedora pkgs. wes should only have to spin our own for backports 16:39:40 so just those would have to be signed 16:40:10 cool and you could get those from fedora ga/updates? or would everything need to be in one repo? 16:41:11 we'd like to keep them separate if possible so that adding to the pool is a controlled operation 16:41:25 nirik: we need in a single repo sothat we can specify specific nvra during compose. If we use regular updates repo, latest may not be what we wnat 16:41:26 want* 16:41:30 pulling from the regular repos is an issue since they can change from under us 16:42:26 we'll be using lockfiles anyway, though separating them gives us more control/certainty 16:42:56 jlebon, ksinny_ : Okay, makes sense 16:42:56 hum, ok. but you don't care if they are signed by fedora keys or other key? 16:43:46 the backport ones? i think we'd rather they be signed by the fedora keys if possible :) 16:44:05 ok, if we can version the tags then we can know what fedora key to sign them with 16:44:32 For regular package, having them signed by Fedora key will be good. Does these packages needs to get resigned when we are just adding a new tag? 16:44:32 * dustymabe reads scrollback 16:44:54 i think we'd like to have the tags unversioned and control things via the lockfile 16:44:58 ksinny_: not if they are already signed, but if they aren't they would have to be 16:45:13 dustymabe: then what key do we sign things with? 16:45:44 nirik: could we just switch the default key at some cutover point? 16:45:53 alternatively we could have several tags 16:46:07 dustymabe: several tags? 16:46:22 i.e. fXX-coreos-pool that then aggregate into fedora-coreos-pool 16:46:31 using inheritances 16:46:31 also, it would be nice if we used inheritence here... since you mostly want the base fedora one? 16:47:32 does that make sense? 16:47:35 should I try to restate ? 16:48:15 could signing be done based on the buildroot that was used for the pkg? 16:48:44 then it just automatically switches over when we start backporting things from a newer release 16:48:47 nope, robosign just knows when something was tagged and what the tag was, so it looks at it's rules and does that 16:48:48 jlebon: AFAIK, nope, its based on the tags 16:49:32 nirik: so we could either manually update the rules when we cutover or we need multiple tags that then we aggregate into a generic tag 16:49:52 is that right? 16:50:02 hmm, ok. in that case using subtags + inheritance like dustymabe suggested sounds good. though the issue then is that we have to pester you folks each release to get a new tag created :) 16:50:17 each release probably isn't too bad 16:50:28 jlebon: well, we can just add it to the tags we make on branching 16:50:31 as long as it doesn't get more involved than that 16:50:33 dustymabe, jlebon : We can make them as part of branching 16:50:43 +1 nice 16:51:05 but can we also make these tags children of the updates tag? 16:51:13 nirik: how does inheritance work with dist-repo :? 16:51:27 do I get all the packages in my parent too? 16:51:28 dustymabe: it will follow it unless you pass --no-inherit 16:51:41 ahh cool 16:51:50 --noinherit rather 16:52:03 jlebon: i wonder if we actually want to do something "smart" here 16:52:13 oh, also, do you all need multilib? 16:52:40 i.e. if we inherit from updates then when we make a dist-repo will it make a repo with all the packages from all of fedora or can we limit it to just packages we care about ? 16:52:42 oh, no, nevermind 16:52:56 it would do all of them. 16:53:14 yeah we probably don't want to do that then - for efficiency sake 16:53:22 unless noinherit, then it would do only the ones in that tag 16:53:44 so you would have to add all the packages you care about specifically, and track them... 16:53:55 yeah we plan to have tooling to do that anyway 16:54:08 the tooling will actually look at what is in updates and then tag that into the tags we care about 16:54:29 so it won't use inheritance but it will still use what is in fedora proper 16:54:33 (for the subset of pkgs in updates that we actually care about) 16:54:36 jlebon: ksinny_ ^^ is that right ? 16:54:49 ok. 16:54:50 I guess fxx-coreos-pool and fxx-coreos-release tags inheriting from fxx and generating the dist-repos with the fxx key is what is needed? 16:55:03 dustymabe: yeah, that's my understanding 16:55:11 mboddu: well, they don't want that because that would get everything... all fedora packages 16:55:18 mboddu: let me try to take what you set and modify it slightly 16:55:25 we need: 16:55:41 fxx-coreos-pool - i.e. f30-coreos-pool f31-coreos-pool 16:55:45 nirik: Yes, but the plan is to use --noinherit 16:56:01 fedora-coreos-pool - inherits from fxx-coreos-pool 16:56:11 With inheritance, they dont need to add packages 16:56:14 fedora-coreos-release 16:56:28 mboddu: but they want to. ;) 16:56:30 fedora-coreos-release - we manually tag into fedora-coreos-release 16:56:40 Okay 16:57:13 do those three things I posted make sense? 16:57:22 dustymabe: can you add this to the ticket? I think we are all on the same page now, just good to spell it out there. 16:57:24 dustymabe: yeah, that makes sense to me. then dist-repo on coreos-pool with inheritance. 16:57:28 dustymabe: So, why fedora-coreos-pool? 16:57:50 to aggregate into a single repo 16:58:13 Why not just use fxx-coreos-pool? 16:58:16 mboddu: it's just a big ocean of packages we pull from - we have very fine granular control of what packages (NVR) actually get consumed on the build side 16:58:33 so, in the end we will have only two dist-repos (fedora-coreos-pool and fedora-coreos-release) or dist-repos for all sub-releases? 16:58:50 ksinny_: we don't even need a dist-repo for fedora-coreos-release IMHO 16:58:55 i think we only need a single dist-repo, on fedora-coreos-pool 16:59:01 with inheritance turned on 16:59:11 the only point of fedora-coreos-release IIRC is for extended garbage collection policy 16:59:16 dustymabe: I thought we will need it to replicate FCOS release builds 16:59:38 ksinny_: no I don't think so because those will still be in fedora-coreos-pool too 16:59:41 note: this will also require robosign and koji hub/gc changes. 16:59:47 yeah, right 17:00:17 Okay, one last question, from the ticket "coreos-pool and coreos-release koji tags created (include arches aarch64, ppc64le, x86_64)" 17:00:19 so we will have to change the dist-repo config when you want to change keys 17:00:25 Do you guys need a buildroot? 17:00:45 jlebon: ^^ do you know the answer to that ? 17:01:01 nirik: Seems so 17:01:03 likely not 17:01:15 unless we need to backport something that affects the buildroot for other backports themselves 17:01:46 i guess that *could* happen? though doesn't seem likely 17:01:57 we could punt it down the road until it does. ;) 17:01:58 jlebon: Then we will get to it when we have it 17:02:05 nirik: +1 17:02:19 +1 17:02:23 +1 17:02:23 should I update the ticket? 17:02:38 dustymabe: Yes please and I will create the tags later today 17:02:39 dustymabe: should be add s390x arch as well? 17:02:40 so yeah, it does mean when you want to move from f30 to f31 we will need to change the inheritence and dist-repo config to use the new key 17:02:49 ksinny_: No need or arches 17:02:54 but that shouldn't hopefully be too hard 17:02:56 of* 17:03:19 mboddu: let's focus on that statement for a sec 17:03:27 oops i meant niric 17:03:36 nirik | so yeah, it does mean when you want to move from f30 to f31 we will need to change the inheritence and dist-repo config to use the new key 17:03:46 nirik: ^^ can you elaborate ? 17:04:24 so say we have f30-coreos-pool -> fedora-coreos-pool 17:04:39 and a dist-repo script to make a dist-repo from fedora-coreos-pool. 17:04:40 When stable stream changes from f30 to f31, then fedora-coreos-pool inheritance should change to f31-coreos-pool and change the key for the dist-repo 17:05:02 ok i might be mistaken 17:05:08 ^ is what I think nirik meant 17:05:12 let me illustrate something and you all tell me if it's not possible 17:05:16 right, when you want f31-coreos-pool -> fedora-coreos-pool and the dist repo signed by f31 key, we have to change that 17:05:44 I was thinking we could have f30-coreos-pool and f31-coreos-pool both -> fedora-coreos-pool 17:05:47 it's just a koji command and a commit to the tag2repo config. 17:06:15 i.e. fedora-coreos-pool is just an aggregate 17:06:31 of both 17:06:43 I don't think thats possible. 17:06:51 a tag has a option for 'parent' 17:07:34 so perhaps the versioned ones are not of use then? 17:07:54 maybe - jlebon and ksinny_ were you thinking the same thing I was ? 17:08:07 dustymabe: yeah 17:08:20 * jlebon reads scrollback 17:08:33 right yeah 17:08:41 you can chain them... ie, f31-coreos-pool -> f30-coreos-pool -> coreos-pool 17:08:42 can't we inherit from multiple versioned tags? 17:08:59 nirik: you can chain but one will overwrite the other right ? 17:09:02 there might be some way I am not aware of. ;) 17:09:11 i.e. systemd in f30 would override systemd in f31 ? 17:09:14 yes, last one wins 17:09:25 ok so let's go back to the original problem 17:09:52 wanted: a tag that contains all the packages we would consume for FCOS (in various streams at the same time) 17:09:59 I can ask koji folks... I might just be unaware... 17:09:59 dustymabe: You can chain them but the latest of them winds 17:10:06 wins* 17:10:15 problem: if we build backports we want the backport signed with the tag for that release of fedora 17:10:29 so that problem led us to versioned tags 17:10:50 so.. we could do this: 17:11:04 we have versioned tags that are used to stage builds 17:11:16 how about have no inheritance then, and just tag them into fedora-coreos-pool manually? 17:11:19 and then we have a process that just tags them over into fedora-coreos-pool after they are signed 17:11:38 dustymabe: :) 17:11:45 robosign can do that sure. 17:11:51 oh nice 17:11:59 i suspect backports will be occasional, so it shouldn't be an issue for us 17:12:03 nirik: can you describe that process in just a little more detail? 17:12:04 and it's something we could automate 17:12:12 foo-1.0.fc30 just was tagged into f30-coreos-pool. Sign it, move it to fedora-coreos-pool 17:12:29 so robosign can do that automatically for us based on configuration? 17:12:35 tag2distrepo sees that and makes dist repo 17:12:49 yes. but it needs to know what key to sign with 17:12:55 oh, that's great! 17:13:09 but f30-coreos-pool likely gives the hint. :) 17:13:10 then it can use the originating tag name for that, right? 17:13:15 yeah I think that solves our exact problem 17:13:47 any gaps I'm not aware of ? 17:14:08 jlebon: Nope, it cant use tag names, its based on the config - https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/robosignatory/files/robosignatory.production.py#n116 17:14:39 dustymabe: One more question 17:14:40 mboddu: right gotcha, that should work fine for us too 17:14:44 we probibly need to see how dist-repo handles multiple keys 17:14:52 assuming this is a good option for us - can we discuss the naming of the versioned tags ? 17:15:00 I see that it can... but not sure how it works, never used it 17:15:27 dustymabe: you want to name them f30-dustymabe-rules? :) 17:15:32 So, when we change from f30 to f31, do we need to untag all the builds in fedora-coreos-pool? 17:15:46 Which were f30 based 17:16:08 nirik: so, foo-1.0.fc30 and foo-1.0.fc31 can co-exist in fedora-coreos-pool tag and fedora-coreos-pool dist-repo, correct? 17:16:09 nirik: I would suggest something like f30-coreos-builds-unsigned or something like that - not f30-coreos-pool 17:16:40 ksinny_: yes. I think so. 17:16:42 dustymabe: f30-coreos-pending? 17:16:48 mboddu: we'd prefer to have both foo-1.0.fc30 and foo-1.0.fc31 at the same time as ksinny_ mentioned 17:16:49 cool 17:16:54 yeah, pending is usually what we use or signing-pending 17:16:56 mboddu: f30-coreos-pending works for me 17:17:02 or f30-coreos-builds-signing-pending 17:17:40 Sorry, f30-coreos-signing-pending 17:17:49 mboddu: that works 17:18:02 mboddu: regarding 'mboddu | So, when we change from f30 to f31, do we need to untag all the builds in fedora-coreos-pool?' 17:18:07 i think we don't want to untag anything 17:18:19 could also be f30-coreos-backports since it'll only be used for backports 17:18:20 we want them all in there 17:18:26 dustymabe: Yeah, I got that 17:18:39 jlebon: but the tag gets removed after the rpm is signed 17:18:46 backports might be more descriptive 17:18:55 but yeah 17:19:10 cool deal 17:19:14 gotcha 17:19:17 any outstanding open items ? 17:19:21 dustymabe: Another question, from f30 to f31 move 17:19:41 dustymabe: Ahhh, nvm, I am good 17:19:51 so whats your process for building a backport? commit to dist-git, build against whatever, tag into fNN-coreos-signing-pending ? 17:20:01 :) - feel free to ask - this is the time where we need to work out kinks :) 17:20:09 nirik: yeah I think that's what we'll do 17:20:28 nirik: we will/should have branches within dist-git we can use 17:20:40 dustymabe: Nope, I am good, I might ask couple more questions when I am doing it (my brain kicks in while doing something) :D 17:20:51 and we should be able to target a release using those branches I think 17:20:56 something = work 17:21:03 nirik: we should be able to just do `fedpkg build --target=fNN-coreos-signing-pending`, right? 17:21:12 nirik: this is where we have been discussing that: https://github.com/coreos/fedora-coreos-tracker/issues/79 17:21:18 dustymabe: ok. sure, although note that we don't allow deleting branches there, so it would exist forever. 17:21:30 jlebon: Nope, thats why I asked about the buildroot 17:21:33 jlebon: yep 17:21:41 wait, no, mboddu is right 17:21:53 you need to build against the regular fedora target and tag it in 17:21:54 fNN-coreos-signing-pending is just a tag 17:21:56 gotcha, so we'll have to tag as a separate step them 17:22:01 nirik: right - it would exist forever 17:22:44 nirik aren't there rules somewhere about 'if this dist-git branch then use this build-target/tag' ? 17:22:55 jlebon: But creating a build target is not hard 17:23:00 dustymabe: Its part of fedpkg 17:23:18 mboddu: it'd be nice, since it'd mean we wouldn't pollute the regular tags 17:23:35 yeah we definitely don't want to piss off maintainers 17:23:37 re. fNN-coreos-backports, i think it'd be cool if those pkgs remained tagged, so we have an easy way to see what we've backported 17:23:53 can't robosignatory just add a tag instead of removing *and* adding? 17:24:19 nirik: mboddu could we get you to look through https://github.com/coreos/fedora-coreos-tracker/issues/79 to see if you have any sage advice for us ? 17:24:20 sorry, meant fNN-coreos-signing-pending 17:24:23 jlebon: "can't robosignatory just add a tag instead of removing *and* adding?" what do you mean? 17:24:24 maybe after the meeting 17:24:46 dustymabe: Sure 17:24:46 higher up, dustymabe said: 17:24:47 13:18:39 < dustymabe> jlebon: but the tag gets removed after the rpm is signed 17:25:03 i interpreted that to mean it gets untagged when it's moved to fedora-coreos-pool 17:25:18 jlebon: yes that is my interpretation as well 17:25:26 I don't know if it can do that... would have to look. 17:25:56 dustymabe: sure. Will look at some point... likely not now. ;) 17:25:57 cool, if so, then i think fNN-coreos-backports would be the best name. otherwise, then it's really just to hack around signing, so signing-pending is fine 17:26:00 jlebon: we could do that another way - we could add backport to the release of each rpm we backport 17:27:05 dustymabe: not sure i follow 17:27:09 i.e. foo-2.backport.fc29 17:27:19 just add it into the NVRA 17:27:56 should also make it clearer if people report bugs for the maintainers to identify it is an issue in a backport package 17:27:58 ahh gotcha. yeah, i think we need to do that regardless 17:28:11 in that case what you are suggesting isn't needed I don't think 17:28:30 note that if you do that as a dist tag you need to be carefull as many specs use that in conditionals. 17:28:33 anyway, didn't mean to distract from the important items :) 17:28:48 nirik: use what ? 17:28:54 so I think we have everything, mboddu: you good? 17:29:19 nirik: Yes 17:29:32 dustymabe: Are you going to comment on the ticket on what is needed? 17:29:33 dustymabe: dist = fc30, so you if you replace/override that with 'dist=backport.fc30' conditionals in specs could break. :) 17:29:45 but I guess you were talking about just manually adding it. 17:30:14 yeah, i don't think we would override the dist rpm macro 17:30:15 yeah I was talking about rather than updating the release from 1 to 2 we update it from 1 to 2.backport 17:30:30 but hadn't given it a lot of thought 17:30:35 mboddu: yes I'll update the ticket 17:30:53 dustymabe: Thanks 17:31:09 Okay, one ticket and 1:30 min 17:31:27 I should definitely close this meeting, we are way over time 17:31:40 the ticket was #8294 correct? 17:31:42 We can discuss it in #fedora-releng if anything more is needed 17:31:58 dustymabe: Yes, https://pagure.io/releng/issue/8294 17:32:05 Thanks everyone for joining 17:32:07 thanks all for all the help! 👍 17:32:13 mboddu++ 17:32:13 jlebon: Karma for mohanboddu changed to 16 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:32:14 thanka! 17:32:16 nirik++ 17:32:16 jlebon: Karma for kevin changed to 42 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:32:20 #endmeeting