16:30:52 #startmeeting fedora_coreos_meeting 16:30:52 Meeting started Wed Jun 3 16:30:52 2020 UTC. 16:30:52 This meeting is logged and archived in a public location. 16:30:52 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:52 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:52 The meeting name has been set to 'fedora_coreos_meeting' 16:30:56 #topic roll call 16:30:57 o/ 16:30:58 .hello2 16:30:59 jlebon: jlebon 'None' 16:31:01 .hello2 16:31:02 cyberpear: cyberpear 'James Cassell' 16:31:06 .hello sohank2602 16:31:07 skunkerk: sohank2602 'Sohan Kunkerkar' 16:31:12 .hello2 16:31:12 .hello2 16:31:15 kparal: kparal 'Kamil Páral' 16:31:17 .hello2 16:31:17 .hello2 16:31:18 coremodule: coremodule 'Geoffrey Marr' 16:31:21 davdunc: davdunc 'David Duncan' 16:31:25 slowrie: slowrie 'Stephen Lowrie' 16:31:38 .hello2 16:31:38 bgilbert: bgilbert 'Benjamin Gilbert' 16:31:39 well hello all you beautiful faces out there 16:31:40 .hello sumantro 16:31:41 sumantro: sumantro 'Sumantro Mukherjee' 16:31:42 .hello2 16:31:44 dustymabe: dustymabe 'Dusty Mabe' 16:31:46 .hello2 16:31:47 .hello2 16:31:48 .hello2 16:31:50 darkmuggle: darkmuggle 'None' 16:31:51 .hello2 sumantrom 16:31:52 lucab: lucab 'Luca Bruno' 16:31:55 bcotton: bcotton 'Ben Cotton' 16:31:58 this might be a record 16:31:58 sumantro: sumantro 'Sumantro Mukherjee' 16:32:20 (kparal coremodule and sumantro are here to discuss the upcoming test day) 16:32:30 #chair cyberpear jlebon skunkerk kparal coremodule davdunc slowrie bgilbert sumantro darkmuggle lucab bcotton 16:32:30 Current chairs: bcotton bgilbert coremodule cyberpear darkmuggle davdunc dustymabe jlebon kparal lucab skunkerk slowrie sumantro 16:32:46 * dustymabe smiling 16:33:11 i'll give a few more moments for people to join 16:33:50 #topic Action items from last meeting 16:34:06 I don't see any actions from laste meeting: https://meetbot-raw.fedoraproject.org/teams/fedora_coreos_meeting/fedora_coreos_meeting.2020-05-27-16.30.txt 16:34:07 .hello2 16:34:08 jdoss: jdoss 'Joe Doss' 16:34:11 #chair jdoss 16:34:11 Current chairs: bcotton bgilbert coremodule cyberpear darkmuggle davdunc dustymabe jdoss jlebon kparal lucab skunkerk slowrie sumantro 16:34:24 thanks lucab for running the meeting last week 16:34:45 #topic meeting agenda 16:35:05 it looks like we have some fedora qa representatives here today to discuss the pending test day - so we'll definitely add that to the agenda 16:35:11 we also have the meeting tickets 16:35:31 #link https://github.com/coreos/fedora-coreos-tracker/labels/meeting 16:36:15 any other topics we'd like to add to the agenda ? 16:36:35 big topics preferrably - other things can wait for open floor 16:37:20 rojig 16:37:43 * cyberpear can wait Open Floor 16:37:47 cyberpear: +1 16:37:59 #topic Fedora Test day for our `next` stream (Fedora 32) 16:38:04 #link https://github.com/coreos/fedora-coreos-tracker/issues/491 16:38:17 #link https://fedoraproject.org/wiki/Test_Day:Fedora_32_CoreOS_2020-06-08 16:38:18 #info we have just moved our `testing` stream over to Fedora 32 16:38:28 #link http://testdays.fedorainfracloud.org/events/84 16:38:54 kparal: thanks! I see you have an open question for some of us in https://github.com/coreos/fedora-coreos-tracker/issues/491#issuecomment-638025251 16:39:15 I have not had a chance to look at the test cases yet 16:39:33 ok, has anyone else looked at them yet? 16:39:36 but after these releases go out (should be done soon after this meeting) I'll definitely look at them 16:39:59 please do review the test cases and don't be afraid to edit them on the spot 16:40:05 those are your test cases after all :) 16:40:09 kparal: i.e. edit directly in the wiki? 16:40:12 sure! 16:40:29 ok cool - anyone else want to take an action to review the test cases ? 16:40:35 dustymabe: can you verify whether you see an Edit button on those test case pages? 16:40:46 e.g. https://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_CoreOS_virtual_install 16:40:52 you need to be logged in 16:41:08 i'll take a look too 16:41:11 I guess it might require CLA+1 group, which I assume most of you should have 16:41:33 kparal: I see an edit button! 16:41:36 great 16:41:46 I see an edit button 16:42:10 also, please don't be afraid to create new test cases if you want to cover more use cases. We tried to prepare some basics, but it's really up to you what exactly you want to see covered 16:42:30 according to the audience, where you want to get UX feedback from users, etc 16:42:36 kparal: we did just move our `testing` release over to f32, but testing against `next` should work too (same content for now) 16:42:46 so we can just leave the test cases referring to `next` and it should be fine 16:42:52 ok, sounds good 16:43:10 Do Test Days usually look at the branched release, or current release? 16:43:15 though if there are any upgrade tests we may want to start from testing.. i'll try to keep that in mind when looking at the tests 16:43:32 cyberpear: usually test days are for the upcoming major release 16:43:37 but we're a bit behind here 16:43:45 the test day is on Jun 8th, so we have time to finalize the test cases until then. I'd like to move the testcases to permanent URLs (out of Sumantro's home) on Friday at the latest 16:43:58 but we can still edit them even after that of course 16:44:18 +1 16:44:24 but if you tell us some of the test cases should be dumped, we can dump them without renaming them to the "proper" wiki names 16:44:28 cyberpear: does that make sense? 16:44:33 so that's the reason why they are still with temporary names 16:44:45 yep, makes sense 16:45:21 if you create new test cases, just update your github ticket or our pagure ticket and I'll add them to the testday results page (the last #link) 16:45:33 #action dustymabe and jlebon (and anyone else who wants to) will review test cases linked from the various test day pages and try to fixup things that need fixing 16:45:51 kparal: regarding communications around the test day - how do we want to handle that ? 16:46:13 I instructed the visitors to join #fedora-coreos channel 16:46:25 that's mentioned on the test day wiki page 16:46:35 so everything would be discussed in there 16:46:36 ahh yep that's good, but I was wondering about blog posts or fedmag article 16:46:53 not required for you to do that, just wondering if there was something planned 16:46:56 sumantro: that's a question for you :) 16:46:57 kparal, why aren’t we using the #fedora-test-day that we usually use? 16:47:52 coremodule: usually a dedicated irc is not available, and this will help join the audience from people coming just for the test day and the regular users who are already connected or familiar with #fedora-coreos 16:47:54 fedmag blogpost to happen in an hr 16:48:01 and so is commblog 16:48:07 just finishing the draft 16:48:09 also, dustymabe said earlier they'd prefer #fedora-coreos iirc 16:48:16 kparal: coremodule: either way works for me 16:48:35 the announcement went out to @test @test-annouce already 16:48:38 we'll set up a #topic on #fedora-test-day that will redirect people to #fedora-coreos 16:48:38 we agreed during the last meeting that keeping it on #fedora-coreos was a sensible thing to do 16:48:47 SGTM 16:49:04 wfm! 16:49:10 sumantro: (please remember to set up that #topic if I forget, thanks) ^^ 16:49:28 kparal, I will take care of it :) 16:49:51 dustymabe, here's the call for testers announcement : https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/HL4BSOGIAH6AB5H7KW3IOKOB4MBE7V5N/ 16:49:52 thanks so much kparal and sumantro for continuing to follow up and help us organize this 16:49:59 dustymabe: you can review sumantro's announcement draft before he sends it out, if you want, just talk to him 16:50:26 also, please spread the announcement in your own channels, both internally in RH and in your FCOS or CL-related communities 16:50:36 +1 16:50:37 that's important, because we don't know how to reach the right audience, thanks 16:51:05 we can add it as a topic on #fedora-coreos maybe? 16:51:10 #info FCOS test day announcement on the qa fedora list: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/HL4BSOGIAH6AB5H7KW3IOKOB4MBE7V5N/ 16:51:28 i'll touch on possible other communications here soon 16:51:42 dustymabe, thanks +1 16:51:48 #topic F32 rebase tracker for changes discussion 16:51:52 #link https://github.com/coreos/fedora-coreos-tracker/issues/372 16:52:03 ok this is our generic ticket for discussing the rebase to f32 16:52:30 i think it's worth calling out known issues 16:52:46 we did just switch `testing` to f32 16:53:22 we have a reported known issue in https://github.com/coreos/fedora-coreos-tracker/issues/512 16:53:34 dustymabe: I think it's worth having some kind of "fallout/f32" label on GH, mind if I tag them? 16:54:02 #info known issue in f32 testing release with selinux labeling absolute symlinks after ignition run https://github.com/coreos/fedora-coreos-tracker/issues/512 16:54:05 lucab: SGTM 16:54:50 #info known kernel issue when testing our nodes on AWS https://github.com/coreos/fedora-coreos-tracker/issues/507 16:55:03 #undo 16:55:03 Removing item from minutes: INFO by dustymabe at 16:54:50 : known kernel issue when testing our nodes on AWS https://github.com/coreos/fedora-coreos-tracker/issues/507 16:55:14 #info known kernel issue in f32 testing when testing our nodes on AWS https://github.com/coreos/fedora-coreos-tracker/issues/507 16:56:02 despite these two issues we have decided to go ahead and switch the `testing` stream to f32. The symlink selinux issue is easily worked around and the AWS kernel issue we think is harmless (happens on shutdown) 16:56:34 I'm thinking we maybe should do some sort of coreos-status post that details a few things 16:56:44 - we switched `testing` to f32 and it's coming to stable soon 16:56:54 - highlight above mentioned known issues that we are investigating 16:57:01 - draw attention to the test day 16:57:30 +1 16:57:35 +1 because of point 2 16:57:58 any NACKs ? 16:58:37 #agreed we'll send out a coreos-status post with some relevant information about the rebase to f32 and possible issues people might be seeing 16:59:12 bgilbert: actually I wonder if we also need to mention to people that they'll probably want to bump their `coreos-installer` versions they are using (if they aren't using the install media) 16:59:37 dustymabe: yeah, because of the late key bump 16:59:42 good thought 16:59:45 +1, we won't have that problem in the future 16:59:56 but this time it's good to call it out 17:00:05 dustymabe: https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+label%3Afallout%2Ff32+ 17:00:25 bgilbert: jlebon: either of you want to send that coreos-status post. If not I'll do it later today 17:00:35 feel free :-P 17:00:49 * dustymabe wonders if we should copy the fedora devel@ list as well 17:01:02 dustymabe: go ahead :) 17:01:24 #action dustymabe to send coreos-status post with relevant information about the change to fedora 32 17:01:47 #topic RFC: Merge coreos/ignition-dracut into coreos/ignition 17:01:53 #link https://github.com/coreos/fedora-coreos-tracker/issues/511 17:02:07 darkmuggle: want to set the stage here? 17:02:16 Sure 17:02:49 So for a while we have had Ignition and Ignition's Dracut support in two separate Github repositories 17:03:30 One of the things that has come up a few times is that there are features in Ignition that require Dracut support 17:03:48 A great example is the forth-coming support for complex devices (i.e. LUKS) 17:04:22 We have danced around the idea of merging the dracut code into the Ignition code base itself. 17:04:57 After it recently came up again, I am proposing that we merge the two code bases together into Ignition itself. 17:05:08 would it make any sense to merge it into dracut instead? 17:05:12 or is it very ignition-specific? 17:05:27 Its very Ignition specific. 17:05:40 the backstory here is that ignition-dracut was forked from CL's bootengine repo, which contained many more dracut things 17:06:01 added a comment here: https://github.com/coreos/fedora-coreos-tracker/issues/511#issuecomment-638329716 17:06:07 we've mostly pruned everything away now, and we can move the few remaining things into fedora-coreos-config 17:06:25 TL;DR this might make it slightly harder to pick up just changes in ignition-dracut files without pulling in all upstream changes in ignition itself 17:06:38 but that's the only downside I really see 17:07:26 The upsides is that it makes introducing new features a bit easier, and should help with correlating changes in the dracut modules to the Ignition version. 17:07:26 dustymabe: I mean technically it would be the same where we could still just bump the ignition hash (though it's better to keep it tied to release tags) 17:07:50 slowrie: right, I see bumping the ignition hash (and building from master) as a bit more risky 17:08:12 whereas for ignition-dracut (separate repo) I didn't view it as quite as risky 17:08:43 another minor complication here is that ignition-dracut's spec2x is still used by RHCOS. though hopefully not for too much longer 17:08:56 bgilbert: lucab: I'm interested in your opinions on this since you know more of the history all the way back to the bootengine days 17:09:05 slowrie: as well 17:09:33 jlebon: in principle we could merge ignition-dracut spec2x into ignition spec2x, but yeah, probably easier to wait 17:09:38 Not agreeing either way here but I would say that with the context of not wanting to include non-release tag'd changes I don't think it's feasible with the current ignition release process 17:09:54 So we'd need to scope in work to get that more automated 17:09:55 slowrie: how so? 17:09:59 signing 17:10:11 I don't follow 17:10:13 slowrie: if we wanted to pick up changes we'd just add a patch to the rpm 17:10:19 ^ 17:10:34 which is probably a more "honest" approach than bumping a sidecar repo anyway 17:10:40 ^ 17:10:45 I guess that's fair 17:10:53 dustymabe: at this point I'm +1 to merging 17:10:54 one caveat 17:11:19 the premise so far has been that ignition-dracut is "optional" 17:11:19 (I don't have much input here. For afterburn we keep the dracut part in-repo but there isn't much variance across distros.) 17:11:31 i.e. distros might just implement their own thing instead 17:11:47 at this point, there's so much complexity in ignition-dracut that I don't think that's feasible 17:11:59 #proposed Merging ignition-dracut into ignition seems to have clear advantages and the disadvantages seem to be minor or easily worked around. 17:12:08 bgilbert: even if we include it in repo isn't it still optional in that they still have to package it and can choose to include the binary and their own bits? 17:12:22 bgilbert: right, i think distros can still just ignore that part of the source code if they'd like 17:12:26 I second slowrie's comment. 17:12:31 well sure, but we'll feel free to have complex dependencies between them 17:12:35 free-r 17:12:36 dustymabe: that proposal is more of an observation than a plan of action :) 17:12:46 so anyway, if we merge, we should be more careful not to have *COS-specific stuff in the dracut modules 17:12:50 which I'm +1 to anyway 17:12:53 but we should still work with Suse (other known users) to try to do this in a non-painful way for them (gives us some insight into making it easier for other distros as well) 17:12:58 bgilbert: yup, agreed 17:13:21 We already have the current separation with some ignition related units being in fedora-coreos-config 17:13:22 bgilbert: is *COS-specific stuff basically ostree bits? 17:13:35 I would make the proposal: Merge Ignition specific Dracut modules into Ignition and *COS modules into Fedora CoreOS Config. 17:13:37 #proposed Merging ignition-dracut into ignition seems to have clear advantages and the disadvantages seem to be minor or easily worked around. We'll merge them together unless clear blockers come up when attempting the transition. 17:13:44 bgilbert: or is there more you see as a concern? 17:14:02 Can we add a note that it'd be nice to delay that work a bit 17:14:05 lucab: anything tied to *COS-specific behavior that happens to touch Ignition 17:14:10 I'd like to not do more rebases :) 17:14:11 lucab: see https://github.com/coreos/fedora-coreos-tracker/issues/511#issuecomment-637629796 17:14:19 we've had some leakage in the past 17:14:32 #proposed Merging ignition-dracut into ignition seems to have clear advantages and the disadvantages seem to be minor or easily worked around. We'll merge them together in the future (when it's safest to do so) unless clear blockers come up when attempting the transition. 17:14:52 dustymabe: +1 17:14:52 slowrie: better ^^ ? 17:14:52 jlebon: great list, thanks 17:14:53 +1 17:15:08 +1 17:15:26 any opposed? 17:15:27 yeah wfm, I'm hoping to get the PRs up for the LUKS stuff soon-ish (this week or next) and just don't want to have to deal with merging more stuff & rebasing again 17:15:28 +1 17:16:06 i'm thinking we at least get this f32 transition out of the way and maybe switching RHCOS over to the new ignition version/spec and then do it, but I'm not tied to that 17:16:33 #agreed Merging ignition-dracut into ignition seems to have clear advantages and the disadvantages seem to be minor or easily worked around. We'll merge them together in the future (when it's safest to do so) unless clear blockers come up when attempting the transition. 17:16:59 thanks for bringing that up darkmuggle - and for everyone having a fruitful discussion 17:17:00 i think it's fine to not block on RHCOS moving over. we could just delete the master branch and make the default branch spec2x to make it obvious 17:17:07 getting RHCOS over to current ignition makes life a lot easier and gives much more freedom 17:17:20 I'd prefer not to delete any code in repos :) 17:17:32 yeah, never delete code 17:17:53 #topic network interface name differs between Fedora CoreOS and RedHat CoreOS 17:17:57 fair -- switch the default branch then :) 17:18:00 #link https://github.com/coreos/fedora-coreos-tracker/issues/484 17:18:31 worst case, "GitHub Archive" it... I hate even when projects delete the all code in the repo in a final "we're done" commit, leaving the history if you dig for it 17:18:33 (though git history is another unfortunate casualty of this) 17:18:38 looks like this was discussed last week - do we want to formalize our plans ? 17:18:55 +1 to archiving it 17:19:01 you can include the history during the move with git subtree 17:19:16 sorry, let's move on to $topic 17:19:50 i had a question in https://github.com/coreos/fedora-coreos-tracker/issues/484#issuecomment-637008004 17:20:16 dustymabe: that's correct 17:20:27 an obvious bug in #topic with easy fix, but handling existing systems will be the tricky part 17:20:37 i think it's likely other "migration" things like this will come up in the future 17:20:57 I think I was still trying to put systemd-naming-scheme pinning in the same ticket last week, but I think it's unreasonable, so jlebon's plan should be good 17:22:28 #proposed For this issue we'd like to correct the behavior but realize it would be a disruption for existing nodes that upgrade. We propose to add a barrier node in our update graph that will keep behavior for updating nodes the same, while fixing the bug and causing new behavior for newly installed nodes. 17:23:10 lucab: when we move over, should we still make sure we pin to some scheme so it doesn't change under us in the future? 17:23:31 jlebon: longer term I think so 17:23:51 jlebon: but I don't think it's as urgent as moving out of eth* 17:24:01 ack 17:24:07 +1 to proposal 17:24:20 eth* makes sense for cloud, not hardware 17:24:46 but I guess users can request that at install time 17:24:57 any opposed to the current proposal? 17:25:06 please ack/nack 17:25:15 +1 17:25:24 ack 17:25:59 ack 17:26:08 #agreed For this issue we'd like to correct the behavior but realize it would be a disruption for existing nodes that upgrade. We propose to add a barrier node in our update graph that will keep behavior for updating nodes the same, while fixing the bug and causing new behavior for newly installed nodes. 17:26:45 I assume for $timing on when to make this change we should either let the f32 changes propagate out safely and then revisit on when to make this change 17:26:56 OR do it at the same time as the transition to f32 for stable 17:27:10 i think probably the former for now since we've got a lot going on already 17:28:01 sounds reasonable to me 17:28:21 ballpark ETA was "before OKD goes GA" 17:28:26 ok it looks like we're running out of time so i'll skip the last meeting ticket 17:28:29 #topic open floor 17:28:33 cyberpear: I think you had a topic 17:28:35 anyone else? 17:28:37 o/ 17:28:39 go for it 17:29:11 * cyberpear waits for bcotton 17:29:19 thanks cyberpear 17:29:35 * lucab waves back to bcotton 17:30:07 so when we set up the BZ component for FCOS, we talked about cc'ing the FAS group on bugs. it appears that you *can* in fact email FAS groups, so is that something we want to pursue or let it sit for now? 17:30:34 SGTM - i have to check what fas groups even exist for us 17:31:00 * cyberpear should join the FCOS fas group 17:31:07 in general we should try to make it so that it doesn't send to $everyone, (a few people should be enough) 17:31:15 but a single person is a single point of failure too 17:31:31 for some context - the BZ component is just there to redirect people to our issue tracker 17:31:45 the other option would be to have people watch that component themselves 17:31:48 but some people will probably still open some issues and we don't want that to go into /dev/null 17:32:16 bcotton: that might be ok too 17:32:28 I'd have to remind myself how to watch a component in bugzilla 17:33:00 but if we could get a few of us to volunteer (the job is basically to try to redirect people to the issue tracker, or point them at issues that already exist for what they reported) then that would work 17:33:04 I seem to be in a "coreos" FAS group 17:33:12 https://bugzilla.redhat.com/userprefs.cgi?tab=component_watch 17:33:38 and what's the component again? 17:33:51 gosh, i have to know everything :p 17:33:55 :) 17:34:12 it's why we love you 17:34:40 is it possible to have a custom template that points to the tracker? 17:34:50 anyone else want to volunteer to watch the component and do the described $job from above ? 17:34:56 jlebon: yeah I think he already set that up for us 17:34:58 * jlebon raises hand 17:35:04 ahh cool, very nice 17:35:20 Product: Fedora, Component: fedora-coreos 17:35:51 ok I watched that component 17:35:54 added ✔️ 17:36:13 pheww, no bugs in there yet 17:36:16 sounds like cyberpear wanted to too 👀 17:36:41 ok any other topics for open floor ? 17:36:51 we're kind of at the end of time 17:37:00 but I wanted to talk about rojoig briefly 17:37:13 is this on the near or medium-term roadmap? 17:37:27 yum mirroring infrastructure is widespread, but most tools don't know about ostree 17:37:37 and it seems that pulp3 doesn't support ostree either 17:37:47 ahh like rpm-ostree rojig ? 17:38:12 yes, rojig "rpm-ostree jigsaw", would allow riding on the back of all the existing yum mirroring solutions 17:38:25 previous discussion at https://github.com/coreos/fedora-coreos-tracker/issues/23 17:38:28 for offline and "disconnected" use cases where the network isn't on the internet 17:38:39 I share this sentiment on "how should I mirror FCOS ostree repo" 17:38:52 OpenShift solves it by shoving the ostree into a container, but that's very heavy weight w/ lots of duplication 17:39:31 lucab: but it's pretty easy to mirror with `ostree pull` for individual use cases 17:39:39 cough it's a quay.io stress-test plan cough 17:39:43 on that topic, see https://blogs.gnome.org/alexl/2020/05/13/putting-container-updates-on-a-diet/ 17:40:01 there's working code at https://github.com/flatpak/flatpak/pull/3606 17:40:13 yes, that's good work, but there's still not a good container "registry" mirroring solution, only repo-by-repo 17:41:33 cyberpear: so to answer your literal question, it is not currently on the roadmap AFAIK 17:41:37 cyberpear: I don't really have a good answer for you. It would take a decent amount of work to get from where we are to delivering things via rpm-ostree rojig. 17:41:46 I think rpm-ostree rojig is the "best thing since sliced bread" in terms of leveraging existing infrastructure, but shrinking container deltas is also very helpful 17:42:08 Thanks... good enough for today. I know we're over time. 17:42:09 dustymabe: I'm not saying it isn't, but everything I've seen so far are single-use artisanal recipes 17:42:29 lucab: so you want an enterprise mirroring tool to support it basically? 17:42:36 if it were an/the native way of doing it, users wouldn't be creating the rojig, only consuming it 17:43:09 hmm, AFAIK pulp does support ostree 17:43:17 (re. something you said earlier) 17:43:22 pulp2 does, pulp3 does not, afaik 17:43:36 pulp2 is maintenance mode 17:43:38 ahh i see. interesting 17:43:38 yeah they may have ripped it out for some reason 17:43:41 dustymabe: maybe, or some turn-key container? I don't exactly know, but I was having a similar sentiment that it is an under-appreciated areaa 17:44:05 probably ripped it out since RH doesn't need it since their only use of RHCOS is OCP and they ship containers w/ ostree there 17:44:11 cyberpear: it might be worth a ticket and more discussion 17:44:23 not so much "ripped it out" as "didn't build it" since pulp3 is very plugin-focused 17:44:47 probably no pulp_ostree plugin for 3, but I didn't look too closely in the past month 17:45:03 I'll try to open a ticket 17:45:06 ok i'm going to close out the meeting soon since we're 15 minutes over 17:45:12 thanks, all! 17:45:24 thanks! 17:45:36 #endmeeting