17:01:57 <sgallagh> #startmeeting ELN (2023-12-01)
17:01:57 <zodbot> Meeting started Fri Dec  1 17:01:57 2023 UTC.
17:01:57 <zodbot> This meeting is logged and archived in a public location.
17:01:57 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
17:01:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:01:57 <zodbot> The meeting name has been set to 'eln_(2023-12-01)'
17:01:57 <sgallagh> #meetingname eln
17:01:57 <zodbot> The meeting name has been set to 'eln'
17:02:03 <sgallagh> #topic Init Process
17:02:08 <sgallagh> .hi
17:02:09 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:02:09 <tdawson> Hello
17:03:26 <sgallagh> .hello tdawson
17:03:27 <zodbot> sgallagh: tdawson 'None' <tdawson@redhat.com>
17:03:42 <sgallagh> I'll give folks a couple minutes to wander in.
17:04:11 <sgallagh> Even if we don't have quorum, I still want to discuss the CS10 import and mass-rebuild plans, since we won't have another meeting before that happens.
17:04:21 <sgallagh> So we can still get some things on the record today.
17:06:41 <sgallagh> #topic Agenda
17:06:58 <sgallagh> #info Agenda Item: CentOS Stream 10 import and ELN mass-rebuild
17:07:18 <sgallagh> #info Agenda Item: Strategy for importing packages whose git history fails git-fsck
17:07:50 <sgallagh> Any other topics for today?
17:07:51 <neil> .hi
17:07:53 <zodbot> neil: neil 'Neil Hanlon' <neil@shrug.pw>
17:08:52 <Son_Goku> .hello ngompa
17:08:53 <zodbot> Son_Goku: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:09:51 <sgallagh> Thanks for joining, folks.
17:10:14 <sgallagh> OK, first up, let's talk about the big CS10 import
17:10:22 <sgallagh> #topic CentOS Stream 10 import and ELN mass-rebuild
17:11:12 <sgallagh> So, as I assume most people are aware, we are coming up on our end-game for the CS10 bootstrap.
17:11:17 <smooge> hello
17:11:19 <decathorpe> .hi
17:11:20 <zodbot> decathorpe: decathorpe 'Fabio Valentini' <decathorpe@gmail.com>
17:12:20 <tdawson> It's the final countdown ...
17:12:27 <sgallagh> #info CentOS Stream 10 will break inheritance from Fedora ELN on 2024-02-06, co-incident with the Fedora 40 Branch from Rawhide.
17:13:01 * neil plays music
17:13:55 <sgallagh> To that end, we've been working these last few months to get the last bits in place to enable development directly on CentOS Stream 10.
17:14:28 <sgallagh> #info There will be a separate announcement made to identify when public contributions to CentOS Stream 10 will begin.
17:15:22 <Son_Goku> wait, that's not happening at the same time?
17:15:27 <sgallagh> There will almost certainly be a gap between the inheritance break and when it opens to the public. The details of this are still being worked out and may slip depending on technical hurdles.
17:16:04 <Son_Goku> I'm confused, because the c10s branches are already there in gitlab.com/redhat/centos-stream
17:16:10 <sgallagh> @Son_Goku To clarify, when I say "public" here, I mean to the general public AND Red Hatters not part of the bootstrap team.
17:16:15 <sgallagh> Not just external people.
17:16:49 <neil> I was interpreting that as "hold until we know things are OK so we aren't overloading the bootstrap team", basically
17:17:10 <tdawson> neil: Correct ... and worded very well.
17:17:14 <sgallagh> Exactly. We want this gap to be as short as possible, but I'm not committing to same-day :)
17:17:32 * neil is clearly chomping at the bit to contribute changes to C10S :P
17:17:40 <neil> like adding wayland back in /s
17:17:51 <neil> X.. oops
17:17:52 <sgallagh> The `c10s` branches are there, but only the ELNBuildSync tool can push to them at present.
17:17:54 <neil> i can't even make a joke goodly
17:18:55 <sgallagh> As @Son_Goku notes, the branches exist and we've been slowly importing packages as we go, but we need to make the final push and get the remainder of the buildroot packages imported in the near future.
17:19:15 <sgallagh> We've been waiting on the resolution to three major issues:
17:19:49 <sgallagh> 1) We discovered a bug in RPM 4.19 that was causing builds to fail on i686 under certain conditions that were unfortunately common on RHEL builders.
17:20:22 <sgallagh> That was fixed upstream recently and I pushed it to Rawhide, ELN and CS10 yesterday.
17:20:55 <tdawson> And it's working ... I've tested 30 of the failed builds with success ... er ...e xcept one that was unrelated.
17:21:12 <neil> Question (which I can hold until the appropriate time): do we have a calendar of these event dates and if not, do we want to record them anywhere  (a page on the wiki with "Important Dates" or so)
17:21:20 <sgallagh> The other two issues are related to CS10 carrying the full Fedora history into git (as requested by the community)
17:22:28 <sgallagh> @neil We don't have a public calendar yet, mostly because the Powers That Be have asked us not to make any official announcements about CentOS Stream 10 dates until we're ready to commit to them.
17:22:46 <sgallagh> Aside from the inheritance break at F40 Branch, which has been public information for a long time now.
17:22:48 <neil> Ack, I grok that on a spiritual level
17:23:14 <Son_Goku> sgallagh: what are the two issues?
17:23:57 <sgallagh> Issue 2) RPMAutoSpec - in CS9, the way we did the history import had the side-effect of bypassing the need for build-time processing of the `%autorelease` and `%autochangelog` macros.
17:24:16 <sgallagh> s/history import/non-history import/
17:24:42 <sgallagh> So we needed to get the processing support added to CS10 and RHEL 10, which proved harder than expected.
17:25:32 <sgallagh> Specifically, the Koji plugin was rejected from inclusion on the builders for a variety of reasons too esoteric to attempt to enumerate here.
17:26:03 <sgallagh> So we have been reworking it such that the processing will now happen as a mock plugin instead of directly on the builder hosts.
17:26:07 <sgallagh> #link https://github.com/rpm-software-management/mock/pull/1253
17:26:40 <sgallagh> That's just about ready to merge, finally. We'll get some testing done on it and get it deployed in CS Koji as soon as possible.
17:26:50 <Son_Goku> so that has the side effect that it works on anything that uses mock now
17:26:54 <Son_Goku> including copr, right?
17:26:58 <sgallagh> yes
17:27:31 <Son_Goku> good... if we weren't going to have smart things like tracking rebuilds properly, then having it as a koji plugin was more overhead than it was worth
17:27:49 <sgallagh> However, this is /not/ a blocker for the CS 10 import, as the macros we have available there will just set the release field to `1` for the initial imports.
17:28:05 * Son_Goku still wishes that we could have an automatic rebuild counter thing for rpmautospec...
17:28:33 <sgallagh> But without a plan for it, subsequent builds would have significant issues, so we needed to have an answer ready before we get too far.
17:28:54 <Son_Goku> yes
17:28:57 <sgallagh> The third issue is the topic of the other agenda item, so I'll switch the topic.
17:29:00 <Son_Goku> okay
17:29:09 <nils> Son_Goku, Caveat about COPR and rpmautospec: As I understand it, COPR repos don’t have meaningful commit logs
17:29:18 <sgallagh> #topic Strategy for importing packages whose git history fails git-fsck
17:29:22 <Son_Goku> yes, there's other reasons copr and rpmautospec are broken
17:30:35 <sgallagh> There are sixteen packages in Fedora ELN that currently we cannot import into Gitlab for CentOS Stream 10 due to its automatic rejection of any commit that fails `git-fsck`, which it runs on all pushes.
17:31:08 <sgallagh> They're all the result of the same committer in 2008 having a spurious extra '<' in their email address.
17:31:17 <Son_Goku> oh ooof
17:31:23 <Son_Goku> and we can't fix that, can we
17:31:26 <Son_Goku> wait
17:31:28 <Son_Goku> 2008?
17:31:35 <sgallagh> Gitlab committed to having a fix for this in October, but they have not delivered.
17:31:35 <Son_Goku> we were using CVS back then, weren't we?
17:32:03 <sgallagh> Yeah, but the switch to dist-git still predates the git-fsck check for this issue.
17:32:14 <Son_Goku> oh ooof
17:32:30 <Son_Goku> is there a public GitLab issue for this?
17:32:41 <sgallagh> Yes, one minute while I dig it up
17:33:11 <sgallagh> #link https://gitlab.com/gitlab-org/gitlab/-/issues/246750
17:34:09 <sgallagh> Gitlab has gone radio-silent on my requests for updates here, which isn't great.
17:34:14 <Son_Goku> oh hey I remember this issue
17:34:41 <sgallagh> So the time has come to start looking at less-great options.
17:35:40 <Son_Goku> could we fix the existing Fedora repos and archive the original refs?
17:35:44 * neil digs out the time machine
17:35:45 <sgallagh> I've come up with three options, listed here in order of least-to-greatest order of preference (by me; YMMV)
17:36:09 <Son_Goku> ah okay, what are those options?
17:36:22 <sgallagh> @Son_Goku That's... not on my list because I pretty much assumed the answer would be "no", but maybe you have more information than I there.
17:36:54 <sgallagh> Maybe I should direct that at smooge, actually
17:37:10 <Son_Goku> we've had to do this before, mostly when people screw up and commit huge blobs that need to be "pruned"
17:37:37 <Son_Goku> as long as you're not doing `git clone --mirror`, you get to mostly ignore them after that
17:37:41 <sgallagh> I thought that we only allowed that if it was discovered right after that.
17:37:53 <Son_Goku> we have no rubric or guidelines for it
17:37:58 <sgallagh> Interesting...
17:38:04 <sgallagh> OK, so we have four options... :-)
17:38:08 <sgallagh> (maybe)
17:38:25 <Son_Goku> I've asked for it once when I discovered a whole multi-gigabyte source tarball was committed into a repo
17:38:35 <Son_Goku> and it was making checkouts unbearably slow
17:38:35 <neil> five including my idea of going back in time to prevent the mistake to begin with ;)
17:38:36 <sgallagh> The biggest issue I see with that is that for these sixteen packages, we'd have to make sure anyone who has a clone of them currently re-pulls it.
17:38:46 <sgallagh> Since the HEAD for the `rawhide` branch would change
17:39:14 <sgallagh> neil: No, still four because I went back in time and smacked you upside the head for suggesting it ;-)
17:39:16 <Son_Goku> how many of those 16 have forks?
17:39:18 <tdawson> Wait ... it's only changing HEAD ?  I thought you said it changed all the git hashes.
17:39:33 <Son_Goku> it changes all the hashes because the whole chain leading to HEAD changes
17:39:38 <sgallagh> tdawson: Both things happen.
17:39:43 * neil says ouch in the past
17:39:44 <tdawson> Ahh ... ok
17:39:48 <Son_Goku> the problem is that since it was 2008, that means that it basically affects the whole git history ever done in git
17:40:02 <Son_Goku> we didn't switch to git until 2011
17:40:04 <sgallagh> Current checkouts will be pointing at the old hashes and will need manual effort to get back on track
17:40:31 <sgallagh> Son_Goku: You understand why I am grouchy at Gitlab :)
17:40:45 <Son_Goku> yes, I've always understood that :P
17:41:16 <Son_Goku> you're talking to the guy who ran multiple GitLab instances and had several high severity tickets with them that took >1 year or longer to resolve
17:41:18 <sgallagh> OK, so Option 1) Do for these sixteen packages what we did in CS 9 and don't import the history.
17:41:34 <sgallagh> Just take a snapshot with a commit message that references where it came from in Fedora's dist-git
17:42:49 <sgallagh> This would essentially break inheritance for these packages immediately, because the sync tool from ELN -> CS10 no longer has the built-in support for doing that, and re-adding it (and supporting it on a per-package level) would be prohibitive given the remaining time.
17:42:57 <Son_Goku> oof
17:43:07 <Son_Goku> are any of these 16 packages using rpmautospec?
17:43:10 <sgallagh> This is my least-preferred option.
17:43:21 <sgallagh> I don't believe so, but I haven't checked specifically. Why?
17:43:35 <Son_Goku> if they did, that option becomes even less appealing
17:43:54 <Son_Goku> because manual deconversion has been very error-prone so far
17:44:11 <sgallagh> The affected packages: antlr, apache-commons-cli, apache-commons-codec, apache-commons-exec, apache-commons-io, apache-commons-parent, beust-jcommander, bsf, jdom, jsch, log4j, maven-antrun-plugin, maven-archiver, maven-assembly-plugin, maven-shade-plugin, mojo-parent
17:44:13 <Son_Goku> and "rpmautospec revert" is still not a thing
17:44:19 <Son_Goku> lol of course
17:44:21 <Son_Goku> java packages
17:44:30 * neil rolls his eyes at the universe
17:44:47 * neil hands Son_Goku, sgallagh, tdawson all drinks
17:45:01 <sgallagh> Keep pouring
17:45:06 <neil> say when
17:45:14 * sgallagh doesn't
17:45:18 * Son_Goku doesn't either
17:45:18 * neil nods
17:45:41 <neil> * a piano begins playing Billy Joel *
17:46:14 <tdawson> Let's move on to the next proposal ..
17:46:17 <sgallagh> Ah, sorry. Looking at my notes, that was Option 2). The original Option 1) was that, but with automation like in CS9. But we determined that can't be done.
17:46:46 <sgallagh> Option 3) was similar to Neal's suggestion, except we don't do it in Fedora.
17:47:03 <sgallagh> We rewrite the history and manually import that over to CS10, breaking inheritance early.
17:47:16 <Son_Goku> I figured that was one of the options
17:47:19 <sgallagh> We keep the intent of the history, but we lose the fast-forward merge ability (and therefore auto-sync)
17:47:45 <Son_Goku> I wonder if we should run git-fsck on pushes in pagure
17:47:48 <Son_Goku> we currently don't...
17:47:53 <neil> Son_Goku++
17:48:03 <neil> i'd just like to note for the record I dislike _all_ of the ideas... (I know we all do, probably)
17:48:04 <sgallagh> Son_Goku: We don't? I'm pretty sure git itself is doing it.
17:48:16 <Son_Goku> we use libgit2 as the handler
17:48:22 <Son_Goku> so we're not actually running git
17:48:57 <Son_Goku> currently pagure only uses git itself for generating per file history, since libgit2 doesn't give you that feature
17:48:59 <sgallagh> Right, so I don't much like any of these choices either. But that's why these are contingency plans, not plans :)
17:49:12 <Son_Goku> yeah, I definitely don't like doing any of these
17:49:17 <sgallagh> Unfortunately, we have to pick one and run with it, because time is running out.
17:49:38 <Son_Goku> so we should ask infra if we can fix them in Fedora first
17:49:47 <Son_Goku> because this is actually a pretty annoying problem to hit
17:50:26 <Son_Goku> and we'll keep hitting it every three years without fail if we don't fix it here
17:50:50 <sgallagh> FWIW, given that this is so far back in the history, rewriting from there is effectively doubling the size of the repo for each of these packages. (Unless compression works better than I expect)
17:51:02 <Son_Goku> no, you're basically right
17:51:28 <Son_Goku> unless the git repos are on btrfs or xfs and being reflinked, it'll double space
17:52:06 <sgallagh> It's fortunately a very small number of packages, at least.
17:52:10 <Son_Goku> it sucks, but it also just doesn't matter unless you do `git clone --mirror`
17:52:56 <Son_Goku> and I don't think that's what you're doing for the fedora-eln -> centos-stream sync
17:53:21 <sgallagh> We're pulling the commit associated with the build (and its history) and pushing that to the c10s branch
17:53:27 <Son_Goku> right
17:53:36 <Son_Goku> which is _not_ the same as mirroring all data
17:53:38 <sgallagh> So yeah, that won't impact CS10 at least
17:54:44 <sgallagh> Alright, if you think Infra/Releng will be amenable, that's probably the best of the bad choices.
17:54:45 <tdawson> My opinion (and this isn't just because I want less work) ... if we can do the change in Fedora, let's do it.  It's a one time thing.
17:54:49 <Son_Goku> assuming none of the repos are rpmautospec'd, then fixing this is a question of getting all the refs regenerated to get rid of this problem (or at least the rawhide one) and having archived branches with the "old" history
17:55:14 <Son_Goku> then make a new commit and build them in rawhide so that the history _sticks_ and syncs
17:55:15 <sgallagh> In the event they say "no", I suggest Option 3) as the least-worst follow-up
17:55:19 <Son_Goku> yeah
17:55:29 <Son_Goku> it sucks but it would be automatable
17:55:49 <Son_Goku> it's garbage but what can you do
17:56:12 <sgallagh> It's a garbage CAN, not a garbage CANNOT!
17:56:17 <neil> sgallagh++
17:56:17 <zodbot> neil: Karma for sgallagh changed to 7 (for the release cycle f39):  https://badges.fedoraproject.org/tags/cookie/any
17:56:18 <tdawson> Well ... for two months .... and 16 packages that don't get updated much ... I think even manual (but scripted) isn't too bad.
17:56:45 <Son_Goku> we should also probably figure out a server-side hook in dist-git to enforce git-fsck after fixing them
17:56:50 <Son_Goku> and reject pushes that fail
17:56:53 <sgallagh> Sure, but unless Gitlab steps up, we'd have to deal with it in RHEL 11 again too
17:56:55 <Son_Goku> because we currently do not do that
17:57:41 <tdawson> sgallagh: It's not like it would take GitLab *another* three years to fix ths </sarcasm>
17:57:52 * neil drinks to that
17:57:59 <Son_Goku> tdawson: you clearly haven't worked with gitlab before /s
17:58:05 <tdawson> :)
17:58:11 * neil anxiously checks when his license renewal is...
17:58:17 * Son_Goku still remembers that it took them three years to fix MR page view performance issues they introduced in GitLab 11
17:58:20 <neil> anyways i like gitea :P
17:58:51 <Son_Goku> that issue is why I got into pagure in the first place :)
17:59:05 <neil> for the second time this meeting, i feel that spirtually
17:59:06 <Son_Goku> at least I can understand and hack on the codebase and it makes sense when I work on it
17:59:29 <neil> yep yep. not trying to convince Product that it should be done
17:59:38 <Son_Goku> it may not be the prettiest, but there's a lot of value in something that I can reason
17:59:41 <tdawson> For me, I like option 4 (fix in fedora) the best, and option 3 (fix it on sync to cs10) the next best.
17:59:48 <sgallagh> #info We will ask Fedora Infra/Releng if we can rewrite the history of these sixteen packages and archive the old branch.
18:00:02 <Son_Goku> shouldn't that be agreed?
18:00:05 <Son_Goku> not info?
18:00:09 <sgallagh> #info Failing that, we will rewrite the history manually for import to CS 10 and maintain it by hand until the handoff.
18:00:18 <sgallagh> #undo
18:00:18 <zodbot> Removing item from minutes: INFO by sgallagh at 18:00:09 : Failing that, we will rewrite the history manually for import to CS 10 and maintain it by hand until the handoff.
18:00:19 <sgallagh> #undo
18:00:19 <zodbot> Removing item from minutes: INFO by sgallagh at 17:59:48 : We will ask Fedora Infra/Releng if we can rewrite the history of these sixteen packages and archive the old branch.
18:00:24 <sgallagh> #agreed We will ask Fedora Infra/Releng if we can rewrite the history of these sixteen packages and archive the old branch.
18:00:29 <sgallagh> #agreed Failing that, we will rewrite the history manually for import to CS 10 and maintain it by hand until the handoff.
18:00:34 <sgallagh> Thanks, you're right
18:00:58 <Son_Goku> yeah I think this is probably the best of the bad options
18:01:41 <sgallagh> Thanks everyone for working through this. I appreciate it.
18:02:06 <sgallagh> We're technically over time now, but:
18:02:06 <sgallagh> #topic Open Floor
18:02:06 <sgallagh> Anything for Open Floor?
18:03:02 <Son_Goku> I have something to ask for you to bring up internally
18:03:53 <Son_Goku> I don't expect anything to happen anytime soon on this, but I would really like you folks to figure out a better way to logically organize the stuff in content-resolver-input for stuff like GNOME
18:04:17 <Son_Goku> it's really confusing and hard to follow when related components are split up across different yamls and it makes the lookups more difficult
18:04:40 <sgallagh> That's a fair critique
18:04:53 <smooge> sgallagh: sorry i had to go into a meeting. i will scroll over and see if I can answer your ping from meeting
18:05:17 <Son_Goku> I'm trying to keep track of these things for EPEL related stuff, and it's really difficult because of the structure
18:05:41 <sgallagh> @Son_Goku Realistically, maybe the best option would be for us to do a big cleanup when ELN switches over to EL11 in February.
18:05:49 <Son_Goku> that would be great
18:06:21 <Son_Goku> if you look at what we do for kde, it's really easy to identify everything related to that workload since it's all in one or two yamls
18:06:40 <Son_Goku> which btw, tdawson we need to fix now that plasma 6 is in rawhide 😅
18:06:58 <sgallagh> Yeah, I suspect that there's a fair amount of "Well, the package is already in the set somewhere, so I don't need to include it" going on
18:07:22 <Son_Goku> even if you've got the whole sst thing, maybe having a unified yaml as a duplicate would be helpful
18:07:27 <tdawson> Son_Goku: Yep ... but it's a bit farther down on my priorities ... so it's going to be a couple weeks.
18:07:37 <Son_Goku> because remember that this tooling was originally made for workloads, not for package ownership
18:07:51 <sgallagh> @Son_Goku Sorry, I can't parse "having a unified yaml as a duplicate".
18:08:09 <Son_Goku> that is, a gnome workload yaml that duplicates the content of all the sst gnome/desktop thingies
18:08:21 <sgallagh> oh, hmm
18:08:25 <Son_Goku> that way I have a single thing I can look at in tiny.distro.builders
18:09:40 <sgallagh> That's a solution that could work from a technical perspective, but I'm not sure I like it.
18:10:03 * Son_Goku shrugs
18:10:09 * yselkowitz moves to table this until after february, it won't happen before then
18:10:12 <sgallagh> I need to think on it more when I'm not holding a meeting open :)
18:10:32 <sgallagh> Thank you for raising it.
18:10:46 <smooge> option 5) you create new packages with each of the ones listed (and any others in Fedora which have this problem) which we do a 'fix' of the problem in all the history. The old packages get a commit saying 'package is now at' and the new packages take over
18:10:51 <smooge> or some variant of that
18:11:15 <sgallagh> #info We need to revisit the way that Content Resolver workloads are broken up. Currently they are more aligned to "who owns it" than "where the package belongs".
18:11:19 <smooge> oh look I forgot to page down
18:11:33 <Son_Goku> smooge: it's a valid option, yes
18:11:57 <Son_Goku> an easy hack is just we start prefixing those packages with `java-*` with new sources and just do the thing
18:12:02 <Son_Goku> and retire the old ones
18:12:10 <smooge> the reason being is that this will cause some sort of problem if Fedora moves to another repository system anyway
18:12:13 <sgallagh> The only functional difference there that I see is the need to do sixteen package reviews.
18:12:14 <Son_Goku> we can request new dist-git repos without history and push fixed history
18:12:22 <Son_Goku> sgallagh: no, we don't
18:12:32 <Son_Goku> we can request this as an exception in fedpkg request-repo
18:12:35 <yselkowitz> renames require re-review
18:13:12 <Son_Goku> well fine, just assign me all 16 and I'll review them sgallagh :P
18:13:20 <smooge> these are java packages so re-reviwing them is probably a good idea anyway
18:13:41 <sgallagh> These are java packages, so re-reviewing them is likely to be painful and time-consuming :-P
18:13:56 <yselkowitz> the java maintainer(s) might have something to say about that too?
18:14:05 <smooge> yes they would
18:14:24 <smooge> the main thing is that if Fedora has to move SCM's in the next year or so.. its going to come up
18:14:46 <sgallagh> smooge: I'm not sure if this solves the issue, though.
18:15:05 <sgallagh> We still have to have the old repos available since they were used to build published packages.
18:15:24 <sgallagh> So we'll need to find some way to bring them over (or else archive current dist-git somewhere, I guess)
18:15:59 <smooge> yep.. glad its not my problem anymore
18:16:01 <Son_Goku> even with the assumption we're not moving
18:16:06 <Son_Goku> we still should fix this
18:16:14 <Son_Goku> because things like remote pull requests are basically broken for these repos
18:16:20 <Son_Goku> because people can't host them anywhere else
18:16:54 <Son_Goku> if someone hosted a fork from a derivative distro elsewhere but wanted to send changes back to fedora, that workflow should absolutely work for every single package
18:17:14 <Son_Goku> and git repos failing git-fsck is a great way to have that workflow fail
18:18:08 <yselkowitz> this sounds like a fesco issue
18:18:15 <Son_Goku> probably
18:18:16 <yselkowitz> not (just) an eln one
18:18:28 <Son_Goku> I wasn't personally aware we had repos broken like this
18:19:02 <sgallagh> I don't know how many others are affected; these are the sixteen that exist in ELN
18:19:05 <Son_Goku> I know that when I did downstream work for $DAYJOB-1, I did use the remote pull request feature at times so it's valuable
18:19:18 <sgallagh> It's quite possible there are many others in Fedora.
18:19:26 <Son_Goku> we probably need to do a scan
18:20:44 <sgallagh> I'm not aware of any more efficient approach than to check out every repo in a loop and run `git fsck` manually on them. That will be fun...
18:21:07 <yselkowitz> you have a very odd definition of fun ;-)
18:21:18 <sgallagh> (Well, okay, we can parallelize it somewhat, but it's still going to take a long time)
18:23:03 <sgallagh> Alright, we're well over time now. I'm going to close things up.
18:23:20 <sgallagh> Thank you all for coming.
18:23:26 <sgallagh> #endmeeting