16:30:19 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:19 <zodbot> Meeting started Wed May 13 16:30:19 2020 UTC.
16:30:19 <zodbot> This meeting is logged and archived in a public location.
16:30:19 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:19 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:23 <cyberpear> .hello2
16:30:24 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:30:26 <jdoss> .hello2
16:30:27 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:30:29 <jlebon> .hello2
16:30:30 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:32 <lorbus> .hello2
16:30:33 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:30:36 <dustymabe> #topic roll call
16:30:52 <dustymabe> #chair cyberpear jdoss jlebon lorbus
16:30:52 <zodbot> Current chairs: cyberpear dustymabe jdoss jlebon lorbus
16:30:58 <dustymabe> glad to see the ol' jdoss
16:31:11 * jdoss waves
16:31:13 <bgilbert> .hello2
16:31:14 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:34 <jdoss> I am still alive. Just been getting firehosed at my new job.
16:32:01 * gilliard__ listening (satellit)
16:33:21 <dustymabe> jdoss: it happens :)
16:33:39 <dustymabe> #chair bgilbert
16:33:39 <zodbot> Current chairs: bgilbert cyberpear dustymabe jdoss jlebon lorbus
16:33:47 <dustymabe> #topic Action items from last meeting
16:33:55 <dustymabe> no action items to speak of specifically :)
16:34:07 <dustymabe> #topic topics for this meeting
16:34:08 <lucab> .hello2
16:34:10 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:34:29 <dustymabe> any topics anyone would like to discuss during this meeting? we have one meeting ticket and then we can discuss other topics
16:34:39 <dustymabe> otherwise we'll skip to open floor after the meeting ticket
16:34:41 <dustymabe> #chair lucab
16:34:41 <zodbot> Current chairs: bgilbert cyberpear dustymabe jdoss jlebon lorbus lucab
16:35:28 * dustymabe waits another minute for topic suggestions
16:36:11 <dustymabe> #topic F32 rebase tracker for changes discussion
16:36:15 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/372
16:36:38 <dustymabe> ok I'm re-using this ticket for "let's talk about the mechanics of switching to f32 for our testing/stable streams"
16:36:48 <dustymabe> I updated the ticket with our proposed timeline for switching to f32
16:37:16 <dustymabe> which IIUC means our next `testing` release is when we switch to f32
16:37:30 <jlebon> +1
16:37:51 <jlebon> i just linked https://github.com/coreos/fedora-coreos-config/pull/394 to that ticket
16:38:23 <dustymabe> any other things we need to consider?
16:38:34 <dustymabe> lorbus: how do we look from an OKD perspective ?
16:39:15 <lorbus> It should work out of the box as long as podman doesn't break :) We're still explicitly using cgroupsv1 there, too
16:39:50 <dustymabe> yes, we're still on cgroups v1
16:39:53 <lorbus> That explicit config will go away soon, but as long as we don't switch to cgroupsv2 with FCOS now, that won't be an issue
16:40:01 <dustymabe> lorbus: there is a iptables/nft change
16:40:24 <dustymabe> I'd feel much more comfortable if we could get you or vadim to try an OKD cluster on `next`
16:40:44 <lorbus> do you have a link? I thought there is a compat layer
16:40:45 <lucab> dustymabe: I still need to close on https://github.com/coreos/fedora-coreos-tracker/issues/468, I'll find some time before the end of the week
16:41:00 <dustymabe> with our currently proposed schedule we've got 3 weeks to fix any bugs we find
16:41:20 <dustymabe> lorbus: https://github.com/coreos/fedora-coreos-tracker/issues/372#issuecomment-588368597
16:41:28 <dustymabe> that links to the change proposal
16:42:02 <lorbus> +1 to the idea to test OKD on next
16:42:05 <lorbus> I'll see to that
16:42:33 <dustymabe> #action lorbus to try out OKD on our `next` stream so we can work out any kinks before switching `stable` to f32
16:43:00 <dustymabe> lorbus: vadim may have tried it already, so maybe work together with him on that Action Item
16:43:31 <lorbus> yep, definitely
16:43:55 <dustymabe> lucab: thanks! I think I need to respond upstream on that fstrim issue as well
16:44:18 <dustymabe> any other things we need to do or mechanics we need to discuss regarding the switch to f32?
16:44:36 <dustymabe> anyone here running next? did you rebase to it or did you start fresh?
16:45:05 <dustymabe> I'm running it for my IRC server
16:46:27 <dustymabe> I think we've also said this before.. we need to create an update barrier so that all upgrades from f31 -> f32 go through the same path
16:46:29 <dustymabe> correct ?
16:46:40 <jlebon> dustymabe: upgraded from f31? re. IRC server
16:46:48 <jlebon> yes, indeed
16:47:06 <jlebon> https://github.com/coreos/fedora-coreos-streams/issues/99#issuecomment-625291969
16:47:29 <dustymabe> jlebon: I think my irc server started as rebased from stable to next (f32), but I've redeployed it since
16:48:08 <dustymabe> jlebon: lucab: ok so we agreed to that barrier.. should we also limit the paths to f32 ?
16:48:08 <jlebon> yeah, that's probably the bit we should test the most -- upgrade testing
16:48:31 <dustymabe> so let's say the final f31 release on stable is A
16:48:50 <dustymabe> we know we will filter all previous releases of f31 to A before allowing them to upgrade to f32
16:49:07 <lucab> I did a bunch of reabses (including stable-testing-next), I only spotted the Zincati fragment out of place
16:49:08 <dustymabe> but will we allow for A->B A->C A->D
16:49:23 <dustymabe> or will we only allow one path from f31 to f32
16:49:28 <dustymabe> i.e. only A-B exists
16:49:40 <dustymabe> and then you can update B->C or B->D etc..
16:49:48 <lucab> dustymabe: only one, that's the barrier
16:50:31 <dustymabe> lucab: and if B isn't the latest release we still make them go through B?
16:50:58 <jlebon> lucab: hmm, interesting. is there a way to get the other behaviour if we wanted?
16:51:22 <lucab> not sure I follow
16:51:38 <dustymabe> lucab: what i'm talking about is a double barrier essentially
16:52:11 <dustymabe> let's say A is the barrier release (the last release of stable f31)
16:52:14 <jlebon> lucab: does a barrier affect both inbound and outbound edges, or just inbound?
16:52:31 <dustymabe> what paths are available to systems currently on A?
16:52:34 <lucab> jlebon: only inbound
16:53:19 <lucab> dustymabe: whatever are defined by further rollouts and barriers
16:53:24 <dustymabe> lucab: right
16:53:33 <jlebon> that's what dustymabe is asking :)   i'm not sure if we *need* to have that
16:53:39 <dustymabe> so i think jlebon and I are asking if we can control it such that there is only one available path
16:53:52 <dustymabe> so A->B is the only upgrade path that exists for A
16:54:05 <dustymabe> it would limit our upgrade testing matrix
16:54:30 <dustymabe> but not sure how practical it is
16:54:34 <lucab> we can, with two barriers
16:54:42 <dustymabe> lucab: right. ok that's what I was thinking
16:55:02 <dustymabe> whether we want to do that or not can be another discussion probably
16:55:18 <lucab> your other idea/approach makes a system who has to guess about future nodes
16:55:57 <dustymabe> the end result is what I was looking for.. creating two barriers achieves the goal
16:56:07 <dustymabe> ok any other things to discuss for moving to f32 ?
16:56:14 <lucab> (I can think about it a bit more, but I am not thrilled)
16:56:35 <dustymabe> lucab: yeah we may not want to do that, was an idea
16:57:00 <lucab> dustymabe: in general we shouldn't have barriers, unless we know we have to force a chokepoint for a migration script
16:57:25 <lucab> even this F31->F32 is not really needed
16:57:53 <dustymabe> lucab: but it's useful in that it allows us to have users follow a more tested path?
16:57:55 <lucab> it just makes the model easier to reason about for us humans
16:58:21 <dustymabe> i.e. we probably aren't going to spin up FCOS from january to test upgrading to f32
16:58:45 * jlebon has to drop off for other meeting, but overall i think i lean more towards not having a double barrier
16:58:50 <bgilbert> it might make sense for upgrade _testing_ to specifically test an F31 origin
16:59:00 <lucab> dustymabe: right, it trims the space of unknown things
16:59:04 <dustymabe> cool
16:59:05 <bgilbert> and then avoid barriers on the user side
16:59:05 <jlebon> we need to trust our CI testing, and expand it if we feel nervous
16:59:16 <dustymabe> i think we are all saying the same thing :)
16:59:23 <bgilbert> I don't think so?
16:59:40 <dustymabe> oh, k let me dig
16:59:53 <bgilbert> I tend to agree with Luca that barriers should exist for specific technical reasons
17:00:09 <dustymabe> bgilbert: "F31 origin" meaning the original gangsta ?
17:00:14 <dustymabe> the first release?
17:00:17 <lucab> bgilbert: the next CI run on `testing` will test that (but only from last F31 release)
17:00:28 <bgilbert> "origin" in the sense of the starting point for the upgrade test
17:00:36 <bgilbert> so probably one of the last F31 releases
17:00:59 <dustymabe> right, which is what I thought we were proposing.. introduce an update barrier so that we force users to follow the path that our CI tests
17:01:21 <bgilbert> I'm proposing that every future release should be upgrade-tested from F31
17:01:26 <bgilbert> and then we don't need a barrier
17:01:50 <dustymabe> bgilbert: I agree, but I think what you are saying is that we don't need a double barrier
17:01:59 <bgilbert> I'm saying we don't need a single barrier either
17:01:59 <dustymabe> we still need the single barrier
17:02:15 <cyberpear> bgilbert: do you want every F31 release, even from Jan to be able to upgrade to every single F32 release?
17:02:21 <bgilbert> yup!
17:02:23 <bgilbert> it's the more aggressive option, to be sure :-)  and if it turns out to be a disaster, we can use barriers for future releases
17:02:45 <bgilbert> but if we set the precedent, it'll only get harder to do barrierless upgrades later
17:03:03 <dustymabe> bgilbert: hmm. but what is the benefit of doing a barrierless upgrade in this case ?
17:03:14 <lucab> that's fair
17:03:26 <cyberpear> I guess the number of test runs would be "number of F31 FCOS releases in the wild" for each release
17:03:32 <bgilbert> barriers add extra friction
17:03:38 <bgilbert> cyberpear: I don't think we need to test everything -> everything
17:03:48 <lucab> dustymabe: less reboots for a new node created on an old release
17:03:49 <bgilbert> pick a representative F31 release and test upgrading from it
17:03:50 <cyberpear> just everything -> proposed
17:03:54 <dustymabe> I feel like it makes me feel better to know this upgrade path was tested for the f31 to f32 rebase
17:03:55 <walters> we're talking about unknown unknowns, but my instinct says that there aren't any bugs that would happen only when upgrading between an earlier version to latest
17:04:14 <bgilbert> cyberpear: no, F31 -> proposed, forever
17:04:15 <cyberpear> or "oldest F31" -> current and "newest F31" -> current
17:04:25 <bgilbert> cyberpear: sure
17:04:48 <lucab> bgilbert: I think the idea was more or less "representative == barrier"
17:04:56 <bgilbert> in my experience, upgrade bugs are path-dependent anyway
17:04:59 <walters> the cases that exist are likely in things like moby/podman stored containers&images but I think they need to deal with even old images in general anyways
17:05:21 <bgilbert> "this range of F31 releases wrote this file that we can't read anymore" type of thing
17:06:05 <bgilbert> so I guess what I'm saying is, I'm +1 to more CI, but think the barrier is too cautious
17:06:20 <bgilbert> the things that will break are not the things we think will break
17:07:03 <dustymabe> yeah I'm not sure where I stand
17:07:39 <dustymabe> seemed like a no-brainer to me. we've never had the ability to force an upgrade path before and match a known tested major upgrade path
17:07:55 <lucab> for this specific F31->F32, I do not disagree
17:08:16 <lucab> last time we used a barrier was because we needed a real migration script
17:08:17 <dustymabe> but I think you're probably right that it's not absolutely necessary and we'd probably be fine
17:08:51 <lucab> this time we are not aware of any, so there wouldn't be any strict need to
17:09:02 <bgilbert> we will hopefully never have fewer users than right now who might be broken if we try being aggressive
17:09:15 <bgilbert> so now's the time :-P
17:09:24 <cyberpear> normal Fedora upgrades don't go thru any barrier
17:09:39 <dustymabe> cyberpear: because they don't have that ability
17:09:41 <dustymabe> we do
17:10:06 <lucab> bgilbert: OTOH, I'm still scared by the "let's go back and fingerprint every grub since day 0"
17:10:36 <bgilbert> it worked, didn't it?  :-D
17:10:38 <bgilbert> but yeah
17:10:45 <bgilbert> ...or, put a different way, not using a barrier won't create a precedent, but using one will.
17:10:48 <lucab> (grub is a bad example here)
17:11:11 <dustymabe> bgilbert: i don't think we'll have to create a barrier in the future (say f32->f33) if we do use one now
17:11:45 <bgilbert> dustymabe: there will be psychological pressure to.  it'll be the safe, conservative approach.
17:11:57 <bgilbert> we'll have more users then than we did during 31->32, etc.
17:12:14 <bgilbert> we'll be considered more stable then than now
17:12:53 <bgilbert> I've talked too much here, I'll stop
17:12:56 <cyberpear> I vote for "no barrier unless known to be technically required"
17:13:24 <dustymabe> ok maybe let's pick up this discussion again here in the next week (or maybe we start a separate tracker issue to capture the discussion)
17:13:40 <lucab> bgilbert: I think I'm personally fine with establishing a "barrier between majors" rule
17:14:07 <lucab> let's move to GH, we have till next week to come up with a decision
17:14:27 <dustymabe> bgilbert: does "let's continue to have the discussion" sound good? so we can move on to other topics?
17:14:51 <bgilbert> sure
17:14:53 <bgilbert> one other thought I just had
17:15:01 <lucab> (we can also retroactively put a barrier if we need to)
17:15:03 <bgilbert> barriers, by their nature, force us to live with destination-side bugs forever
17:15:24 <bgilbert> i.e., if the target side of the barrier has a kernel bug that causes boot problems on some boxes
17:15:30 <bgilbert> we're stuck with it, or have to retarget the barrier
17:15:35 <cyberpear> can we re-write that barrier later?
17:15:39 <bgilbert> yes
17:16:13 <bgilbert> but in principle the barrier becomes an artifact in its own right that might require maintenance
17:16:17 <bgilbert> EOF
17:16:39 <dustymabe> #action dustymabe to create a ticket where we discuss appropriate update barrier approach for major upgrades
17:17:01 <dustymabe> #topic podman corner case bug: when do we backport fix problems ?
17:17:13 <dustymabe> #link https://github.com/containers/libpod/issues/5950
17:17:18 <dustymabe> ok so mini story here
17:17:24 <dustymabe> i'm running `next` on my irc server
17:17:39 <dustymabe> it starts a rootless podman container via systemd on boot
17:17:53 <dustymabe> that spawns weechat inside of tmux (too much detail)
17:18:20 <dustymabe> anywho - after the last upgrade to `next` my podman containers stopped working
17:18:35 <dustymabe> https://github.com/containers/libpod/issues/5950#issuecomment-625450333
17:18:48 <dustymabe> it only applies to running a rootless container via systemd
17:18:55 <dustymabe> so it's a bit of a corner case
17:19:15 <dustymabe> but i'm wondering if that is something we should consider backporting a fix for in the future
17:19:24 <dustymabe> so far we haven't had any users other than me report the issue
17:19:36 <bgilbert> does it only apply to `next`?
17:19:36 * dustymabe notes we should add a CI test for this specific problem
17:19:53 <dustymabe> bgilbert: good question. I think it's a podman 1.9 bug - so I think it applies to all our current streams
17:20:12 <bgilbert> is it a regression?
17:20:15 <dustymabe> but I need to confirm that
17:20:26 <bgilbert> (for us)
17:20:33 <dustymabe> bgilbert: yes, my system was working.. went down for upgrade and then stopped working
17:20:44 <bgilbert> whoops, sorry, so you said
17:21:09 <bgilbert> has it actually landed in the other streams?
17:21:22 <dustymabe> IIUC podman 1.9 is in all our streams right now
17:21:41 <dustymabe> it took us a 3 weeks to do a new release of `next`
17:21:58 <cyberpear> what do you mean by backport? -- can't we just push a regular update and include it in the next release?
17:21:59 <dustymabe> so I didn't catch the bug when it was in `testing`, but not yet in `stable`
17:23:08 <bgilbert> IMO container runtime regressions are the sort of fix we should prioritize
17:23:28 <dustymabe> bgilbert: right, i agree
17:23:39 <dustymabe> but it does slightly depend on the case
17:23:50 <dustymabe> so in this case it's only a problem if you're starting a rootless container via systemd
17:23:50 <bgilbert> with the current FCOS model (less priority on stability) I agree with cyberpear that we should tend to pick up new packages
17:24:03 <bgilbert> maaaaaybe an actual backport to `stable` but meh
17:24:15 <bgilbert> dustymabe: understood
17:24:17 <dustymabe> if I was running `testing` I would have caught this before it hit stable
17:24:27 <dustymabe> but I was trying to get some coverage on `next` so i missed it :(
17:24:38 <bgilbert> in CL I think we we would have rolled out-of-cycle releases on all channels
17:25:03 <dustymabe> so our options:
17:25:04 <bgilbert> (streams)
17:25:18 <dustymabe> my understanding is that podman is going to do a new release very soon with the fix in
17:25:50 <dustymabe> should we try to respin current testing with that new podman so that we can get stable fixed in the next round of releases (next week)?
17:26:09 <dustymabe> or do we just not do anything since no one has reported an issue ?
17:26:33 <dustymabe> it's hard for me to gauge if it's a real problem for people without having anyone complain about it:0
17:26:55 <bgilbert> dustymabe: if you give me a second I can solve your dilemma :-P
17:27:00 <cyberpear> I count 1 person who complained :P
17:27:06 <dustymabe> in the very least we *should* make sure the fix goes into the testing release that is cut next week
17:28:14 <dustymabe> bgilbert: :)
17:28:25 <bgilbert> all I can offer is my CL experience, which says: these things are judgment calls, and this feels like a case we care about
17:28:26 <bgilbert> complaints or no
17:28:37 <bgilbert> so I'd vote respin
17:28:47 <dustymabe> bgilbert: ok so you'd be a fan of respinning testing to get the new release into it?
17:28:48 <bgilbert> YMMV
17:28:49 <dustymabe> +1
17:28:50 <cyberpear> I'd also vote respin
17:28:56 <bgilbert> dustymabe: yup
17:29:16 <dustymabe> ok. i'll ping the podman guys on the release and try to respin testing
17:29:44 <dustymabe> #action dustymabe to get new podman release into testing release so we can fix stable in next weeks releases
17:29:54 <bgilbert> +1
17:29:58 <dustymabe> bgilbert: as part of that I will also 100% confirm that the bug does affect stable and testing
17:30:04 <bgilbert> cool
17:30:09 <dustymabe> #topic open floor
17:30:09 <bgilbert> also, better to exercise the machinery
17:30:37 <cyberpear> I think for F33, we should rebase "next" in time for F33 Beta, then have the first F33-based "stable" based on exactly F33 final content
17:30:55 <bgilbert> +1 to rebasing earlier, that was always the plan
17:31:10 <bgilbert> what's the benefit to stable based on exactly F33?
17:31:13 <lorbus> +1 to that
17:31:18 <bgilbert> it'd be two weeks after F33 lands, of course
17:31:44 <bgilbert> two+ weeks
17:31:54 <dustymabe> hmmm
17:32:10 <cyberpear> understood, just think would be good to have a very-well defined "point-in-time" snapshot, and that one's been pre-defined
17:32:14 <dustymabe> I think the delay we have could be shortened slightly but probably not by much (my opinion)
17:32:27 <dustymabe> +1 to having a next stream earlier
17:32:29 <bgilbert> "+1 to rebasing earlier" = the next stream, not testing/stable
17:32:33 <bgilbert> yup
17:32:34 <dustymabe> based on f33
17:32:38 <lorbus> we're lagging behind rawhide quite a bit, so the earlier we move the next stream to it, the better imo
17:32:58 <bgilbert> cyberpear: we're effectively a rolling distro, though?
17:33:36 <cyberpear> yes, but if we eventually needed a barrier or double-barrier, the GA content would make a good place for it, IMO
17:33:37 <bgilbert> if the exact package set matters, I feel like we're doing something wrong
17:33:47 <dustymabe> there are a lot of 0day fixes that land after f33
17:34:02 <dustymabe> honestly the GA content is mostly about "does what's delivered on the media work right"
17:34:09 <cyberpear> so let's define some FCOS release criteria and have them part of GA content?
17:34:55 <cyberpear> also, part of our artifacts are ISOs...
17:35:11 <dustymabe> cyberpear: we can certainly get some more hooks into the releng processses such that bugs that affect us are considered with higher priority
17:35:43 <bgilbert> I don't think what happened this cycle is representative
17:35:49 <bgilbert> we were running to catch up
17:35:53 <dustymabe> yep
17:35:57 <bgilbert> we'll get better at this
17:36:04 <cyberpear> yep, just trying to plan for the future
17:36:10 <bgilbert> +1 to hooks into releng where it makes sense.  also better CI.
17:36:16 <bgilbert> but: cyberpear, I'm not clear on what problem you're trying to solve
17:36:20 <dustymabe> cyberpear: yep, and you can help us with that too
17:36:35 <bgilbert> haphazard release process? something else?
17:37:46 <lorbus> I gotta drop. thanks all, and thanks for hosting dustymabe!
17:37:52 <cyberpear> "in 3 years, I want to go back and reproduce my system as it was on F33 FCOS" -- I know it's not a priority for most here, but having a release based on GA content would give it a good chance of succeeding, even w/ overlaid content
17:38:05 <dustymabe> lorbus++
17:38:05 <zodbot> dustymabe: Karma for lorbus changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:38:40 <bgilbert> cyberpear: the 33.20201105.3.0 release artifacts will still exist
17:38:46 <cyberpear> the GA content RPM set is kept forever; everything inbetween until EOL is discarded once there's an update made
17:38:47 <bgilbert> presuming you've saved the URLs :-/
17:38:56 <dustymabe> cyberpear: and the git history in the fcos configs have the exact rpm NVRAs
17:39:29 <bgilbert> cyberpear: and we protect those NVRAs from GC for some period of time
17:39:38 <cyberpear> good to know
17:39:52 <cyberpear> anyway, nothing actionable on this today, I think
17:39:54 <bgilbert> honestly I wouldn't trust that the release can be rebuilt from parts in 3 years
17:39:59 <walters> (and the source is always saved)
17:40:22 <bgilbert> even if you had the RPMs.  you'd pin to an old cosa, which might have who-knows-what bugs with your 3-year-newer kernel etc.
17:40:31 * dustymabe notes time
17:40:56 <cyberpear> bgilbert: that's why I'd like to see FCOS become part of the compose process so the GA artifices are also preserved along w/ the RPMs
17:41:02 * cyberpear also sees we're over time
17:41:05 <bgilbert> cyberpear: thanks for bringing it up though.  release processes _always_ need improvement :-)
17:41:05 <dustymabe> will end meeting in two minutes
17:41:28 <bgilbert> cyberpear: outside the FCOS bucket, you mean?
17:42:00 <cyberpear> I mean, have the F33-GA-based FCOS be sent to the mirror network, as if it were part of the GA compose
17:42:04 <dustymabe> the problem here is mostly that we don't have a specific GC contract for FCOS artifacts that users can rely on
17:42:10 <bgilbert> dustymabe: true
17:42:27 <bgilbert> cyberpear: I'm still really really hoping no one ever references our artifacts except from stream metadata
17:42:37 <bgilbert> continued use of old releases = bad
17:42:43 <bgilbert> meanwhile, in the real world...
17:42:47 <cyberpear> (hence, sending it to the mirror network, so it can take advantage of the existing processes in place)
17:42:59 <bgilbert> (which is why I wouldn't be happy about sending it out to mirrors)
17:43:12 <bgilbert> that one might be a losing battle though
17:43:21 <cyberpear> (yeah, real world I find myself occasionally needing a RHEL 5 VM or container, to test something out for someone who's stuck on it for some reason)
17:43:26 <bgilbert> yeah :-(
17:43:41 <dustymabe> #endmeeting