16:00:12 <sgallagh> #startmeeting ELN (2023-06-02)
16:00:12 <zodbot> Meeting started Fri Jun  2 16:00:12 2023 UTC.
16:00:12 <zodbot> This meeting is logged and archived in a public location.
16:00:12 <zodbot> The chair is sgallagh. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:12 <zodbot> The meeting name has been set to 'eln_(2023-06-02)'
16:00:12 <sgallagh> #meetingname eln
16:00:12 <zodbot> The meeting name has been set to 'eln'
16:00:12 <sgallagh> #topic init process
16:00:12 <sgallagh> .hi
16:00:13 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
16:00:40 <bstinson> .hi
16:00:41 <zodbot> bstinson: bstinson 'Brian Stinson' <brian@bstinson.com>
16:02:35 <sgallagh> Hmm, not many people turning up today.
16:04:15 <bstinson> quiet friday
16:04:32 <sgallagh> Seems so.
16:04:55 <sgallagh> odd, though. I know of at least two people who indicated they'd be coming for today's topic
16:05:28 <sgallagh> I'll give folks a few more minutes, then I guess we can try again at the next meeting
16:05:36 <bstinson> that works
16:06:05 <tdawson> Hi, I'm here too
16:06:20 <tdawson> It might be the F38 release party
16:07:25 <sgallagh> Ah, that's a fair point.
16:07:43 <davide> .hello dcavalca
16:07:44 <zodbot> davide: dcavalca 'Davide Cavalca' <davide@cavalca.name>
16:07:52 * davide was distracted by the release party :)
16:09:03 <sgallagh> I'll kick things off at 10 past the hour, but I suspect it will be a brief meeting
16:10:08 <sgallagh> #topic x86_64 baseline and multiarch support
16:10:33 <sgallagh> We have a special guest today to discuss this topic: the inimitable Brian Stinson.
16:10:37 <yselkowitz[m]> .hello yselkowitz
16:10:37 <sgallagh> Take it away, Brian!
16:10:38 <zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
16:11:26 <bstinson> hey! so we'd really like to get rid of 32bit packages, because the year 2038 is coming a lot quicker than some of us would like
16:11:53 <bstinson> we aren't disabling that arch in the buildsystem yet because there's a lot to do
16:12:52 <davide> I can tell you we have folks internally that still need and use 32 bit packages
16:13:04 <bstinson> I put in a MR to the compose configs, though, with the intent to remove multilib: https://pagure.io/pungi-fedora/pull-request/1175
16:13:14 <bstinson> (it didn't take for the latest compose, though)
16:13:17 <davide> last time I looked it was folks doing android development, android OS development and embedded stuff with Yocto
16:13:33 <bstinson> davide: interesting, some data on that would be great
16:13:45 <davide> there's also a non-trivial amount of garbage-quality vendor software that still needs 32 bit packages unfortunately
16:13:47 <bstinson> i plan on opening a topic on Fedora discord to discuss this
16:14:01 <davide> yeah, I can dig it up, last time I did this was a couple of years ago
16:14:16 <sgallagh> bstinson: What do you mean, "it didn't take"?
16:14:17 <bstinson> oh there is certainly some relationship work to do with various vendors
16:14:24 <sgallagh> (I haven't looked, yet)
16:14:30 <bstinson> sgallagh: i'm seeing i686 packages in today's compose
16:14:39 <davide> is the plan to shop shipping 32 bit packages for C10 ?
16:14:39 <bstinson> but i haven't had the chance to dig further yet
16:14:45 <davide> * stop shipping
16:14:55 <bstinson> davide: yes the plan is to stop shipping 32bit packages for CentOS Stream 10
16:15:07 <sgallagh> bstinson: When did you look? Because the compose prior to an hour ago were from two days ago
16:15:14 <sgallagh> (Thanks to the outages)
16:15:20 <smooge> davide most of the 32 bit stuff on android and yocto was arm based versus i686
16:15:26 <bstinson> sgallagh: i'm looking at this compose: https://odcs.fedoraproject.org/composes/production/Fedora-ELN-20230602.1/compose/metadata/
16:15:41 <sgallagh> OK
16:15:49 <davide> oh we definitely had folks doing that on i686 unfortunately
16:16:01 <davide> but I can definitely get more current data and get back to you by the next meeting
16:16:06 <bstinson> i might need to do something with the multilib blacklists in case pungi is caching something
16:16:17 <sgallagh> hmm, ok
16:16:34 <sgallagh> Davide Cavalca: Thanks, that would be helpful
16:16:46 <davide> bstinson: ok my request here would be to announce this publicly and loudly as early as possible
16:16:53 <bstinson> certainly
16:16:56 <davide> as I expect a lot of folks will have to do damage control
16:17:09 <sgallagh> Yep. This meeting is Step 1 of that
16:17:48 <bstinson> tell me more about the damage that you foresee
16:18:15 <bstinson> (announcing publicly and loudly is indeed the intent behind doing things this way)
16:18:36 <davide> in most instances I think it'll be folks that don't realize they depend on 32 bit stuff, or worse that have vendor deps they can't change that depend on 32 bit stuff
16:19:13 <davide> in my specific case (and in HPC / research settings I expect) it'll be "we still need to do builds for this horrible thing that was released six years ago and has 32 bit deps in it"
16:20:32 <davide> the other thing I recommend thinking about: if someone were to propose a CentOS SIG to do 32 bit builds, would that be possible/ok?
16:20:43 <smooge> it is not easy
16:20:56 <davide> to be clear, this is not something I'm planning to do, but I suspect someone might want to, so it'd be good to think through the scenario
16:21:11 <bstinson> i think folks would find that tedious, but it's not something we'd block
16:21:12 <smooge> this was looked in Fedora at one point and needs a lot of 'extra' stuff
16:21:26 <davide> bstinson: oh, also, this will break wine I think?
16:21:31 <bstinson> not any more!
16:21:56 <bstinson> that was one case we looked at, wine now bundles it's 32bit layer and doesn't require multilib
16:22:03 <sgallagh> bstinson: "Not any more" in response to which comment?
16:22:34 <Eighth_Doctor> .hello ngompa
16:22:35 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
16:22:42 <bstinson> sgallagh: wine no longer requires OS multilib for its 32bit functionality
16:23:04 <davide> bstinson: my other question here is what's the plan for Fedora? do you/someone plans to put up a Change for F40 to drop 32 bit packages?
16:23:04 <sgallagh> That's surprising, but not unpleasantly so
16:23:16 <Eighth_Doctor> it still needs several components in 32-bit in order to do things
16:23:32 <davide> because that would break Steam I think, which will probably make it a non-starter for most Fedora users
16:23:35 <bstinson> davide: we have no plans to influence fedora here
16:23:38 <Eighth_Doctor> it is something they're working on, but they are far from done
16:23:44 <smooge> davide: one issue I found in science stuff and 3rd party 32bit apps is that they tend to 'break' if you aren't also using all the libraries and tools from around the time it was built. AKA they worked better if you just kept it to say EL5 versus trying to keep updating it to ELN
16:24:15 <davide> yeah that's one option, and probably what I'd end up doing for our stuff as well if push comes to shove
16:24:52 <Eighth_Doctor> the biggest damage is that VST for audio production will break
16:25:11 <Eighth_Doctor> since a lot of them are 32-bit, whether they're Linux native or bridged through Wine
16:25:14 <bstinson> there are lifecycle options available that still include 32bit libraries if folks need them (some of them will still be in support for many years!)
16:25:15 <davide> oh damn I completely forgot about that
16:25:35 <smooge> personally i think if you need 32 bit stick to EL8/El9
16:25:45 <Eighth_Doctor> lots of people got burned when macOS killed 32-bit in the audio space
16:25:51 <bstinson> smooge's recommendation is a good one
16:26:22 <Eighth_Doctor> well, it does nobody any good unless you have some way to influence the big users of 32-bit to do something
16:26:28 <smooge> if you need newer stuff start making containers of the thing you built it with
16:26:45 <yselkowitz[m]> can we limit the packages which are built for 32-bit though, to save time and resources?
16:26:57 * Eighth_Doctor wishes we had a 32-on-64 thunking system
16:27:09 <davide> I just ran a query on my production fleet, and https://paste.centos.org/view/b27c55f0 are the i686 arch packages I see installed
16:27:36 <smooge> in Fedora, I have no idea what the plan is.. I expect that they will still want it working in 2040 with some builders set to negative time to keep it working
16:27:38 <davide> (ballkpark, this dataset isn't super accurate)
16:27:48 <bstinson> i should mention: one thing we are doing is work in the toolchain to allow building true 32bit binaries (think bootloaders) in a 64bit buildroot
16:27:52 <yselkowitz[m]> ...either by being more proactive wrt https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval, or by using a special target for those packages which actually need 32-bit, while the default target omits it?
16:28:03 <sgallagh> "thunking?
16:28:13 <Eighth_Doctor> Stephen Gallagher: sorry, that shows me being old
16:28:26 <bstinson> davide: of the surveys i've done, lots of folks that still have 32bit stuff on their fleets cargo-culted install scripts from older releases
16:28:28 <Eighth_Doctor> I remember 32-on-16 and 16-on-32 from the Windows 3.x and Windows 95 days
16:28:52 <davide> this is stuff actively used (we have logic to reap it if it isn't)
16:29:28 <Eighth_Doctor> the idea is that you transparently convert symbol and pointer calls so that things that ordinarily use 32-bit stuff would use 64-bit variants
16:31:28 <bstinson> but data like this is really helpful, especially if the actual workload can be identified
16:31:40 <davide> from a cursory look, this stuff is used by 1) interactive development systems 2) build infra for android OS stuff 3) build infra for mobile phone apps 4) build infra for fpga stuff 5) build infra for chromium
16:32:03 <Eighth_Doctor> Android is ripping out 32-bit support with the next version, so that shouldn't be a problem
16:32:06 <davide> plus some other stuff I can't name in a public channel
16:32:23 <bstinson> this is all in scope for the discussion on discourse, but if any other folks don't feel comfortable discussing in the open you can contact me at bstinson@redhat.com
16:32:26 <Eighth_Doctor> Chromium is made of evil, but is mostly 64-bit these days
16:32:41 <davide> Conan Kudo: the problem is that when you build hardware you have to support released products, and those are gonna be stuck on some ancient AOSP
16:32:42 <yselkowitz[m]> "mostly"?
16:32:57 <Eighth_Doctor> Davide Cavalca: yeah I know
16:33:04 <Eighth_Doctor> believe me, I know 😦
16:33:47 <Eighth_Doctor> yselkowitz: yes, "mostly"... though hopefully it doesn't matter for us
16:33:48 <Eighth_Doctor> the big ones I know of are Wine and audio production
16:33:48 <Eighth_Doctor> Wine is further along than audio production
16:33:58 <Eighth_Doctor> audio is sadly a huge 32-bit environment
16:35:06 <bstinson> what other questions do we have about the 32bit change?
16:36:15 <Eighth_Doctor> are you planning to do anything to push companies to stop depending on 32-bit things?
16:36:32 <Eighth_Doctor> because this is going to be extremely painful if you don't
16:36:47 <Eighth_Doctor> and "staying on old RHEL" isn't an acceptable answer for most people, they'll just move away from RHEL entirely
16:36:53 <bstinson> yes Red Hat is working with some of our partners on this to start
16:36:55 <sgallagh> Arguably, disabling 32-bit support is a pretty strong push
16:37:08 <Eighth_Doctor> well, no, that's just a destructive change
16:37:18 <Eighth_Doctor> that doesn't mean anything if you're not doing some ecosystem work too
16:37:45 <smooge> i think it should be a 'we are going to offer you a 32 bit environment for 100x what you pay for the 64bit environment'
16:37:47 <davide> yeah I don't know how many folks do embedded development using RHEL/CentOS as a host, but that kind of work will become much more painful
16:37:53 <Eighth_Doctor> the Valve/Ubuntu situation is something we should avoid in the RH ecosystem
16:38:19 <Eighth_Doctor> we're already not doing as well as we could be ecosystem wise, let's not make it worse
16:38:26 <davide> smooge: one possible suggestion here would be to offer an unsupported repo with a very small subset of 32 bit package builds for c10
16:38:31 <davide> and then drop it altogether for c11
16:39:20 <smooge> the issue is that it isn't easy to do. There is an increasingly amount of 'this does not compile on 32bit' for core libraries
16:39:27 <davide> I'm not sure if the concern here from the RH side is support or build-side workload; if it's the latter this obviously wouldn't help
16:39:40 <smooge> which then take some patching and work to make
16:39:52 <bstinson> not really, because keeping 32bit support for Red Hat implies keeping 32bit support and maintaining ABI with RHEL8 possibly for up to 13 years after RHEL 10 goes GA
16:39:55 <sgallagh> Also, let's not leave out the 2038 problem
16:40:56 <bstinson> basically what sgallagh said. we have to think about the long tail of RHEL 10, not just the beginning here
16:41:05 <Eighth_Doctor> that's not entirely coupled with 32-bit since you can use 64-bit time_t with 32-bit APIs, but there are ABI concerns
16:41:15 <davide> yeah that's fair; please make sure you word the reasoning explicitly when you announce this if you can though
16:41:25 <smooge> davide it is that reason I think that the solution will be 'keep an EL-8 environmnet and patch it yourself'
16:41:44 <davide> yeah I don't think you can work around the ABI issue, that's totally fair
16:42:54 <Eighth_Doctor> I personally don't have an issue other than games and audio production
16:43:17 <Eighth_Doctor> fortunately it seems most native gamedev is 64-bit these day
16:43:20 <Eighth_Doctor> *days
16:43:23 <davide> Conan Kudo: would it be worth to loop in the VFX folks early on this?
16:43:25 <smooge> for the people I have known doing 32bit, they are still on EL7 because EL8/EL9 already didn't have enough 32bit for them
16:43:26 <Eighth_Doctor> yes
16:43:30 <Eighth_Doctor> Davide Cavalca: absolutely
16:43:43 <smooge> the others I know are Debian people
16:43:53 <Eighth_Doctor> the creative industry likely doesn't know how much 32-bit they depend on
16:44:10 <bstinson> that's something we're working on from the Red Hat side
16:44:41 <davide> alright, I'll do a more detailed survey on my end, and try to give you some specifics around areas of impact (and potentially impacted vendors, if I can find that out)
16:44:57 <davide> hopefully that helps with planning
16:45:02 <bstinson> it does very much
16:45:12 <Eighth_Doctor> and probably talking to Codeweavers about CrossOver on a pure 64-bit RHEL would be a good idea
16:45:19 <davide> really appreciate the early heads up here, knowing this now makes my life a lot easier
16:45:31 <Eighth_Doctor> I don't know if they're a Red Hat partner, but regardless, it's worth talking to them
16:45:33 <bstinson> and i'll point out that this is exactly what we hope to do by putting things through Fedora ELN and CentOS Stream early
16:46:59 <bstinson> anything else for 32bit, or should we check in on x86_64-v3 ?
16:48:56 <bstinson> x86_64-v3!
16:49:27 <bstinson> This is happening too, in fact the baselines have already been changed. I expect to do another discourse thread on this one to gather similar feedback
16:50:02 <bstinson> i've heard a few concerns about this, but i wanted to check with the group here to see what we think the impact is on ourselves
16:50:46 <davide> I'm pretty sure we still have some hardware that doesn't meet that baseline
16:51:27 <davide> I'd need to check the specifics on this as well
16:51:50 <bstinson> i'm curious what hardware refresh cycles are like these days
16:51:57 <davide> the only scenario I'm mildly concerned about is systems deployed in locations that make them difficult to replace
16:52:10 <davide> (think about edge caches or cache boxes in ISP cages)
16:53:39 <bstinson> yeah, i'm thinking again about long-tail hardware refresh cycles
16:54:40 <davide> I can tell you in my environment we have some pretty gnarly long tail hardware
16:55:22 <davide> especially as systems often get repurposed for other uses throughout their life (e.g. production systems might become development servers after a refresh)
16:55:50 <yselkowitz[m]> ** 5 minute warning **
16:56:11 <davide> and some things are especially painful to cycle (e.g. rack switches and other networking stuff, or the things at the edge that I mentioned earlier)
16:56:49 <davide> I can't find a dataset to check arch baselines right now, but I'll track this down as well and gauge what the impact would be
16:58:16 <smooge> to be honest this again is where 'you can run something old on that til 2030'
16:58:26 <bstinson> to give a similar rationale on this, remember that we're trying to build out what we think hardware support will look like in a couple of years time
16:59:44 <bstinson> and then hold it for 13+ years
16:59:50 <davide> smooge: this is a bit different than the 32 bit case, as for 32 bit you can just containerize, while in this case you'd be stuck supporting an old OS release train just for these systems
17:00:04 <davide> but then again, it might well turn out to be a non issue with the timeline we're looking at
17:01:53 <davide> anyway, I think we're out of time
17:01:59 <bstinson> similar call to action here, i'd love to talk (on discourse or in private) with anyone who will tell me things about how their hardware cycles work
17:02:32 <bstinson> links will go in the #fedora-eln channel when those are up, and we'll mention them at the next meeting too
17:03:54 <sgallagh> Thanks, Brian.
17:04:06 <sgallagh> Unfortunately, we are now over time for this meeting.
17:04:16 <davide> sounds good, thank you
17:04:24 <sgallagh> Thanks everyone who came.
17:04:46 <sgallagh> #action bstinson to create Discourse discussion threads for the x86_64-v3 and 32-bit multilib topics.
17:04:55 <sgallagh> #endmeeting