2025-01-30 17:30:39 <@tflink:fedora.im> !startmeeting fedora-ai-ml-sig
2025-01-30 17:30:39 <@meetbot:fedora.im> Meeting started at 2025-01-30 17:30:39 UTC
2025-01-30 17:30:39 <@meetbot:fedora.im> The Meeting name is 'fedora-ai-ml-sig'
2025-01-30 17:30:44 <@tflink:fedora.im> !topic welcome
2025-01-30 17:30:53 <@tflink:fedora.im> !hello
2025-01-30 17:30:54 <@zodbot:fedora.im> Tim Flink (tflink)
2025-01-30 17:31:03 <@man2dev:fedora.im> !hi
2025-01-30 17:31:04 <@zodbot:fedora.im> Mohammadreza Hendiani (man2dev)
2025-01-30 17:31:29 <@mystro256:fedora.im> !hi
2025-01-30 17:31:30 <@zodbot:fedora.im> None (mystro256)
2025-01-30 17:32:13 <@trix:fedora.im> !hi
2025-01-30 17:32:14 <@zodbot:fedora.im> Tom Rix (trix)
2025-01-30 17:32:22 <@xanderlent:fedora.im> !hi
2025-01-30 17:32:23 <@zodbot:fedora.im> Alexander Lent (xanderlent) - he / him / his
2025-01-30 17:32:57 <@tflink:fedora.im> welcome everyone, let's get this party started
2025-01-30 17:33:04 <@tflink:fedora.im> !topic npu in Fedora
2025-01-30 17:33:13 <@tflink:fedora.im> @trix take it away
2025-01-30 17:33:39 <@trix:fedora.im> yes, general discussion about npu this last week. it would be good to get into f43 and be a feature
2025-01-30 17:34:44 <@trix:fedora.im> not somethign i can do myself but maybe Alexander Lent can push forward 
2025-01-30 17:34:51 <@tflink:fedora.im> are there specific npus that you're talking about or is this a general thing?
2025-01-30 17:35:06 <@tflink:fedora.im> also, is an npu different from onboard gpus? I think they are
2025-01-30 17:35:07 <@trix:fedora.im> there is intel and amd's that i know of. maybe arm.
2025-01-30 17:35:33 <@tflink:fedora.im> I thought that amd's npu didn't work on linux, did that change?
2025-01-30 17:35:40 <@trix:fedora.im> part of microsoft's directML 
2025-01-30 17:35:59 <@trix:fedora.im> 6.14 i think will have support.
2025-01-30 17:36:12 <@tflink:fedora.im> cool, glad to be wrong on that one :)
2025-01-30 17:36:28 <@mystro256:fedora.im> has the code been accepted yet?
2025-01-30 17:36:35 <@mystro256:fedora.im> it was pending when I last checked
2025-01-30 17:37:20 <@trix:fedora.im> i _think_ david a. pushed it as part of drm-next, but i may be wrong.
2025-01-30 17:37:43 <@tflink:fedora.im> !info the hope is to get support for at least some NPUs into fedora in the f43 timeframe
2025-01-30 17:37:44 <@trix:fedora.im> F43 is 6 months away.
2025-01-30 17:38:29 <@trix:fedora.im> we are all about enabling hw that hasn't hit the streets!
2025-01-30 17:38:33 <@tflink:fedora.im> !info kernel support for amd npus is close if it isn't in 6.14
2025-01-30 17:38:40 <@xanderlent:fedora.im> The ones I know of are Intel's AI Boost NPU, AMD's XDNA NPU, the Qualcomm Hexagon NPU, and a few assorted ones found in ARM SoCs. 
2025-01-30 17:38:40 <@xanderlent:fedora.im> 
2025-01-30 17:38:40 <@xanderlent:fedora.im> tflink: These are discrete accelerators designed for accelerating ML/Tensor/Matrix ops generally. 
2025-01-30 17:39:40 <@tflink:fedora.im> !info NPUs are discrete accelerators designed for accelerating ML/Tensor/Matrix ops but not full GPUs. Examples of this include Intel's AI Boost NPU, AMD's XDNA NPU and Qualcomms Hexagon NPU
2025-01-30 17:40:23 <@tflink:fedora.im> I assume that qualcomm hasn't open sourced any support for their NPU, is that an option for Fedora?
2025-01-30 17:40:25 <@xanderlent:fedora.im> The goal seems to be efficient offloading of various ML math tasks; for the first-gen Intel NPU in Meteor Lake, Intel notes that the GPU is strictly faster but the NPU consumes less power.
2025-01-30 17:41:02 <@xanderlent:fedora.im> Last I heard, it was much like the rest of the Qcom Snapdragon X platform: Linux support is coming soon, when they get around to it
2025-01-30 17:41:30 <@tflink:fedora.im> is intel's ML stack in a shape where it can use NPUs in Fedora?
2025-01-30 17:41:59 <@tflink:fedora.im> ie, is it reasonable to plan for anything other than AMD's XDNA for f43?
2025-01-30 17:42:27 <@xanderlent:fedora.im> I think so, at least if we package the Intel NPU Level Zero driver and OpenVINO, people will be able to use models through OpenVINO.
2025-01-30 17:42:54 <@tflink:fedora.im> that might be a tall order, though. unless the stack has improved since I last talked to folks about it
2025-01-30 17:43:29 <@trix:fedora.im> i believe david said npu != oneapi
2025-01-30 17:43:50 <@trix:fedora.im> same a xdna != rocm
2025-01-30 17:44:16 <@tflink:fedora.im> ah, ok. if it doesn't have the same kernel module and llvm problems that oneapi has, my concern may be incorrect
2025-01-30 17:44:50 <@xanderlent:fedora.im> On the Intel side OpenVINO uses the Level Zero API to talk to the NPU driver, but other than the loader it's an entirely different driver implementation than Intel's GPU Level Zero, and it doesn't use SYCL at all.
2025-01-30 17:45:12 <@xanderlent:fedora.im> Correction: involve SYCL at all
2025-01-30 17:45:50 <@xanderlent:fedora.im> From my perspective it kind of seems like they shoehorned the npu operations into level zero since that was the hot new API at Intel at the time. They wrote a whole set of custom extensions to the API that do most of the neural related ops.
2025-01-30 17:46:33 <@trix:fedora.im> reaching for the antacids :|
2025-01-30 17:46:39 <@tflink:fedora.im> but it sounds like the software and kernel interfaces needed to make intel's NPU work already exist upstream?
2025-01-30 17:47:28 <@xanderlent:fedora.im> Yes, kernel module has been upstream since I think 6.6?
2025-01-30 17:47:33 <@tflink:fedora.im> cool
2025-01-30 17:47:58 <@tflink:fedora.im> !info the software needed to make Intel's NPU should exist in upstreams already
2025-01-30 17:48:24 <@tflink:fedora.im> is the proposal to add NPU support to pytorch? or something like vllm/ollama/something else I'm not thinking of?
2025-01-30 17:48:45 <@man2dev:fedora.im> onnx ?
2025-01-30 17:49:12 <@man2dev:fedora.im> I think that has npu support too
2025-01-30 17:49:28 <@trix:fedora.im> if onnx is possible, that would good, it is a more vendor neutral 
2025-01-30 17:50:28 <@xanderlent:fedora.im> I haven't yet looked into ONNX and the Intel NPU.
2025-01-30 17:51:47 <@xanderlent:fedora.im> I'm not sure I can get the Intel NPU pytorch support ready for F43, it will require some Python and Rust deps to be packaged.
2025-01-30 17:51:47 <@xanderlent:fedora.im> 
2025-01-30 17:51:47 <@xanderlent:fedora.im> As for pytorch, support is through a custom set of Python libraries that go on top of/alongside pytorch. Namely intel-npu-acceleration-library and IPEX or ipex-llm.
2025-01-30 17:52:22 <@trix:fedora.im> ipex ? intel pytorch extension ?
2025-01-30 17:52:56 <@tflink:fedora.im> so we're looking at pytorch and/or onnx for intel but that might need to wait. did I understand correctly?
2025-01-30 17:53:04 <@trix:fedora.im> yeahhh, we don't have intel gpu in basic pytorch yet.
2025-01-30 17:53:09 <@tflink:fedora.im> what about for AMD? is there a new non-rocm stack?
2025-01-30 17:53:23 <@man2dev:fedora.im> is this it https://github.com/intel/ipex-llm
2025-01-30 17:53:59 <@trix:fedora.im> yes, i look at this when i was a hatter.
2025-01-30 17:57:05 <@tflink:fedora.im> I might be more dense than usual today but I'm still having trouble figuring out exactly what the proposal is for or if it's reasonable to expect it to happen for F43.
2025-01-30 17:57:42 <@tflink:fedora.im> I wonder if it would be best to save more discussion until the next meeting and start a discourse thread or wiki page to start hashing out the details before the next meeting
2025-01-30 17:58:00 <@trix:fedora.im> yes. i need to drop sooo..
2025-01-30 17:58:47 <@tflink:fedora.im> if that works, do folks want to do wiki or discourse?
2025-01-30 17:59:41 <@tflink:fedora.im> or something else, I don't have strong feelings. if nobody has preferences, I'll start a discource thread after the meeting
2025-01-30 18:01:20 <@man2dev:fedora.im> I tried updating maturin to 1.8.1 for python-safetensors was not very successful though 
2025-01-30 18:01:20 <@man2dev:fedora.im> oh small note:
2025-01-30 18:01:51 <@xanderlent:fedora.im> tflink: I think part of the problem with these NPU stacks, at least on Linux, is that they are all pretty custom at this point.
2025-01-30 18:01:51 <@xanderlent:fedora.im> 
2025-01-30 18:01:51 <@xanderlent:fedora.im> I'm not sure there is currently a single high-level API we could ship that covers every npu device. Microsoft is working on DirectML on their side, but support for that is still experimental on a per vendor basis, even on current Windows.
2025-01-30 18:02:29 <@tflink:fedora.im> you could say the same thing about all the scientific stacks, honestly
2025-01-30 18:02:42 <@tflink:fedora.im> not quite to the same extent but it's definitely a thing :)
2025-01-30 18:03:39 <@tflink:fedora.im> anyhow, if there are no objections, I'll take the action item to create a discourse thread to hash out the "NPU for F43+" details
2025-01-30 18:04:25 <@tflink:fedora.im> !action tflink to start a thread on discourse to work out the details on NPU support in Fedora
2025-01-30 18:05:21 <@music:fedora.im> Mohammadreza Hendiani: Yeah, maturin is a large-ish project and usually requires some work on its dependencies for a new version. But - why 1.8.1? I only see `requires = ["maturin>=1.0,<2.0"]` glancing at https://github.com/huggingface/safetensors/blob/f05a1ec483c470202a3413772eef8eab6c3a8ba0/bindings/python/pyproject.toml#L92.
2025-01-30 18:05:55 <@tflink:fedora.im> anything else on the NPU topic? it seems like some folks have moved on to a new topic
2025-01-30 18:06:05 <@man2dev:fedora.im> yeah its a really hard package to build
2025-01-30 18:06:48 <@tflink:fedora.im> is there more on the maturin topic?
2025-01-30 18:07:27 <@man2dev:fedora.im> if i recall it was the minimal dependency that python-safetensors needed
2025-01-30 18:07:41 <@tflink:fedora.im> ... is that a yes?
2025-01-30 18:08:02 <@tflink:fedora.im> !topic maturin packaging
2025-01-30 18:08:03 <@man2dev:fedora.im> no
2025-01-30 18:08:05 <@man2dev:fedora.im> im done
2025-01-30 18:08:10 <@tflink:fedora.im> !undo
2025-01-30 18:08:28 <@tflink:fedora.im> there are no other topics on the agenda so it's time for
2025-01-30 18:08:31 <@tflink:fedora.im> !topic open floor
2025-01-30 18:08:37 <@tflink:fedora.im> anything else that folks would like to bring up?
2025-01-30 18:11:41 <@man2dev:fedora.im> music: Oh i remembered why it was 1.8 it just happened to be the last stable release
2025-01-30 18:13:23 <@tflink:fedora.im> ok, thanks for coming, everyone
2025-01-30 18:13:32 <@tflink:fedora.im> !endmeeting