Understanding NPUs with OpenVINO: Real Capabilities, Limitations & ML Use Cases
OpenVINO™ is Intel’s official toolkit for optimizing and deploying AI models efficiently across multiple hardware targets including CPU, GPU, and NPU. The toolkit is widely used in edge, desktop, and cloud environments to accelerate inference for a variety of model types.
This article dives deeper into the often-asked questions:
- What can an NPU really do?
- Can it train models or only run inference?
- Which models are good fits and which are not?
I've combined official OpenVINO documentation with hands-on experience testing Hugging Face models to provide a realistic view of the state of NPUs in 2025.
What Is Intel’s NPU?
The Neural Processing Unit (NPU) in Intel systems, such as Intel Core Ultra processors, is a hardware accelerator designed to offload neural network inference tasks from the CPU and GPU. Official OpenVINO documentation lists the NPU alongside CPU and GPU as a supported inference device.
The NPU device is supported directly through OpenVINO’s runtime and is enabled via a dedicated plugin.
How NPUs Fit into OpenVINO’s Inference Stack
OpenVINO supports running models on the following devices:
- CPU
- GPU
- NPU
Developers can explicitly select devices such as "CPU", "GPU", or "NPU", or use "AUTO" and heterogeneous execution modes where parts of a model are scheduled across different devices for optimal performance.
This hardware flexibility is powerful, but it comes with important constraints, particularly on the NPU.
Example: Running a Small Language Model with OpenVINO
The following example demonstrates how to run a small language model using OpenVINO and Hugging Face Transformers. This code is useful for validating an OpenVINO installation and observing device behavior in practice.
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
model_name = "distilgpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = OVModelForCausalLM.from_pretrained(
model_name,
export=True,
device="AUTO" # CPU/GPU/NPU selection handled by OpenVINO
)
inputs = tokenizer("Hello from OpenVINO!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
In Practice: What We Observed
When testing language-generation models such as distilgpt2 and LLaMA, the following behavior was observed:
- Running inference on CPU works
- Running inference on GPU works
- Running inference on NPU fails with compiler errors, typically related to missing upper bounds on tensor shapes
This occurs because models like GPT and LLaMA rely on dynamic shapes and autoregressive generation. Current NPU compilers require shapes to be fixed and known at compile time, a limitation repeatedly highlighted in Intel forums and OpenVINO documentation.
In summary:
NPUs are highly efficient at executing fixed-shape graphs but struggle with models that require dynamic computation and adaptive sequence lengths.
Why Dynamic Shapes Matter
Many state-of-the-art models, especially language models, rely on dynamic input and output shapes. Sequence length changes at runtime and cannot be known ahead of time.
While OpenVINO supports dynamic shapes on CPU and GPU, the NPU compiler requires static shapes in order to generate optimized execution graphs.
Official OpenVINO discussions confirm that Hugging Face models often include dynamic shapes by design. This prevents them from running on NPUs unless they are reshaped to static sizes. Currently, no official workflow exists for converting generic Hugging Face language models into static-shape, NPU-compatible graphs.
What NPUs Do Well
Static Inference Workloads
NPUs perform best when the following conditions are met:
- Input shapes remain constant
- The model graph is fixed
- No autoregressive loops are present
Examples of supported and well-performing scenarios include:
- Image classification models such as ResNet and MobileNet
- Object detection models such as YOLO variants
- Fixed-input vision tasks such as image segmentation
- BERT-style transformer models with fixed sequence lengths
Although most Hugging Face LLMs use dynamic shapes by default, many vision and classification models can be compiled and executed successfully on the NPU once converted to OpenVINO IR format.
Model Support and Verification
Official OpenVINO model compatibility lists include multiple models verified to run on the NPU through supported workflows.
The key takeaway is:
Models that require dynamic sequence lengths or autoregressive output generation are not currently supported on NPUs through OpenVINO Python workflows.
Hands-On Testing Confirmations
Practical experiments confirm the following:
- Hugging Face models using
generate()trigger NPU compiler errors - CPU and GPU inference works reliably for dynamic LLMs
- Static-shape models such as image classifiers execute correctly on NPU
- Converting dynamic language models to static shapes is non-trivial and lacks official tooling
These findings align with Intel Community discussions highlighting the absence of official tutorials for reshaping Hugging Face LLMs for NPU execution.
Where NPUs Shine
Vision Models
NPUs provide significant performance and efficiency gains for vision workloads once models are converted to OpenVINO format and compiled with fixed batch sizes.
BERT and Embeddings
Transformer models designed around fixed sequence lengths, such as BERT with padded inputs, can often be executed successfully on the NPU after proper preprocessing.
Heterogeneous Inference
OpenVINO supports heterogeneous execution where different subgraphs of a model can be assigned to different devices. For example, convolutional feature extraction layers can run on the NPU while control-heavy logic remains on the CPU.
Where NPUs Are Still Limited
Generative Language Models
Dynamic prompts, autoregressive decoding, and key-value cache handling prevent direct execution of GPT-style language models on NPUs in standard Python workflows using Transformers, Optimum, and OpenVINO.
This limitation has been consistently observed across community experiments and discussions.
Training
NPUs are inference-only devices. Training workloads requiring gradient computation, backpropagation, and optimizers are not supported.
Realistic Expectations for 2025
OpenVINO continues to evolve with improvements in generative model support, batching behavior, and device integration. Recent release notes highlight:
- Expanded generative AI tooling in OpenVINO
- Improved batch handling for NPUs
- Enhanced device integrations, including support for Triton Inference Server
These developments suggest gradual expansion of NPU capabilities. However, fully dynamic LLM inference on NPUs remains a specialized and advanced workflow.
Final Thoughts
Intel’s NPU with OpenVINO in 2025 is a capable inference accelerator that enables:
- Static vision and classification models
- Efficient non-autoregressive neural networks
- Multi-device orchestration through heterogeneous execution
However, it does not yet support fully dynamic LLM generation workflows out of the box.
Understanding these strengths and limitations allows developers to design realistic AI pipelines that balance performance, efficiency, and flexibility.