Skip to content

[NPU] Support NPUW for text-embedding models #3088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[NPU] Support NPUW for text-embedding models #3088

wants to merge 3 commits into from
+169 −55

Conversation

@mengweiguo
Copy link
Contributor

@mengweiguo mengweiguo commented Dec 1, 2025 •
edited
Loading

Description

The benefits handled by prefill-chunk in NPUW:

  • Support long context
  • Performance improvement

Note:

  1. NPUW PR: [NPUW] Support prefill-chunk for text-embedding model openvino#33076
  2. The tests has been verified to work with both NPUW and GenAI updates.

CVS-177453

Checklist:

  • Tests have been updated or added to cover the new code.
  • This patch fully addresses the ticket.
  • I have made corresponding changes to the documentation.

Copilot AI review requested due to automatic review settings December 1, 2025 07:21
@github-actions github-actions bot added category: llm_bench Label for tool/llm_bench folder no-match-files category: RAG RAG pipeline components labels Dec 1, 2025
Copilot AI reviewed Dec 1, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for NPUW (Neural Processing Unit Workload) optimization for text embedding models, enabling long context support and performance improvements through prefill-chunk handling.

Key changes:

  • Added NPU-specific compilation path for text embedding models with dynamic shapes
  • Introduced new configuration parameter emb_pad_to_max_length to control padding behavior
  • Refactored NPU compilation logic to support both LLM and text embedding model types

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tools/llm_bench/llm_bench_utils/ov_utils.py Fixed parameter name and moved padding configuration to support new padding control
tools/llm_bench/llm_bench_utils/model_utils.py Added mapping for new emb_pad_to_max_length parameter
tools/llm_bench/benchmark.py Added command-line argument for embedding padding control
src/cpp/src/utils.hpp Declared new function for NPU text embedding compilation
src/cpp/src/utils.cpp Refactored NPU compilation logic and added text embedding-specific configuration
src/cpp/src/rag/text_embedding_pipeline.cpp Implemented NPU compilation path for dynamic text embedding models

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mengweiguo mengweiguo mentioned this pull request Dec 1, 2025
Open
Copilot AI review requested due to automatic review settings December 2, 2025 02:50
Copilot AI reviewed Dec 2, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mengweiguo mengweiguo force-pushed the qwen3-embedding-pr branch 2 times, most recently from 1b7e667 to 3120bdf Compare December 2, 2025 03:16
Copilot AI review requested due to automatic review settings December 2, 2025 03:16
Copilot AI reviewed Dec 2, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mengweiguo mengweiguo force-pushed the qwen3-embedding-pr branch 2 times, most recently from f713558 to a591398 Compare December 2, 2025 10:14
Copilot AI review requested due to automatic review settings December 2, 2025 10:14
Copilot AI reviewed Dec 2, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mengweiguo mengweiguo changed the title Support NPUW for text-embedding models [NPU] Support NPUW for text-embedding models Dec 3, 2025
@mengweiguo mengweiguo marked this pull request as ready for review December 3, 2025 05:47
Copilot AI review requested due to automatic review settings December 3, 2025 05:47
Copilot AI reviewed Dec 3, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Wovchena Wovchena requested a review from as-suvorov December 3, 2025 07:20
as-suvorov
as-suvorov requested changes Dec 3, 2025
View reviewed changes
@as-suvorov
Copy link
Collaborator

as-suvorov commented Dec 3, 2025

@mengweiguo You applied formatting for updated files. We have several PRs to enable formatting. I'm a bit concerned if your changes would match our fromatting rules.
If you want to apply formatting for updated files please align your PR with #3008 pre commit rules. Otherwise please revert formatting changes.

@mengweiguo
Copy link
Contributor Author

mengweiguo commented Dec 4, 2025

Otherwise please revert formatting changes.

Reverted. Thanks.

as-suvorov
as-suvorov approved these changes Dec 4, 2025
View reviewed changes
tools/llm_bench/llm_bench_utils/ov_utils.py
pooling_type = kwargs.get("emb_pooling_type")
max_length = kwargs.get("emb_max_length")
padding_side = kwargs.get("embedding_padding_side")
padding_side = kwargs.get("emb_padding_side")
Copy link
Collaborator

@as-suvorov as-suvorov Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sbalandi argument names were not aligned, thus option was lost. Consider arguments review and potentially introduce types.
@mengweiguo thanks!

@as-suvorov
Copy link
Collaborator

as-suvorov commented Dec 4, 2025

@mengweiguo we also need tests for NPUW, could you please implement simple test at: https://github.com/openvinotoolkit/openvino.genai/blob/master/tests/python_tests/test_rag.py

@as-suvorov as-suvorov self-requested a review December 4, 2025 13:55
@mengweiguo
Copy link
Contributor Author

mengweiguo commented Dec 5, 2025

could you please implement simple test

Sure, will do that.

Copilot AI review requested due to automatic review settings December 5, 2025 06:26
Copilot AI reviewed Dec 5, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@as-suvorov as-suvorov Awaiting requested review from as-suvorov

Assignees

No one assigned

Labels

category: llm_bench Label for tool/llm_bench folder category: RAG RAG pipeline components no-match-files

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mengweiguo @as-suvorov