Skip to content

Add OpenVINO backend #15307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 114 commits into
base: master
Choose a base branch
from
Draft

Add OpenVINO backend #15307

wants to merge 114 commits into from
+4,254 −5

Conversation

wine99
Copy link

@wine99 wine99 commented Aug 14, 2025

No description provided.

YangleiZouIntel and others added 30 commits August 14, 2025 17:00
@YangleiZouIntel @wine99
Add ggml-openvino base files
355da54
@zhanmyz @wine99
add openvino as optional backend for Llama.cpp ggml
4453fc3
@zhanmyz @wine99
* Configure the device(default CPU) that uses OpenVINO to compile th…
e5035ff 
…e model

 * Add OpenVINO ADD operator to Llama.cpp. The output is somewhat abnormal and needs further debugging.
@zhanmyz @wine99
Solve the issue of abnormal model output caused by using OpenVINO ADD…
1f0600f 
… operator
@zhanmyz @wine99
Add OpenVINO MUL operator to GGML of Llama.cpp.
22aac2e
@zhanmyz @wine99
Add compile options
75e55c0
@zhanmyz @wine99
add OpenVINO frontend convert process steps
f1821d1
@zhanmyz @wine99
add get openvino available ops function
290249c
@yumengbo @wine99
Add PoC of integration of openvino frontend. Main changes: ggml-ov-fr…
46a5662 
…ontend-utils, GraphIterator, Decoder
@yumengbo @wine99
Implement GgmlOvDecoder. Add dump functions.
6bf6b7a
@yumengbo @wine99
Convert subgraph with add, sub, mul, div op to ov model and do infer …
3c12e98 
…on openvino device
@yumengbo @wine99
Add GGML_OV_FRONTEND option. Add readme.
cf29563
@yumengbo @wine99
Change output for infer request to set output tensor. Support scale, …
870cdb7 
…view op.
@zhanmyz @wine99
add GET_ROWS operator of OpenVINO to GGML of llama.cpp
6942273
@zhanmyz @wine99
Update build.md and add operation mapping(GGML to OpenVINO)
4fa1177
@zhanmyz @wine99
add the rms_norm operator implemented using OpenVINO to the GGML back…
77c74da 
…end of llama.cpp
@yumengbo @wine99
Fix issue for output memory copy of infer request
11edfa6
@yumengbo @wine99
Change to implementation following pytorch frontend
98c9350
@yumengbo @wine99
Add support for UNARY SILU op . Fix pytorch impl bugs.
e6729a8
@yumengbo @wine99
Support Softmax op
d48cb65
@yumengbo @wine99
Support Softmax op
d8c05a6
@yumengbo @wine99
Support ROPE op.
8715fd2
@zhanmyz @wine99
Add support for RMS_NORM OP
e8cf08f
@zhanmyz @wine99
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML ba…
9711f27 
…ckend
@zhanmyz @wine99
Move CPY from GGML OV Backend to OV Frontend
d88e931
@zhanmyz @wine99
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops
3023596
@zhanmyz @wine99
add implementation of CPY when the output tensor is non-contiguous
a59a1c1
@zhanmyz @wine99
add tmp source code files
d9e8387
@zhanmyz @wine99
Execute singel CONT operator is OK
5e3369a
@zhanmyz @wine99
Execute CONT & VIEW operators in OV Frontend is OK
9667b1d
@wine99 wine99 marked this pull request as draft August 14, 2025 09:09
@github-actions github-actions bot added documentation Improvements or additions to documentation testing Everything test related devops improvements to build systems and github actions ggml changes relating to the ggml tensor library for machine learning labels Aug 14, 2025
@SearchSavior
Copy link

SearchSavior commented Aug 19, 2025 •
edited
Loading

Hello,

in this repo https://github.com/yangsu2022/GGUF-to-OpenVINO and the article https://blog.openvino.ai/blog-posts/openvino-genai-supports-gguf-models only a small set of models are supported.

Will this feature in llama.cpp offer wider gguf coverage via something like the parameter mapping described here,

https://github.com/yangsu2022/GGUF-to-OpenVINO/blob/405a95e300f8307fb4b779a12d46cf86adf5a441/convert_llama3.1_gguf_to_torch.py#L14

A few other questions:

  • What parts of OpenVINO feature set are intended to be brought into llama.cpp?

  • Is this PR trying to bring in only performance from openvino runtime to support llama.cpp usecase?

  • Pipeline parallel is coming in next release (I think), will that be implemented here for heterogeneous execution in llama.cpp?

Thank you for your work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

@ggerganov ggerganov Awaiting requested review from ggerganov ggerganov is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
devops improvements to build systems and github actions documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning testing Everything test related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@wine99 @SearchSavior @YangleiZouIntel @zhanmyz @yumengbo @virajwad @ravi9 @cavusmustafa