Skip to content

Refactor OpenVINO backend: move and clean up parameter handling#240

Open
zhaixuejun1993 wants to merge 7 commits into
ravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/refacter-v0
Open

Refactor OpenVINO backend: move and clean up parameter handling#240
zhaixuejun1993 wants to merge 7 commits into
ravi9:dev_backend_openvinofrom
zhaixuejun1993:xuejun/refacter-v0

Conversation

@zhaixuejun1993

@zhaixuejun1993 zhaixuejun1993 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

This pull request refactors and extends the OpenVINO decoder integration in ggml-decoder.cpp and its header to improve maintainability, flexibility, and support for dynamic dimension inference. The main changes include introducing new utility methods for tensor shape/stride/type queries, significantly refactoring dynamic dimension inference to use a centralized mapping, and simplifying model input handling.

API Extensions and Refactoring:

  • Added new virtual methods to GgmlOvDecoder for querying input/output tensor shapes, strides, type sizes, block sizes, and dynamic dimension information, making the decoder interface more expressive and modular. (ggml-decoder.cpp, ggml-decoder.h) [1] [2] [3] [4]

  • Refactored the handling of model inputs and extra inputs to use a simplified and more flexible structure, reducing code duplication and improving clarity. (ggml-decoder.cpp) [1] [2] [3]

Dynamic Dimension Inference Improvements:

  • Overhauled the compute_node_dynamic_dims logic to use a centralized dynamic_dim_infer_map (from the new openvino/op_table.h), replacing the large switch statement with a more maintainable and extensible approach. This makes it easier to add or modify dynamic dimension inference rules for new ops. (ggml-decoder.cpp) [1] [2] [3] [4]

Cleanup and Dependency Management:

  • Removed unused OpenVINO header includes and added the new openvino/op_table.h dependency, ensuring only necessary headers are included. (ggml-decoder.cpp) [1] [2]

These changes collectively improve the decoder’s extensibility and maintainability, especially for handling dynamic shapes and new operations.This pull request refactors how model inputs and extra inputs are handled in the GGML-to-OpenVINO translation layer, improving type safety and code clarity. Instead of storing model inputs and extra inputs as generic OpenVINO nodes, the code now uses explicit structures (ModelInputInfo and ModelExtraInputInfo) to describe their properties. This change also centralizes the logic for creating OpenVINO parameters and constants, and updates related interfaces and usages throughout the codebase. Additionally, the codebase is cleaned up with improved documentation and removal of unused utilities.

Key changes:

Refactoring of Model Input Handling:

  • Replaces storage of model inputs and extra inputs from std::shared_ptr<ov::Node> to new structures ModelInputInfo and ModelExtraInputInfo, providing explicit type and shape information. (ggml-decoder.h, openvino/decoder.h, ggml-decoder.cpp, [1] [2] [3] [4] [5] [6] [7]

  • Updates the get_model_inputs() and get_model_extra_inputs() interfaces to return the new info structures, and removes the now-unnecessary get_model_extra_input_values() method. (ggml-decoder.h, openvino/decoder.h, [1] [2]

Centralized Parameter and Constant Creation:

  • Introduces helper functions create_parameter and create_extra_input to consistently create OpenVINO parameter and constant nodes from the new info structures. Updates the translation session to use these helpers when constructing the OpenVINO model. (openvino/translate_session.cpp, [1] [2]

API and Utility Improvements:

  • Updates get_ov_input_tensor to use the new ModelExtraInputInfo structure for constructing input tensors. (utils.cpp, ggml/src/ggml-openvino/utils.cppL840-R843)
  • Adds or improves documentation for several utility functions and classes, making the codebase easier to understand and maintain. (openvino/translate_session.h, openvino/utils.h, [1] [2] [3]

Cleanup and Removal of Unused Code:

  • Removes unused or redundant utility functions such as getCurrentTime, non_cont_dim, and various template utilities from utils.cpp and utils.h. (utils.cpp, utils.h, [1] [2]

Minor Fixes:

Additional information

Requirements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant