Enable loading of ORT format model graph runtime optimizations #9901
Conversation
…e_optimization_load
…e_optimization_load
…e_optimization_load
…e_optimization_load
…e_optimization_load
…e_optimization_load
…e_optimization_load
…icrosoft/onnxruntime into edgchen1/sat_runtime_optimization_load
…e_optimization_load
onnxruntime/test/testdata/transform/runtime_optimization/qdq_convs_gen.py
Outdated
Show resolved
Hide resolved
…e_optimization_load
…e_optimization_load
onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc
Show resolved
Hide resolved
onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.h
Outdated
Show resolved
Hide resolved
onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc
Show resolved
Hide resolved
onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc
Show resolved
Hide resolved
| #if !defined(ORT_ENABLE_ORT_FORMAT_RUNTIME_GRAPH_OPTIMIZATION) | ||
| ORT_RETURN_IF(saving_runtime_optimizations, "Saving runtime optimizations is not supported in this build."); | ||
| #endif // !defined(ORT_ENABLE_ORT_FORMAT_RUNTIME_GRAPH_OPTIMIZATION) | ||
|
|
There was a problem hiding this comment.
I would have expected ORT_ENABLE_ORT_FORMAT_RUNTIME_GRAPH_OPTIMIZATION to only be applied to a minimal build to add the code for applying the saved optimizations.
In a full build you should always be able to save the optimizations, with that being turned on via config setting.
There was a problem hiding this comment.
I took the approach of having the general runtime optimization functionality be optional. We could have it always be enabled in a full build.
I guess we could also do something like this:
#if !defined(ORT_MINIMAL_BUILD) || (defined(ORT_EXTENDED_MINIMAL_BUILD) && defined(ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD))
Not sure that's better though.
There was a problem hiding this comment.
I'm not sure of a scenario where you don't want it to be included in a full build. We'd certainly need it enabled in any pre-built package so that can be used to create the saved optimizations.
Does ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD imply ORT_EXTENDED_MINIMAL_BUILD so things can be simplified?
How many places would we need to use this check though? AddPredefinedTransformers is not enabled at all in a minimal build currently.
…_RUNTIME_OPTIMIZATION_REPLAY_IN_MINIMAL_BUILD.
…e_optimization_load
tools/ci_build/github/azure-pipelines/linux-cpu-minimal-build-ci-pipeline.yml
Outdated
Show resolved
Hide resolved
Description
Initial implementation of load/replay of runtime optimizations in an ORT format model.
Motivation and Context
Enable some graph optimization at runtime in ORT format models.