Remove f16, bf16 from node's evaluate methods v2 #22674
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since this change, most of operation that override Node::evaluate will stop instantiating evaluate methods for f16 and bf16. There are few exceptions - we still keep f16 (and/or bf16) evaluates for the following operations: - Ceiling - Convert - FakeConvert Primary reason for that is to reduce binary size. The change saves us around 200 KB. The change is transparent to the caller, so you can still evaluate f16/bf16 operations, but internally they'll be executed with f32 precision. Ticket: CVS-108489
mateusztabaka
force-pushed
the
remove_f16_bf16_evaluates_v2
branch
from
cef8252 to
2c14955
Compare
mateusztabaka
force-pushed
the
remove_f16_bf16_evaluates_v2
branch
from
2d3c74c to
a9a0849
Compare
praasz
reviewed
Feb 15, 2024
praasz
approved these changes
Feb 16, 2024
iefode
reviewed
Feb 19, 2024
| #include "ov_ops/type_relaxed.hpp" | ||
|
|
||
| const ov::element::TypeVector& ov::util::unsupported_types() { | ||
| static const ov::element::TypeVector types{ov::element::f16, ov::element::bf16}; |
There was a problem hiding this comment.
just idea: As I know, some plugins use conversion from i64 to i32. Will it be useful to apply the same approach (fp32 -> fp32) for the mentioned element types?
There was a problem hiding this comment.
The goal was to remove the f16, and bf16 from core ops to reduce bin-size but constant folding and shape inference (before apply the plugin conversion) still requires these precisions to do calculations and this is reason to apply convert to f32 without changing the model. In core when operator do calculations on f16, bf16 values are converted to float and then back (element by element)
The i64 and i32 are common types when used for shapes calculations (constant fold and shape inference) and apply conversion will may introduce data copies native support I think is better.
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
itikhono
approved these changes
Feb 21, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 21, 2024
Since this change, most of operation that override Node::evaluate will stop instantiating evaluate methods for f16 and bf16. There are few exceptions - we still keep f16 (and/or bf16) evaluates for the following operations: - Ceiling - Convert - FakeConvert Primary reason for that is to reduce binary size. The change saves us around 200 KB. The change is transparent to the caller, so you can still evaluate f16/bf16 operations, but internally they'll be executed with f32 precision. Ticket: CVS-108489
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
rkazants
reviewed
Feb 21, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 18, 2024
…ding is omitted (#26756) Details: It's a modification of #22674 f16 LLM (llama was tested) compilation time on ARM is unreasonable huge. Perf report shows that every ConstantFolding transformation takes several seconds even if the graph is not modified. The root cause is util::convert_to_supported_precision call even if constant folding is skipped. The suggested fix is to skip util::convert_to_supported_precision call if folding is not applied. Tickets: CVS-152428 --------- Co-authored-by: Aleksandr Voron <[email protected]> Co-authored-by: Andrii Staikov <[email protected]>
CuriousPanCake
added a commit
to CuriousPanCake/openvino
that referenced
this pull request
Nov 6, 2024
…ding is omitted (openvinotoolkit#26756) Details: It's a modification of openvinotoolkit#22674 f16 LLM (llama was tested) compilation time on ARM is unreasonable huge. Perf report shows that every ConstantFolding transformation takes several seconds even if the graph is not modified. The root cause is util::convert_to_supported_precision call even if constant folding is skipped. The suggested fix is to skip util::convert_to_supported_precision call if folding is not applied. Tickets: CVS-152428 --------- Co-authored-by: Aleksandr Voron <[email protected]> Co-authored-by: Andrii Staikov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since this change, most of operation that override Node::evaluate will stop instantiating evaluate methods for f16 and bf16. There are few exceptions - we still keep f16 (and/or bf16) evaluates for the following operations:
Primary reason for that is to reduce binary size. The change saves us around 200 KB. The change is transparent to the caller, so you can still evaluate f16/bf16 operations, but internally they'll be executed with f32 precision.
Ticket: CVS-108489