Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[FEATURE] Fusing Leaky Relu operator in Fully Connected #19971

Merged
merged 1 commit into from
Apr 22, 2021
Merged

[FEATURE] Fusing Leaky Relu operator in Fully Connected #19971

merged 1 commit into from
Apr 22, 2021

Conversation

agrabows
Copy link
Contributor

@agrabows agrabows commented Mar 1, 2021

Description

This change will enable fusing Leaky Relu operator and Fully Connected into single operator supported by oneDNN backend for both int8 and fp32 data types.

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage

Changes

  • Fusing FC with leaky relu for oneDNN backend

@mxnet-bot
Copy link

Hey @agrabows , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [clang, sanity, centos-cpu, centos-gpu, miscellaneous, edge, website, windows-cpu, windows-gpu, unix-gpu, unix-cpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@lanking520 lanking520 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Mar 1, 2021
@agrabows agrabows changed the title Fusing Leaky Relu operator in Fully Connected [FEATURE] Fusing Leaky Relu operator in Fully Connected Mar 1, 2021
@lanking520 lanking520 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress labels Mar 1, 2021
@agrabows agrabows force-pushed the v1.x_fc_fuse_gelu branch from 4e8806d to dbd09ba Compare April 1, 2021 08:29
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Apr 1, 2021
@agrabows
Copy link
Contributor Author

@mxnet-bot run ci [unix-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [unix-gpu]

@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Apr 15, 2021
szha
szha reviewed Apr 19, 2021
View reviewed changes
src/operator/subgraph/mkldnn/mkldnn_fc_property.h
Comment on lines +107 to +113
if (param.act_type == leakyrelu::kLeakyReLU ||
param.act_type == leakyrelu::kELU ||
param.act_type == leakyrelu::kGELU) {
matched_list_.push_back(&new_node);
status_ = kSuccess;
return true;
}
Copy link
Member

@szha szha Apr 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full set of supported activation types are {'elu', 'gelu', 'leaky', 'prelu', 'rrelu', 'selu'}. What about the rest of the activation types? Will they be supported by onednn in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those were activation types that can be easily fused with FC in this PR. We plan to enable other activation fuses in the next PR as they need different approach.

@szha szha merged commit 54aae29 into apache:v1.x Apr 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@szha szha szha left review comments

@bgawrych bgawrych bgawrych approved these changes

Assignees
No one assigned
Labels
pr-awaiting-review PR is waiting for code review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@agrabows @mxnet-bot @szha @bgawrych @lanking520 @agrabow