Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Safe accumulation type for nd.norm #14126

Closed
eric-haibin-lin opened this issue Feb 12, 2019 · 1 comment
Closed

Safe accumulation type for nd.norm #14126

eric-haibin-lin opened this issue Feb 12, 2019 · 1 comment
Assignees
@szha
Labels
FP16 Operator

Comments

@eric-haibin-lin
Copy link
Member

For FP16, reduction operators tend to loss precision if the accumulation data type remains fp16. Instead, the accumulation dtype should be in fp32. i.e.

dtype = fp16 -> acc_type = fp32
dtype = fp32 -> acc_type = fp64
dtype = fp64 -> acc_type = fp64

We should do it for the norm op.

Reference impl for softmax: #14098
@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Feature

@eric-haibin-lin eric-haibin-lin mentioned this issue Feb 12, 2019
Closed
@szha szha self-assigned this Feb 15, 2019
@szha szha mentioned this issue Feb 23, 2019
Closed
5 tasks
@haojin2 haojin2 mentioned this issue Apr 4, 2019
Merged
9 tasks
@szha szha closed this as completed Nov 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@szha szha

Labels
FP16 Operator
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

use safe accumulation for norm szha/mxnet
3 participants
@szha @eric-haibin-lin @mxnet-label-bot