Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mixed-precision quantization milestone1: naive_intNwo + eval/benchmark framework #531

Merged
merged 12 commits into from
Aug 1, 2024
Prev Previous commit
Next Next commit
edit the sqnr threshold
  • Loading branch information
Hanxian97 committed Jul 30, 2024
commit f4fccf3dc70bde806f68569faf3e35821604e773
4 changes: 2 additions & 2 deletions test/quantization/test_mixed_precision.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ def test_weight_only_quant(quantization_bit=2, symmetric=False):
y_wo = m(x)
sqnr = compute_error(y_ref, y_wo)
#SQNR_dB can be approximated by 6.02n, where n is the bit width of the quantization
#e.g., we set sqnr threshold = 44 for 8-bit, so that 6.02 * 8= 48.16 fullfills
assert sqnr > 44.0-(8-quantization_bit)*6.02, "sqnr: {} is too low".format(sqnr)
#e.g., we set sqnr threshold = 42 for 8-bit, so that 6.02 * 8= 48.16 fullfills
assert sqnr > 42.0-(8-quantization_bit)*6.02, "sqnr: {} is too low".format(sqnr)


# test if the asymmetric and symmetric quantization API works with different bit widths
Expand Down
Loading