Norm RMS Epsilon

[Nexesenex]

While it crosses my mind..

@ikawrakow : a while ago, you made some measurements with variations of Norm RMS Epsilon which showed some little benefits to offset it for <2bpw quants. It was on L2 I believe, and I wonder if it applies to other arches, and if yes, if there's some sort of "formula" which would come with it to improve the low bitrate quants themselves.

Just beotian thoughts.

And merry XMAS btw, if you celebrate it!

[ikawrakow]

I'm travelling, so just quickly from the phone.

Yes, there is a small benefit from increasing rms_eps also for LlaMA-3, but only for very low-bit quants (IQ2_XXS). No, I haven't done any kind of systematic investigation.