Normalize storage in Embeddings constructor.

[sebpuetz]

If no norms are passed to Embeddings, normalize embeddings and add norms to the embeddings.

[sebpuetz]

Probably a bad idea, we'd have to treat quantized and array storages differently. What do you think @danieldk ?

[danieldk]

Probably a bad idea, we'd have to treat quantized and array storages differently.

Not only quantized storages, but also mmap'ed storages.

finalfusion embeddings should already be l2-normalized according to the spec. We have some historical embeddings with are normalized, but do not have a norms chunk. And it would be senseless to renormalize them again and then store all-1 norms.

It's should be part of the interface that callers only provide l2-normalized embeddings. Ideally, we would check this as an invariant, but the invariant is too expensive (and also goes against mmap'ing embeddings). I guess we could consider validate the initial n embeddings as a sanity check.

[sebpuetz]

Probably a bad idea, we'd have to treat quantized and array storages differently.

Not only quantized storages, but also mmap'ed storages.

For mmap'ed storages it'd at least raise an exception, for quantized storages it just wouldn't do anything to the storage but still store the norms. It'd be reconstructed by virtue of __array__(), normalized and thrown away. The Norms would be stored anyways. So definitely don't want to introduce this kind of unexpected behaviour.

It's should be part of the interface that callers only provide l2-normalized embeddings. Ideally, we would check this as an invariant, but the invariant is too expensive (and also goes against mmap'ing embeddings). I guess we could consider validate the initial n embeddings as a sanity check.

Might add something to the docs, but I don't think going further makes much sense. The storage array is writable and so is the norms array, we can't really enforce normalization unless the arrays are made immutable (I think there's also a flag for in-memory arrays to prevent modification).