Wasn't Llama 3.0 extremely sensitive to quantization? Quality went down horribly iirc. Ollama defaults not to the best quality. Check against fp16.

Yeah, checked my notes: cosine runs around 55% slower than dot product. Not so much difference between Euclidean and dot product though on x86. However on my ARM device, Euclidean was ~ 30% slower than dot product. Fun stuff...

So your intention to rule out Euclidean completely? Some datasets require it... Otherwise, sure, for cosine / dot product, the only drawback will be loosing a bit of precision. Not sure if users would expect to get back the same values they stored in your use case.

Is this a special use case? For non-normalized you cannot know the maximum values and thus you can only normalize per vector. This makes distance functions other than cosine more expensive as these need to be de-normalized. Which piece am I missing?

🤯 indeed

I really like the polygon bear - is it hand made?

Just for fun, I ran it on a 16 core CPU... Haha. 😄