David Espejo
@davidmirror.bsky.social
Avid learner. OSS maintainer.Books, running, DistSys, and dogs. Proud father and husband.
TIL: weight tying improves performance by sharing(tying) the embedding and softmax matrixes as they both can be represented by vectors with low cosine distance between each other.
A comparison:
A comparison:
December 5, 2024 at 11:00 AM
TIL: weight tying improves performance by sharing(tying) the embedding and softmax matrixes as they both can be represented by vectors with low cosine distance between each other.
A comparison:
A comparison: