Cool idea, could you also extract networks at each level to see how similarities in token embeddings evolve through each attention layer? with the right tokens as inputs you could see tokens initially cluster around literal meaning, then change into clustering around abstract meaning

Basically none, we might need larger institutional change before that happens: psycnet.apa.org/record/2023-...