you're welcome! and no worries, I figured :)

so I have a 12GB VRAM AMD GPU and 64GB of memory. tho it's not actually using the full 12GB of VRAM because i'm running with `-cmoe` (offloading MoE experts to CPU). getting around ~20t/s in, 7t/s out but since it uses linear attention the rate is basically the same regardless of context length

linear attention is a wondrous thing

no idea how it compares to the proprietary western models, I don't use those abominations, but this is definitely good enough to be very useful for me

this is the first model where like, I'll be reading its internal monologue and think "ah fuck I know what's causing that bug," only to have the model itself come to the exact same conclusion in the next paragraph and fix the problem all on its own

oookay i think we have officially gone off the rails

GENIUS

but how will I know if 30-50 feral hogs break into the pantry