SGLang has specforge, too, which seems promising. I mostly want a small standalone model rather than EAGLE-3 so that I can accelerate with llama.cpp, which doesn't have EAGLE-3 support yet.

How meta, a Halloween-ish joke that makes me want to say "Boooooo" lol

First time hearing about Francis Petrussewicz, really cool detail of Texas history / Goliad. Excellent episode!

Poetry

That's awesome tbh, a "But Doctor, I *am* Pagliacci!" moment lol

If you're doing inference on an aarch64 CPU, there's also the q4_0_4_4 quantization, which enables ARM-specific instructions and should perform roughly the same as q4_k_m.

I like their Muffulettas!

Ah sorry, didn't see your second post on directory-based solutions being no good. You could parametrize a nix shell with CUDA and python versions, but I can't vouch for that personally.

I like pyenv for managing different Python versions, but I'm not sure if venvs + the pypi CUDA packages work well enough. Nix (nixos.wiki/wiki/CUDA) might work well for isolation, but that would be more work than you'd likely want for a one-off/debugging.

Eh, it's hard to believe it's not all self-serving and disingenuous when comparing some of the things he wrote there vs. campaigning for Trump. I liked the nerdy, painfully earnest account of hard work and sacrifice when I read it, but that person (if he ever existed) seems to be long gone.

I tried refactoring some code recently with dyn trait args, but ran into trouble bc I also wanted it to be Serialize-able with serde and that needs a Sized type.