The user can select the Metal device by setting the environment variable GGML_METAL_DEVICE=X where X is an integer representing the device number as displayed when llama is run. The patch for selecting the GPU works well however the speed of token generation was abysmal.

llama.cpp doesnt really go in for providing backward compatibility so i developed a patch to the objective-c module that can be enabled at build time. If you find you need it (see next few posts first though) you can find it here gist.github.com/nbx3/b367a2e...

First issue with llama.cpp's implementation is that it supports only the first detected GPU which is always the internal one, in my case an Intel UHD Graphics 630, which was is useable but very slow.

Snowy Surrey

forgejo.org is another good option for self hosting if it's git you need. But you can get free app/static site deployments and a usable URL from both digital ocean and cloud flare pages.

Great to see so many positive life changing drugs reach stage 3 and nice science journalism