Olivier Chafik
@ochafik.bsky.social
OpenSCAD & llama.cpp contributor.
Used to dream in Scala, now waiting for a reason to learn Rust.
ex-Googler (views expressed = my own)
he/him 🏳️🌈
Also found at: @ochafik@fosstodon.org
(Slowly moving out of https://x.com/ochafik)
Used to dream in Scala, now waiting for a reason to learn Rust.
ex-Googler (views expressed = my own)
he/him 🏳️🌈
Also found at: @ochafik@fosstodon.org
(Slowly moving out of https://x.com/ochafik)
Runs anywhere (incl. Raspberry Pi 5).
On a Mac:
brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
Still fresh / lots of bugs to discover: feedback welcome!
Shoot out to @ggerganov and @ngxson for the patient reviews and general amazing work!
EOT🧵.
On a Mac:
brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
Still fresh / lots of bugs to discover: feedback welcome!
Shoot out to @ggerganov and @ngxson for the patient reviews and general amazing work!
EOT🧵.
February 1, 2025 at 1:45 PM
Runs anywhere (incl. Raspberry Pi 5).
On a Mac:
brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
Still fresh / lots of bugs to discover: feedback welcome!
Shoot out to @ggerganov and @ngxson for the patient reviews and general amazing work!
EOT🧵.
On a Mac:
brew install llama.cpp
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
Still fresh / lots of bugs to discover: feedback welcome!
Shoot out to @ggerganov and @ngxson for the patient reviews and general amazing work!
EOT🧵.
Shout out to @ggerganov and the amazing contributors to his llama.cpp adventure for creating such a welcoming and technically thrilling project. One of the most rewarding places to invest hobby time in :-)
github.com/ggerganov/ll...
🧵 4/4
github.com/ggerganov/ll...
🧵 4/4
github.com
December 4, 2024 at 4:40 PM
Shout out to @ggerganov and the amazing contributors to his llama.cpp adventure for creating such a welcoming and technically thrilling project. One of the most rewarding places to invest hobby time in :-)
github.com/ggerganov/ll...
🧵 4/4
github.com/ggerganov/ll...
🧵 4/4
Note that llama.cpp already has best-in-class JSON schema constraints support, which some already use for tool calling / which my PR builds on (it's been a fun year of PRs!):
github.com/ggerganov/ll...
🧵 3/4
github.com/ggerganov/ll...
🧵 3/4
llama.cpp/grammars at master · ggerganov/llama.cpp
LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.
github.com
December 4, 2024 at 4:40 PM
Note that llama.cpp already has best-in-class JSON schema constraints support, which some already use for tool calling / which my PR builds on (it's been a fun year of PRs!):
github.com/ggerganov/ll...
🧵 3/4
github.com/ggerganov/ll...
🧵 3/4
Forked this off my PR that brings fully-grammar constrained tool call to *all* models (with native prompting style for a few of them):
github.com/ggerganov/ll...
🧵 2/4
github.com/ggerganov/ll...
🧵 2/4
Tool call support (Llama 3.x, Functionary v3, Hermes 2 Pro, Mistral Nemo, generic) w/ lazy grammars & minimalist Jinja engine by ochafik · Pull Request #9639 · ggerganov/llama.cpp
This supersedes #6389 (now using a fully C++ approach), #5695 (first attempt at supporting Functionary) and #9592 (more recent Python wrapper).
Background
It tackles two main problems related to to...
github.com
December 4, 2024 at 4:40 PM
Forked this off my PR that brings fully-grammar constrained tool call to *all* models (with native prompting style for a few of them):
github.com/ggerganov/ll...
🧵 2/4
github.com/ggerganov/ll...
🧵 2/4
To notch it up one bit, you can also specify your own JSON schema (to, say, a list of at between 5 and 10 strings, each conforming to a specific regexp), we've got one of the best support out there
github.com/ggerganov/ll...
github.com/ggerganov/ll...
llama.cpp/grammars at master · ggerganov/llama.cpp
LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.
github.com
December 4, 2024 at 4:34 PM
To notch it up one bit, you can also specify your own JSON schema (to, say, a list of at between 5 and 10 strings, each conforming to a specific regexp), we've got one of the best support out there
github.com/ggerganov/ll...
github.com/ggerganov/ll...