How’d it do?

Qwen is probably best out there right now: ollama.com/library/qwen...

If you want a nicer UI, check out OpenWebUI. It presents a nice ChatGPT-esque web UI with history and etc….

My hunch is that they can write machine code right now well enough. I've never seen any evals on it, though.

One thing to consider is portability. Machine code is denser than source code, but I'd bet cross compiling source code to 50 distros is far cheaper from a compute perspective.

But yeah I guess bottom line, rag can get you far. Won’t know where it breaks until it does unfortunately. I look forward to a world where RAG systems can monitor themselves and signal to a user “hey it might be time to do some fine tunings!”

Depends on the use case. If the query is “what is my most controversial opinion across all my notes?” then rag can easily fall over unless you anticipated it ahead of time in the indexing pipeline. That’s admittedly an extreme example, but the spectrum between that & simple fact retrieval is blurry

Yeah I get what you’re saying. But I’d caution against dismissing people because they don’t speak for _everyone_.

I am an expert 😂 and while I trust LLMs for many things, me and most of my friends very much would not trust an LLM machine code output.

What is total dataset size in bytes? If complex reasoning across the whole set of notes is required for your use case — it could be! — RAG will fall over on you.

Have you done any experiments with your benchmarks going from 1 to 100 examples to see if accuracy regresses?

I think it can[1] but we don’t do it because:

1) we don’t trust the LLM enough. We want to review the code.
2) high level languages give you a higher density of expression per token. i.e. it takes less tokens so you get faster answers

[1] chatgpt.com/share/674db3...

I work in it so I’m in a bit of a bubble. What are some of the most egregious lies you see?

Ok very cool.

Do you run any benchmarks against your default prompt templates, and have you published them so others can compare different models or prompt/template tweaks?

Do you fine tune any of your models much or do you just work with prompt templating?

Long story short I think the change is a 10 year horizon. Not 2.

Only just recently have the models with long enough context length and recall across context to make retrieval work.

It’s completely transformed how I work: writing code, tests, design docs; less time scouring stackoverflow or fighting with plantuml/mermaid making diagrams. I’m far more productive.

But I’m a special case.

I think the real unlock is going to be agents. This promise still hasn’t been realized.

How this? Can’t tell if it’s underdone.

Flowers enjoyed. Very nice flowers.

🙋🏻‍♂️