We built a thing! The Databricks Reranker is now in Public Preview. It's as easy as changing the arguments to your vector search call, and doesn't require any additional setup.

Read more: www.databricks.com/blog/reranki...

Excited to launch Agent Bricks, a new way to build auto-optimized agents on your tasks. Agent Bricks uniquely takes a *declarative* approach to agent development: you tell us what you want, and we auto-generate evals and optimize the agent.

www.databricks.com/blog/introdu...

Apache Spark 4.0 is out with some huge improvements across the board. SQL’s much more powerful, Spark Connect makes it easier to run apps, new languages and more. It’s amazing to see the community still growing fast and releasing over 5000 patches in 4.0. www.databricks.com/blog/introdu...

#MLSys 2025 is next week! You can still register at mlsys.org.

Nice results on never-ending learning for code editing. We believe that a lot of AI applications will be customizable this way (to every company's codebase, users, etc). The combined AI serving, data and MLOps environment on Databricks makes these easy to build.
www.databricks.com/blog/power-f...

🎥 New Video: Get Hands-On with MLflow Tracing!

In this video, @danliden.com walks through how #MLflow Tracing boosts observability in #GenAI apps—great for debugging, experimentation & organizing data workflows.

Watch now ➡️ www.youtube.com/watch?v=iRbB...

#opensource #oss

Really cool result from the Databricks research team: You can tune LLMs for a task *without data labels*, using test-time compute and RL, and outperform supervised fine-tuning! Our new TAO method scales with compute to produce fast, high-quality models. www.databricks.com/blog/tao-usi...

The #MLSys2025 program is up and registration is open! Check out accepted papers at mlsys.org/virtual/2025... and sign up to attend at mlsys.org/Register.

Exciting news—MLflow 2.21.0 is live! 🎉 This release includes significant features, enhancements, and bug fixes to improve documentation, #GenAI prompt management, tracing & more.

🔗 Explore all the new features & improvements: mlflow.org/releases/2.2...

#opensource #oss #mlflow

🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs!

We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.

We're probably a little too obsessed with zero-shot retrieval. If you have documents (you do), then you can generate synthetic data, and finetune your embedding. Blog post lead by @jacobianneuro.bsky.social shows how well this works in practice.

www.databricks.com/blog/improvi...

We're bringing in a new era of enterprise data management and agentic AI with SAP Business Data Cloud with Databricks.

✅ Unifies your SAP and non-SAP data

✅ Natively embeds Databricks technology

✅ AI agents streamline workflows

Learn more: sap.to/sapbdc

Sponsor registration is open for #MLSys 2025. We have the most submissions ever to MLSys so it promises to be a great conference! mlsys.org/Sponsors/spo...

Researchers open source Sky-T1, a ‘reasoning’ AI model that can be trained for less than $450

"Sky-T1-32B-Preview, our reasoning model that performs on par with o1-preview on popular reasoning and coding benchmarks."
That was quick! Is this already the Alpaca moment for reasoning models?
Source: novasky-ai.github.io/posts/sky-t1/

Congrats to Meta on releasing Llama 3.3, a 70B model that matches the performance of Llama-405B! Open weight models are advancing so rapidly and the cost to get this performance is quickly going down. We're thrilled to let users serve & customize this on Databricks. huggingface.co/meta-llama/L...

Compound AI Systems, Inference-time Compute Meetup @ NeurIPS 2024, with many AI luminaries as panelists. Poster submissions are open: lu.ma/q5r8b67t