4 cont.) It saves me hours, days, or weeks, depending on how messy or gross that data is.

These are the use cases to try and find within an organization. Manual toil. Stare and Compare. They are high leverage and have measurable impact.

4) Automation is the way forward.

Giant corpus of financial reports in PDFs that I need structured? Easy peasy. Pile of images I need to search through to classify? Trivial

These things used to be very hard. I can build a pipeline with a finely-tuned prompt and guidance in minutes.

3 cont.) This has become the default I use. It's still Augmented Generation, but much more controlled and finely-tuned. It's cost effective, often times requires less prompts, and enables things like Ensembles.

That's not to say embeddings don't have other uses! I like them for classification.

3 cont.) Last year, there was no way around RAG - input contexts were limited and costs were prohibitive. You simply had to do it - and we got good results in our projects. But those pale in comparison to building a very detailed context or simply putting a whole document/text into the prompt.

3) RAG has changed significantly.

Last year, the default pattern was splitting a document up and generating embeddings, then using semantic similarity to build context for specific solutions - to improve accuracy and deal with hallucinations.

We've largely abandoned the embeddings and similarity.

2 cont.) ... and they ALL behave differently - either because they can do so much more or because they were updated with new data, etc. etc. So the way you were using the tools 6 months ago vs. now is a consistently shifting goal. So you have to provide constantly updating guidance and training.

2) When you deploy LLM technology, you create a massive training problem.

Everyone is now a prompt engineer, organization-wide. Mileage will vary in how different roles and individuals are able to leverage it. Additionally, we've found that you transition LLMs about every 6 months...

1 cont.) Chatbots do work well when there is a constraint - specifically on the knowledge base the agent is responsible for (think HR and Corporate policies), but this is... like... fancy search at that point. It's a nice experience, but not going to drive tons of business - and their are risks.

1 cont.) Most users aren't stopping what they're doing and asking a person or agent about some nuance on a regular basis, and when they do, its often a very complicated technical nuance that is hard for an LLM to answer correctly.

1 cont.) In a lot of enterprises, I've seen chatbots or Agents (which is a chatbot with better PR) deployed quickly to mostly shrugs from the business and users. I think that goes back to a lack of understanding about how these folks do their jobs.

1) Think long and hard about whether you need another chatbot.

As someone who worked really hard to make Chatbots "a thing" in my past, there are just some fundamental issues with the user experience, even with techniques like RAG and Long Context.