That why recently I so often add to a prompt "don't rush, take your time, do deep research, gather all details before run in conclusion, gather all available information and don't make assumption". Just stop it from falling to the most obvious but wrong answer.

Each time when you pick a answer from a LLM and return it back in a training sample for a model, LLM will likely silently collapse to the answer and instead of high variation of other possible response will prefer this one. It could be plausible answer from the LLM but it shouldn’t be the only one.

Vectors are not enough because:
- doesn't capture structure
- similarity <> relevance
- lack of explainability

I'd like to tag Jesús Barrasa and Stephen Chin but I couldn't find them on bsky 🤔

pypi.org/project/ace-... btw lib at pypi looks like empty placeholder so even if this particular case could be false alarm but vulnerability drove by LLM agent is very likely