Laura Dietz
@laura-dietz.bsky.social
Computer Science Professor with focus in Information Retrieval and Text-centric Machine Learning
"Look at this for 10 minutes, then go an rewrite your peer-review"
December 26, 2024 at 12:27 AM
"Look at this for 10 minutes, then go an rewrite your peer-review"
Publishing is cruel, because we are reviewing our peers AND are destined to disagree.
December 26, 2024 at 12:24 AM
Publishing is cruel, because we are reviewing our peers AND are destined to disagree.
Also, so far I have not seen any shred of work on how to evaluate the topical coherence of a response.
(I am not referring to a linguistic "fluency", but whether the response makes sense)
(I am not referring to a linguistic "fluency", but whether the response makes sense)
December 3, 2024 at 1:33 AM
Also, so far I have not seen any shred of work on how to evaluate the topical coherence of a response.
(I am not referring to a linguistic "fluency", but whether the response makes sense)
(I am not referring to a linguistic "fluency", but whether the response makes sense)
I also still think we don't have good metrics for evaluating RAG.
Neither one-big-relevance-prompt-grading (like UMBRELLA), nor breaking it into finer nuggets/grading rubrics are a done deal. We are still figuring out too many pieces.
Neither one-big-relevance-prompt-grading (like UMBRELLA), nor breaking it into finer nuggets/grading rubrics are a done deal. We are still figuring out too many pieces.
December 3, 2024 at 1:32 AM
I also still think we don't have good metrics for evaluating RAG.
Neither one-big-relevance-prompt-grading (like UMBRELLA), nor breaking it into finer nuggets/grading rubrics are a done deal. We are still figuring out too many pieces.
Neither one-big-relevance-prompt-grading (like UMBRELLA), nor breaking it into finer nuggets/grading rubrics are a done deal. We are still figuring out too many pieces.