Likewise, great projects like this allow for testing and further refinement:

github.com/mhumphries23...

Thanks very much! I agree that the OCR space is moving quickly, and new models/tools are impressive.
@wjbmattingly.bsky.social is really doing cool work in this area with finetuned v-LLM models for OCR in low frequency languages.

I want to thank the editors and team at the @jdighist.bsky.social for a great experience, and especially the peer reviewers for their invaluable feedback. Thanks as well to @abegibson.bsky.social, @wjbmattingly.bsky.social, Patrick Wadden and Ian Crowe for comments on versions of this article.

8/8

I conclude by assessing how source criticism can ground historical uses of AI. We need to assess the real opportunities and risks of these techs, particularly as they are embedded within our digital lives. Historians have expertise to contribute to ongoing debates about AI's impact on our world.
7/8

I then focus on AI as tools for data preparation, using oral history transcriptions and OCR correction as case studies. In both domains AI models show promise in languages and content well-represented in their training data. But for data that falls outside of their training, LLMs often falter.
6/8

In more open-ended benchmarks AI performs far less impressively, indicating strengths in factual recall but weak capacities for historical interpretation. Assessing these techs requires careful design, and historians have expertise to contribute in evaluating this technologies.

5/8

One approach to assessing LLM performance is on benchmarks of historical knowledge. In some domains LLMs now match “expert” level knowledge in tests like A.P. history, per the MMLU benchmarks. However, such performance may owe less to true historical fluency than "benchmark leakage."

4/8

As Frédéric Clavert notes, LLMs are also a “grid of interpretations" informed by the nature of their training data. My article maps this space through applying @emollick.bsky.social's concept of generative AI's “jagged frontier” of uneven competencies.

www.uni.lu/c2dh-en/arti...

3/8

Using a framing from @tedunderwood.com, I explore LLMs as a form of “latent space” that allows for novel exploration of our collective past. I argue we need new critical approaches to better understand the true potentials of these techs, as well their costs.

hcommons.org/deposits/ite...

2/8