Barry Haddow
@bazril.bsky.social
Some details in our HPLT report arxiv.org/abs/2503.10267 . Code released here: github.com/hplt-project...
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present ...
arxiv.org
May 9, 2025 at 5:11 PM
Some details in our HPLT report arxiv.org/abs/2503.10267 . Code released here: github.com/hplt-project...
We also release MultiHPLT, with a total of 1275 language pairs (not English-centric). opus.nlpl.eu/MultiHPLT/en...
A note on formats: TMX files contain only unique translation units. Moses downloads include all non-empty alignment units including duplicates. Token counts for each language also include duplicate sentences and documents.
opus.nlpl.eu
February 28, 2025 at 2:03 PM
We also release MultiHPLT, with a total of 1275 language pairs (not English-centric). opus.nlpl.eu/MultiHPLT/en...
We will shortly be adding document alignments to this data. This means that we will release sets of aligned complete documents, with additional details of the sentence alignments that we found in the documents.
February 28, 2025 at 1:34 PM
We will shortly be adding document alignments to this data. This means that we will release sets of aligned complete documents, with additional details of the sentence alignments that we found in the documents.