Could you give an example of the input/output you're looking for on which function call (encode, tokenize, etc)? And maybe which tokenizer it's inheriting from 😅 (looks like maybe the OPT models inherit from a GPT2Tokenizer?)

Happy to talk about any of these topics and more!

I will also likely end up talking a lot about my pride and joy (my dog).

And if you think sentence-level machine translation is good-enough, I encourage you to run your systems on our evaluation data (ctxpro, an extension to ContraPro and other similar evaluation datasets)

github.com/rewicks/ctxpro

Most recently I've released the ParaDocs dataset which reconstructs document annotations on large, parallel machine translation datasets. Contextual information is integral to machine translation, but often overlooked!

Data: huggingface.co/datasets/jhu...