I’m not Japanese but I do live in Japan.
I’m not Japanese but I do live in Japan.
```
const segmenter = new Intl.Segmenter("ja", { granularity: "word" });
const segments = [...segmenter.segment(text)].map(seg => seg.segment);
console.log(segments);
```
But It may over eagerly split words that aren’t in its dictionary.
```
const segmenter = new Intl.Segmenter("ja", { granularity: "word" });
const segments = [...segmenter.segment(text)].map(seg => seg.segment);
console.log(segments);
```
But It may over eagerly split words that aren’t in its dictionary.
1. It’s fewer tokens?
2. Cursor/etc may get confused by some types of “semicolon babysitting” www.epicweb.dev/your-code-st...
1. It’s fewer tokens?
2. Cursor/etc may get confused by some types of “semicolon babysitting” www.epicweb.dev/your-code-st...