Sorry for reviving a dead thread to nerd out but allegedly, in some old Japanese documents, they were described as a harem (before being retconned to be bodyguards) so I would not be surprised. Although, as mentioned, its only alleged, I'm yet to find these docs for myself.

So true