How would the character lose memories? On damage seems too punishing. On death seems like a better idea, turning into a soulslike with the skills returning when you find your last death point.

Purpose* if it isn't evident already, english isn't my main language so sorry if that causes confusion.