What lands are you thinking about? The Overseas Territories (as they are now called) were mostly settled, not conquered, and - although I accept Chagos is different - the OTs are free to choose whether they want to stay or go.