Les résidences secondaires ne rentrent pas dans le périmètre des logements vacants pour l’Insee

My guess would be aging population (compounded by upper classes higher life expectancy) and rising prices locking out potential ypung owners. By the time they have the means to buy (or inherit) they probably have a well-established suburban life.