At one time it housed approximately 6,000 veterans, but Napoleon took over a large part of it for the Musee de l'Armee (Army Museum).
At one time, 6,000 veterans lived there until Napoleon intervened.
I believe it's a mistake to represent the majority of the population's behavior off of a few extraordinary individuals.