Why should the population be higher just because we can support more people? The resources a human actually needs compared to the available resources of our solar system let alone galaxy are, for all intensive purposes, finite. The issue is our systems keep allowing more and more immigrants and are grossly inefficient and as you said cause environmental damage because of that inefficiency