Time is also a huge separator.

There could've been entire civilizations that have conquered galactic travel and died out before we even existed.

And there could be other civilizations out there that will come around long after we've gone extinct.