Our minds aren't infinite in memory, so much of what we sense and experience is eventually discarded and forgotten, especially if it's not very important.
It's only what the mind finds important that we actually remember.