That the passenger broke no law- up until he refused to leave- has nothing to do with it. If you're on someone else's property they may insist that you get off. You may (or may not) have legal recourse when you're forced to leave, but leave you must.

Especially when the police show up and order you out.