It is all about the math.

$1MM plus costs to purchase this piece of land.  Presume you spend another $20,000 on plans, bribes to neighbors and variance fights to get a 1200 sq foot house approved (~max size for the lot).  Then you have to demolish the dog house and build a new house.  Presume with the tight foot print etc it costs you $300 a sq. ft.  That means you are now into the property $1,360,000 BEFORE carried interest / financing costs.  Now, tell me you can get say $1.5MM for a 1200 sq ft. new home with no parking on a micro lot.  

Even if you cheap out and build marginally at code it is going to cost you $200 a sq. ft. by the time you are done.. $1,200,000 for a tiny three story bowling alley of a house?  .Utter madness.