Tax law could also play a role.  It's astonishing how much the value of a house jumps when it has that historic exemption.  Since it's hard to get back the money you've invested in improvements, perhaps our ratio of tax on land vs. improvements is currently set too low.  The construction industry would benefit from changing this, since anything that makes a building project more cost-effective will help them.  This could also address part of the gap between rental value and construction cost that you mention.  Since this could be rolled out over decades, it could be gradual and planned out, giving people time to adjust and build in response.  

Also, would it be reasonable for developers to build a bunch of buildings that have the smaller unit sizes?