The house was just a house. A mansion, really. A prison. But then an artists' colony, a getaway, a refuge. Then a mansion, and a family compound, and a place of secrets and rumors and private dramas. And through all the years and iterations, the residents of the house have forged their own communities of varying functionality. 
 The story is told in reverse, mostly: present, 1950s, 1920s, 1900. it works, answering questions that you've only just developed, though I'd have liked more of a refresher at the end to tie everything forward.