its foundations remained amongst the weeds to show where the walls and keeps had stood , but the local smallfolk had long ago made off with most of the stones to raise their barns and septs and holdfasts .