although there were extensive grounds and trails , we decided the ruins of the massive stone building-which even the park referred to as the `` castle '' -were the place to start , seeing as that 's what our clue directly connected to .