the besiegers could set up camp around the walls of a fortress and mindlessly throw boulders at the walls for years , while the besieged could spend those same years happily piling rocks against the inside of those walls .