for as long as he had known it , castle black had been a place of silence and shadows , where a meagre company of men in black moved like ghosts amongst the ruins of a fortress that had once housed ten times their numbers .