he remembered the story very well : brian fraser , a bastard of old lord lovat 's , had stolen ellen mackenzie from her brothers colum and dougal , the mackenzies of castle leoch , and had ended up at last with her at lallybroch , the pair of them more or less disowned by both clans but at least left alone by them .