Not including Neidpath, who owned the other castle that Sir Simon Fraser had for most of its existence?
Oliver Castle: Oliver Castle was a medieval tower house, located in the upper Tweed Valley in the Scottish Borders.
i don't know