The Blackfish was considered an outlaw south of the Neck, and the Riverlands were in the hands of the Lannisters. Even if they were liberated, the one in charge would be his foolish nephew, who had even had a child with his Frey bride. Try as he might, Brynden could simply not bring himself to look favourably upon the cursed fruit of a union with a Frey, conceived during the Red Wedding.