Volumnia, despite her other faults, is neither shy nor retiring, not the sort of person likely to have isolated herself with her child.
The patrician Volumnia would at least have had the usual slave girl to raise him.