"Don't throw the baby out..."
[DOC] [TLE] (Don't) Throw The Baby out with the Bath Water - JStor [PAR] of English Proverbs by citing the very slightly different form of the proverb "Don't 
throw the baby out with the bathwater" (Mieder.