"Don't throw the baby out..."
I have heard one explanation of its origin that seems a... [DOC] [TLE] Don't Throw the Baby Out with the Bath Water  - Mr. Nussbaum [PAR] This idiom stresses to be careful what you throw away and to not throw away 
something valuable or important along with things unimportant.