Motherhood never brought Hermione the joy she was assured by so many that it would. The guilt she felt at not wanting to be a mother only increased when she was around her only child. She did love him, but always felt he would've been better off with someone else as his mother. Maybe recognizing that fact was proof that she did care about the boy. He deserved more than he was given.