Your child's birth father and your husband's children could be part of the problem.
Reading between the lines, it may be that the child's biological father and your husband's own offspring-situation are part of the equation.