He doesn't like spending time with her and even though he's her favorite he pretty much wants nothing to do with her.  That is not caring.  That is the opposite of caring.  But at least he's honest for what that's worth.  He just wants her to be placed where he isn't.