Don’t understand him wrong. He Xuan was also interested in Malfoys, but that’s more about their role in what the voice called "British magical society". The former Ghost-King can’t confidently call those magicians his family (who clearly preferred He Xuan to his twin and were so careless, that their own servant was able to steal their child from their own house). He Xuan doesn’t know how it sounds for someone external, but personally for him, that is not good material for parents. Not when he remembers Potters, who gave their lives for them or his first family, who was with him and supported every step. Right now, Malfoys are felt more like a group of distant relatives, which you need to see and smile to a few times a year on the big festivals. 