Q:

Are women better math teachers for little children?

Once in a discussion a colleague told me that he thinks:

It is better to use women to teach maths to little children including preschoolers and children in elementary school. Also it is better to use men to teach math courses in high school and university. 

Note that in some sense he divided students by two categories, children and adults.    
I have no idea about truth or falsity of his claim. Although I believe that in some cases and for special communities using a male/female math teacher could have a positive or negative influence on students.   
Question 1: Is there any research to show that there is a meaningful relevance between gender of a math teacher and quality of teaching maths (for different levels of students) in any sense?
Question 2: Is above claim really true?   

A:

Most of the research on gender and math education is focused on student gender differences. However, a few references can be found that focus on the what differences there may be based on the gender of the teacher. 
One thing that appears to be common among some of these studies appears to be that perception of student performance varies based on gender of teacher and student. For instance, Ehrenberg, Goldhaber, and Brewer (1995) found that white female teachers evaluated their white female students more highly than did white male teachers. Yet they did not find learning differences. 
A different analysis of the same data (Dee, 2006) appears to agree that perception differences of student performance exist. And the analysis found performance distances as well. However, the analysis also found that there were confounding circumstances in that same data set that led the author to exclude math classes from his conclusion that the gender of the teacher does matter in school. Dee referred to differences that (he determines) were probably attributable female teachers being assigned more difficult students. 
In other words, there is a perception in line with what your colleague suggested, and it appears to influence the outcomes in this particular data set, namely the National Education Longitudinal Survey (NELS) from 1988. Teachers were assigned the less advanced math students, resulting in a finding of lower achievement for students of female teachers of math. 
Other studies have noted: 

differences in perception of student performance based on teacher gender; 
differences in teachers assessment of student achievement based on student gender, but inconclusive links between belief and teacher gender; 
some evidence that female teachers, in some secondary school mathematics classrooms, interact more with male students;
But not much to make a case that there is a clear link between teacher gender and performance.

I think we can safely say that this perception you refer to exists, but the basis for it is dubious.
I wonder what reputable evidence people would refer to if challenged in this assumption. And, of course, anyone making such a claim should be challenged to produce some sort of evidence. Based on a look at what's out there, I can say that I would not want to be the one having to defend a claim like this.
Cited:
Dee, Thomas S. (2006). The Why Chromosome. Education Next. Retrieved from http://educationnext.org/the-why-chromosome/
Ehrenberg, R. G., Goldhaber, D. D., & Brewer, D. J. (1995). Do teachers’ race, gender, and ethnicity matter? Evidence from the National Education Longitudinal Study of 1988 [Electronic version]. Industrial and Labor Relations Review, 48(3), 547-561.

A:

A study I have read indicates that female teachers' math anxiety (negatively) affects girls' math achievement at the elementary level, which would run counter to your claim that women are "better" for math teachers at the elementary level. (However, as a woman myself, this to me means that we need to do better at countering math anxiety in our female elementary teachers!). 
As others mentioned, this effect could possibly be due to contributing to stereotype threat. As far as I know, there is no study that indicates that women are objectively better at teaching elementary math and men objectively better at teaching upper-level math, or vice versa, or really much of a comparison at all. My anecdotal experience would suggest otherwise, as well (my best math teachers/professors were female), but while some slight overall correlation might exist between math teaching ability and gender I doubt we currently have a means to indicate whether it is due to some innate gender difference as opposed to social and cultural phenomena. The idea of innate vs. constructed gender differences is an ongoing and heated discussion.

