My advisor talks about me to his colleagues in their native language I am in a country where English is not the native language. I have some proficiency in that language but none of the team members know this since I have always communicated with them in English. 

Recently, I have been struggling with certain tasks. During a lunch, my advisor was talking about my progress to his colleagues in their native language, and how I was unable to get some basic facts correct. What he said is technically true but I feel insecure to be exposed like that and humiliated that he is willing to talk about me behind my back.

The idea of responding in their native language came across my mind, but confrontation is too scary to me.
Without knowing more about what was actually said, or the tone of it, I would dismiss it as your advisor concerned about how best to advise you. I think these sorts of conversations are relatively common when a prof, especially an inexperienced one, has a problem helping a student that s/he can't resolve.

Unless you think they were joking about you or implying you weren't adequate to the task, you should probably just forget it. But discussing your troubles with colleagues can be a valid way for the professor to actually find the best way to be of assistance. 

There are worse interpretations, of course, but don't assume the worst unless it is really necessary. 

I've had students, actually, who missed some essential things in their early education. They can be hard to advise since there isn't time for them to go back and fill in those gaps. It is an important problem.