What's the best way to measure mathematical ability? Very soft question I admit, but it's something that's been bothering me for a while.  

I've been thinking that being self taught has the problem of accreditation.  You can't evaluate a mathematician in a vacuum.  You need an accredited mathematician to decide whether or not someone else is also a mathematician worthy of accreditation.  Well, who evaluated the other mathematician?  Other accredited mathematicians.  It's sort of like becoming a member of an exclusive club.  

We put the job of accreditation on our universities.  But what if some person was discovered, off-the-grid so to speak, who had taught themselves mathematics from library textbooks.  

How could such a person evaluate themselves?  How do you know if you're making progress when you study?  

It's tricky.  It's like language learning.  Do I speak German more fluently now than I did yesterday?  I've no idea.  Who can say?  

It's like playing with Lego.  How do you know if you're getting better with Legos?  You build more complicated things.  But who's to say one person's Lego helicopter is better than another's Lego Enterprise?  What's the goal with Legos?  Is there one?  Should there be one?

I know already that this question will be deleted almost immediately, but I think these are important questions and many people visiting this site are in fact self-taught and I'm sure these questions show up as massive roadblocks.

Thanks for reading.
Your question is philosophical. As far as I know, there is no normed linear space of all possible mathematicians on which we have a metric to compare elements in the space!

My personal opinion is that there is an infinite amount of mathematics. If there isn't, then it is extremely large. This means any knowledge you have will be woefully less than what is possible to know. (Put some measure on the infinite set of all possible mathematics?) 

Additionally, you will always find a person, either from the past or present, and possibly the future, who knows more mathematics or knows more about some particular branch of mathematics. While it is true that you can make comparisons with other people's mathematical abilities, that is often demoralizing. For most people, anyways, they aren't "at the top".

Of course, there is a threshold I am assuming one is beyond. There is clearly a difference between the majority of calculus students and math PhD students. Here, yes, you should be comparing yourself to see if you can master undergraduate and graduate material to know if you can, say for instance, do research in mathematics.

But all that aside, it is somewhat pointless and self-serving if you do mathematics merely for status. You should enjoy doing it as you should enjoy doing anything difficult. And mathematics is genuinely beautiful. I suggest you are doing well if you continually are surprised at what you learn, if you are able to find satisfaction in meeting non-trivial personal goals, and if you surprise yourself by going beyond your expectations.

In particular, improve your ability to solve problems, improve your ability to propose problems, and improve your ability to explain mathematics. 

If you work on a difficult problem and you solve it, then you've surpassed your expectations! Otherwise it would not have been a difficult problem--the doubt of solving it makes it difficult. If you can propose new problems, you can enjoy the excitement of the novelty of new (to you at least) mathematics. If you can visualize the structure of mathematics and express it clearly to others, you can appreciate the beauty of mathematics and in fact share it as well. 

In order to do any of this, you do need to know what you don't know. Otherwise you can't envision any improvement. There is some value in making your own personal maps of what the mathematical landscape looks like. This is very much in line with the saying that the wise man knows what he does not know.

That's my 2 cents.
Take an exam. It is objective and unbiased (usually).