Self studying and letters of recommendations I got the following response from a mathematics professor at UT Austin:


  Gaining admission is more about the positive things in your file
  than about checking all the right boxes. If you haven't taken all of the
  standard courses, but have learned it all on your own, that's OK. However,
  there has to be a very good reason why you're better than another
  applicant who HAS taken all of the standard courses. Your letter-writers
  have to make a convincing argument that there is something very, very
  special about you. If your physics professors are better able to present
  these arguments than you math professors, then it's OK to have most of
  your letters come from physicists. But those letters have to be very, very
  good.


My situation is such that since I am not a mathematics major, I haven't done a lot of standard math courses, which I intend to/or have studied on my own. Clearly, I need to win the graduate admissions committee's over by making sure my letter writers write good letters for me.

My question is:


  Generally, what can students like myself do to make sure they get the best possible letters from their professors? That is, how should one structure the self study plan to ensure maximum possible output and in return, the best possible letter from an instructor. More so, I'm not sure what qualifies as a "very, very special letter." What makes a letter "special?" More so, how can students like myself make sure that they are able to get such letters.
How best to organise your self-study? Pick your 10 favourite PhD programmes and look at their admissions requirements. Those are the courses you need to cover in your self study.

Then read the email you received carefully. Notice that it's not the letter that needs to be special-- it's you. You've already hinted at your drive and motivation to continue studying your subject at graduate level; why else would you devote so much of your (presumably) otherwise free time to studying it outside of classes? This effort may carry some weight with the admissions committee. Think of it as your unique selling point.

Now the tricky part: really making sure that the admissions people understand your USP before they meet you. This is your referees' job, and to help them do it successfully, it's your job to make sure they know and understand you as a student, as a mathematician and as a person. 

In this task, I think communication is the key. Talk to your instructor; tell her your plans for self-study, discuss what topics interest you the most, ask for feedback on your work, chat about relevant literature, where you want to go for your PhD, whether the moon really is made of cheese-- you name it. 

What I'm getting at is that for your letter to stand out from the no doubt tens or hundreds of other generic letters the admissions committee read, you need to come across as a real person with a genuine interest in your subject and the requisite skills and knowledge to back it up. And the only way that will happen is if your referee knows you as such.

(Disclaimer: I've never written or read a reference letter, but I've had some small recent successes in the hunt for a PhD place, largely (I think) due to my reference letters.)