 What are your criteria for hiring?
pj: I think 10 recommendations seems excessive.  Good programmers wouldn't waste their time working for a company that required them to bother that many people, just so you could figure out if they were a good programmer.Think of it as a barrier to entry.  Good programmers want good opportunities that don't make them bend over backwards to get.  Path of least resistance.If you make it hard for them to work for you -- they won't.Seriously, who wants to call up 10 people and ask them if they mind if some random person calls them?It's a waste of time and any good programmer is going to know this and have plenty of better places to work that don't require 10 recommendations.Also, what a programmer works on doesn't say anything about how smart (s)he is.  If you want to know how smart they are, ask them about as many random things as you can.  How broad is their knowledge base?  Smart programmers know a lot of stuff outside the world of programming.Ask her about philosophy, beer, or cards.  Ask about poker, pizza and travel.Smart people have big minds that expand way beyond personal projects and portfolios. If they have a portfolio /at all/ then they are spending more time on marketing themselves than they are on programming and that's not such a good sign.  I know you don't mean they have a binder with photos of work they did, or maybe you do, but either way, they should be spending more time on programming than marketing their programmer skills.  If they harp on how awesome they are at programming, they probably don't know much about programming.I'd say irrelevant projects done for fun are almost a better sign of smartness than those that were only relevant.Ask the potential candidate about how the projects they have worked on could be improved.  What went wrong, where were the bottlenecks, what kind of hardware was it, did they find any limitations with the software, programming languages, or environments they worked in?Ask them how to build world peace and solve the financial crisis.  See how rational they are.  See how honest they are.Introspection, Honesty and Rationality are probably the three most important characteristics I look for in programmers.  If they can communicate with the "real world" around them as well, that's a plus.Most people in the real world can't handle the amount of thought, blunt truth, and rationality that programmers dish out, so it becomes the job of the programmer to figure out how to communicate with others, so that just brings me back to the 10 recommendations thing...  Probably not a good idea.  I hope that was hyperbole.