How will an application for graduate admissions be affected if psychological issues prevent student from getting strong recommendations? What if a student is unable to or has difficulty getting strong recommendation letters (for graduate admissions in mathematics) due to psychological factors, such as autism?

Will graduate schools even consider these problems?
A graduate school probably will not make any exceptions for psychological issues: by the time that you are at the level of graduate school, people are generally evaluating you as a "whole package."  If a person has cognitive or psychological issues (and many successful people do), what will likely matter is whether those impair them to the degree that they will have trouble succeeding. 

For example, consider a person on the autism spectrum: this covers a huge range of different issues and many different levels of functionality and types of coping strategy.  One person might have a strong focus and coping strategies that cause others to feel just that they are "socially awkward" and get strong recommendations regardless. Another with less adaptive coping strategies might instead find themselves perceived as argumentative and hard to work with and have a hard time getting good recommendations.

In the end it boils down to this: a person has to have people willing to advocate for them.  If they can't get advocates, then it doesn't matter whether it's because of bad grades, bad attitude, mental illness, or anything else (unless you are dealing with a clear case of discrimination against a specially protected category, e.g., in the U.S., religion, race, or gender).