Is it legal in the US for a student to download a copy of a textbook, to study? (this is coming out of a comment thread regarding this question.)

The USA has a Fair Use legal provision restricting its copyright law:


  17 U.S.C. § 107 
  
  Notwithstanding the provisions of sections 17 U.S.C. §
  106 and 17 U.S.C. § 106A, the fair use of a copyrighted work,
  including such use by reproduction in copies or phonorecords or by any
  other means specified by that section, for purposes such as criticism,
  comment, news reporting, teaching (including multiple copies for
  classroom use), scholarship, or research, is not an infringement of
  copyright. In determining whether the use made of a work in any
  particular case is a fair use the factors to be considered shall
  include: 
  
  
  the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes;
  the nature of the copyrighted work; 
  the amount and substantiality of the portion used in relation to the copyrighted work as a whole;
  and the effect of the use upon the potential market for or value of the copyrighted work. 
  
  
  The fact that a work is unpublished shall not itself bar a finding of fair use if such finding is made upon consideration of all the above factors.


Obviously there might be other relevant statutes and legal precedents, but  still, 


Is it legal, considering the above, for a university/college student to download a copy of a textbook (say, in a course s/he is taking), for studying the course material? 
Same question, for a student not enrolled in a class for which the downloaded book is a textbook?
Same question, for self-study outside of Academia?


Notes:


"Legal" in two senses: First, the sense of very small chance of being convicted of a criminal act or obligated to pay money; second, the sense of there being a solid argument for the legality of this act which is likely to hold up if you appeal and appeal and maybe get to the supreme court.
Assume it's been established that the downloader did not publish the copy elsewhere, did not cite from it extensively, etc. etc.
I'm not asking whether it's moral or ethical to download textbooks, only about the legality. I believe it is moral and ethical, but that discussion is not what this question is about; please don't start it.
In general, it is not considered fair use to download the entirety of a textbook, regardless of whether or not you're a student or enrolled in a class using the textbook. Fair use normally is considered to extend to making copies of small excerpts of larger works. For instance, you could copy a particularly relevant figure from a book, or a quote from a textbook or reader for use in a class discussion.