Completely crazy idea for an image classification system?
pgbovine: you might want to take a look at the Shazam song classification system, which essentially does a 'brute force' search like the one you sketched out, except with a lot of cleverness to make it run super-fastresearch paper:
http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdfblog entry summarizing paper:
http://laplacian.wordpress.com/2009/01/10/how-shazam-works/