Q:

Measure similarity between quantized grayscale images

I want to measure the similarity between quantized grayscale images (8 levels) that mostly depict organic patterns found in nature, say pigmentation patterns in animals (zebra pattern, leopard spots,..). I cannot use some kind of trained model, and I need a continuous similarity value. Feature scale tolerance is not important, but differences in rotation and density of the features could be large. Ideally two different flat textures of zebra patterns should have a similarity of 1.0 and say zebra-tiger a similarity of 0.7. I understand how impossible that sounds, but I hope someone could point me to relevant current research or methods somewhat suitable to what I'm looking for.
EDIT: More info about the desired usage.
Examples of images undergone grayscale conversion and quantization:
(512x512 pixels, 8 gray levels). I do not own these images.

These are then used as templates for a procedural pattern generation technique I am developing. Images are randomly generated, generation settings that produce high similarity are kept to be further tweaked. The procedurally generated images could be anything from random pixels to intricate shapes, slowly converging at the image of highest possible similarity.

A:

A classification of the repertoire of the patterns that can be exhibited by reaction diffusion models has been attempted and this map might already be close to what you are after.
Other than this, if you absolutely have to create a metric of similarity that emerges from the images themselves then my suggestion would be to look at some form of invariant or universal feature.
You could for example look at Mutual Information between the two images and more specifically, LZ Complexity (which is not reciprocal, so $f(x,y) \neq f(y,x)$ where $f$ is the metric). The difference here is that mutual information essentially characterises the differences between the distributions that give rise to signals (therefore, the processes where they came from).
Also, looking at the sort of "shapes" the model creates, it seems to swing between relatively broad line strokes and point / circle structures. So, you could apply a simple threshold to the resulting image and then run a Hough Transform on it and try to estimate a ratio of "line-icity" (i.e. amount of sharp tall points in the image) to "circleicity" (double pairs of points at a specific configuration). I am not sure how sensitive or accurate would that be for a small change in the model's parameters though.
Hope this helps.

