Given 2 colors, how to calculate the mix amount between them? As an input I have two colors, let's say red (RGB = 1,0,0) and magenta (RGB = 1,0,1).
Now I have an image which includes additive mixes between these two colors, for example purple (RGB = 0.5,0,1).
I want to calculate the mix amount between these two colors where 0 is 100% the first color (red) and 1 100% the second color (magenta). In the example with purple, the mix would be 0.5.
Of course input colors may be as complicated as possible, but it is always ensured that they are indeed mixable.
I know how to calculate this for grayscale colors, but not for arbitrary input colors.
A visualized input and output would be like this, when the output will be used as the alpha channel for the second color:


Things became much easier to me when I thought of the RGB components as XYZ coordinates of a 3-dimensional point, and that the lengths between those is the key to success!
To get the mix-amount between two colors, you require some basic vector maths:


*

*We have the background color A and foreground color B. C is the mixed color. All 3 have R, G and B components.

*Compute the square length between A and B. With the formula for retrieving the square length between two 3-dimensional vectors, do the following:
$$
    \left \| AB \right \|^2 = (A_{r}-B_{r})^2+(A_{g}-B_{g})^2+(A_{b}-B_{b})^2
$$

*Calculate the square length between A and C (the mixed color). For C it is ensured that it is a point on the track between A and B (and if it isn't, we don't care, as the provided A and B is wrong and it's not our fault):
$$
    \left \| AC \right \|^2 = (A_{r}-C_{r})^2+(A_{g}-C_{g})^2+(A_{b}-C_{b})^2
$$

*Divide the square length of A to B by the square length of A to C. If you take the root of the result, you'll get the "alpha denominator" (you'll see shortly why): 
$$
    d = \sqrt{\frac{\left \| AB \right \|^2}{\left \| AC \right \|^2}}
$$

*The alpha value eventually is 1 divided by the denominator:
$$
    a = \frac{1}{d}
$$


The resulting a is exactly 0 if the color is completely the background color A. It is 1 if the color is completely foreground color B. Anything between is background color A + foreground color B * a.
I'm sorry if any of my math formatting is incorrect, I'm new to LaTeX and not really a mathematician :)
