The final goal of this paper is to show that the evaluation procedure must be adapted from how it is currently performed.
First, many evaluation protocols require to provide separate results for each protocol, and we have done the same mistake in our previous evaluation [1]}, [2]}.
For example, the Celebrities in Frontal-Profile (CFP) dataset [3]} provides two protocols, a frontal-frontal and a frontal-profile protocol, and more than 94 % accuracy is reported in [4]}.
When evaluating both protocols separately, a different score threshold is selected for both protocols.
This is, however, not how face recognition works in practice where a single threshold is used independently of the type of image (frontal or profile) at hand.
To highlight the difference, we repeat the evaluation of the different poses from Fig. fig:poses by selecting a separate threshold per sub-protocol (one threshold for each face pose) on the dev set and plotting the FNRM on the eval set in Fig. fig:thresholds:optimal.
Clearly, the error rates drop drastically when a separate threshold is computed.
Second, in most datasets and evaluation protocols, ROC curves are plotted that show the performance of the evaluated system only on the eval set (aka. the test set).
How well a threshold selected on this test set translates to previously unseen subjects is not clear, but from comparing Fig. fig:thresholds:eval with Fig. fig:thresholds:optimal we can see that there is a trend to reduce error rates when selecting the threshold on the eval set directly.
We believe, splitting the protocols into dev and eval is critical to evaluate the algorithm on data that has not been seen at any stage of the process.
