1. A three-dimensional shape measurement apparatus that measures a three-dimensional shape of a specimen in a non-contacting manner, the three-dimensional shape measurement apparatus comprising: a score calculator that calculates a score for evaluating the reliability of measurement data for each measurement point by use of information obtained during a process of estimating a height of the specimen or the estimated height; and a data processor that processes the measurement data according to a result of evaluating the measurement data for each of the measurement points by use of the score.