We propose a variant of a well-known machine translation (MT) evaluation metric, HyTER <cite>(Dreyer and Marcu, 2012)</cite> , which exploits reference translations enriched with meaning equivalent expressions. We show that although <cite>the metric</cite> obtains good results on small and carefully curated data with both manually and automatically selected substitutes, it achieves medium performance on much larger and noisier datasets, demonstrating the limits of <cite>the metric</cite> for tuning and evaluation of current MT systems.