This means that the more you ignore a challenger, the more he or she will try harder to detract you from your routine and go one-on-one with them.
However, they admit that you are the man or women to beat.