Mishkat al-Masabih 2658
Usama b. Sharik said :
I went out with God’s messenger to perform the pilgrimage, and the people were coming to him. One would say, “Messenger of God, I ran between as-Safa and al-Marwa before going round the Ka'ba,” or “I did something after its proper time, or “I did something before its proper time.” He would reply, “No harm will come, except to one who defames a Muslim acting wrongfully. That is the one who will be in trouble and will perish.”

Abu Dawud transmitted it.
عَن أُسامةَ بنِ شرِيكٍ قَالَ: خَرَجْتُ مَعَ رَسُولِ اللَّهِ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ حَاجًّا فَكَانَ النَّاسُ يَأْتُونَهُ فَمِنْ قَائِلٍ: يَا رَسُولَ اللَّهِ سَعَيْتُ قَبْلَ أَنْ أَطُوفَ أَوْ أَخَّرْتُ شَيْئًا أَوْ قَدَّمْتُ شَيْئًا فَكَانَ يَقُولُ: «لَا حَرَجَ إِلَّا عَلَى رَجُلٍ اقْتَرَضَ عِرْضَ مُسْلِمٍ وَهُوَ ظَالِمٌ فَذَلِكَ الَّذِي حَرِجَ وهَلِك» . رَوَاهُ أَبُو دَاوُد