Martin had just decided that he would linger for a minute more, to not look suspicious. To excuse his early departure before Lord Malfoy would only make him suspicious so he would forego it and risk being thought of as rude.