Dudley had given his parents Harry’s phone number in case of an emergency. In this case, the emergency was Dudley having failed to inform his parents of their arrival to the magical household and it had been three days since they got there. He did not want to know what was waiting for him on the other end of that line.