Similarly, if the port number suggests a higher level protocol that can wait (like FTP) it would be fair to let that packet wait.
Filtering on the IP address to favour your own services would be like an OS vendor favouring the products from their own applications division.