Is it unethical/unscientific to omit outlier data in a publication when they are in FAVOR of your argument? I have a scatter(log-)plot with some 10,000 data points that plots the running time of some algorithm against the input on random instances of some problem.

I have a lot of these plots, and due to space constraints I can't exactly dedicate a large amount of space to them. The plots are 1-2 inches tall. The trouble is that a handful (maybe 5 or so) of these samples have taken a ridiculously short amount of time to complete (say, a few milliseconds), whereas pretty much all of the other data points have taken 2-3 orders of magnitude longer.  

I'm trying to show that my algorithm is fast, so I figure it shouldn't hurt to just omit these handful of data points and generate more samples, right? 

I feel like mentioning anything would unnecessarily confuse the reader, and keeping them would annoy the reader since the plot would have a large amount of blank space. And obviously it's not like I'm trying to suppress evidence against my research or something---the discarded data is only in favor of my algorithm.

Would I be violating some ethical code here if I just discard those samples without mentioning anything so that my plots look nicer? Is it unscientific? And if so would anyone actually care?
Science aims at revelation and insight.  Before you can even consider dropping these samples, you need to understand why they exist.  

The reason is that unexpectedly "good" data can be just as much a sign of problems with your theory as unexpectedly bad data.  Are these data points telling you that you've got a bug in your algorithm?  Are they saying that the instrumentation you were using for timing was unreliable or not calibrated properly?  Or is it just that under certain circumstances your random problems happen to be exceedingly easy to solve?  There may be other possibilities as well.

If you can't determine why the outliers exist, then you must include them, in order to help the reader evaluate your work.  If you determine they exist for a problematic reason, then, well, you're not ready to publish yet.  If you determine they exist for a benign reason, then you can drop them from the figure, but you must explain exactly what you've done in the text and why, or else you risk misleading the reader.

At the end of the day, the data you got is the data you got, and you need to deal with it honestly.