If your brick is dusty or dirty, use a scrub brush to scrub at any surface-level dirt or dust. If you have a vacuum cleaner handy, turn it on and use the brush attachment to blow away residual debris.  If you don't brush away all of the dirt, don't worry. You'll be able to scrub away the rest as you wash the brick. Avoid wire brushes, which can leave metal scratches on the brick and cause rust or corrosion over time.
Vacuum or brush away any loose dirt.