1. A system for identifying unique and duplicate messages, comprising: a database of messages; an extractor module to extract a header and a message body from each message; a parser module to calculate a hash code for each message over at least part of the header and the body of that message and to group the messages having matching hash codes; a deduper module to randomly select one message in each group with two or more messages as a unique message and to mark the remaining messages in the group as exact duplicate messages; an attachment parser module to calculate a hash code over at least a portion of an attachment to two or more of the messages; and a concatenator module to generate a compound hash code for each of the two or more messages by concatenating the hash code for that message and the hash code for the attachment; and a processor to execute the modules.