1. An apparatus for identifying candidate duplicate documents of a document contained in a document corpus, the apparatus comprising: a processor; and a non-transitory computer-readable storage medium having instructions stored thereupon which are executable by the processor and which, when executed, cause the apparatus to create a bitset optimized inverted index comprising bitsets associated with tokens contained in documents in the document corpus, wherein each bitset comprises a number of binary digits equivalent to the number of documents in the document corpus, and wherein each binary digit in each bitset corresponds to a different document in the document corpus, receive a request to identify the candidate duplicate documents of the document contained in the document corpus, tokenize the document to identify tokens in the document, search the tokens contained in the documents in the document corpus to identify tokens that are in the document and also in at least one document in the document corpus, and utilize the bitsets associated with the identified tokens to select documents in the document corpus as the candidate duplicate documents of the document.