I'm trying to detect if an article or forum post is a duplicate entry within the database. I've given this some thought, coming to the conclusion that someone who duplicate content will do so using one of the three (in descending difficult to detect):simple copy paste the whole textcopy and paste parts of text merging it with their owncopy an article from an external site and masquerade as their ownPrepping Text For AnalysisBasically any anomalies; the goal is to make the text as pure as possible.For more accurate results, the text is standardized by: Stripping duplicate white spaces and trimming leading and trailing.Newlines are standardized to \n. HTML tags are removed. Using a RegEx called Daring Fireball URLs are stripped. I use BB code in my application so that goes to.()ccented and foreign (besides Enlgish) are converted to their non foreign form.I store information about each article in (1) statistics table and in (2) keywords table.(1) Statistics TableThe following statistics are stored about the textual content (much like this post)text lengthletter countword countsentence countaverage words per sentenceautomated readability indexgunning fog scoreFor European languages Coleman-Liau and Automated Readability Index should be used as they do not use syllable counting, so should produce a reasonably accurate score.(2) Keywords TableThe keywords are generated by excluding a huge list of stop words (common words), e.g., 'the', 'a', 'of', 'to', etc, etc.Sample Datatext_length, 3963letter_count, 3052word_count, 684sentence_count, 33word_per_sentence, 21gunning_fog, 11.5auto_read_index, 9.9keyword 1, killedkeyword 2, officerskeyword 3, policeIt should be noted that once an article gets updated all of the above statistics are regenerated and could be completely different values.How could I use the above information to detect if an article that's being published for the first time, is already existing within the database?I'm aware anything I'll design will not be perfect, the biggest risk being (1) Content that is not a duplicate will be flagged as duplicate (2) The system allows the duplicate content through.So the algorithm should generate a risk assessment number from 0 being no duplicate risk 5 being possible duplicate and 10 being duplicate. Anything above 5 then there's a good possibility that the content is duplicate. In this case the content could be flagged and linked to the article's that are possible duplicates and a human could decide whether to delete or allow.As I said before I'm storing keywords for the whole article, however I wonder if I could do the same on paragraph basis; this would also mean further separating my data in the DB but it would also make it easier for detecting (2) in my initial post.I'm thinking weighted average between the statistics, but in what order and what would be the consequences...
What algorithms can I use to detect if articles or posts are duplicates?
algorithms
There are many algorithms that deal with document similarity in NLP. Here's a seminal paper describing various algorithms. Also wikipedia has a larger collection. I favor the Jaro Winkler measure and have used it for grad school projects in aglomerative clustering methods.