Better duplicate identification
Tichy: Maybe a good way to identify duplicates could be a comparison of word frequencies (or frequencies of 3 word sentences or something like that)? Perhaps that way one could eliminate the "layout" and identify the core text.I would like to create a library for that, if I could find the time :-/ Maybe then sites like HN would have an easier time to include the functionality.Not sure if counting frequencies would be enough to identify the duplicates that have been filtered through translation. Interesting stuff.