Better duplicate identification
brk: Makes sense to me.  It also seems that a good part of the dupes come from the fact that the full URL is used as the identifier.  And often times that URL has a lot of referrer garbage and similar data in it.  It would seem there could be some logic to try and pare URLs down somewhat in the comparator code.