user2210: After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting. 
To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.
How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually. 
Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?

user3772: You can try Video Comparer (shareware).
http://www.video-comparer.com
It quickly detects similar videos with image transformations and supports split videos into multiple CDs.

user4983: I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is http://duplicatevideosearch.com

user32436: This seems related to : https://video.stackexchange.com/a/33521/32436

You could check out
https://theophanemayaud.github.io/video-simili-duplicate-cleaner/
(macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.
The idea is to extract a few (eg 2, the first and the last) frames,
and compare them. If the duration is close, and the frames look
similar : you've got a duplicate.

You can "manually compare the extracted frames", or use thresholds to configure some automatic behaviors, and trash the duplicates.

It uses opencv for frame comparison, and ffmpeg for frame extraction
(ffmpeg is also for video metadata which is very useful when trying to
tell which video to keep !). Since it's open source, you can improve
it to fit your needs !

I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !

user-1: Video recognition software is I would say, a niche market which means you will either find very expensive software and/or software that targets government, movie industry and so forth.
It works by "finger printing" video so it can recognize the video despite different formats, compressions, cropping etc. This ability is not so common in the non-forensic world (at least not yet).
I did try to find something that is targeting consumers but was not able to find any. There do exists APIs (programming interfaces) that provides technology for doing this, and even free, but you will need to develop the software to use it yourself which I am fairly certain is beyond the scope of what you're asking for.
So in conclusion, unless you want to spend a noticeable amount of money on specialized software you will need to go through the stack manually. However, in my link above you will find a trial version of such software so you can see if that is what you're looking for and if it works for your purpose.

user3134: I'd recommend a hybrid approach using both computers and people.

Bucket the videos by their length (round to the nearest second)
For each bucket, use ffmpeg to generate thumbnails at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.

You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse.  Good luck!

2210: <p>After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting. </p>

<p>To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.</p>

<p>How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually. </p>

<p>Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?</p>
 3772: <p>You can try Video Comparer (shareware).</p>

<p><a href="http://www.video-comparer.com" rel="nofollow">http://www.video-comparer.com</a></p>

<p>It quickly detects similar videos with image transformations and supports split videos into multiple CDs.</p>
 4983: <p>I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is <a href="http://duplicatevideosearch.com" rel="nofollow">http://duplicatevideosearch.com</a></p>
 32436: <p>This seems related to : <a href="https://video.stackexchange.com/a/33521/32436">https://video.stackexchange.com/a/33521/32436</a></p>
<blockquote>
<p>You could check out
<a href="https://theophanemayaud.github.io/video-simili-duplicate-cleaner/" rel="nofollow noreferrer">https://theophanemayaud.github.io/video-simili-duplicate-cleaner/</a>
(macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.</p>
<p>The idea is to extract a few (eg 2, the first and the last) frames,
and compare them. If the duration is close, and the frames look
similar : you've got a duplicate.</p>
</blockquote>
<p>You can &quot;manually compare the extracted frames&quot;, or use thresholds to configure some automatic behaviors, and trash the duplicates.</p>
<blockquote>
<p>It uses opencv for frame comparison, and ffmpeg for frame extraction
(ffmpeg is also for video metadata which is very useful when trying to
tell which video to keep !). Since it's open source, you can improve
it to fit your needs !</p>
</blockquote>
<p>I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !</p>
 -1: <p>Video recognition software is I would say, a niche market which means you will either find very expensive software and/or software that <a href="https://web.archive.org/web/20130813180741/http://www.forensic-pathways.co.uk/products-and-services/videntifier" rel="nofollow">targets government</a>, movie industry and so forth.</p>

<p>It works by "finger printing" video so it can recognize the video despite different formats, compressions, cropping etc. This ability is not so common in the non-forensic world (at least not yet).</p>

<p>I did try to find something that is targeting consumers but was not able to find any. There do exists APIs (programming interfaces) that provides technology for doing this, and even free, but you will need to develop the software to use it yourself which I am fairly certain is beyond the scope of what you're asking for.</p>

<p>So in conclusion, unless you want to spend a noticeable amount of money on specialized software you will need to go through the stack manually. However, in my link above you will find a trial version of such software so you can see if that is what you're looking for and if it works for your purpose.</p>
 3134: <p>I'd recommend a hybrid approach using both computers and people.</p>

<ol>
<li>Bucket the videos by their length (round to the nearest second)</li>
<li>For each bucket, use ffmpeg to <a href="http://blog.prashanthellina.com/2008/03/29/creating-video-thumbnails-using-ffmpeg/">generate thumbnails</a> at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)</li>
<li>Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.</li>
</ol>

<p>You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse.  Good luck!</p>
