user2362: Using ffprobe shows that both videos have the same framecount, still if I extract the frames as images the compressed one has an extra frame at the beginning (which is just the duplicated first frame).
This causes me a lot of problems since I have to compute the frame differences between the two videos and having an extra frame makes everything out of sync.
Input
ffmpeg -i "720p50_mobcal_ter.avi" -an -f mpeg2video -y "720p50_mobcal_ter.mpg"

Output
ffmpeg version N-76684-g1fe82ab Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55.  6.100 / 55.  6.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 15.100 /  6. 15.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, avi, from '720p50_mobcal_ter.avi':
  Metadata:
    encoder         : Lavf57.14.100
  Duration: 00:00:10.08, start: 0.000000, bitrate: 552974 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720, 554059 kb/s, SAR 1:1 DAR 16:9, 50 fps, 50 tbr, 50 tbn, 50 tbc
Output #0, mpeg2video, to '720p50_mobcal_ter.mpg':
  Metadata:
    encoder         : Lavf57.14.100
    Stream #0:0: Video: mpeg2video, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc
    Metadata:
      encoder         : Lavc57.15.100 mpeg2video
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
frame=   41 fps=0.0 q=31.0 size=     984kB time=00:00:00.78 bitrate=10330.5kbits/frame=   80 fps= 78 q=31.0 size=    1323kB time=00:00:01.56 bitrate=6948.1kbits/frame=  124 fps= 80 q=31.0 size=    1725kB time=00:00:02.44 bitrate=5790.0kbits/frame=  168 fps= 81 q=31.0 size=    2084kB time=00:00:03.32 bitrate=5142.8kbits/frame=  212 fps= 81 q=31.0 size=    2482kB time=00:00:04.20 bitrate=4841.4kbits/frame=  255 fps= 82 q=31.0 size=    2840kB time=00:00:05.06 bitrate=4597.2kbits/frame=  296 fps= 82 q=31.0 size=    3133kB time=00:00:05.88 bitrate=4364.5kbits/frame=  338 fps= 82 q=24.8 size=    3453kB time=00:00:06.72 bitrate=4209.2kbits/frame=  382 fps= 82 q=31.0 size=    3723kB time=00:00:07.60 bitrate=4013.4kbits/frame=  426 fps= 83 q=31.0 size=    4005kB time=00:00:08.48 bitrate=3869.1kbits/frame=  470 fps= 83 q=24.8 size=    4276kB time=00:00:09.36 bitrate=3742.5kbits/frame=  504 fps= 83 q=31.0 Lsize=    4469kB time=00:00:10.06 bitrate=3639.3kbits/s
video:4469kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%

My current solution is to trim the compressed video before computing the differences, but this involves re-encoding and quality loss, making the results useless.
To make the question a bit more complete, here's the commands that I'm using to trim and compute the differences:
ffmpeg -y -i "720p50_mobcal_ter.mpg" -an -f mpeg2video -vf select=gte(n\,1) "cut-720p50_mobcal_ter.mpg"


ffmpeg -y -i "720p50_mobcal_ter.avi" -i "cut-720p50_mobcal_ter.mpg" -filter_complex "blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg"

The question is: how can I avoid having the first duplicated frame? If this can't be avoided, is there a way to remove it without re-encoding the video or just skipping it while computing the differences?
EDIT:
Following @Mulvya's answer, I tried his suggestion but it doesn't work somehow. I understand that it's hard to find the problem without having the video file so, instead, I'm posting a screenshot with the first 4 frames of each video to help you visualize my issue.

As you can see, the compressed video has a grey third frame that messes up the differences.
The reason behind this edit is that I'm sure that the given answer should work, at least from my understanding on how filter mappings work, but it probably doesn't work because the issue here is not what I thought it was.
I expect to have an almost black third frame on the differences video, and not the full frame from the original one.
EDIT 2: Tried the new solution which somehow worked the first time but now it doesn't anymore.
Here's the list of the commands that I'm using, the result is still the same of the screenshot but now the problem has moved to the second frame instead of the third one (basically it's like how the screenshot would be without the 00000.png frames)
ffmpeg  -i "720p50_mobcal_ter.avi" -an -f mpeg1video -y "compressed.mpg" 
ffmpeg  -i "720p50_mobcal_ter.avi" -vf fps=50 "original\image-%05d.png"
ffmpeg  -i "compressed.mpg" -vf fps=50 "compressed\image-%05d.png"
ffmpeg  -y -i "720p50_mobcal_ter.avi" -i "compressed.mpg" -filter_complex "[1:v]trim=start_frame=1,setpts=PTS-STARTPTS[cut];[0:v][cut]blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg" 
ffmpeg  -i "differences.mpg" -vf fps=50 "differences\image-%05d.png"

The source video that I'm using is here just in case you may want to try it yourself.

user1871: You can skip that first frame during the computation process.
ffmpeg -y -i "720p50_mobcal_ter.avi" -i "720p50_mobcal_ter.mpg" -filter_complex "[1:v]trim=start_frame=1,setpts=PTS-STARTPTS[cut];[0:v][cut]blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg"

Edit: as requested, added the commands I used to reproduce the behaviour but I didn't experience the problem.
ffmpeg -i "720p50_mobcal_ter.y4m" -c:v copy "720p50_mobcal_ter.avi"
ffmpeg -i "720p50_mobcal_ter.avi" -c:v mpeg2video "720p50_mobcal_ter.mpg"
ffmpeg -i "720p50_mobcal_ter.avi" -i "720p50_mobcal_ter.mpg" -filter_complex blend=all_mode=difference,hue=s=0 -c:v libx264 -crf 18 "differences.mp4"

ffmpeg -i "720p50_mobcal_ter.avi" avi\frames%03d.png
ffmpeg -i "720p50_mobcal_ter.mpg" mpg\frames%03d.png

My ffmpeg build, taken from Zeranoe
ffmpeg version N-77380-g2dba040 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 11.100 / 55. 11.100
  libavcodec     57. 18.100 / 57. 18.100
  libavformat    57. 20.100 / 57. 20.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 21.100 /  6. 21.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100


2362: <p>Using <em>ffprobe</em> shows that both videos have the same framecount, still if I extract the frames as images the compressed one has an extra frame at the beginning (which is just the duplicated first frame).</p>

<p>This causes me a lot of problems since I have to compute the frame differences between the two videos and having an extra frame makes everything out of sync.</p>

<p><em>Input</em></p>

<pre><code>ffmpeg -i "720p50_mobcal_ter.avi" -an -f mpeg2video -y "720p50_mobcal_ter.mpg"
</code></pre>

<p><em>Output</em></p>

<pre><code>ffmpeg version N-76684-g1fe82ab Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55.  6.100 / 55.  6.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 15.100 /  6. 15.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, avi, from '720p50_mobcal_ter.avi':
  Metadata:
    encoder         : Lavf57.14.100
  Duration: 00:00:10.08, start: 0.000000, bitrate: 552974 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720, 554059 kb/s, SAR 1:1 DAR 16:9, 50 fps, 50 tbr, 50 tbn, 50 tbc
Output #0, mpeg2video, to '720p50_mobcal_ter.mpg':
  Metadata:
    encoder         : Lavf57.14.100
    Stream #0:0: Video: mpeg2video, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc
    Metadata:
      encoder         : Lavc57.15.100 mpeg2video
Stream mapping:
  Stream #0:0 -&gt; #0:0 (rawvideo (native) -&gt; mpeg2video (native))
Press [q] to stop, [?] for help
frame=   41 fps=0.0 q=31.0 size=     984kB time=00:00:00.78 bitrate=10330.5kbits/frame=   80 fps= 78 q=31.0 size=    1323kB time=00:00:01.56 bitrate=6948.1kbits/frame=  124 fps= 80 q=31.0 size=    1725kB time=00:00:02.44 bitrate=5790.0kbits/frame=  168 fps= 81 q=31.0 size=    2084kB time=00:00:03.32 bitrate=5142.8kbits/frame=  212 fps= 81 q=31.0 size=    2482kB time=00:00:04.20 bitrate=4841.4kbits/frame=  255 fps= 82 q=31.0 size=    2840kB time=00:00:05.06 bitrate=4597.2kbits/frame=  296 fps= 82 q=31.0 size=    3133kB time=00:00:05.88 bitrate=4364.5kbits/frame=  338 fps= 82 q=24.8 size=    3453kB time=00:00:06.72 bitrate=4209.2kbits/frame=  382 fps= 82 q=31.0 size=    3723kB time=00:00:07.60 bitrate=4013.4kbits/frame=  426 fps= 83 q=31.0 size=    4005kB time=00:00:08.48 bitrate=3869.1kbits/frame=  470 fps= 83 q=24.8 size=    4276kB time=00:00:09.36 bitrate=3742.5kbits/frame=  504 fps= 83 q=31.0 Lsize=    4469kB time=00:00:10.06 bitrate=3639.3kbits/s
video:4469kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
</code></pre>

<p>My current solution is to trim the compressed video before computing the differences, but this involves re-encoding and quality loss, making the results useless.</p>

<p>To make the question a bit more complete, here's the commands that I'm using to trim and compute the differences:</p>

<pre><code>ffmpeg -y -i "720p50_mobcal_ter.mpg" -an -f mpeg2video -vf select=gte(n\,1) "cut-720p50_mobcal_ter.mpg"


ffmpeg -y -i "720p50_mobcal_ter.avi" -i "cut-720p50_mobcal_ter.mpg" -filter_complex "blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg"
</code></pre>

<p>The question is: how can I avoid having the first duplicated frame? If this can't be avoided, is there a way to remove it without re-encoding the video or just skipping it while computing the differences?</p>

<p>EDIT:</p>

<p>Following @Mulvya's answer, I tried his suggestion but it doesn't work somehow. I understand that it's hard to find the problem without having the video file so, instead, I'm posting a screenshot with the first 4 frames of each video to help you visualize my issue.</p>

<p><a href="https://i.stack.imgur.com/dErAF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dErAF.png" alt="enter image description here"></a></p>

<p>As you can see, the compressed video has a grey third frame that messes up the differences.</p>

<p>The reason behind this edit is that I'm sure that the given answer should work, at least from my understanding on how filter mappings work, but it probably doesn't work because the issue here is not what I thought it was.</p>

<p>I expect to have an almost black third frame on the differences video, and not the full frame from the original one.</p>

<p>EDIT 2: Tried the new solution which somehow worked the first time but now it doesn't anymore.</p>

<p>Here's the list of the commands that I'm using, the result is still the same of the screenshot but now the problem has moved to the second frame instead of the third one (basically it's like how the screenshot would be without the <em>00000.png</em> frames)</p>

<pre><code>ffmpeg  -i "720p50_mobcal_ter.avi" -an -f mpeg1video -y "compressed.mpg" 
ffmpeg  -i "720p50_mobcal_ter.avi" -vf fps=50 "original\image-%05d.png"
ffmpeg  -i "compressed.mpg" -vf fps=50 "compressed\image-%05d.png"
ffmpeg  -y -i "720p50_mobcal_ter.avi" -i "compressed.mpg" -filter_complex "[1:v]trim=start_frame=1,setpts=PTS-STARTPTS[cut];[0:v][cut]blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg" 
ffmpeg  -i "differences.mpg" -vf fps=50 "differences\image-%05d.png"
</code></pre>

<p>The source video that I'm using is <a href="https://media.xiph.org/video/derf/y4m/720p50_mobcal_ter.y4m" rel="nofollow noreferrer">here</a> just in case you may want to try it yourself.</p>
 1871: <p>You can skip that first frame during the computation process.</p>

<pre><code>ffmpeg -y -i "720p50_mobcal_ter.avi" -i "720p50_mobcal_ter.mpg" -filter_complex "[1:v]trim=start_frame=1,setpts=PTS-STARTPTS[cut];[0:v][cut]blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "differences.mpg"
</code></pre>

<p><strong>Edit</strong>: as requested, added the commands I used to reproduce the behaviour but I didn't experience the problem.</p>

<pre><code>ffmpeg -i "720p50_mobcal_ter.y4m" -c:v copy "720p50_mobcal_ter.avi"
ffmpeg -i "720p50_mobcal_ter.avi" -c:v mpeg2video "720p50_mobcal_ter.mpg"
ffmpeg -i "720p50_mobcal_ter.avi" -i "720p50_mobcal_ter.mpg" -filter_complex blend=all_mode=difference,hue=s=0 -c:v libx264 -crf 18 "differences.mp4"

ffmpeg -i "720p50_mobcal_ter.avi" avi\frames%03d.png
ffmpeg -i "720p50_mobcal_ter.mpg" mpg\frames%03d.png
</code></pre>

<p>My ffmpeg build, taken from Zeranoe</p>

<pre><code>ffmpeg version N-77380-g2dba040 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 11.100 / 55. 11.100
  libavcodec     57. 18.100 / 57. 18.100
  libavformat    57. 20.100 / 57. 20.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 21.100 /  6. 21.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
</code></pre>
