user-1: I'm compressing some videos for archiving, and I came to this configuration which gives me a nice overall quality on a small size:
ffmpeg -threads 4 -y -hide_banner -i input.mp4 -vf scale=-2:480 -c:v libx264 -b:v 200k -preset slow -tune animation -movflags faststart -pix_fmt yuv420p -an -f mp4 -pass 1 NUL
ffmpeg -threads 4 -y -hide_banner -i input.mp4 -vf scale=-2:480 -c:v libx264 -b:v 200k -preset slow -tune animation -movflags faststart -pix_fmt yuv420p -pass 2 2-pass-x264.mp4

This output quality is acceptable for my needs:

But I would like to enjoy hardware acceleration, it is way much faster.
So I tried:
ffmpeg -hwaccel nvdec -y -hide_banner -i input.mp4 -vf scale=-2:480 -movflags faststart -pix_fmt yuv420p -c:v h264_nvenc -b:v 200k -maxrate 2M -bufsize 1M -rc-lookahead:v 30 -rc:v vbr_hq -preset slow -profile:v high -level 5.1 -an h264_nvenc.mp4

But this gives me an output with worse overall quality, and in complex parts, things gets worse:

Is it possible to achieve near same overall quality (for low bitrate) using h264_nvenc encoder?

user34942: Specify bitrate as -b:v 0 made the trick forme, the overall bitrate bumped from 2m to 16m.
ffmpeg -i in.mp4 -c:v h264_nvenc -b:v 0 out.mp4


user19112: h264_nvenc is able to encode in constant quality, at least from Pascal onwards. I had to do quite some digging around the internet to figure out how to properly use it, but finally found a working formula.
A command like below will record in constant quality, and the CQ value 26 will produce about the same but slightly better quality, measured with VMAF, as a libx264 encode with the following options: -vcodec libx264 -level 4.1 -profile:v high -tune film -preset medium -crf 23. The H264_Nvenc bitrate was about 19% higher than the libx264 encode, but it had a slightly better VMAF score. I made these tests on some 1080p60 game recording, but also on some 1080p movie trailers.
ffmpeg -i input.mkv -c:v h264_nvenc -preset p6 -profile:v high -tune hq -rc-lookahead 8 -bf 2 -rc vbr -cq 26 -b:v 0 -maxrate 120M -bufsize 240M output.mkv

Important:

Unless you set the Max rate and buffer size via -maxrate 120M -bufsize 240M, the CQ mode doesn't behave accordingly and lowering its value won't yield better results.
The Constant Quality mode is activated by specifying Variable Bitrate rate control together with a CQ parameter, a bitrate set to zero, and the above two parameters set to some large value which won't limit the output. Don't be afraid of those values, as bitrate will be quite good.
The limits above will force the encode to produce a Level 5.1 encoding. Even higher limits like 300/600 produced a video with Level labeled 6.2, which had trouble playing correctly in NLE's as Vegas Pro. If you require a certain maximum Level, ensure the maxrate and buffer size ar within the level's limits and check the resulted output.


-1: <p>I'm compressing some videos for archiving, and I came to this configuration which gives me a nice overall quality on a small size:</p>

<pre><code>ffmpeg -threads 4 -y -hide_banner -i input.mp4 -vf scale=-2:480 -c:v libx264 -b:v 200k -preset slow -tune animation -movflags faststart -pix_fmt yuv420p -an -f mp4 -pass 1 NUL
ffmpeg -threads 4 -y -hide_banner -i input.mp4 -vf scale=-2:480 -c:v libx264 -b:v 200k -preset slow -tune animation -movflags faststart -pix_fmt yuv420p -pass 2 2-pass-x264.mp4
</code></pre>

<p>This output quality is acceptable for my needs:</p>

<p><a href="https://i.stack.imgur.com/jyEYX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jyEYX.png" alt="x264 encoded frame"></a></p>

<p>But I would like to enjoy hardware acceleration, it is way much faster.</p>

<p>So I tried:</p>

<pre><code>ffmpeg -hwaccel nvdec -y -hide_banner -i input.mp4 -vf scale=-2:480 -movflags faststart -pix_fmt yuv420p -c:v h264_nvenc -b:v 200k -maxrate 2M -bufsize 1M -rc-lookahead:v 30 -rc:v vbr_hq -preset slow -profile:v high -level 5.1 -an h264_nvenc.mp4
</code></pre>

<p>But this gives me an output with worse overall quality, and in complex parts, things gets worse:</p>

<p><a href="https://i.stack.imgur.com/b7udX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b7udX.png" alt="h264_nvenc encded frame"></a></p>

<p>Is it possible to achieve near same overall quality (for low bitrate) using h264_nvenc encoder?</p>
 34942: <p>Specify bitrate as <code>-b:v 0</code> made the trick forme, the overall bitrate bumped from <code>2m</code> to <code>16m</code>.</p>
<pre><code>ffmpeg -i in.mp4 -c:v h264_nvenc -b:v 0 out.mp4
</code></pre>
 19112: <p>h264_nvenc is able to encode in constant quality, at least from Pascal onwards. I had to do quite some digging around the internet to figure out how to properly use it, but finally found a working formula.</p>
<p>A command like below will record in constant quality, and the CQ value 26 will produce about the same but slightly better quality, measured with VMAF, as a libx264 encode with the following options: <code>-vcodec libx264 -level 4.1 -profile:v high -tune film -preset medium -crf 23</code>. The H264_Nvenc bitrate was about 19% higher than the libx264 encode, but it had a slightly better VMAF score. I made these tests on some 1080p60 game recording, but also on some 1080p movie trailers.</p>
<pre><code>ffmpeg -i input.mkv -c:v h264_nvenc -preset p6 -profile:v high -tune hq -rc-lookahead 8 -bf 2 -rc vbr -cq 26 -b:v 0 -maxrate 120M -bufsize 240M output.mkv
</code></pre>
<p>Important:</p>
<ul>
<li>Unless you set the Max rate and buffer size via <code>-maxrate 120M -bufsize 240M</code>, the CQ mode doesn't behave accordingly and lowering its value won't yield better results.
The Constant Quality mode is activated by specifying Variable Bitrate rate control together with a CQ parameter, a bitrate set to zero, and the above two parameters set to some large value which won't limit the output. Don't be afraid of those values, as bitrate will be quite good.</li>
<li>The limits above will force the encode to produce a Level 5.1 encoding. Even higher limits like 300/600 produced a video with Level labeled 6.2, which had trouble playing correctly in NLE's as Vegas Pro. If you require a certain maximum Level, ensure the maxrate and buffer size ar within the level's limits and check the resulted output.</li>
</ul>
