LtRoyalShrimp

Guides, tutorials and more…

Generic post 

How to get the best stream quality with an NVIDIA GPU (NVENC)

ltroyalshrimp

Having toyed around in the streaming world for 10+ years and always trying to maximize stream quality given the resolution, frame rate, and bit rate restrictions, I’m happy to say that I’ve arrived at settings that will be powering my stream from now on.

NVIDIA released the NVENC SDK version 10 some time ago and with it came new NVENC tuning options, allowing us to squeeze every inch of quality out of the NVENC encoder. You can read more about the changes here – https://developer.nvidia.com/blog/introducing-video-codec-sdk-10-presets/

Note: Starting with OBS Studio 28.1 Beta, the improved NVIDIA encoding settings are built into OBS Studio. The guide below is no longer required.

The trick to getting the best quality is by replacing the FFMpeg version that OBS uses. As of OBS Studio 27.1.3 an older version of FFMpeg is used that is not capable of using the new NVIDIA SDK changes, so we need to manually update it.

Requirements:

  • OBS Studio 27.1.3
  • StreamFX 11.0
  • NVIDIA RTX 20-series GPU or newer with the Turing encoder (7th gen)
    • More details: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

Set up steps


Install OBS Studio 64bit

Once OBS is installed, install StreamFX – https://streamfx.xaymar.com/

Keep OBS Studio closed

Locate where OBS Studio is installed, which by default is C:\Program Files\obs-studio.

Simply right-click on the OBS Studio shortcut and select “open file location”

Navigate to  bin and then 64bit.

Look for a file called avcodec-58.dll. Is it there? Good.

Download FFmpeg version 4.4 shared GPLv3 from BtbN. The file should be called ffmpeg-n4.4-latest-win64-gpl-4.4.zip

Extract the zip file.

Go into the folder and copy all the .dll files.

Then, go back to the OBS Studio 64bit folder and paste the new FFMpeg files. You will be asked if you want replace the existing files – Make sure to select yes!

Now open OBS Studio and open the settings

Click on the Output settings and set the mode to advanced

From the encoder dropdown select NVENC with FFmpeg

Now, here are my recommended settings. These are mostly pulled from Xaymar’s excellent encoding guide found here.

  • Preset: High Quality
  • Profile: High
  • Level: Automatic
  • Mode: Constant bitrate
  • Two Pass: Enabled
  • Look Ahead: 32 frames
  • Adaptive I-frames: Enabled
  • Adaptive B-frames: enabled
  • Target bit rate: Depends on your resolution and frame rate. Refer to Xaymat’s high quality streaming guide here.
  • Buffer size: Needs to be 2x the target bit rate
  • Minimum QP: -1
  • Maxikmum QP: -1
  • Spatial Adative Quantization: Enabled
  • Spatial Adaptive Quantization Strength: -1 (The default is 15)
  • Temporal Adaptive Quantization: Enabled
  • Maximum B-frames: 4
  • B-frame reference mode: Use only Middle B-frames as reference
  • Zero Latency: Off
  • Weighted prediction: off
  • Non-reference P-frames: Enabled
  • Reference Frames: -1

Now here is where we get into what makes the magic happen: Custom command line parameters. Enter these parameters:

-preset=p7 -tune=hq -multipass=2

That’s it. Now you are streaming with the absolute best quality NVENC has to offer.