June 7, 2017

NVidia NVENC settings for H.265 in Live Transcoder

NVidia® Products with the Kepler, Maxwell and Pascal generation GPUs contain a dedicated accelerator for video encoding, called NVENC, on the GPU die.

Nimble Streamer Live Transcoder has full support for NVidia video encoding and decoding hardware acceleration. Having the hardware capable of the processing and drivers properly installed, our customer can choose NVENC to handle streams' encoding.

HEVC (H.265) codec is supported by multiple NVidia GPUs so now it's also supported in our Live Transcoder. You can use Transcoder to decode and encode H.265 on your hardware under both Linux and Windows.

You can take a look at the list of NVidia GPUs capable of hardware encoding acceleration. To make HW acceleration work, you need to install the graphic card drivers into the system. Use this link to download and install them. If you haven't yet installed Nimble Streamer transcoder, use this page to find proper setup instruction.

Notice that NVENC has a couple of known issues, please check Troubleshooting section below in this article.

Setting encoder

The transcoding scenarios are created using our web UI. You can check this YouTube playlist to see how various use cases are defined. Takes just a couple of minutes to complete.

Scenarios setup page
Example of HEVC encoding scenario

To set up NVENC settings you need to open encoder settings dialog and choose "nvenc" as the Encoder and select h265 as Codec.

After that you can add various parameters and set up specific values to tune up your encoding process. Please find full list of available encoding parameters below.


Specifies H.265 preset.
  • hp - high performance
  • default - tradeoff between performance and quality
  • hq - high quality
  • llhp - low latency high performance
  • ll - default low latency preset and the quality and speed is midway of the two other presets
  • llhq - low latency high quality
  • lossless - default lossless preset
  • losslesshp - lossless high performance
  • bd - BlueRay disk quality


Specifies H.265 profile.
  • main
  • auto


Specifies the level of the encoded bitstream.
  • auto
  • 1
  • 2
  • 21
  • 3
  • 31
  • 4
  • 41
  • 5
  • 51
  • 52
  • 6
  • 61
  • 62


Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on.

If you set it to "auto" then transcoder will choose the least busy GPU.


Number of pictures within the current GOP (Group of Pictures).
  • 1 - only I-frames are used


Specifies maximum number of B frames between non-B-frames.
0 - no B-frames
1 - IBP
2 - IBBP


Specifies the DPB size used for encoding.

Setting it to 0 will let driver use the default dpb size. The low latency application which wants to invalidate reference frame as an error resilience tool is recommended to use a large DPB size so that the encoder can keep old reference frames which can be used if recent frames are invalidated.

fps_n, fps_d

Set output FPS numerator and denominator. It only affects num_units_in_tick and time_scale fields in SPS.
If fps_n=30 and fps_d=1 then it's 30 FPS
If fps_n=60000 and fps_d=2002 then it's 29.97 FPS
Source stream FPS or filter FPS is used if fps_n and fps_d is are set.


Sets bitrate type.
  • cqp - Constant QP mode
  • vbr - Variable bitrate mode
  • cbr - Constant bitrate mode
  • vbr_minqp - ariable bitrate mode with MinQP
  • ll_2pass_quality - Multi pass encoding optimized for image quality and works only with low latency mode
  • ll_2pass_size - Multi pass encoding optimized for maintaining frame size and works only with low latency mode
  • vbr_2pass - Multi pass VBR


Sets bitrate in Kbps.


Sets max bitrate in Kbps.


Specifies the VBV(HRD) initial delay in Kbits.
0 - use the default VBV initial delay


Specifies the VBV(HRD) buffer size in Kbits.
0 - use the default VBV buffer size

qpi, qpp, qpb

Specifies the initial QP to be used for encoding, these values would be used for all frames if in CQP mode.


Specifies the minimum QP used for rate control.


Specifies the maximum QP used for rate control.


Specifies the initial QP used for rate control.


Target Constant Quality level for VBR mode (range 0-51 with 0-automatic);


If your input stream is anamorphic you might need to save its SAR parameter in the output as well, especially if you’re using a 'scale' filter in your Transcoder pipeline while DAR = SAR x Width / Height. Nimble supports keeping input SAR using keep-sar parameter set to true for encoder in its ‘Video output’ section. SAR/DAR/PAR correlation is described in this article.


  • 0 - disable
  • 1 - enable


Specifies the frame/field mode.


Specifies the desired motion vector prediction precision.


Enable Spatial adaptive quantization.
  • 0 - disable
  • 1 - enable


Specifies AQ strength.
AQ strength scale is from 1 (low) - 15 (aggressive).


Specifies Temporal adaptive quantization.
  • 0 - disable
  • 1 - enable


Set to enable to minimize GOP-to-GOP rate fluctuations.
  • 0 - disable
  • 1 - enable


Enable lookahead with depth <lookaheadDepth>.


Maximum depth of lookahead with range 0-32 (only used if enableLookahead=1)


Disable adaptive I-frame insertion at scene cuts (only has an effect when lookahead is enabled).
  • 0 - none
  • 1 - disable adaptive I-frame insertion


Disable adaptive B-frame decision (only has an effect when lookahead is enabled)
  • 0 - none
  • 1 - Disable adaptive B-frame decision


Specifies the minimum size of luma coding unit.
  • auto
  • 8 - 8x8
  • 16 - 16x16
  • 32 - 32x32
  • 64 - 64x64


Specifies the maximum size of luma coding unit. Currently NVENC SDK only supports maxCUSize equal to 32.
  • auto
  • 8 - 8x8
  • 16 - 16x16
  • 32 - 32x32
  • 64 - 64x64


Constrained intra prediction.
  • 0 - disable(default)
  • 1 - enable


loop filtering across slice boundary
  • 0 - enable(default)
  • 1 - disable


Write Access Unit Delimiter syntax:
  • 0 - disable
  • 1 - enable(default)


Use of long term reference pictures for inter prediction.
  • 0 - disable(default)
  • 1 - enable


Gradual decoder refresh or intra refresh. If the GOP structure uses B frames this will be ignored.
  • 0 - disable
  • 1 - enable


Specifies the interval between successive intra refresh if enableIntrarefresh is set.
Requires enableIntraRefresh to be set.
Will be disabled if NV_ENC_CONFIG::gopLength is not set to NVENC_INFINITE_GOPLENGTH.


Specifies the length of intra refresh in number of frames for periodic intra refresh.
This value should be smaller than intraRefreshPeriod.


Specifies the LTR operating mode.
Set to 0 to disallow encoding using LTR frames until later specified.
Set to 1 to allow encoding using LTR frames unless later invalidated.


Specifies the number of LTR frames used.
  • If ltrTrustMode=1, encoder will mark first numLTRFrames base layer reference frames within each IDR interval as LTR.
  • If ltrMarkFrame=1, ltrNumFrames specifies maximum number of ltr frames in DPB.
  • If ltrNumFrames value is more that DPB size(refs) encoder will take decision on its own.


sliceMode in conjunction with sliceModeData specifies the way in which the picture is divided into slices
  • sliceMode = 0 CTU based slices,
  • sliceMode = 1 Byte based slices,
  • sliceMode = 2 CTU row based slices,
  • sliceMode = 3, numSlices in Picture
  • When sliceMode == 0 and sliceModeData == 0 whole picture will be coded with one slice.


Specifies the parameter needed for sliceMode. For:
  • sliceMode = 0, sliceModeData specifies # of CTUs in each slice (except last slice)
  • sliceMode = 1, sliceModeData specifies maximum # of bytes in each slice (except last slice)
  • sliceMode = 2, sliceModeData specifies # of CTU rows in each slice (except last slice)
  • sliceMode = 3, sliceModeData specifies number of slices in the picture. Driver will divide picture into slices optimally.


Specifies the max temporal layer used for hierarchical coding.

These are the parameters which you can use already in order to control NVidia video encoding hardware acceleration.


Linux drivers

Linux version of NVENC has some known issues with HEVC support for outdated driver versions.
If you see the following message in nimble.log: 
failed to lock output bitstream encoder=0x3d871f0, status=8
then you need to update your driver to version 381.22 or higher. Version 378.13 has issues described in this discussion.

GTX 950

Linux version of NVENC does not allow more than one HEVC decoding session for GTX 950. This is also a known issue which is reproduced for ffmpeg. This problem might not exist on other cards.

Other issues

Please check Transcoder troubleshooting article covering most frequent issues.

We keep improving our transcoder feature set, so please contact us for any questions.

Related documentation

NVIDIA, the NVIDIA logo and CUDA are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and/or other countries. 

No comments:

Post a Comment

If you face any specific issue or want to ask some question to our team,

This will give much faster and precise response.
Thank you.

Note: Only a member of this blog may post a comment.