Live video overlays and videowall with Nimble Live Transcoder

In the dynamic landscape of media streaming, enhancing content through creative overlays has become an indispensable aspect of media delivery. Nimble Streamer Live Transcoder emerges as a powerful tool in this realm, offering a versatile platform for superimposing text, images, animations, or live video streams while preserving and enhancing your brand identity. Here, we will talk about overlaying video or pictures on top of the video.

Prerequisites

Before using the Transcoder for Nimble Streamer, you must have a basic subscription to WMSPanel, as Transcoder does not work for trial servers. The Transcoder license costs 50 USD per server per month, there is no trial version for it.

Please find more details on installing and registering an instance of Transcoder on the following installation page.

You should also have a basic knowledge of Live Transcoder setup, please refer to our documentation reference.

Basics of operation

First, a base layer that determines the output resolution to create an overlaid video must be defined. This can be a live video or an image.

Then, all other input sources will be stacked on top of this main layer via the Overlay filter. You may also need the Scale filter to adjust the size of the other sources to place and fit them on the base layer.

The next concept is that you must define one of the sources as the Main stream. Once the scenario is started, the Transcoder will start equalizing all other streams’ timestamps to the Main stream to provide a stable source for Overlays.

Please also check an extended video tutorial about creating mosaic video wall using overlays at the end of this article.

Let's go through the details of this scenario setup below.

Sources for overlays: live streams

All supported source types can be used as the main source for video or backing layer: live streams, image or video files, and NDI sources.

However, the Overlay filter has some limitations for using NDI streams at the moment, more about it  are provided below.

Put a Video source decoder block on a timeline, and the Add video source dialogue will appear.

This will allow choosing the source type: Stream, File, NDI. First, we describe the regular live stream setup.

Type in an application and choose the stream to auto-fill the stream’s name. If the streams are registered in the panel, they will appear for selection. Choose the required one.

Next, you must also choose the purpose of this source: it could be either the Main stream or the PTS Adjusted stream.

Checking the Main stream box will mark the stream as main. For your convenience the main input will be marked with a bull's eye:

To equalize any other stream's timestamps with the Main's timestamps, choose the PTS adjustment enabled checkbox in its source’s settings.

Notice that NDI doesn’t have a timestamp adjustment option at the moment, so it can be used as ‘Main stream’ only.

Since the Main and PTS Adjustment are defined, you can safely chain several Overlays (or any other filters) in a Transcoder’s scenario. This versatility will help create distinctive visual effects for your video output.

As for the Audio, the Main is the only stream that will have perfectly synced audio. We do not advise taking audio from other streams, as we cannot guarantee the audio sync in such a case.

If the source live stream is over or stopped, the decoder will hold the previously available frame in its output, making the next in a chain Overlay filter operational.

In case the Transcoder scenario is just starting and one of the live sources participating in the Overlay chain is missing, the Scenario will fail to start. We advise using hot-swap failover to ensure the source always has some viable frames to process.

Source for overlay: video file, static, or animated image files

Move on to the next type of decoder: File Source decoder. Although we already have an article about decoding from files, here we will notice the crucial points.

As the name implies, it uses files on the same server Nimble Streamer is running as input. As you specify the path to some file and the scenario is running, then the File Decoder generates looped output with stable timestamps which can be assigned as Main. This is true for both video or image files. As you might have spotted, there’s no PTS Adjustment checkbox. It is considered that if the Main stream is not specified, such sources will be forced as PTS adjusted.

The video file formats supported are MP4, MOV and MKV. The codecs are listed in our codecs reference. Videos with alpha channel are not supported at the moment. Feel free to ask our support team if you plan using them.

Supported image formats include PNG, APNG, JPEG, GIF, TIFF and BMP. The alpha channel or GIF transparency is available. GIF and APNG animations are supported too.

In case of a single frame image, the file will be re-loaded as fast as the file is modified or overwritten. Notice that reloading will not work for video files or files with a sequence of frames, like APNG animations.

All the above makes the File Decoder a handy source for animated or updated Overlays like moving logos or bar ads.

Notice that path considerations are different between Windows and Linux.

For more information on File decoder, please read Binding un-synced video and audio sources article.

Single Overlay filter setup

Now let's move on to some practical setup that will be a basis for cases like video wall or mosaic.

We will combine a backing static layer with some live stream.

Suppose we require a standard 1080p output and on top of it a live video will be played with a resolution half that of a frame.

Prepare an empty 1080p resolution image, that will define a frame and put it in some folder accessible by Nimble. It could be a solid color, but for clarity, the color bars are used. We advise using PNG, as it is a more versatile format.

Put Video Source, select File, and type in path and filename. Set FPS to 30.

Add another Video Source and this time specify a received live stream by its application and stream names. We'll choose live/bunny here.

Next, apply a Scale filter to ensure that the live stream does not fill the background layer. Let’s make it 540p as it sounds applicable. Drag a line from the Live Stream Source decoder to the Scale filter.

Next goes the Overlay filter. Drop it on a timeline closer to File Source decoder. Connect the file decoder and Overlay filter by dragging a line between them. Specify the position of an overlaid video.

Set the output encoder by dropping Video Source to the timeline, name the output stream (like overlays/out) and choose any settings you like, default values are acceptable for checking.

One thing to note: Ensure that the Main video is received from: option refers to a File Source, as this specifies the source for the backing layer with a higher resolution.

Now it is time to decide which of the inputs to consider primary for timestamp adjustment.

The concern here is that the output stream could only have one audio source that would be synced with video and it must be marked as Main. So in this simple scenario, the preference naturally leans toward the live stream if we add its audio as an Audio source.

However, if syncing sound isn't necessary - let's say you just want background music - then specify the File source as the Main stream. In this case, the timestamps will be produced by the Transcoder itself and all other stream’s timestamps will be adjusted to them.

Click the Save button and in a few moments, you will have overlaid scaled video in the output stream via any Nimble-supported streaming protocol. Click Live Streams above the save button to jump from the Scenario to the List of Live Streams.

Multiple Overlay filter setup (videowall)

Now we're moving to a more advanced setup: chaining more video sources.

Add another Video Source (we use a file here for simplicity, but it's up to you), Scale, and Overlay. Use the same params as above, and chain them as shown in the picture. In the Overlay’s filter settings, make sure Main video is received from: is set as a previous Overlay filter.

You may add more video inputs to overlay them to form a video frame like below:

Hardware accelerated Overlay and other filters

Is the hardware acceleration available for this feature set? Absolutely, but just for Ubuntu Linux users. Both NVENC and QuickSync have hardware-accelerated versions of overlay filters. It's better to use NVENC-ffmpeg or NVENC-quicksinc decoders with these filters for performance reasons. This will avoid getting and putting frames to the system’s RAM and losing CPU cycles by using additional hwupload and hwdownload filters.

However, these decoders come with certain limitations. While they allow you to decode H.264 or HEVC (if supported by the GPU) mp4/mkv video files, they are not suitable for handling static images or animated files such as GIFs.

Remember to consider the hardware-accelerated scale_npp (scale_cuda) and vpp_qsv (scale_qsv) filters for resizing a video source. NVENC-ffmpeg decoder allows scaling in its settings by the resize option. Explore additional information about NVENC-only scenarios in the following article.

If you require more details on creating an NVENC-only scenario, please let us know. We have plans to publish an article specifically addressing this topic.

Troubleshooting

Here we’ll describe some tips and common errors you may face while while using Transcoder.

Although we did an extensive job of improving the stability of the Overlay filter, anyway we recommend using the out-of-process mode to provide better stability for the Nimble Streamer server. This mode will ensure that a single scenario with an issue won’t crash Nimble Streamer. Read about it in this article.

If you’re changing source type, let's say from File to Live, scenario restart is considered. It can be performed while editing the Scenario by the corresponding icons near its name.

Some common messages in the log:

• [video_decoder] E: failed to open stream /some/path/file.ext - You’ll get this error in the log if the specified file name is not found or Nimble has issues accessing it (e.g. due to permission restrictions).
• [video_filter] E: failed to parse filter graph - This means something is wrong with a filter chain from the parameter or there are filter name incompatibility errors between the decoder and filter. Other messages are needed for details.
• E: encoder too slow’  for encoder or decoder in most cases it means you are running out of processing power. If the multicore CPU is used, and you observe that some cores are not loaded when the error is present, this may be improved by allowing more cores for decoding or encoding.
• [video_filter] E: reset video filter on pts gap(prev=XXX, cur=YYY, timescale=1/1000), filter=XXXXX, stream=larix/2 - It's not an error, but the indication that due to the timestamp jumps, the filter was re-created.
• [video_decoder] E: decode interval gap detected, decode_interval=XXXX for [_some_path_] - it very likely is a decoding performance issue or the inconsistent system time.

Mosaic video wall tutorial

We've made an extended video tutorial about creating mosaic video wall using overlays in Live Transcoder