Zabbix monitoring of Nimble Streamer, SRT streams and NVidia GPU

A continuous monitoring is needed if you want to ensure the correct functioning of any server. Getting basic metrics such as RAM, CPU or bandwidth usage helps rapidly responding to performance issues.

We’d like to show a way for our customers to add Nimble Streamer server-specific metrics as well as SRT streams statistics to an open-source Zabbix monitoring system. This will allow you to keep extended metrics in your monitoring system, which sometimes are not even available in WMSPanel, like RAM cache status for certain periods of time.

SRT metrics will also be helpful not only for stream health checks but also for analysis of how this new protocol works for your environment.

TLDR: You can find Zabbix templates and agent configs for Nimble Streamer general and SRT monitoring in this GitHub repo.

1. Monitoring features and metrics

We provide two Zabbix templates:

1. General server metrics
2. SRT metrics

Each template is based on certain calls of Nimble Streamer native API which we'll describe in later sections.

Take a look at this quick demo video showing all steps and brief capabilities overview

1.1 General metrics

The first TemplateNimbleServer.yaml in our github collects Nimble Streamer server statistics, which are obtained via /manage/server_status method of Nimble Streamer native API. This template includes some graphs for the RAMCache and FileCache monitoring.

You can find a detailed description of each parameter on server_status API method page.

1.2. SRT metrics

The second TemplateNimbleSRT.yaml on our github is made for monitoring the SRT streams statistics using /manage/srt_sender_stats and /manage/srt_receiver_stats API methods respectively. It uses Low Level Zabbix’s Discovery (LLD) to dynamically discover incoming and outgoing SRT streams metrics. For each connection (either sender or receiver) an Application with detailed statistics is created. Brief description is available if you move a mouse over [?] icon.

Please note, that the Nimble native API doesn’t send full scope of session metrics from SRT lib, but you can find full list at SRT library docs page as a reference.

Nimble Streamer returns the following list of values by API call that are also used in the Nimble SRT template:

• SRT Receiver/Sender Window Flow
• SRT Receiver/Sender Window Flight
• SRT Receiver/Sender Window Congestion
• SRT Receiver/Sender Total Packets Received 
• SRT Receiver/Sender Total Packets Lost
• SRT Receiver/Sender Total Packets Dropped 
• SRT Receiver/Sender Total Bytes Received
• SRT Receiver/Sender Status 
• SRT Receiver/Sender Session Time
• SRT Receiver/Sender RTT
• SRT Receiver/Sender Retry Count
• SRT Receiver/Sender Rate
• SRT Receiver/Sender Packets Belated
• SRT Receiver/Sender NAKs Sent 
• SRT Receiver/Sender Estimated mbpsMaxBandwidth
• SRT Receiver/Sender Estimated mbpsBandwidth
• SRT Receiver/Sender bytesLost
• SRT Receiver/Sender bytesDropped 

At the moment we don't have any SRT-specific Graphs or Triggers or Screens with the data we gather using these templates, but we'll probably add some improvements in future versions. We also have some plans on extending Nimble API capabilities to show more metrics and protocols, let us know if you're interested.

1.3 NVidia GPU monitoring

If you have NVidia GPU you can gather its metrics in Zabbix via Nimble’s Native API /manage/server_status method. We provide a Zabbix NVidia template and related config that relies on it to gather statistics. For temperature monitoring, the nvidia-smi utility is used, so make sure you can run it successfully from the console. Nimble Server NVidia template is actually the same, but with the addition of NVidia GPU discovery rules. To install it, follow the same steps and prerequisites as for Nimble Server Template.

Here are examples of stats.

1.4 Pre-requisites

The templates that we provide are made using Zabbix version 5.2.6, but the latest version of Zabbix should work fine too.

Notice that we used Ubuntu Linux 20.04 for Nimble Streamer server instance and setup details are provided for that OS. There should not be any issues with any Linux distro if you change package manager commands and repo names.

We didn’t check the provided templates with Nimble Streamer and Zabbix Agent running on Windows but it should also work as soon as you're able to install and run some releavant version of jq and cURL on it. Contact us if you succeed using this monitoring method for Windows version of Nimble.

We also assume that you already have the Zabbix server installed. If not, please find setup details on Zabbix's comprehensive documentation web pages. Zabbix provides many installation choices. The virtual appliance works best for us but you may choose the one that best suits your environment. If you have specific issues with it, ask Zabbix support or the Zabbix community forum.

2. Link Templates and Gather Data

Here are the steps you need to complete in order to add Nimble-specific metrics to your Zabbix server:

1. Enable Nimble Streamer HTTP API for the server instance which you want to monitor
2. Install cURL if it is not yet installed
3. Import provided Templates to your Zabbix Server
4. Install Zabbix Agent to your Nimble server
5. Update Zabbix Agent configs to make templates work
6. Add Nimble Streamer host to Zabbix server
7. Apply Nimble Streamer Templates to a specific host

Here are the details of each step.

2.1 Enable Nimble Streamer API

To enable Nimble native API, add ‘management_listen_interfaces = *’ to /etc/nimble/nimble.conf file. We have used the default port 8082 to make Zabbix Agent receive metrics from Nimble. Please follow this documentation page. to find out more about enabling Nimble API.

2.2. Install cURL

Make sure that cURL is installed on Nimble server. It can be used to make sure API is set up fine and it will be used further.

2.3 Import templates

Next, import Nimble Templates into Zabbix.

Download template files from respective GitHub repo to some location. From the Zabbix’s pop-up menu, go to Configuration -> Templates and click the Import button at the right top of the Templates page.

At the Import page, make sure that the checkboxes look like on the following screenshot and choose the template file to import.

Complete importing process by clicking the Import button. "Imported successfully" message should appear at the top of the page.

Navigate to the Templates page again, type "Nimble" at Name field and click Apply to set filter for Nimble templates only. You will notice two Nimble templates available.

2.4 Install Zabbix agent

You need to install Zabbix Agent on your Nimble Streamer server to send your server’s metrics to Zabbix Server. The easiest method is to install the agent using official Zabbix repositories by executing the following commands:

wget https://repo.zabbix.com/zabbix/5.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.2-1+ubuntu20.04_all.deb

sudo dpkg -i zabbix-release_5.2-1+ubuntu20.04_all.deb

sudo apt update 

sudo apt install zabbix-agent jq

sudo systemctl enable zabbix-agent

The "jq" JSON parser is needed to process data from Nimble API responses.

2.5 Update Zabbix agent configs

Now complete the setup of Zabbix Agent by editing the /etc/zabbix/zabbix.conf file with your favorite editor to specify a Zabbix Server IP or Domain name via"Server=" option.

Please also add hostname of the server at "Hostname=" option and use the same name as Host name field when adding a host to Zabbix.

Download Zabbix Agent configs for Nimble from GitHub repo and copy them to /etc/zabbix/zabbix_agentd.d/.

wget https://raw.githubusercontent.com/WMSPanel/zabbix/main/zbx_nimble.conf zbx_nimble.conf

wget https://raw.githubusercontent.com/WMSPanel/zabbix/main/zbx_nimble_srt.conf -O zbx_nimble_srt.conf 

sudo cp *.conf /etc/zabbix/zabbix_agentd.d/

Check if your firewall allows incoming connection to port 10050 and start Zabbix Agent:

sudo systemctl start zabbix-agent

2.6 Add Nimble Streamer host to Zabbix

Switch to Zabbix web interface and add your Nimble Streamer server by clicking Create Host at Configuration -> Hosts page as usual. 

Select Interface type as Agent, and specify your Nimble server’s IP or domain name to connect to. Use Nimble group name and the same Host name as you defined in zabbix.conf file.

2.7 Apply Nimble Streamer Templates to a host

At the Templates tab select Host group Nimble and select Templates.

Click Add button after selecting Templates to add host to Zabbix. The added host will appear at the top of the page.

If you did everything correctly, you will notice availability status (ZBX) will become green after a few moments:

You may get the following error if the hostname in zabbix_agent.conf file is not matching the hostname you’ve entered when adding Hosts:

The setup is finished. Check if the metrics are being received by Zabbix at Monitoring -> Latest data menu. 

Now you can use this data to compare with data received by other templates or methods, create unique Graphs to add to the Dashboard or custom Screens, and add some Triggers to track if your server’s health goes down.

That's it. We'll continue improving our Zabbix monitoring templates and configs, stay tuned for updates and contact us if you have any questions or suggestions.

Follow us in social media to get updates about our new features and products: YouTube, Twitter, Facebook, LinkedIn, Reddit, Telegram

Related documentation

Nimble Streamer native API, Nimble Streamer configuration methods, Softvelum github