Showing posts with label robustness. Show all posts

December 25, 2013

Dispersa: a distributed streams availability check service

WMSPanel team proudly presents a new product for improving streaming media quality of service.

It's Dispersa, a service for distributed streams availability checking. This is a first step towards creating a service for checking the quality of experience.

If you're running a streaming media business you need to bring proper quality of service. This requires some significant effort. One of the basic but efficient practices is the stream availability check. This one needs some external point of view at your streaming infrastructure hence the additional efforts.

Currently you can enter a URL of the stream and make it checked over for free.

The following protocols are supported:

HTTP Live Streaming (HLS);
HTTP Dynamic Streaming (HDS);
Smooth Streaming;
MPEG-DASH;
RTSP;
RTMP;
Icecast;
Shoutcast.

The stream is being checked in parallel from all of existing checkpoints. A checkpoint is a server with Nimble Streamer installed. Nimble is performing required check and sends results to WMSPanel which shows all performed actions on the monitoring dashboard.

We will be expanding this distributed network of checkpoints by adding new servers in various geographical points. You may also participate in this: you may add checkpoint at your own locations and be listed in our checkpoints page. Later on we'll add the ability to add checkpoints in your private networks to make all check-overs in closed environment.

Now make your stream availability check now for free.

Also try our 24/7 streams monitoring feature set with streams monitoring history. If you'd like to have same statistics within your monitoring or analytic tools, use Dispersa alerts push API to get notified about streams going offline and online.

Take a look at our screencast describing the service:

Q: My streaming network is protected with WMSPanel hotlinking protection. Can Dispersa work here?

Yes, just set up allow list for Dispersa checkpoints IP range.

Here's what we want to do next:

Profiling.
Quality of experience metrics.

November 13, 2013

How to make robust streaming with Nimble Streamer

As we mentioned in the previous post, building robust content delivery network means that the content must be delivered regardless of availability of all servers. There we've reviewed the techniques available for anyone to build a simple re-streaming infrastructure with content origin, streaming edges and DNS failover.

Let's build this delivery network using DigitalOcean hosting and Nimble Streamer. We've already used this stack for quick start of VOD streaming, so let's continue using these two for building our first CDN prototype. You can use any other hosting provider best fit for you, like Amazon or Rackspace.

UPDATE: WMSPanel team released the article describing the recommended approach to load balancing for HTTP-based protocols like HLS, DASH, Icecast etc. Please use the technique described there to build your own balancing solution rather than approach shown below.
The DNS failover approach used below is considered obsolete. However, Nimble Steamer installation and the re-streaming setup are correct and may be used for setting up your environment.

OK, so we have a content origin which accepts requests.

We need to make it robust and scalable. The DNS failover technique will be used for that. We'll make a set of re-streaming caching edge servers to provide failover and to allow easily adding new edges in this farm.

Building robust and scalable re-streaming infrastructure.

Load balancing and failover with Nimble Streamer

Nowadays HTTP based protocols become more and more popular for many reasons and about a half streaming media has switched to use them by now. This movement will proceed and this is why it was decided to implement Nimble Streamer, a light-weight media server for HTTP protocols re-streaming and VOD content processing. So it is intended to be used as a basis for content delivery over HTTP.

Building robust content delivery network means that the content must be delivered regardless of availability of all servers. Let's see how Nimble Streamer can be used for these conditions.

UPDATE: WMSPanel team released an article describing the recommended approach to load balancing for HTTP-based protocols like HLS, DASH, Icecast etc. Please use the technique described there to build your own balancing solution rather than HTTP redirect-based approach shown below. The text below is considered obsolete.

A typical CDN consists of two major parts - content origins and delivery edge servers. Origins are typically stores the content, makes transcoding, re-packaging and transmuxing. Origins are heart of any CDN and we recommend using Wowza Media Server as an origin especially in case of live video.

Edges are required to:

off-load the origins, e.g. they might be too busy with encoding,
pass around transmission bandwidth limitations as any server has its throughput limitations,
improve robustness in case of server failures.

Having several edges, we cannot point a player to some certain server. We should use domain name in this case and once any of the edges goes down, we need to make sure that all new connections will go to properly functioning servers. DNS failover (DNS round-robin) technique can be used for having some single domain name as well as providing a robust failover since DNS allows mapping several IP addresses to one domain.

Origin and Nimble Streamer edges with DNS failover.

Let's see a possible network architecture to cover live streaming use case.

1. Origin

We have an origin server that provides a stream which is packaged with HTTP Live Streaming (HLS, or Cupertino), Microsoft SmoothStreaming (Smooth) or progressive download. It may use any streaming media software like Wowza Media Server, Nimble Streamer, nginx, Microsoft IIS etc.

2. Edges

We will take Nimble Streamer as edge software. Nimble will take requests for contents from the player and will request that contents from origin. As soon as the chunk is started to be downloaded, it's being given to the player and then stored in cache - either RAM cache for live streaming or ROM for VOD streaming. Once a content chunk is available in cache, it will not be requested from the origin.

Re-streaming routes are easily set up in WMSPanel. The most convenient feature for this case is that you may describe a route and apply it to multiple servers. So it doesn't matter how many edges you have, you may be sure that all edges will have identical routing settings.

3. DNS

Now let's make sure our content from the origin will always be available for any player via single robust access point.

DNS failover is based on using multiple A-type DNS records for the same domain name which point to a set of IP addresses. We use DNS failover for providing robust balancing for our WMSPanel cloud service. The records would look like this:

video.ultimatemediastreaming.com A 162.243.88.159
video.ultimatemediastreaming.com A 162.243.88.131

It's all set.

4. Streaming

Now you may use URLs like this for using in a web or desktop player:

http://video.ultimatemediastreaming.com:8081/live/playlist.m3u8

You can be sure that any new connection will be taken care of using either of those 2 mentioned edges. If one of them goes down, the second one will keep processing incoming connections and give the content back.

Now you can try it yourself. Just follow installation instructions and contact us if you need help.

Check how to build robust streaming infrastructure with DigitalOcean and Nimble Streamer.

You can also read "The Paranoid’s Guide to Internet Video Streaming" by Thomas Gires to see real-life example of using pay-per-view feature set along with Nimble Streamer as light-weight edge for HLS re-streaming.

March 24, 2013

Improving availability and robustness with DNS failover

Hi,

WMSPanel gathers data from hundreds of Wowza servers and processes data about hundreds of thousands of streams. So having the service that must be reliable and available 24/7 we improve our robustness all the time.

From the very beginning we use cloud hosting and this allows easily scaling our computing resources in order to make them more reliable. But every virtual cloud hosting is based on some hardware. And we know that every hardware fails sooner or later. So there are several way to dodge it.

We use several virtual servers for each specific part of data processing and storage. As example, we have 2 virtual servers which process sync packets coming from customers' servers. Each virtual server is located at different hardware unit. Thus the load is spread across 2 machines each having 4 CPU cores. So even if the load increases and at the same time one of servers fails at one of the physical hardware units, we'll have enough computing power at different physical instance to handle incoming requests. And we can easily bring up more servers to keep up the processing.

The same applies to UI processing. Each incoming request from your browser goes to load balancer and the is forwarded to either of 2 servers that work as front-end. So regardless of the load and the availability of any of those servers, we always have some entity which will respond when you browse the statistics reports.

All other part of our system are designed the same way to make less points of failure.

There was just one part which did not have its "hot backup". It's a load balancer. If it failed, we would be in deep trouble because every request from either web browser or Wowza agent goes through it.

We decided to use DNS failover based on multiple DNS A records. Those records point at 2 identical load balancers which then normally route the traffic according to its target. WMSPanel has 2 sources of incoming requests: users' browsers (both PC and mobile) and Wowza servers agents.

Let see what happens in case of failure of the load balancer at the primary IP which is set up in DNS record.

A browser will just try secondary IP as it should have it in the local DNS cache. So this will work seamlessly for the end user and it's a default behavior for all web browsers.
In the same situation the Wowza agent will work the same way. It will check DNS for secondary IP and send its sync-ups over there. Note that you need to have an agent version at least 1.3.0.0 to handle it correctly.

There may be more than just two DNS records so when we need another balancer, even located in the different data center (e.g., Latin America or Asia), we'll just add a new record there and it will work perfectly.

Read a complete blog post about WMSPanel architecture and design to see how we provide high availability and robustness, with DNS failover being a part of it.

If you haven't yet used our service, don't wait and try it now for 2 weeks free of charge.

Pages