December 24, 2019

Introducing Apple Low Latency HLS in Softvelum products

The HLS of RFC8216 is currently a de-facto standard for live and VOD delivery. Playlists and chunks delivered over HTTP/1.1 provide the simplicity of implementation, wide platforms availability, adaptive bitrate support and scalability. These advantages allowed it to get the biggest share of customer base. However it has some disadvantages for live streaming, primarily because chunks delivery means many seconds of delay at the start of a live stream. The industry has been making some improvements on top of HLS, and then Apple released its spec update to address existing issues.


Notice: with latest updates of LL HLS spec draft from Apple, our products now have the outdated implementations for described features. We do not recommend using them.
Full support is coming soon, stay tuned.


Low Latency HLS


Low Latency HLS (LL-HLS) is the next generation of Apple's HTTP Live Streaming protocol introduced in early 2019.
Several key features improve it over regular HLS for live delivery:
  • Partial segments (parts) can be accessed before full chunks of content are available.
  • Server side can use HTTP/2 Push for sending parts.
  • Holding playlist requests for obtaining latest parts as soon as they appear.
Softvelum team added LL-HLS into the bundle. We provide customers with playback capabilities using SLDP Player for iOS, and Nimble Streamer allows generating LL-HLS for all supported containers such as fMP4, audio-only and MPEGTS.

1. SLDP Player to play LL-HLS


Apple still has LL-HLS in beta stage as of iOS 13.3 at the end of December of 2019, so there are a number of shortcomings in its implementation. The main concern is the fact that iOS native player implementation cannot be published into AppStore yet. Lack of browsers' and other platforms' availability is also a big blocker so far. So the only way to try the playback for development purposes is to build your own app for that.

SLDP Player SDK for iOS allows having full-featured Low Latency HLS playback on iOS devices. It covers live streams from any source capable of LL-HLS like Wowza Streaming Engine and Nimble Streamer, and it also supports regular HLS from any available source.

If you'd like to build your own low latency playback app, you can get player SDK from our team for further test and integration. Once the LL-HLS technology goes from Apple beta to production (in early 2020 as per Apple), you'll be able to have full-featured app and publish it under your brand.

2. Nimble Streamer for LL-HLS transmuxing


Nimble Streamer software media server allows taking any supported live input streams and re-packaging them into Low Latency HLS. Here are the steps you need to follow in order to make it work.

2.1 HTTP/2 and config setup


2.1.1. LL-HLS uses HTTP/2 via SSL as a transport protocol. So you need to enable it before performing any further setup.
Please follow this HTTP/2 setup article to make this protocol working for Nimble Streamer.

2.1.2. In addition to that, you need to add this parameter into nimble.conf and restart Nimble Streamer, read this page for config setup details:
hls_add_program_date_time = true
If a client tries to access LL-HLS stream via HTTP/1.1, or if HTTP/2 is not properly set up, then player will fall back to regular-legacy HLS and will not use any advantages of LL-HLS.

You can check if Nimble Streamer delivers HTTP/2 by checking access log. To enable access logs, add this parameter into nimble.conf the same way you've done it for other parameters:
log_access = file
Once you re-start Nimble, you'll be able to view the log. In Ubuntu it's located in /var/log/nimble/access.log by default. Now when you try to get your regular HLS live stream via https:// via curl or HTTP/2-capable player, you'll get this kind of record in access log:
Dec 24 17:43:09 nimble[5500]: [nimble-access.log] 192.18.1.2 - - "GET /livell/stream/playlist.m3u8 HTTP/2" 200 84 1114 372 "-" "AppleCoreMedia/1.0.0.17C54 (iPhone; U; CPU OS 13_3 like Mac OS X; en_us)"
You can see HTTP/2 there which means it's working. In other cases it will have HTTP/1.1 and this will mean you need to check what's wrong. Contact us in case of issues.

2.2 Live streaming setup


Now you need to set up transmuxing settings via WMSPanel web service. If you are not familiar with live streaming setup of Nimble Streamer, please refer to live streaming digest page, or respective input protocol pages, such as RTMP streaming. Please make sure you have a correctly set up live stream (like regular-latency HLS or SLDP) before trying to use LL-HLS.

Once you have a live stream set up in WMSPanel, go to Nimble Streamer top menu and select Live streams settings. You will see Global setting tab for selected server (and you may create application-specific setting as well).


Currently, Nimble Streamer supports all 3 containers available for HLS, you can see their respective checkboxes on the screenshot above:
  • HLS - HLS with audio-only container. Audio-only is optimized for audio delivery having a reduced size. The ID3 tags are also inserted in each audio part.
  • HLS (MPEGTS) - MPEG-TS: the only container with video and audio support for LL-HLS
  • fMP4 - fragmented MP4. Notice that fMP4 container playback has a couple of issues related to current Apple implementation of their player as of iOS 13.3, please refer to section "3. Known issues" below for more information.
Once you select either of those containers, WMSPanel will show Enable Apple's Low Latency HLS checkbox and you need to select it. It will also show HLS part duration edit box to define parts' duration in milliseconds, we recommend using default value of 1000ms, see section "3. Known issues" for details.

Once LL-HLS is enabled, you need to re-start the input stream so Nimble Streamer could start producing LL-HLS output stream.

2.3 Workflow and playlists


Now as the set up has been made, you can use the player to consume the stream using the usual playback URL. The main playlist will have proper chunklists which will have a content according to LL-HLS spec, as shown in the example below.
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="audio.fmp4?nimblesessionid=1"
#EXT-X-TARGETDURATION:7
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.536
#EXT-X-PART-INF:PART-TARGET=0.512
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:29:02.609Z
#EXTINF:5.995,
a_6_6016_1.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_5.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_6.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_7.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_8.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_9.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_12011_2_10.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.363,URI="a_6_12011_2_11.fmp4?nimblesessionid=1"
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:29:08.604Z
#EXTINF:5.995,
a_6_12011_2.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_5.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_6.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_7.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_8.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_9.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_18006_3_10.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.362,URI="a_6_18006_3_11.fmp4?nimblesessionid=1"
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:29:14.599Z
#EXTINF:5.994,
a_6_18006_3.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_5.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_6.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_7.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_8.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_9.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_10.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.384,URI="a_6_24000_4_11.fmp4?nimblesessionid=1"
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:29:20.593Z
#EXTINF:6.016,
a_6_24000_4.fmp4?nimblesessionid=1


Parts in chunklist. Comparing to regular HLS, you see a lot of lines representing parts like this:
#EXT-X-PART:DURATION=0.512,URI="a_6_24000_4_0.fmp4?nimblesessionid=1"
The full chunk which contain these parts will be described after all parts' lines:
a_6_24000_4.fmp4?nimblesessionid=1
All parts within chunks are numerated starting from zero. So "a_6_18006_3_0.fmp4" mean its the first part of chunk number 3.

Part length. This attribute declares a designated size of upcoming parts:
#EXT-X-PART-INF:PART-TARGET=0.512
In this example it's 512 milliseconds.

Can block reload
. Check this line:
#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.536
The "CAN-BLOCK-RELOAD" declares that media server allows holding playlist request.

Hold playlist request. LL-HLS allows requesting the server to hold sending out the playlist until a specific chunk and/or part is available for the stream.
So a player may request some part which is going to be available within a few seconds from now, then Nimble Streamer will check if that part is available. Once the requested part is ready, Nimble will return a playlist.

Check this request example:
curl -k "https://localhost:8443/livell/stream/chunks.m3u8?nimblesessionid=1&_HLS_msn=59&_HLS_part=5"
The highlighted _HLS_msn=59 and _HLS_part=5 parameters indicate that the server must hold the request until Nimble Streamer has part number 5 of chunk number 59 or later and then it could return a playlist. You can use only _HLS_msn=59 parameter, in this case the playlist will be sent out only once full chunk is available.

The resulting chunklist will look like this:
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="audio.fmp4?nimblesessionid=1"
#EXT-X-TARGETDURATION:7
#EXT-X-MEDIA-SEQUENCE:55
#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.536
#EXT-X-PART-INF:PART-TARGET=0.512
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:34:26.599Z
#EXTINF:5.994,
a_6_330006_55.fmp4?nimblesessionid=1
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:34:32.593Z
#EXTINF:6.016,
a_6_336000_56.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_5.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_6.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_7.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_8.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_9.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_342016_57_10.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.363,URI="a_6_342016_57_11.fmp4?nimblesessionid=1"
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:34:38.609Z
#EXTINF:5.995,
a_6_342016_57.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_5.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_6.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_7.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_8.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_9.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_348011_58_10.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.363,URI="a_6_348011_58_11.fmp4?nimblesessionid=1"
#EXT-X-PROGRAM-DATE-TIME:2019-12-23T02:34:44.604Z
#EXTINF:5.995,
a_6_348011_58.fmp4?nimblesessionid=1
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_0.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_1.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_2.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_3.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_4.fmp4?nimblesessionid=1"
#EXT-X-PART:DURATION=0.512,URI="a_6_354006_59_5.fmp4?nimblesessionid=1"
You can see it ends with part a_6_354006_59_5.fmp4 - it's part number 5 of the upcoming chunk 59. That chunk will be available only a few seconds later, but the player can already perform the playback, this helps a lot with reducing the latency.

Push the requested part. In addition to requesting specific part upon its arrival, a player may request Nimble Streamer to make HTTP/2 Push of that part to reduce the playback latency even further. This will be made by adding "_HLS_push=1" parameter in URL. If we look at Nimble Streamer access logs, we'll see the following actions:

Dec 24 18:43:04 nimble[5500]: [nimble-access.log] 192.18.1.2 - - "GET /livell/stream/chunks.m3u8?nimblesessionid=18&_HLS_msn=9&_HLS_part=0&_HLS_push=1 HTTP/2" 200 84 1114 372 "-" "AppleCoreMedia/1.0.0.17C54 (iPhone; U; CPU OS 13_3 like Mac OS X; en_us)"
Dec 24 18:43:04 nimble[5500]: [nimble-access.log] 192.18.1.2 - - "PUSH /livell/stream/l_4_27008_9_0.aac?nimblesessionid=18 HTTP/2" 200 0 49896 662 "-" "AppleCoreMedia/1.0.0.17C54 (iPhone; U; CPU OS 13_3 like Mac OS X; en_us)"

Dec 24 18:43:05 nimble[5500]: [nimble-access.log] 192.18.1.2 - - "GET /livell/stream/chunks.m3u8?nimblesessionid=18&_HLS_msn=9&_HLS_part=1&_HLS_push=1 HTTP/2" 200 84 1180 341 "-" "AppleCoreMedia/1.0.0.17C54 (iPhone; U; CPU OS 13_3 like Mac OS X; en_us)"
Dec 24 18:43:05 nimble[5500]: [nimble-access.log] 192.18.1.2 - - "PUSH /livell/stream/l_4_27008_9_1.aac?nimblesessionid=18 HTTP/2" 200 0 49828 568 "-" "AppleCoreMedia/1.0.0.17C54 (iPhone; U; CPU OS 13_3 like Mac OS X; en_us)"

As you can see the player is sending hold-playlist request (described earlier) with specific part number and _HLS_push=1 parameter. Nimble Streamer returns that playlist in response, as well as making HTTP/2 Push for the requested part.

Performance. With all these specific actions Nimble Streamer generates and serves parts within HLS stream with high efficiency. From resource consumption perspective, LL-HLS processing costs the same as handling regular-latency playlists and chunks.

3. Known issues and troubleshooting


At the moment Apple native player on iOS 13.3 has the following problems with LL-HLS implementation which may affect end-user experience.

1. fMP4 video+audio. If you use fMP4 container then you will be able to get either video or audio component working. Video+audio fMP4 streaming is not working properly yet. You can try using MPEGTS container for video+audio instead.

2. Part duration. If you set part duration to less than 1000 ms then video will not work at all. So we recommend setting part duration as "1000".

We are sure those issues will be fixed in Apple's upcoming releases. Meanwhile on iOS 13.3 you'll have to test it with aforementioned limitations.

3. Interleaving compensation. If you have video+audio stream you may have issues with playback due to interleaving as described in this article. This kind of issues becomes even more severe in case of low latency playback. In order to fix this you can try enabling interleaving compensation with Min. delay set to zero, see the image below.





Feel free to try Nimble Streamer with Low Latency HLS and buy SLDP Player SDK to get your hands on the iOS playback.

Let us know if you have any questions.

No comments:

Post a Comment

If you face any specific issue or want to ask some question to our team,
PLEASE USE OUR HELPDESK

This will give much faster and precise response.
Thank you.