Table of Contents
1. Streaming Protocols
graph LR subgraph EN["Encoder"] end subgraph MS["Media Server"] end subgraph PL["Player"] end EN --> MS MS --> PL
1. Traditional Video Streaming Protocols:
- Adobe Real-Time Messaging Protocol (RTMP). transport audio and video data between a dedicated streaming server and the Adobe Flash Player.
- RTSP/RTP : standard in many surveillance and closed-circuit television (CCTV) IP cameras.
- RTSP is a presentation-layer protocol (“television remote control”), RTP is the transport protocol.
2. Adaptive HTTP-Based Streaming Protocols: progressive downloads sent via regular web servers. Using adaptive bitrate streaming.
- Apple HTTP Live Streaming (HLS): with adaptive bitrate streaming.
- Low-Latency HLS.
- MPEG-DASH.
- Low-Latency CMAF for DASH.
- Microsoft Smooth Streaming in 2008 for use with Silverlight player applications.
- Adobe HTTP Dynamic Streaming (HDS).
3. New Technologies:
| Benefits | Drawbacks | Latency | Transport | |
|---|---|---|---|---|
| RTMP | Low-latency and requires no buffering | Not optimized for quality of experience or scalability | 5 seconds | TCP |
| RTSP/RTP | Low-latency and supported by most IP cameras | No longer used for video delivery to end users | 2 seconds | UDP/TCP |
| Apple HLS | Adaptive bitrate and widely supported | Quality of experience is prioritized over low latency | 6-30 seconds | TCP |
| Low-Latency HLS | Low latency, scalability, and high quality, backward compatibility | As an emerging spec, vendors are still implementing support | 2 seconds or less | TCP |
| MPEG-DASH | Vendor-independent, international standard for adaptive bitrate | Not supported by iOS or Apple TV | 6-30 seconds | TCP |
| CMAF | Low latency meets HTTP-based streaming | As an emerging spec, vendors are still implementing support | 3 seconds or less | |
| Microsoft | Adaptive bitrate and supported by iOS | Proprietary technology, doesn’t compete with HLS and DASH | 6-30 seconds | |
| Adobe HDS | Adaptive bitrate technology for Flash | Proprietary technology with lacking support | 6-30 seconds | |
| SRT | High-quality, low-latency video over suboptimal networks | Not widely supported for video playback | 3 seconds or less | UDP |
| WebRTC | Super fast and browser-based | Designed for video conferencing and not scale | Sub-500-millisecond delivery | UDP/TCP |
StackOverflow:
Others:
- GStreamer: a pipeline-based multimedia framework, good choice for applications that need to perform complex media processing tasks.
- GStreamer Encoding Latency analysis.
- Robot streaming : WebRTC is the best choice.
2. Streaming Projects
Moonlight Game Streaming Project:
graph LR XX[["Encoder"]] --> MS["Media Server TCP & UDP"] MS --> PL["Decoder HEVC/H264/etc"]
- Uses both raw TCP and UDP, reference.
- Encoder/Decoder : uses HEVC.
- Android Media Codec Helper. keyword: HEVC.
- H264 vs. H265. H.265 is superior for high quality streaming.
- Archived version uses FFMPEG H264.
- Android Media Codec Helper. keyword: HEVC.
graph LR EN["Encoder HEVC"] --> MS["Media Server TCP & UDP"] MS --> PL[["Decoder"]]
- Uses both raw TCP and UDP : UDP/TCP map
- RTSP rtsp.h.
- keyword: HEVC.
graph LR EN["Encoder HEVC&H264"] --> MS["Media Server TCP & UDP"] MS --> PL["Decoder HEVC&H264"]
- FFMPEG H.265 & H.264. FFmpeg wiki.
- TCP or UDP.
- FFMPEG server :
ffmpeg -vaapi_device /dev/dri/renderD128 -f x11grab -s 2560x1440 -i :0 -vf format=nv12,hwupload -c:v h264_vaapi -qp 35 -framerate 60 -f rtp rtp://127.0.0.1:8004. - WebRTC client.
SRS(Simple Realtime Server) made a high level usage of all the protocols.
3. Development
- Make UDP streamer from scratch:
- Make a UDP socket server/client communication. while raw delay is large, and images has large jitter.
- Buffer need to build, frames check need to update, message package need to be smaller.
- Use FFMPEG RTP, delay about 1 second.
- Adjust encoder parameter with reference
-preset p1 -delay 0, reduce dalay to ~100ms.
- Adjust encoder parameter with reference
- Try SRS : Low Latency Live Stream, RTMP Latency.