A design on the webcodec based live streaming project.
Context: I will need to build a web based video conference that could support screen sharing.
In this post, I’d like to discuss a live conference design built upon http and webcodec. The word web, in the content we will discuss below, means the tech on the web page.
Live Conference is, as you know, composed of videos and audios. As you watch and listen, tech behind this work is a big project composed of networking codecs and quality. When we got a meeting started, you could see the media from all people, maybe the topology is like this.
Alice
/ \
/ \
/ \
Bob ------- Carol
In RTC realm, we call it P2P(Peer to Peer/Port). The webrtc tech, as you know, is good at this connection pattern. To setup a p2p connection, you will need to know the physic port of the other end/peer, which means there should be some port reflecting server(STUN), or proxy(TURN) if reflecting is blocked. All sockets are of UDP type, and socket reusing is a challenge. After reflecting, peers need to exchange socket informations, for that, we will also need a ICE server. So there’s at lease 2~3 servers needed to be implemented, and you need to have sufficient sockets for the UDP communication(Most webrtc framework dont reuse sockets at all).
Another RTC design pattern is called SFU(Selective Forwarding Unit), or the Streming HUB pattern. Just like normal RTSP/RTMP way, you will need to push your stream to the server, then the client shall pull them as a upstream copy. RTSP/RTMP is relatively simple and less socket consuming, without STUN or TURN. But RTSP/RTMP is not allowed on web, thanks to the web security restrictions.
The SFU mode topology is something like this:
In most cases, if people are to share rtsp stream on the web, they will need to remux their rtsp/rtmp into hls/mp4dash/webrtc stream feeds, which are supported via the w3c APIs, and get their stream played on a Video/Audio tag/class/object. In this cases, there’s some hard truth needed to be understood:
- There’s at least 2 data copying phrases (say depacketizing rtp to h264-annexb, then packetize h264-annexb to ts/fmp4 format) copying introduce lag and cpu burden.
- When repackezing into hls or fmp4, the format will make sure at lease one GOP(Group of Pictures) are collected, one GOP will be at lease 1~2 seconds of screen falling behind, that is fatal when dealing with some devices dealing with high speed cars/drones/rockets etc, so it could just be used for some Non-interactive scenario.
- In some scenarios such as surveillance/security, packet loss/reorder/duplicating shall sabotage the repacketizing phrase so we need to harness the stream integrity, and that’s a big topic then.
So what the current web media tool is:
- If we are to adopt webrtc, it is relatively difficult (multiple servers) and socket wasting.
- If we stay with rtsp/rtmp or webrtc SFU, we will need at least a pushing server(Stream Hub), then we will need to remux and maybe do some extra work to format the video stream into hls/dash/webrtc friendly datas, which introduces lag, server burden, integrity work, etc,etc.
It seems to me there’s no perfect or suitable protocal for web conference at all, just downscale from some mature framework.
That’s only just for networking, how about codecs?
When we talk about codecs, we are referring to not only the h26x stream, but also the system design. In rtsp/rtmp cases, the media producer is a camera, and the web responsibility is no more than a player. So a live conference must be built into some apps which could do some so/dylib binding to include the media sdk from a camera producer. A web based one plays no more than a watcher role here, some tech such as MSE(Media Source Extension) are designed to buffer/stream for the player role. While in webrtc case, it provides a complete solution, binding all the codec, networking and integrity work behind webrtc apis and related media apis. That sound good, but in this way a developer is not able to manipulate the details within the webrtc framework unless you rebuild it, the source code alone is 10G or so if I remember correctly?
It seems to me the web is not designed as a camera content producer? Althrough it got camera/microphone capturing capabilities.
But not exactly, I recently notice a tech called webcodec emerges, it empower the web to encode the media content, not by handling all the heavy lifting below the surface such as webrtc, but get the codec api exposed to developers. Enable a developer to weave the logic they need. There’s a couple of example out there for your quick review. Both videos and audios are included, and there’s a codec supporting list.
Yes, you may notice some web apis, such as writable_stream is still experimental(or marked as Baseline Widely available), that’s a production ready promise, but let’s just focus what we could do for now.
We now got webcodec and related web api for h26x/vpx/aac/opus encoded content producing, but lacks suitable protocal for live conferencing
So why not build our own protocal? A protocal that handle stream integrity and play well with webcodecs? The protocal Quic sound like the perfect solution. I accept that, but currently most http component such as firewall load balance etc are built upon http2.0 or http1.1 protocals. In most cases, TCP means more reliable but not flexible, while UDP means stateless, non-blocking but unstable. We need to deal with some cases when http3 is not acceptable with a fallback solution.
So maybe here’s my design, we will need webcodec for codes and http3/quic for networking, when http3 is impossible, fallback to http-keepalive with streaming
Yes, when falling back, it is possible to encounter head-blocking issue. My solution is pretty straight forward, when blocking happens, switch to other low-quality streams😜. Or consider this, use multiple tcp/http connections, one connection for control/feedbacking(like RTCP), others send normal rtps.
Something like this,
SERVER
══════════════════════════════════════════════════════════════════════
+--------------------------------+
| Media Scheduler (MTL) |
|--------------------------------|
| Frame Window |
| Frame Database |
| GOP Dependency Graph |
| Path Scheduler |
| Quality Controller |
+--------------------------------+
│ │ │ │
HTTP #1 │ HTTP #2│ HTTP #3│ RTCP-like
───────────────────┼────────┼────────┼──────────────
│ │ │
▼ ▼ ▼
TCP1 TCP2 TCP3 TCP4
══════════════════════════════════════════════════════════════════════
NETWORK
══════════════════════════════════════════════════════════════════════
Then on the client side:
══════════════════════════════════════════════════════════════════════
NETWORK
══════════════════════════════════════════════════════════════════════
TCP1 TCP2 TCP3 TCP4
HTTP Conn1 HTTP Conn2 HTTP Conn3 Control Conn
│ │ │ ▲
└──────┬────────┴───────────────┘ │
▼ │
+----------------------------------------------+
| Frame Reassembler |
+----------------------------------------------+
│
▼
+-------------------------+
| Receiver Window |
|-------------------------|
|100 ✓ |
|101 ✓ |
|102 ✗ |
|103 ✓ |
|104 ✓ |
+-------------------------+
│
Missing Frame Detector
│
▼
RTCP-like Feedback
│
▲
Request Frame102
│
▲
Drop blocking TCP, starting new ones
Frankly, this is a Quic+webrtc hybrid solution, it got limitaions – it does not perform well when dealing with poor network, so choose if it suits. There’s a tons of better solutions out there, FYI.
Now that we got customized protocal for quality, webcodec ready for live videos, maybe mostly covered, what else?
In traditional RTSP or webrtc protocals, a RTP with a MTU (<1500Bytes) is designed to allow minimal cost retransmission. And there’s fixed format to do this, such as H264.
But that’s wrapping up raw h264 streams with meta such as SPS/PPS, the RTP format is not designed for web, plays better with UDP realm. In our webcodec related scenario, we will need a more ‘webcodec’-friendly packetizing/depacketizing.
In the webcodec world, the most important thing is the RFC6381 format. It is a necessary encoder/decoder initializer.
// For a video encoder example:
const encoder = new VideoEncoder(init);
const encoderConfig = {
codec: "avc1.4d4028",
width: 800,
height: 600,
framerate: 30,
avc: { format: "annexb" }
};
encoder.configure(encoderConfig);
This string ‘avc1.4d4028’ tells webcodec api to initialize a AVC(H264) decoder, with profile ‘0x4d’, level of ‘0x28’ and ‘0x40’ as the constrains. Then the ‘avc’ part tell the decoder/demuxer that the data feed is in a annex-b format(with prefix 0x00 0x00 0x00 0x01), if in avc format, then you will need extra data specified. When dealing with these informations, I would prefer not to split the data into RTP MTU units, they are too complicated. I would need just 4 types, they are:
- Video configuration, let’s call it
vcnfto align the mp4 box naming pattern 😀, they are created by host the stream pusher, cached and when a client try to pull the stream, this info will help init their decoder. - Audio configuration, alias
acnf, same as video configuration, config dont have to worrysamplerate, cause the ‘duration’ field inafrmlater will come to the rescue. - Video frame, alias
vfrm, represent the real h26x stream datas. - Audio frame, alias
afrm, the audio datas.
If we are to adopt the multiple tcp transmission, each TCP connection must got a unique name, I call it feeds, each TCP got a feed ID. So when blocking happens, controller shall know where to deal with the zombie tcps.
It looks like:
SERVER
══════════════════════════════════════════════════════════════════════
+--------------------------------+
| Media Scheduler (MTL) |
|--------------------------------|
| Frame Window |
| Frame Database |
| GOP Dependency Graph |
| Path Scheduler |
| Quality Controller |
+--------------------------------+
│ │ │ │
HTTP #1 │ HTTP #2│ HTTP #3│ RTCP-like
───────────────────┼────────┼────────┼──────────────
│ │ │
▼ ▼ ▼
TCP1 TCP2 TCP3 TCP4
(Feed1) (Feed2) (Feed3)
══════════════════════════════════════════════════════════════════════
NETWORK
══════════════════════════════════════════════════════════════════════
And the Frame data should be like:
vcnf
+------------------------------------------------+
| SEP |
+------------------------------------------------+
| BoxName = vcnf |
+------------------------------------------------+
| PayloadLen |
+------------------------------------------------+
| Width(uint16) |
+------------------------------------------------+
| Height(uint16) |
+------------------------------------------------+
| CodecLen(uint16) |
+------------------------------------------------+
| RFC6381 Codec |
+------------------------------------------------+
| ExtraDataLen(uint32) |
+------------------------------------------------+
| AVC/HEVC Extradata |
+------------------------------------------------+
acnf
+------------------------------------------------+
| SEP |
+------------------------------------------------+
| BoxName = acnf |
+------------------------------------------------+
| PayloadLen |
+------------------------------------------------+
| CodecType |
| |
| RFC6381 codec string |
| |
| Codec Extradata |
+------------------------------------------------+
afrm
+------------------------------------------------+
| SEP |
+------------------------------------------------+
| BoxName = afrm |
+------------------------------------------------+
| PayloadLen |
+------------------------------------------------+
| FEED(uint32) |
+------------------------------------------------+
| Sequence(uint32) |
+------------------------------------------------+
| Timestamp(uint64) |
+------------------------------------------------+
| FrameID(uint64) |
+------------------------------------------------+
| Duration(uint32) |
+------------------------------------------------+
| Payload |
+------------------------------------------------+
Then finally vfrm, vfram is special since it sometimes need fragmenting. Splitting, say, an IDR frame into many fragments, so now a fragmenting id is necessary
+------------------------------------------------+
| SEP |
+------------------------------------------------+
| BoxName = vfrm |
+------------------------------------------------+
| PayloadLen |
+------------------------------------------------+
| FEED(uint32) |
+------------------------------------------------+
| Sequence(uint32) |
+------------------------------------------------+
| Timestamp(uint64) |
+------------------------------------------------+
| FrameID(uint64) |
+------------------------------------------------+
| FragmentID(uint16) |
+------------------------------------------------+
| FragmentCount(uint16) |
+------------------------------------------------+
| FrameType(uint8) |
| |
| 0 = IDR |
| 1 = P |
| 2 = B |
+------------------------------------------------+
| DependencyID(uint64) |
+------------------------------------------------+
| Duration(uint32) |
+------------------------------------------------+
| PayloadLen(uint32) |
+------------------------------------------------+
| NAL Payload |
+------------------------------------------------+
All good to go now, what else?
I will prefer to pickup a tcp reusing framework such as tokio, it handles SO_REUSEPORT work with the help of epoll, enabling us to host more people within a web conference than that of a webrtc.
Summary
We got a tcp and webcodec based web conference designed and built, they are not ideal for poor network, but could plays well. Gladly if it helps, you are welcome to add more designs to refine the idea