FFmpeg: Add Whip Muxer support for subsecond latency streaming by winlinvip · Pull Request #1 · ossrs/ffmpeg-webrtc

winlinvip · 2023-04-16T12:54:10Z

This PR is updated by #18, #19, #24, #43, #44, #45.

This PR has been squashed into a single commit on the patch/whip/v1 branch for easier submission to the FFmpeg community for code review.

Note: Due to the high level of interest in this patch, which has resulted in numerous reviews and discussions, including feedback from ffmpeg developers via email, we will be tracking all of these interactions. To facilitate this process, we will be using GitHub Discussions or Discord: Real-time Broadcast or Discord: SRS. Please feel free to use this platform to discuss the patch or any other questions or suggestions you may have.

WHIP RFC9725 stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers.

Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C.

However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg.

To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 3k lines of C code.

Content

Usage How do I use this patch?
Usage: FFmpeg + SRS How do I use SRS as a WHIP server for FFmpeg?
Usage: FFmpeg + Janus How do I use Janus as a WHIP server for FFmpeg?
Usage: FFmpeg + Pion How do I use Pion as a WHIP server for FFmpeg?
Usage: FFmpeg + AWS-IVS How do I use AWS IVS's WHIP cloud service for FFmpeg?
Usage: FFmpeg + Cloudflare How do I use Cloudflare's WHIP cloud service for FFmpeg?
Usage: FFmpeg + Millicast How do I use Millicast as a WHIP server for FFmpeg?
Usage: FFmpeg + TRTC How do I use TRTC's WHIP cloud service for FFmpeg?
OpenSSL What's the supported version of OpenSSL, and how do I build it?
Debugging How to enable debugging logs of FFmpeg?
Asan and Fate How to build with asan and make with fate?
Simulate Packet Loss How to simulate packet loss in localhost?
Known Issues Regarding known issues and what has been fixed.
Latency What's the process for measuring latency? Typically, it's around 150ms.
Load Certificate File How to import a DTLS certificate and private key from a file?
Authorization How can I authorize using an HTTP header?
Contributors About the authors, contributors, and reviewers for this patch.
Links Useful links about this patch.

Usage

Migrated to #49

Usage: FFmpeg + SRS

Migrated to #37

Usage: FFmpeg + Janus

Migrated to #38

Usage: FFmpeg + Pion

Migrated to #39

Usage: FFmpeg + Millicast

Migrated to #46

Usage: FFmpeg + TRTC

Migrated to #47

Usage: FFmpeg + Cloudflare

Migrated to #41

Usage: FFmpeg + AWS-IVS

Migrated to #42

OpenSSL

Migrated to #33

Debugging

Migrated to #34

Asan and Fate

Migrated to #35

Simulate Packet Loss

Migrated to #36

Known Issues

Migrated to #28

Latency

Migrated to #29

Load Certificate File

Migrated to #30

Authorization

Migrated to #31

Contributors

Migrated to #32

Links

WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/
SRS WHIP: WebRTC: Support WHIP(WebRTC-HTTP ingestion protocol) srs#3170
OBS WHIP: Add WebRTC (WHIP) output support obsproject/obs-studio#7926
metaRTC: feature/metaRTC
For RTP plaintext mode: Add RTP Plaintext Mode for Debugging with Wireshark

…d API.

…_EC_gen.

…cessary ClientHello.

1. Fix OpenSSL build error. 2. Support OpenSSL 1.0.1k and newer versions. 3. Support WHIP authorization via Bearer HTTP header. 4. Change the option default value from 1500 to 1200, to make Pion work. 5. Detect the minimum required OpenSSL version, should be 1.0.1k and newer. 6. Quickly check the SDP answer by taking a glance at the first few bytes.

1. Merge ICE and DTLS ARQ max retry options into a single handshake timeout. 2. Utilize DTLS server role to prevent ARQ, as the peer DTLS client will handle ARQ. 3. Replace IO from DTLSContext with a callback function. 4. Measure and analyze the time cost for each step in the process. 5. Implement DTLS BIO callback for packet fragmentation using BIO_set_callback. 6. Generate private key and certificate prior to ICE for faster handshake. 7. Refine DTLS MTU settings using SSL_set_mtu and DTLS_set_link_mtu. 8. Provide callback for DTLS state, returning errors when DTLS encounters issues or closes. 9. Consolidate ICE request/response handling and DTLS handshake into a single function.

1. Refine WHIP muxer name. 1. Refine SRTP key macros. 1. Refine logging context. 1. Refine SSL error messages. 1. Refine DTLS error messages. 1. Refine RTC error messages. 1. Use AV_RB8 to read integer from memory. 1. Update DTLS curve list to X25519:P-256:P-384:P-521. 1. Refine SRTP profile name for FFmpeg and OpenSSL. 1. Replace magic numbers with macros and extract to functions. 1. Alter log levels from INFO to VERBOSE, except for final results. 1. Use typedef SRTPContext. 1. Refine the ICE STUN magic number. 1. Reposition the on_rtp_write_packet function. 1. Refer to Chrome definition of RTP payload types. 1. Replace magic numbers with macros for RTP and RTCP payload types. 1. Rename to WHIP muxer. 1. Add TODO for OPUS timestamp issue. 1. Refine comments, do not hardcode H.264. 1. Define SDP session id and creator IP as macros. 1. Refine fixed frame size 960 to rtc->audio_par->frame_size. 1. Use h264_mp4toannexb to convert MP4/ISOM to annexb. 1. Address occasional inaccuracies in OPUS audio timestamps. 1. Correct marker setting after utilizing BSF. 1. Remove dependency on avc.h after using BSF.

See ossrs#5 (comment)

…64_mp4toannexb filter only processing MP4 ISOM format.

1. Change the CommonName from ffmpeg.org to lavf. 2. Rename rtcenc.c to whip.c, rtc to whip. 3. Replace av_get_random_seed by AVLFG. 4. Add TODO to support libtls, mbedtls, and gnutls.

winlinvip · 2025-05-15T11:46:40Z

Updated by #18: Feature: refactor DTLS to merge it into tls_openssl.c. This is a big step toward merging the WHIP patch into FFmpeg. I initialized this patch but was unable to finish it. However, after the efforts of FFmpeg developers and maintainers, especially the work from Jack Lau, Steven Liu, Sean Der, Mypopy Zhao, a new patch has been created and completed the last improvements for FFmpeg WHIP. With this patch, we are now ready to submit it to FFmpeg, and we will have a rudimentary WHIP implementation in FFmpeg. Regarding to the new patch, there will still be lots of work to do. We need your help to test and review this patch.

winlinvip force-pushed the feature/rtc-muxer branch 7 times, most recently from 598aecc to cf69b01 Compare April 21, 2023 23:16

winlinvip changed the title ~~WHIP: Support FFmpeg WebRTC muxer via WHP protocol.~~ FFmpeg: Support WebRTC muxer via WHP protocol. Apr 21, 2023

winlinvip changed the title ~~FFmpeg: Support WebRTC muxer via WHP protocol.~~ FFmpeg: Support WebRTC muxer via WHIP protocol. Apr 21, 2023

winlinvip force-pushed the feature/rtc-muxer branch 8 times, most recently from 48d93eb to 92c4187 Compare April 22, 2023 23:23

winlinvip force-pushed the feature/rtc-muxer branch 13 times, most recently from b4da552 to 2ad58fe Compare May 2, 2023 01:02

winlinvip and others added 23 commits October 17, 2023 10:39

WHIP: Remove the macro that suppresses warnings for the SSL deprecate…

2dedc86

…d API.

WHIP: Merge write header to init.

1054bed

WHIP:Support baseline/main/high profile without B frames (#2)

b31e0c0

WHIP: Fix the SSL deprecated warning by replacing EC_KEY_new with EVP…

b3f5c27

…_EC_gen.

WHIP: Refine the code to be shorter.

85c5680

WHIP: Increase the base timeout and thereby reduce the number of unne…

71ee877

…cessary ClientHello.

WHIP: Refine ARQ for DTLS with bug fixed.

726bb67

WHIP: Update muxers.texi for RTC.

6e1fbb5

WHIP: Do not hardcode codec for SDP. (#5)

2ac4100

See ossrs#5 (comment)

WHIP: Parse profile and level from extradata.

5819297

WHIP: Insert SPS and PPS before IDR frames in annexb format due to h2…

0c4a158

…64_mp4toannexb filter only processing MP4 ISOM format.

WHIP: Failed immediately if there is no profile and extradata present.

812932e

WHIP: Refine names and comments.

aaf26ef

1. Change the CommonName from ffmpeg.org to lavf. 2. Rename rtcenc.c to whip.c, rtc to whip. 3. Replace av_get_random_seed by AVLFG. 4. Add TODO to support libtls, mbedtls, and gnutls.

WHIP: Free buffer leak of pb for RTP muxer.

66a064f

WHIP: Use options for passing parameters to the HTTP or UDP context.

9c7a091

WHIP: Enhance security by using BearToken for delete API.

de33fcc

WHIP: Support user specified certificate and key file.

8971a1f

WHIP: Read DTLS certificate file by ffurl.

0bd6867

WHIP: Free bio_in if ENOMEM.

0b95312

WHIP: Refine DTLS, extract DTLS APIs.

63e3a55

winlinvip force-pushed the feature/rtc-muxer branch from df62b77 to 63e3a55 Compare October 17, 2023 02:39

winlinvip added 3 commits October 17, 2023 11:01

Sync with FFmpeg master 5ddab49

172d34b

Extract DTLS to dtls.c

d3561e6

Change some important step log level to info.

3b15ba1

winlinvip changed the base branch from feature/whip to master May 12, 2025 00:18

winlinvip closed this May 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FFmpeg: Add Whip Muxer support for subsecond latency streaming#1

FFmpeg: Add Whip Muxer support for subsecond latency streaming#1
winlinvip wants to merge 60 commits intoossrs:masterfrom
winlinvip:feature/rtc-muxer

winlinvip commented Apr 16, 2023 •

edited

Loading

Uh oh!

winlinvip commented May 15, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

winlinvip commented Apr 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

winlinvip commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

winlinvip commented Apr 16, 2023 •

edited

Loading

winlinvip commented May 15, 2025 •

edited

Loading