jbuf: adaptive playout time calculation #2757

sreimers · 2023-10-09T11:21:49Z

Playout time calculation for jitter buffer. It replaces frame calculation, since it uses a local time reference offset (based on clock rate).

It's based on suggestions from this book: https://csperkins.org/standards/rtp-book.html

Future improvements:

Maybe the spike/jitter algorithm can be improved (we can test with different modes JBUF_ADAPTIVE_SPIKE...)
Currently the jitter and clock skew from soundcard (auplay) is ignored, at least cheap cards/drivers maybe perform badly. A tick based offloading could help to calculate within jbuf. But good skew adaption needs samplerate conversion instead of removing/adding complete frames. Maybe we can use a audio filter module with libsamplerate.
Introduce empty frames (if packet loss is detected a empty frame should scheduled as replacement)

cspiel1 · 2023-10-09T12:44:59Z

src/jbuf.c

-	/* Update sequence number for 'get' */
-	jb->seq_get = f->hdr.seq;
+	/* Check playout time */
+	if (f->playout_time > next_playout) {


Frame completeness is not checked. Is it already ensured by skew and jitter?

Yes, it's ensured by jitter compensation, it's now a time based buffering delay, this should give the lowest delay which is needed to buffer.

okay, we can compare the plots if you are ready.

Sure, give me some time to prepare and ensure the algorithms are working correctly (wrap-around calculations are tough ;-))

cspiel1 · 2023-10-10T08:41:18Z

When jitter and/or skew reduces, then jbuf should return EAGAIN --> seems to be done already in this branch.

Then aubuf will increase number of buffered audio frames. Since jbuf already does the jitter compensation and packet re-ordering these extra frames can be dropped from aubuf. Maybe immediately dropped, or after some short time. This should reduce the latency.

Threads involved for audio:
Thread 1 - main thread, or the RX thread in near future

jbuf_put() is called
jbuf_get(), maybe by a timer. Or maybe by a wait condition (mqueue) from Thread 2.
does also decoding
pushes to aubuf

Thread 2 - audio playback thread

only reads the audio frames from aubuf

Edit: Your PR looks promising for me. I also think that the jitter compensation should be done in jbuf. This will reduce the overall latency because jbuf is able to handle jitter and out-of-order packets.

But unit tests will become difficult or have less coverage because jbuf_get() will depend on timing.

sreimers · 2023-10-10T09:01:09Z

jbuf_get(), maybe by a timer. Or maybe by a wait condition (mqueue) from Thread 2.

I implemented a fallback tmr solution here (fallback since we should avoid to many event loop wakeups): 80b2758

sreimers · 2023-10-10T09:39:17Z

But unit tests will become difficult or have less coverage because jbuf_get() will depend on timing.

I think we can use a virtual clock rate and small frames for tests. But yes this increases test time, at least for some milliseconds.
Edit: Or we can try to mockup tmr_jiffies somehow.

cspiel1 · 2023-10-10T11:22:51Z

But unit tests will become difficult or have less coverage because jbuf_get() will depend on timing.

I think we can use a virtual clock rate and small frames for tests. But yes this increases test time, at least for some milliseconds. Edit: Or we can try to mockup tmr_jiffies somehow.

Okay, let's try this!

Also we could try a from jbuf_put() completely independent timer. E.g. ptime triggered.

alfredh · 2023-10-26T07:01:30Z

please resolve the conflicts ..

alfredh · 2023-12-02T12:05:50Z

what is the status of this PR ?

Can you please rebase it on top of GIT HEAD ?

cspiel1 · 2023-12-04T09:53:04Z

I think we should prefer this approach over mine #2754.

Currently, studying the mentioned book.

sreimers · 2023-12-04T10:08:43Z

I will try to complete/cleanup this work soon. Maybe later this week.

src/core.h

src/rtprecv.c

cspiel1 · 2024-01-09T06:24:35Z

I'll try to fix the sanitizer warning. I think it is not related to this PR.

cspiel1 · 2024-01-09T11:00:52Z

PR for sanitizer warning: #2867

sreimers · 2024-01-12T10:31:11Z

The failing valgrind looks also unrelated: #2873

src/config.c

sreimers changed the title ~~jbuf: adaptive playout time calculation~~ jbuf: adaptive playout time calculation (Work in progress) Oct 9, 2023

cspiel1 reviewed Oct 9, 2023

View reviewed changes

sreimers force-pushed the playout_time branch 3 times, most recently from 73fcdd3 to 80b2758 Compare October 9, 2023 22:50

alfredh added this to the v3.7.0 milestone Oct 12, 2023

alfredh removed this from the v3.7.0 milestone Oct 25, 2023

cspiel1 mentioned this pull request Oct 26, 2023

jbuf frame completeness #2754

Closed

sreimers force-pushed the playout_time branch from c23e8ed to 23ace5b Compare November 6, 2023 17:09

sreimers force-pushed the playout_time branch from 23ace5b to 8f19c65 Compare December 4, 2023 08:43

sreimers force-pushed the playout_time branch from 8f19c65 to e1c3d3d Compare December 8, 2023 16:20

alfredh reviewed Dec 9, 2023

View reviewed changes

src/core.h Outdated Show resolved Hide resolved

alfredh reviewed Dec 9, 2023

View reviewed changes

src/rtprecv.c Show resolved Hide resolved

sreimers force-pushed the playout_time branch 2 times, most recently from 2d8b5d6 to 82b2225 Compare January 8, 2024 11:58

sreimers force-pushed the playout_time branch 2 times, most recently from 840fed9 to ebdda6a Compare January 11, 2024 08:51

sreimers force-pushed the playout_time branch from d713e39 to fb5b8cf Compare January 15, 2024 08:26

cspiel1 reviewed Jan 19, 2024

View reviewed changes

src/config.c Outdated Show resolved Hide resolved

sreimers added 29 commits January 8, 2025 14:11

refactoring

1b5cd49

fix linting

4de6794

fix unused delay warning

9867664

fix ccheck

04ceedf

cleanup and clock skew refactoring

a59721f

add seq late packets

ec9e2bc

fix INT_MIN

3b03960

adapt skew only in adaptive mode

7204a58

fix headers

2f08a0a

fix drift detection (check only within drift window)

30ab8b2

cleanup and reset playout on flush

62409d6

jbuf: check invalid ts_arrive header

7e61b69

fix ts_arrive test

897c58b

rtprecv: fix video max size

ed68f41

jbuf: fix skew drift window handling

aade365

jbuf: extend jbuf_stat with late_lost and current delay

007b045

refacto skew handling, remove unused n_underflow and re-add n_lost stats

fe9abc0

include: refactor config_avt audio/video struct

f13c0db

rename next play handler

f6fc7b7

config: decrease default max latency

8a39ce5

jbuf/stats: add current jitter delay and packets

bbfeec6

jbuf: fix next_play_h usage and use specific errnos

b0159f8

jbuf: improve doxygen for jbuf_next_play

7a389d9

test/jbuf: fix ENOENT empty list return check

53f4537

jbuf: fix wrong offset handling and improve jitter delay handling

dbc53e6

jbuf: add TRACE lost

d4dc011

jbuf: add jbuf_stat generic NACKS

823a955

rtprecv: fix err loop check

4958a28

test/jbuf: fix clang analyze warning

2ad3894

sreimers force-pushed the playout_time branch from 495a8ea to 2ad3894 Compare January 8, 2025 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jbuf: adaptive playout time calculation #2757

jbuf: adaptive playout time calculation #2757

sreimers commented Oct 9, 2023 •

edited

Loading

cspiel1 Oct 9, 2023

sreimers Oct 9, 2023

cspiel1 Oct 9, 2023

sreimers Oct 9, 2023

cspiel1 commented Oct 10, 2023 •

edited

Loading

sreimers commented Oct 10, 2023

sreimers commented Oct 10, 2023 •

edited

Loading

cspiel1 commented Oct 10, 2023

alfredh commented Oct 26, 2023

alfredh commented Dec 2, 2023

cspiel1 commented Dec 4, 2023

sreimers commented Dec 4, 2023

cspiel1 commented Jan 9, 2024

cspiel1 commented Jan 9, 2024

sreimers commented Jan 12, 2024

jbuf: adaptive playout time calculation #2757

Are you sure you want to change the base?

jbuf: adaptive playout time calculation #2757

Conversation

sreimers commented Oct 9, 2023 • edited Loading

cspiel1 Oct 9, 2023

Choose a reason for hiding this comment

sreimers Oct 9, 2023

Choose a reason for hiding this comment

cspiel1 Oct 9, 2023

Choose a reason for hiding this comment

sreimers Oct 9, 2023

Choose a reason for hiding this comment

cspiel1 commented Oct 10, 2023 • edited Loading

sreimers commented Oct 10, 2023

sreimers commented Oct 10, 2023 • edited Loading

cspiel1 commented Oct 10, 2023

alfredh commented Oct 26, 2023

alfredh commented Dec 2, 2023

cspiel1 commented Dec 4, 2023

sreimers commented Dec 4, 2023

cspiel1 commented Jan 9, 2024

cspiel1 commented Jan 9, 2024

sreimers commented Jan 12, 2024

sreimers commented Oct 9, 2023 •

edited

Loading

cspiel1 commented Oct 10, 2023 •

edited

Loading

sreimers commented Oct 10, 2023 •

edited

Loading