This used to be a fixed array size but now is not, and so just acts as
an artificial limitation.
Ref #2241
Change-Id: Ifd5d1951153c9cc7d33a7c5d0d4af7b4ad06efd6
kernelize_one() takes each sink's lock with a trylock, and kernelize() starts
over at once when one is busy. kernelize() is the only place that holds the
locks of two streams at once, and streams that are each other's sinks hold
their own lock while they try the other's, so two kernelize() of one call
start over against each other. After a change of subscriptions all of them
are kernelized on their next packet, at the same moment. With about eight or
more such streams the restarts do not end: the threads spin with the call's
master lock held, the streams' sockets are no longer read, the log fills with
"Kernelizing media stream" lines, and commands for the call are not answered.
A lock in the call, taken ahead of the stream's own lock, runs one
kernelize() of the call at a time. A sink's trylock then only meets a holder
of that single lock, which waits for nothing while it holds it, so the
restart ends. Streams that are each other's sinks are in one call, as
connect, inject and mesh merge their calls, and kernelize() of other calls
does not wait.
A stream with more than RTPE_MAX_FORWARD_DESTINATIONS outputs, two per sink,
was pushed to the kernel anyway: the module answered EINVAL, each output
ENOENT, and the stream was still marked kernelized. It now takes the
no_kernel path.
The limit itself is left as it is. It came with 01a10c0379, where it sized
the array of outputs in struct rtpengine_list_entry; that struct is gone, and
until now the module's check was its only use.
Closes#2241
Change-Id: I8d8edc824652977c7ad8af836038a3495f9025be
ICE=remove in an answer is not a good way to reject an ICE offer, as ICE
might have already been in progress. Drop support for it and require
ICE=reject instead.
Change-Id: If4cda6ce3cd06701e2a841971b41eacf4d236e88
Don't forward STUN packets even if there is no ICE agent (unless
passthrough mode is active). Prevents accidental ICE negotiations
between endpoints.
Change-Id: I4a34de5ba4dd6c1159d256c5cd8c9190945aa4e1
Split up ICE accept/answer and ICE offer logic.
Make sure ICE is not active when it shouldn't be (e.g. stray ICE
attributes in an answer).
Consolidate publish ICE logic.
Update tests: controlling/controlled flag is now not set without ICE.
Closes#2161
Ref #2207
Change-Id: I5074002be3ec17eef7528fdf557f92ba063d4dbe
The state of ICE on the B leg has nothing to do with the state of ICE on
the A leg.
Ref #2161
Ref #2207
Change-Id: I451796dcee10a163c4942cf61aed7c5710413e02
Leaving the destruction of the DTLS objects until shutdown can hide the
non-zero exit code from a failed test
Change-Id: I30bb4c20bc55a163ae625f6a046c218436b957e2
Even though RFC 3711 implicitly says that ROC must not be reset when
rekeying, and that the crypto context is tied to address/port/SSRC, RFC
5764 says in section 3 that each DTLS association carries one SRTP
context per direction. This implies that a new DTLS association comes
with a new SRTP context, which in turn implies that the ROC starts at
zero, even if address/port/SSRC are the same. This also seems to match
what other DTLS-SRTP implementations are doing.
Closes#2240
Change-Id: Ia12ea5dbe6532fb12ab5f41600477fa7ed2e8161
Closes#2238
On a change of source, a fixed egress SSRC carries the sequence numbering
across by re-basing seq_diff on the egress counter (ext_seq). When nothing
has been counted under the SSRC yet, that counter is 0 and the re-base
restarts the sequence at 1. Under the unchanged SSRC the receiver sees a
jump back; with SRTP downstream the replay protection may discard the
stream for a long while.
Re-base only once something has been counted under the SSRC; until then
keep the offset in use. Together with the previous commit an empty counter
now only means a leg that has not sent anything yet.
Change-Id: I05ef221a5a2db016a24099037b20ec91c9cb884d
Closes#2238
Plain RTP output kept the egress index in userspace only when its input was
RTP/AVP (__sh_noop_index); from SRTP or AVPF input (__sh_savp2avp,
__sh_avpf2avp, __sh_savpf2avp) the output handler was a no-op, so such a
leg had no egress counter until the kernel took over.
With a fixed egress SSRC, the re-base on a change of source reads that
counter (ext_seq), found it empty or stale and made the sequence jump under
an SSRC that the next hop cannot see change. With a WebRTC client feeding a
plain RTP leg that is bridged back to another WebRTC client, the jump
reached the far browser, whose SRTP replay protection then dropped audio.
These three handlers now use __shio_noop_index like RTP/AVP input does. The
cost is the index arithmetic on userspace-forwarded packets; the kernel
already keeps the index. Codec-generated packets on such legs (e.g. an
audio player) are indexed too, through the same hook.
Change-Id: Iae8c14b77f5a052cbcc918ce3f903b7b9dc18c88
Closes#2237
__send_timer_rtcp() kept ssrc_out->h.lock across rtcp_send_report(), which
locks other SSRC entries: the receiver-report sources of the sink media and
the sender-report bookkeeping entry on the far media, i.e. ingress entries.
The transcoder locks ingress first and egress second (__ssrc_lock_both).
The two orders met on a call with a transcoded leg and the daemon
deadlocked: a poller in __ssrc_lock_both held the ingress entry and waited
for the egress one, while a send timer in rtcp_sender_report held that
egress entry and waited for the ingress one. Every other thread queued
behind them, including the control socket.
Take the lock only to check and claim next_rtcp, then send the report
without it, as the RTCP_GEN timer in codec.c already does.
Change-Id: Ie9170f71f7ec2336a5ee03f477cc1845a5c947e5
Closes#2204
Ref #21816b3bf829 stopped the explicit DEL after a failed restore of a foreign
call, but call_destroy() can still delete the owner's record: the call
is only flagged foreign once the restore succeeds, while
redis_hosted_db is read from the record (the owner's DB) early on. A
restore that fails after that point, e.g. with "failed to create sfds"
when the local interfaces differ from the owner's, goes through
redis_delete() as an own call and deletes the key in the owner's DB.
Mark the partial replica as never written before destroying it.
Change-Id: Ia64b5f7df9bffa6a6e08548f63a52ab0023a8a4f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes#2213
With the TLS options, the Sentinel thread still spoke plain TCP to the
Sentinels and for the role check of the master they name, so a Sentinel
set with tls-port and port 0 could not be used.
sentinel_connect() now starts TLS with the same context and options. A
Sentinel's certificate must match its host as given in
--redis-sentinel; a master's must match the address the Sentinels gave,
or --redis-tls-server-name. The handshake happens in the thread, never
under r->lock.
Tested by auto-daemon-tests-redis-tls.pl.
Change-Id: I84daf340c5f464765f234b35255cc76238bc8d13
Closes#2213
There is no way to reach a Redis server that only accepts TLS (managed
services, or tls-port with port 0); asked on the mailing list in 2022:
https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ
With --redis-cafile or --redis-capath, every Redis connection uses TLS,
and --redis-certfile with --redis-keyfile adds a client certificate. As
for MQTT, the options exist only in a build with hiredis_ssl. hiredis
checks only the chain, so the name is checked here, in the
subjectAltName: the host as configured, the IP address, or
--redis-tls-server-name. The handshake is bounded by the command
timeout, or by the connect timeout when there is none.
Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to
Build-Depends).
Change-Id: I74a78be0d40c3755fb1071536deb41443c890817
Closes#2235Closes#2229
rtcp_Xr() checks that el->len can hold count * sizeof(report_block)
but does not subtract the fixed header that precedes the report
blocks. A Sender Report has a 28-byte header before the blocks, a
Receiver Report has 8 bytes. Without accounting for this, a packet
just large enough for the header but with count=1 passes the check
and the report block read goes past the buffer.
Pass the header size into the validation so the check covers
header + blocks.
Change-Id: I30f4250b07b788eb28cb030ac8b718e59f0870d5
Closes#2212
Replication is asynchronous, so a master that is replaced takes with it
the writes it acknowledged but had not replicated: a call created then
is missing on the new master, and a call deleted then comes back.
--redis-resync-on-master-change (off by default) keeps the
master_replid of the INFO that redis_connect() already sends. When the
write connection reconnects to another ID, every own call stored at
least once, and the own DELs of the last --redis-resync-delete-window
(at most --redis-resync-delete-max), are queued for the repair thread
of --redis-repair-writes; the calls through the call iterator, one at a
time. No key is scanned. New statistics: redisresyncs, redisresynccalls
and redisresyncdeletes.
Tested by auto-daemon-tests-redis-resync.pl.
Change-Id: I0daac4d2b9ada45a843d27a6de073d8822067a8b
Closes#2212
After a failover the new master refuses writes with NOREPLICAS until it
has a replica. Those writes are counted and logged, but never made
again: a call created then is missing at the next restore, and a call
deleted then comes back.
With --redis-repair-writes (off by default), an own call whose write was
not stored (an error reply, a connection lost before the reply, a
skipped write) is marked in a table keyed by call ID: the call itself,
held, while it is in the call table; its ID and database once it is
deleted, as a deleted call keeps its memory until its last reference
goes. A stored write clears the mark. A marked SET supersedes a marked
DEL of the same ID; if its call was never written, it takes that key
over, so that its own deletion deletes it. A "redis repair" thread
writes the marked calls again, one at a time, through
redis_write_call(), now shared with redis_update_onekey(): a live call
gets its current state, the key of a deleted call a DEL, unless a new
call uses its ID. Signalling writes waiting for r->lock go first, for
100 ms at most. A failed write is retried after a second; at most 100000
calls are kept. New statistic: redisrepairedwrites. A repaired DEL
assumes one writer per database.
Tested by auto-daemon-tests-redis-repair.pl and
auto-daemon-tests-redis-repair-ids.pl.
Change-Id: Ia56a36bba0dd7eed3754694128e2ba6a74827c42
Closes#2212
A subscription made through "sentinel" stayed on the old master after a
failover, and missed the notifications of the new one.
With Sentinel, the notification event loop gets a one-second timer:
when the master changed, it closes the subscription, and
redis_notify_loop() subscribes again on the new master. The
subscription's context also gets a connect callback, since hiredis
frees a context that fails to connect without calling the disconnect
callback. Notifications missed during the switch are not replayed.
Tested by auto-daemon-tests-redis-sentinel-subscribe.pl.
Change-Id: Ia14d48e220bbf98cc48964e76f33ff2a497953e9
Closes#2212
Behind Redis Sentinel, rtpengine stays on the address it was given:
after a failover its writes go to the old master, which refuses them
until it is demoted, then serves them as a replica.
New options --redis-sentinel, --redis-sentinel-master and
--redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in
--redis, --redis-write or --redis-subscribe. A "redis sentinel" thread
asks every Sentinel for the master, takes the highest config epoch
(never an older one than in use), checks the node with ROLE and
publishes it with a generation number. It also runs at once when a
connection meets a replica. redis_check_conn() compares the generation
and reconnects on a change; these connections require the master role.
Without "sentinel", none of this runs.
Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
Change-Id: I7b31b50968fd6a255b9933fbf2c95fc95a8b8408
Closes#2212
--redis-write accepts any role, so after a failover that demotes its
node, rtpengine reconnects to the same address, accepts the replica,
and every later write is refused with READONLY. --redis alone requires
the master at connect, but keeps the connection when its writes are
refused.
The write connection now requires the master role, as --redis alone
does, and a connection that requires it is dropped on a write refused
with READONLY or MASTERDOWN, so that the next operation reconnects and
checks the role. Other refusals keep the connection. The read database
of --redis, when --redis-write is given, still accepts any role.
Tested by auto-daemon-tests-redis-master-only.pl; the READONLY cases
of auto-daemon-tests-redis-errors.pl, which expected the connection to be
kept, now use NOREPLICAS.
Change-Id: Ie164c2c3495c6c8b77553caf9a449e40f8e13699
Cache timestamps from the actual output encoder rather than the DTX decoder context, and retain elapsed time when transcoding resumes after playback or floor blocking. Add repeated-burst coverage for PCMA, Opus and G.722, including timestamp wraparound, and update DTX restart expectations.
Validation: 712 repeated-burst checks, 19 null-packet/DTMF checks and 87 playback checks passed. Extended DTX retains the same 56 baseline failures with no additional failures.
Closes#2191
Change-Id: I3b142a5ea377b94c2a1d16e97dee85569683af86
A fix-up for 4fbef394
Since that commit, the printer adds the `mid` from the egress media
extmap even when the input had none and the older bug (with the missing
room in `codec_add_dtmf_packet()`) started corrupting packets.
Just reserve the extension room in the same way the `codec_add_raw_packet_dup()`
and the `packet_encoded_packetize()` already do.
An example of bug triggered: chrome sending the DTMF with no extensions.
Change-Id: I87a61e1f87e896609e76bbdf87b4c7827027f39d
Expose locally measured RTP loss and jitter in NG query/delete output
even without RTCP, merge kernel-measured per-SSRC loss/jitter from shm
into userspace consumers (NG, MQTT, RTCP RRs, call summary), start the
kernel loss tracker at -1 like userspace, and skip jitter metrics when
no RTP clock rate is known.
Closes#2201Closes#934
Change-Id: Id7a6b1b87ae04def15fd81968d3be07fdf385395
Replace spinning for websocket_service_lock with a short mutex-protected
writable queue. All lws_callback_on_writable() calls are now marshalled
onto the LWS service thread and only run after lws_service() returns,
instead of being issued directly from worker threads while the service
thread may be holding the same lock. The callback queue mutex is
released before calling into libwebsockets, and the amount of queued
work drained per service cycle is bounded to avoid starving other
connections under a connection-abort storm. Pending callbacks are
dropped on teardown before the LWS session storage is freed.
Add an HTTP disconnect-storm regression test (two worker threads
hammering /metrics with aborted connections while a liveness /ping
request keeps proving the service thread is not stuck) to
t/auto-daemon-tests-websocket.py, plus a standalone stress script. The
stress test waits for the listening socket to come up before firing,
avoiding a FileNotFoundError race on CI.
Closes#2206
Change-Id: I81e03c1d7f1a0b5650aa719d00d60be3d3d0213b
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>