Leaving the destruction of the DTLS objects until shutdown can hide the
non-zero exit code from a failed test
Change-Id: I30bb4c20bc55a163ae625f6a046c218436b957e2
Even though RFC 3711 implicitly says that ROC must not be reset when
rekeying, and that the crypto context is tied to address/port/SSRC, RFC
5764 says in section 3 that each DTLS association carries one SRTP
context per direction. This implies that a new DTLS association comes
with a new SRTP context, which in turn implies that the ROC starts at
zero, even if address/port/SSRC are the same. This also seems to match
what other DTLS-SRTP implementations are doing.
Closes#2240
Change-Id: Ia12ea5dbe6532fb12ab5f41600477fa7ed2e8161
Closes#2213
With the TLS options, the Sentinel thread still spoke plain TCP to the
Sentinels and for the role check of the master they name, so a Sentinel
set with tls-port and port 0 could not be used.
sentinel_connect() now starts TLS with the same context and options. A
Sentinel's certificate must match its host as given in
--redis-sentinel; a master's must match the address the Sentinels gave,
or --redis-tls-server-name. The handshake happens in the thread, never
under r->lock.
Tested by auto-daemon-tests-redis-tls.pl.
Change-Id: I84daf340c5f464765f234b35255cc76238bc8d13
Closes#2213
There is no way to reach a Redis server that only accepts TLS (managed
services, or tls-port with port 0); asked on the mailing list in 2022:
https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ
With --redis-cafile or --redis-capath, every Redis connection uses TLS,
and --redis-certfile with --redis-keyfile adds a client certificate. As
for MQTT, the options exist only in a build with hiredis_ssl. hiredis
checks only the chain, so the name is checked here, in the
subjectAltName: the host as configured, the IP address, or
--redis-tls-server-name. The handshake is bounded by the command
timeout, or by the connect timeout when there is none.
Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to
Build-Depends).
Change-Id: I74a78be0d40c3755fb1071536deb41443c890817
Closes#2212
Replication is asynchronous, so a master that is replaced takes with it
the writes it acknowledged but had not replicated: a call created then
is missing on the new master, and a call deleted then comes back.
--redis-resync-on-master-change (off by default) keeps the
master_replid of the INFO that redis_connect() already sends. When the
write connection reconnects to another ID, every own call stored at
least once, and the own DELs of the last --redis-resync-delete-window
(at most --redis-resync-delete-max), are queued for the repair thread
of --redis-repair-writes; the calls through the call iterator, one at a
time. No key is scanned. New statistics: redisresyncs, redisresynccalls
and redisresyncdeletes.
Tested by auto-daemon-tests-redis-resync.pl.
Change-Id: I0daac4d2b9ada45a843d27a6de073d8822067a8b
Closes#2212
After a failover the new master refuses writes with NOREPLICAS until it
has a replica. Those writes are counted and logged, but never made
again: a call created then is missing at the next restore, and a call
deleted then comes back.
With --redis-repair-writes (off by default), an own call whose write was
not stored (an error reply, a connection lost before the reply, a
skipped write) is marked in a table keyed by call ID: the call itself,
held, while it is in the call table; its ID and database once it is
deleted, as a deleted call keeps its memory until its last reference
goes. A stored write clears the mark. A marked SET supersedes a marked
DEL of the same ID; if its call was never written, it takes that key
over, so that its own deletion deletes it. A "redis repair" thread
writes the marked calls again, one at a time, through
redis_write_call(), now shared with redis_update_onekey(): a live call
gets its current state, the key of a deleted call a DEL, unless a new
call uses its ID. Signalling writes waiting for r->lock go first, for
100 ms at most. A failed write is retried after a second; at most 100000
calls are kept. New statistic: redisrepairedwrites. A repaired DEL
assumes one writer per database.
Tested by auto-daemon-tests-redis-repair.pl and
auto-daemon-tests-redis-repair-ids.pl.
Change-Id: Ia56a36bba0dd7eed3754694128e2ba6a74827c42
Closes#2212
A subscription made through "sentinel" stayed on the old master after a
failover, and missed the notifications of the new one.
With Sentinel, the notification event loop gets a one-second timer:
when the master changed, it closes the subscription, and
redis_notify_loop() subscribes again on the new master. The
subscription's context also gets a connect callback, since hiredis
frees a context that fails to connect without calling the disconnect
callback. Notifications missed during the switch are not replayed.
Tested by auto-daemon-tests-redis-sentinel-subscribe.pl.
Change-Id: Ia14d48e220bbf98cc48964e76f33ff2a497953e9
Closes#2212
Behind Redis Sentinel, rtpengine stays on the address it was given:
after a failover its writes go to the old master, which refuses them
until it is demoted, then serves them as a replica.
New options --redis-sentinel, --redis-sentinel-master and
--redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in
--redis, --redis-write or --redis-subscribe. A "redis sentinel" thread
asks every Sentinel for the master, takes the highest config epoch
(never an older one than in use), checks the node with ROLE and
publishes it with a generation number. It also runs at once when a
connection meets a replica. redis_check_conn() compares the generation
and reconnects on a change; these connections require the master role.
Without "sentinel", none of this runs.
Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
Change-Id: I7b31b50968fd6a255b9933fbf2c95fc95a8b8408
Closes#2212
--redis-write accepts any role, so after a failover that demotes its
node, rtpengine reconnects to the same address, accepts the replica,
and every later write is refused with READONLY. --redis alone requires
the master at connect, but keeps the connection when its writes are
refused.
The write connection now requires the master role, as --redis alone
does, and a connection that requires it is dropped on a write refused
with READONLY or MASTERDOWN, so that the next operation reconnects and
checks the role. Other refusals keep the connection. The read database
of --redis, when --redis-write is given, still accepts any role.
Tested by auto-daemon-tests-redis-master-only.pl; the READONLY cases
of auto-daemon-tests-redis-errors.pl, which expected the connection to be
kept, now use NOREPLICAS.
Change-Id: Ie164c2c3495c6c8b77553caf9a449e40f8e13699
Cache timestamps from the actual output encoder rather than the DTX decoder context, and retain elapsed time when transcoding resumes after playback or floor blocking. Add repeated-burst coverage for PCMA, Opus and G.722, including timestamp wraparound, and update DTX restart expectations.
Validation: 712 repeated-burst checks, 19 null-packet/DTMF checks and 87 playback checks passed. Extended DTX retains the same 56 baseline failures with no additional failures.
Closes#2191
Change-Id: I3b142a5ea377b94c2a1d16e97dee85569683af86
Expose locally measured RTP loss and jitter in NG query/delete output
even without RTCP, merge kernel-measured per-SSRC loss/jitter from shm
into userspace consumers (NG, MQTT, RTCP RRs, call summary), start the
kernel loss tracker at -1 like userspace, and skip jitter metrics when
no RTP clock rate is known.
Closes#2201Closes#934
Change-Id: Id7a6b1b87ae04def15fd81968d3be07fdf385395
Replace spinning for websocket_service_lock with a short mutex-protected
writable queue. All lws_callback_on_writable() calls are now marshalled
onto the LWS service thread and only run after lws_service() returns,
instead of being issued directly from worker threads while the service
thread may be holding the same lock. The callback queue mutex is
released before calling into libwebsockets, and the amount of queued
work drained per service cycle is bounded to avoid starving other
connections under a connection-abort storm. Pending callbacks are
dropped on teardown before the LWS session storage is freed.
Add an HTTP disconnect-storm regression test (two worker threads
hammering /metrics with aborted connections while a liveness /ping
request keeps proving the service thread is not stuck) to
t/auto-daemon-tests-websocket.py, plus a standalone stress script. The
stress test waits for the listening socket to come up before firing,
avoiding a FileNotFoundError race on CI.
Closes#2206
Change-Id: I81e03c1d7f1a0b5650aa719d00d60be3d3d0213b
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a codec module for Codec 2, using libcodec2 for decoding and
encoding.
There is no RFC payload format for Codec 2, so this follows the
convention Asterisk uses: encoding name "codec2" at 8000 Hz, mono, with
one or more complete frames per RTP packet, each padded to whole bytes.
The mode is selected with the format parameter "mode=" (3200, 2400,
1600, 1400, 1300, 1200 or 700C) and defaults to 2400, the only mode
Asterisk supports. With the transcode option it can also be given as
the bit rate, e.g. "codec2/8000/1/3200".
Different modes are different bit streams, so payload types only match
when they use the same mode. Modes 1600 and below use 40 ms frames, so
their ptime is rounded up to a multiple of 40 ms. Modes not known to
rtpengine (e.g. 450) are passed through unchanged when both sides
agree, but can't be transcoded.
libcodec2 is optional. It is detected with pkg-config (or from
/usr/include/codec2) by utils/gen-codec2-flags. Without it, Codec 2 is
still recognised for SDP handling and passthrough. In a Debian build
the dependency can be dropped with the pkg.ngcp-rtpengine.nocodec2
build profile.
test-transcode gains Codec 2 tests for both directions in modes 2400,
3200 and 1300, plus passthrough. check_encoder() is no longer limited
to the extended AMR tests, as these use it too.
Closes#2226
Change-Id: I2cad021f48e2043f36c30f5e6290cb7d6eba8c1f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The egress sequence offset of a stream counts the packets rtpengine sends in
place of the sender's: the decoder charges one per packet it consumes and the
encoder pays one back per packet it sends, so the output stays contiguous
through transcoding, DTX and DTMF. The audio player's decoder charges the same
way, but its frames go into the mix, which leaves under the media player's own
SSRC, so nothing pays the charge back: after an injection of N packets the
stream's offset is -N. The kernel module applied it to every packet while the
plain userspace passthrough handler applied it only under a fixed egress SSRC,
so after `inject stop` the receiver saw the sender's numbers from userspace
and then, from the kernel, numbers N lower; under a fixed egress SSRC a number
could repeat after the injection. SRTP derives the packet index from the
sequence number and discards packets behind the replay window: a WebRTC
client, whose window is 1024 packets, went silent for 20 seconds after a
40-second injection.
A decode-only handler, whose output never leaves under the input's egress
SSRC, no longer moves that offset: not for the packets it decodes, not for a
DTX discard and not for a sequence reset of the sender. An injection then
leaves the offset where it was, and the sender's packets leave with the
sender's own numbers from userspace and from the kernel alike, plain RTP or
SRTP, whatever the keys.
The receiver sees one forward gap per injection, the length of the injected
stretch.
Closes#2209
Change-Id: I409b4e51fe85168ec2449cd2602f24333dec539e
A sink that substitutes the SSRC rewrites the header the sender authenticated,
so it re-encrypts the packet. A fixed egress SSRC on a stream forwarded under
the sender's key used to reach the receiver with the sender's authentication
tag over the rewritten header, and the receiver dropped every packet. The
predicate is the one the kernel module already uses for ssrc_subst: echo,
transcoding or a fixed egress SSRC.
The handlers rewrite the packet in place while every sink of the stream is
handed the same received buffer, so a fixed egress SSRC, the sequence offset
and an injected DTMF event used to reach the next sink too, and after an event
the next sink read a header on the handler's stack. The sink loop now starts
every sink from the packet as received and gives each one its own copy unless
it is the last, as it already did for RTCP.
Closes#2209
Change-Id: I265d6b2d3eeac4effcbc76dcd65bd1d3fe8d2ba1
test6: lost / received / lost with forced emissions; the second forced
emission must return the lower queued packet, not the one that arrived
last. test7: a packet exactly one ring size ahead across the 65535->0
wrap-around must be handled as a seq reset.
Both tests fail on the previous sequencer code.
Change-Id: I47c5d0b961856255cb204fbcf2e656cf01b49d7d
Ref #2211
redis, redis-json and redis-subscribe compared raw bytes on their own
sockets, and redis-srtp-index and rollback-redis each carried the same
forked server. They now use NGCP::Rtpengine::FakeRedis, and the forked
server moves into it as redis_serve(). The checks, their names and
their counts are unchanged.
Change-Id: Ieb281e1fe2a772b31a2f928104c741e9c5607297
Ref #2211
A reconnect attempt runs under r->lock, which every ng request that
writes a call waits for (#1865). Against a server that drops packets it
takes the full connect timeout, so signalling stalls for up to that long
at each attempt, for as long as the outage lasts.
Make the attempt without the lock: the thread that reconnects sets
r->reconnecting and owns r->ctx until it clears the flag under the lock.
Other threads wait for it for up to --redis-reconnect-slow, then skip
their operation, as during the pause. A DEL, also the one after a failed
restore, waits for the attempt to end, as it waited for the lock before:
skipped, the key would stay. After the lock was released,
redis_update_onekey() checks again that the call is live.
redis_do_delete() leaves the key to a new call with the same ID, checked
on every DEL, and hands it over to that call if it was never written, so
that its own deletion deletes the key. The CLI and the notification
thread's own connection leave a connection being made alone. A command
on an established connection still holds the lock.
Tested by auto-daemon-tests-redis-reconnect.pl.
Change-Id: Ia98e01c57d4825633e473e3174947daa17afbf13
Ref #2211
The timer that rewrites never-stored SRTP calls can hold a call while it
waits for r->lock. If the call is deleted meanwhile, the deletion,
deciding without the lock that the call was never stored, deletes
nothing, and the timer then writes it: the call is back at the next
restore. Any update racing with a deletion can do the same.
Under r->lock, redis_update_onekey() writes a call only while it is
still in the call table, and redis_do_delete() decides "never stored",
so that a write already past that check is followed by the DEL.
Tested by auto-daemon-tests-redis-destroyed.pl.
Change-Id: I7329a721248b2c8ce64c9e778d2761951db676f9
Ref #2211
cli_handle() takes a str, but most "set" handlers parsed their value
with strtol(), strtod() or atoi() on instr->s and printed it with "%s",
reading up to the next NUL. The ng "cli" command passes a slice of the
request: "set maxsessions 1000" set 10007, "set rediscmdtimeout 0" set
7 ms, and error messages quoted the rest of the request.
Parse with the str_to_X helpers, plus a new str_to_d for maxcpu and
maxload, and print with STR_FORMAT. As with the str_to_X family, a
value with trailing characters ("10x") is refused, and so is a
non-number in the seven handlers that read it as 0. "terminate" now
matches all, own and foreign exactly (str_cmp()), not as a prefix.
Tested by auto-daemon-tests-redis-errors.pl and the new t/test-str.c.
Change-Id: I92ed33a4229688aba513c3d0410b9d77a8bcd516
Ref #2211
On a role mismatch, the reconnect in redis_check_conn() sleeps one
second, asks INFO again and sleeps once more: two seconds under
r->lock, which every Redis update holds before the ng reply (#1865).
After a failover, with --redis on the old master's address, every
update did this, and ng requests timed out.
Make one attempt, with no sleep. After a role failure, or a failure
that took --redis-reconnect-slow (100 ms) or more, skip Redis
operations on that connection for --redis-reconnect-pause (1000 ms),
through restore_tick_us as --redis-allowed-errors does. A quick failure
(connection refused) is retried at once, as before. As with a failed
attempt, a skipped operation is not written; skips are logged once per
pause. The startup connection keeps its eleven attempts.
Tested by auto-daemon-tests-redis-errors.pl.
Change-Id: Ia84288f1c6fe97c4317689fef75d0ff5fc9061f8
Ref #2211
Call updates (SET) and deletions (DEL) are pipelined, and
redis_consume() freed their replies unread: a write refused with
READONLY or NOREPLICAS, as during a Sentinel failover, left no trace in
rtpengine.
Check every reply. An error reply is counted in a new statistic,
rediswriteerrors, and logged at most once a second per connection; the
connection is kept, since a reconnect would get the same answer. A read
error or timeout on the replies drops the connection at once instead of
at the next PING. An encoding failure no longer drops the connection.
Tested by auto-daemon-tests-redis-errors.pl, with a new fake Redis
module, NGCP::Rtpengine::FakeRedis.
Change-Id: I4c49460d7894de0c5eaf5e5c27f5a6460e4115a7
Firstly, play a long media playback towards A with `stop-on-ingress`
and check that the first RTP packet from B ends it. That means,
the player goes silent, the triggering packet itself is not forwarded
and the following packets from B are forwarded to A.
A later `stop media` from the controller is accepted and doesn't disturb the forwarding.
Then, as a counter-check, the same playback with just `block egress`
keeps playing while B's media stays blocked.
Change-Id: I13427c95bb4358549ad52a91b12b715b036c51be
Ports normally come from the port pool, so the caller only learns the
port from rtpengine's reply. A controller that manages the media ports
itself, and has already announced them to the remote side, needs
rtpengine to open exactly those ports.
`local-port` sets the port facing the recipient of the message (the one
in the returned SDP) and `from-local-port` the one facing its sender.
Each takes an integer for the first media section or a list, one per
media section. RTCP takes the next port when not multiplexed. The ports
may be outside of the pool, they are only used when the media's ports
are first opened, and the request fails if they are taken.
`local-port` works in every message that opens ports: offer, answer,
publish, subscribe request and create. `from-local-port` only has a
meaning in offer and answer.
The frequencies list parser becomes a generic integer list parser.
Closes#2216
Change-Id: Ie82338becb8347757c7cc19d90675b927f1edfee
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A controller that allocates the SRTP keys itself needs rtpengine to
encrypt with them, instead of generating keys or copying the ones from
the received SDP.
`SDES-key-SUITE:KEY` (or `key-SUITE:KEY` in the `SDES` list) sets the
master key and salt, base64 encoded as in an a=crypto inline: key, for
that crypto suite on the side facing the recipient of the message. It
works in offers and answers, and also for SRTP to SRTP calls, where
rtpengine then re-encrypts instead of passing the packets through.
Closes#2217
Change-Id: I1a12e4488617cd5daaa49741773a14e330abbfdf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ref #2211
When json_restore_call() fails, it sends "DEL <call-id>" twice: on the
write connection, in its own DB, and on the notification connection, in
whatever DB the last keyspace notification selected. For a foreign call
(a keyspace notification, or the restore of a subscribed keyspace at
startup) that key belongs to the instance that owns the call, which
keeps writing it: a standby that fails to restore one record deletes the
owner's copy, and the call cannot be taken over any more. For an own
call, the DEL on the notification connection can hit a peer's DB.
Only delete an unrestorable record when the call is our own, and only on
the write connection, in our own DB. A foreign record that can never be
restored now stays until its owner rewrites or deletes it, or until it
expires (--redis-expires). Seen in a lab with Redis Sentinel, where a
subscriber's DEL was refused only because its node had just become a
replica.
Tests, in auto-daemon-tests-redis-subscribe.pl: a keyspace notification
for a foreign call whose record cannot be parsed is followed by no
command on either connection, and an own record that cannot be parsed at
startup is deleted once, on the write connection only. Without this
change both get a DEL on both connections.
Change-Id: I5d8a53f3515596cf3fc2c58ea97847476385b876
Clear the matching per-destination suppression state when a completed telephone-event is deduplicated at media scope. Otherwise later listeners suppress speech until the timestamp-discrepancy timeout expires.
Add repeated-burst coverage for AMR-WB and Opus listeners in both destination orders, with blocked DTMF, prompt playback, and an event sink.
Closes#2193
Change-Id: Ia73e9324ecc6122b090be36c384b21000ef94609
monologue_publish() never copied the offered label into the media,
so sdp_create() omitted a=label from the publish answer and from
subscribe requests for that publisher. Add the missing
media_update_label() call.
Closes#2200
Change-Id: Id9aaf4ab09c21f98d3dcaf78632413b99f4a97ff
ssrc_collect_metrics() converted the running RFC 3550 jitter estimate
to milliseconds in place on every call timer run with --measure-rtp,
dividing it by clock_rate/1000 each time. The estimator is kept in
RTP timestamp units and also feeds generated RTCP RRs, which therefore
reported a jitter decaying towards zero. Convert a copy instead.
The RTCP test expected the resulting zero jitter; it now gets the
value actually measured from its back-to-back packets.
Change-Id: Ice29029322764fd76374eb3f7dda517570f7f75b
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy create-answer and subscribe-answer SDP into call-owned storage before parsing, matching offer/answer and publish. Retained stream parameters otherwise reference released NG request buffers and later subscriptions can lose their media type and codecs.
Initialize created-answer monologue session state so generated subscriptions retain a valid SDP origin and session name. Add request-buffer-churn and decoded-audio regressions for created and subscribed endpoints.
Validation on standalone Sipwise master: daemon build, all 89 retained-SDP assertions, all 652 create assertions, and all 932 publish/subscribe assertions pass. The cumulative 26.2.1 release also has isolated browser Opus decoding, privacy, recording, and mixer regression coverage.
Closes#2194
Change-Id: Id8ca0e0ff691356d905ee94c0f41090f973fc70f
Put everything into a dedicated struct
Split out code into separate file
Introduce files for code shared between daemon and kernel module
Update return type to bool
Change-Id: I2cd78dcdef3490bf00eb7e27474303f14cfb1900
Share SSRC dump and restore helpers between ingress and egress, keeping
ssrc_out_table as the outbound list name. Scope each hash lock to its
dump rather than holding both locks throughout media serialization.
Restore records without an outbound list through the original combined
in/out path, including records written before the SSRC hash split.
Adapt the takeover regression test from kbinetruy's PR #2188 to exercise
native and JSON records, sequence rollover, encryption after restart,
and both legacy record layouts. Follow the existing Redis helper,
initializer, iteration, and daemon-test patterns.
Ref #2185
Change-Id: I0ac7e6b351fd3e2bcab38d0318d1787e5332c442
Co-authored-by: Kevin Binetruy <kevin.binetruy@ovhcloud.com>
Since the SSRC contexts were split into separate ingress and egress
entries (cf22d829), only the ingress entries are written to the SSRC
table, and the lines for the outbound index are commented out. The
restore still looks for the out_* keys in the ingress entries, where
they no longer are.
A node that takes over a call therefore starts encrypting from index 0
again. A receiver estimates the ROC from its own state, so if the
sequence number of the call has already wrapped, it rejects every
packet after the takeover.
Write the egress entries to a list of their own, ssrc_out_table, and
restore them from there. Records without the list restore as before.
Fixes#2183Closes#2185
Change-Id: Id28dc86c658f7e94dd87443fc0f09225beaf484e
A call restored from Redis starts with every sfd unconfirmed and every
stream's el_flags at zero, which is EL_DELAYED, regardless of what the
offer asked for or what --endpoint-learning is set to. After a
keyspace-notification failover, all adopted calls therefore re-learn
their peer at the same time, and strict-source has nothing to compare
against until they have.
Write the sfd's confirmed flag and the stream's el_flags and
learned_endpoint to the record, and read them back on restore. The two
stream fields were so far only part of the rollback snapshot. Move them
into the shared stream encoder and decoder, so that both use the same
definition. Records without the new keys restore as before.
Fixes#2182Closes#2184
Change-Id: I6228e55796e7a680b05797df985573e8ab54c653