stats->ext_seq has been the SRTP receive index since srtp_index moved into
shm (08c7f5853d). The transcode sequencer overwrote it after every
emitted packet with its own roc<<16|seq, which lags the received packets
and, whenever the sequencer emits a late or replayed packet with a lower
sequence number, carries a roc that was incremented without a real
wrap-around. packet_index() then estimated the wrong index, and once the
sender's real roc reached 1 none of the fallback guesses matched, so every
packet of the stream failed authentication until the call ended.
packet_index() already maintains the index for every received packet, so
drop the write. Also make the sequencer's own roc increment only on a
forward wrap-around, never on a backwards step.
Closes#2221
Change-Id: Ia38bf8cbc63dd3435717927f1adfd81ac94ea347
test6: lost / received / lost with forced emissions; the second forced
emission must return the lower queued packet, not the one that arrived
last. test7: a packet exactly one ring size ahead across the 65535->0
wrap-around must be handled as a seq reset.
Both tests fail on the previous sequencer code.
Change-Id: I47c5d0b961856255cb204fbcf2e656cf01b49d7d
Treat a packet exactly one ring size ahead across the 65535->0
wrap-around as a seq reset, like the non-wrapped case, instead of
accepting it into the head slot.
Fixes#2220
Change-Id: I94e2fdd3d50d5c8241cfed5da06829097566356e
The decision to stop is taken in the `media_player_run()` with the call
read-locked and the player locked, the same way as the `run_func()` decides
that the playback is finished, so the stop can share the write-locked
part of the finished case, including the check that the player wasn't
stopped or restarted in-between.
That replaces the separate re-check of the `stop-on-ingress` state
and the rescheduling needed when it failed.
Stopping goes through the `call_stop_media()` taking the player directly,
so there is no need to look it up as `MP_DEFAULT` again.
Change-Id: Iaed17c10a0dc6d2b34cdac51fb7f81b7c3a71bb5
When the `run_func()` reports the end of the playback,
then the cleanup is done after dropping the call's read lock and
taking the write lock.
If the `play media` command restarts the player, or the `stop media` stops it
in the meanwhile, the cleanup then clears the `next_run` of the new playback,
which silently stops it and also lifts its `block egress`.
Simplest solution is to count every stop and start/rester of the player
under the write lock and then skip the cleanup if the count has changed
since `run_func()` was called, means the stop has done its own cleanup
and the restart has set up the new playback.
Change-Id: I030b7d32c1b15aaaf81b2f56492bc3299bf6c231
Ref #2211
redis, redis-json and redis-subscribe compared raw bytes on their own
sockets, and redis-srtp-index and rollback-redis each carried the same
forked server. They now use NGCP::Rtpengine::FakeRedis, and the forked
server moves into it as redis_serve(). The checks, their names and
their counts are unchanged.
Change-Id: Ieb281e1fe2a772b31a2f928104c741e9c5607297
Ref #2211
A reconnect attempt runs under r->lock, which every ng request that
writes a call waits for (#1865). Against a server that drops packets it
takes the full connect timeout, so signalling stalls for up to that long
at each attempt, for as long as the outage lasts.
Make the attempt without the lock: the thread that reconnects sets
r->reconnecting and owns r->ctx until it clears the flag under the lock.
Other threads wait for it for up to --redis-reconnect-slow, then skip
their operation, as during the pause. A DEL, also the one after a failed
restore, waits for the attempt to end, as it waited for the lock before:
skipped, the key would stay. After the lock was released,
redis_update_onekey() checks again that the call is live.
redis_do_delete() leaves the key to a new call with the same ID, checked
on every DEL, and hands it over to that call if it was never written, so
that its own deletion deletes the key. The CLI and the notification
thread's own connection leave a connection being made alone. A command
on an established connection still holds the lock.
Tested by auto-daemon-tests-redis-reconnect.pl.
Change-Id: Ia98e01c57d4825633e473e3174947daa17afbf13
Ref #2211
The timer that rewrites never-stored SRTP calls can hold a call while it
waits for r->lock. If the call is deleted meanwhile, the deletion,
deciding without the lock that the call was never stored, deletes
nothing, and the timer then writes it: the call is back at the next
restore. Any update racing with a deletion can do the same.
Under r->lock, redis_update_onekey() writes a call only while it is
still in the call table, and redis_do_delete() decides "never stored",
so that a write already past that check is followed by the DEL.
Tested by auto-daemon-tests-redis-destroyed.pl.
Change-Id: I7329a721248b2c8ce64c9e778d2761951db676f9
Ref #2211
cli_handle() takes a str, but most "set" handlers parsed their value
with strtol(), strtod() or atoi() on instr->s and printed it with "%s",
reading up to the next NUL. The ng "cli" command passes a slice of the
request: "set maxsessions 1000" set 10007, "set rediscmdtimeout 0" set
7 ms, and error messages quoted the rest of the request.
Parse with the str_to_X helpers, plus a new str_to_d for maxcpu and
maxload, and print with STR_FORMAT. As with the str_to_X family, a
value with trailing characters ("10x") is refused, and so is a
non-number in the seven handlers that read it as 0. "terminate" now
matches all, own and foreign exactly (str_cmp()), not as a prefix.
Tested by auto-daemon-tests-redis-errors.pl and the new t/test-str.c.
Change-Id: I92ed33a4229688aba513c3d0410b9d77a8bcd516
Ref #2211
On a role mismatch, the reconnect in redis_check_conn() sleeps one
second, asks INFO again and sleeps once more: two seconds under
r->lock, which every Redis update holds before the ng reply (#1865).
After a failover, with --redis on the old master's address, every
update did this, and ng requests timed out.
Make one attempt, with no sleep. After a role failure, or a failure
that took --redis-reconnect-slow (100 ms) or more, skip Redis
operations on that connection for --redis-reconnect-pause (1000 ms),
through restore_tick_us as --redis-allowed-errors does. A quick failure
(connection refused) is retried at once, as before. As with a failed
attempt, a skipped operation is not written; skips are logged once per
pause. The startup connection keeps its eleven attempts.
Tested by auto-daemon-tests-redis-errors.pl.
Change-Id: Ia84288f1c6fe97c4317689fef75d0ff5fc9061f8
Ref #2211
Call updates (SET) and deletions (DEL) are pipelined, and
redis_consume() freed their replies unread: a write refused with
READONLY or NOREPLICAS, as during a Sentinel failover, left no trace in
rtpengine.
Check every reply. An error reply is counted in a new statistic,
rediswriteerrors, and logged at most once a second per connection; the
connection is kept, since a reconnect would get the same answer. A read
error or timeout on the replies drops the connection at once instead of
at the next PING. An encoding failure no longer drops the connection.
Tested by auto-daemon-tests-redis-errors.pl, with a new fake Redis
module, NGCP::Rtpengine::FakeRedis.
Change-Id: I4c49460d7894de0c5eaf5e5c27f5a6460e4115a7
Firstly, play a long media playback towards A with `stop-on-ingress`
and check that the first RTP packet from B ends it. That means,
the player goes silent, the triggering packet itself is not forwarded
and the following packets from B are forwarded to A.
A later `stop media` from the controller is accepted and doesn't disturb the forwarding.
Then, as a counter-check, the same playback with just `block egress`
keeps playing while B's media stays blocked.
Change-Id: I13427c95bb4358549ad52a91b12b715b036c51be
Document the new `stop-on-ingress` flag in the list of generic flags and
in the `play media` NG command.
Describe what stops the playback, that `block egress` is implied,
which packets are not forwarded, and the kernel forwarding caveat.
Change-Id: If61f7ed0d644737768d559e7e08d116a5d109a48
For the case when the media playback is only meant
to fill a gap between the start of playing media with
the `play_media()` and the time a real remote end-point
starts sending rtp stream.
For example, the ringbacktone is emulated towards the caller
(triggered by previously seen 183 with SDP), while the callee
not actually sending anything (no media coming). In case such
a peer stays silent, keep playing the file, blob or db-id given
file until the peer actually sends at least one rtp packet from
the previously advertised source.
(The same way for the `stop media`, media forwarding resumes).
Important, the flag implies `block-egress`, so the other
parties media stays blocked while the playback is ongoing.
The check runs in the packet path (with the call locked on R),
so it only marks the player and schedules its timer object immediately.
The stop itself then runs from the media player thread
(but this time with the call locked on W).
Unfortunately, those blocked destinations are excluded from kernel forwarding,
because we need to let the daemon see the packets coming. But that's not a big
issue, because as soon as the real end-point exchange comes (final 200OK with SDP),
kernel space processing will be used again.
As the NG caller still sends its own `stop media` afterwards,
a second shutdown of the player becomes common.
Don't flush the send queue then, because without a coder handler, the flush would match
and drop packets that don't belong to the player.
Change-Id: I41c02c1278f380ccd46dd1254f140e306afd4c67
Ports normally come from the port pool, so the caller only learns the
port from rtpengine's reply. A controller that manages the media ports
itself, and has already announced them to the remote side, needs
rtpengine to open exactly those ports.
`local-port` sets the port facing the recipient of the message (the one
in the returned SDP) and `from-local-port` the one facing its sender.
Each takes an integer for the first media section or a list, one per
media section. RTCP takes the next port when not multiplexed. The ports
may be outside of the pool, they are only used when the media's ports
are first opened, and the request fails if they are taken.
`local-port` works in every message that opens ports: offer, answer,
publish, subscribe request and create. `from-local-port` only has a
meaning in offer and answer.
The frequencies list parser becomes a generic integer list parser.
Closes#2216
Change-Id: Ie82338becb8347757c7cc19d90675b927f1edfee
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A controller that allocates the SRTP keys itself needs rtpengine to
encrypt with them, instead of generating keys or copying the ones from
the received SDP.
`SDES-key-SUITE:KEY` (or `key-SUITE:KEY` in the `SDES` list) sets the
master key and salt, base64 encoded as in an a=crypto inline: key, for
that crypto suite on the side facing the recipient of the message. It
works in offers and answers, and also for SRTP to SRTP calls, where
rtpengine then re-encrypts instead of passing the packets through.
Closes#2217
Change-Id: I1a12e4488617cd5daaa49741773a14e330abbfdf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ref #2211
When json_restore_call() fails, it sends "DEL <call-id>" twice: on the
write connection, in its own DB, and on the notification connection, in
whatever DB the last keyspace notification selected. For a foreign call
(a keyspace notification, or the restore of a subscribed keyspace at
startup) that key belongs to the instance that owns the call, which
keeps writing it: a standby that fails to restore one record deletes the
owner's copy, and the call cannot be taken over any more. For an own
call, the DEL on the notification connection can hit a peer's DB.
Only delete an unrestorable record when the call is our own, and only on
the write connection, in our own DB. A foreign record that can never be
restored now stays until its owner rewrites or deletes it, or until it
expires (--redis-expires). Seen in a lab with Redis Sentinel, where a
subscriber's DEL was refused only because its node had just become a
replica.
Tests, in auto-daemon-tests-redis-subscribe.pl: a keyspace notification
for a foreign call whose record cannot be parsed is followed by no
command on either connection, and an own record that cannot be parsed at
startup is deleted once, on the write connection only. Without this
change both get a DEL on both connections.
Change-Id: I5d8a53f3515596cf3fc2c58ea97847476385b876
Ref #2211
redis_set_timeout() receives the timeout in microseconds (the
--redis-cmd-timeout value times 1000) and logged it unconverted, so
--redis-cmd-timeout=1000 was reported as:
Setting timeout for Redis commands to 1000000 milliseconds
Log the value in milliseconds, as the message says. The value passed to
hiredis is unchanged.
No functional change. Seen in the log of every connection made with a
command timeout set; there is no test, as no test reads this line.
Change-Id: Ie4d81e4f8baab1969210a589c98245407b088561
Ref #2211
redis_connect() finds "role:" in the INFO reply and compares what
follows with memcmp(): 11 bytes for "role:master", 19 for
"role:active-replica" and 10 for "role:slave". memcmp() may read all n
bytes, so a reply that ends less than 19 bytes after "role:" (for
example a replica's "role:slave\r\n" with nothing after it) is read
past its end. Real Redis and KeyDB servers send many more lines after
the role, so this does not show in practice, but AddressSanitizer
aborts on it (heap-buffer-overflow in the memcmp interceptor) with a
minimal reply, such as one from a test server or a proxy.
Use strncmp(): hiredis NUL-terminates the reply string, so the compare
stops at its end.
Tested by auto-daemon-tests-redis-errors.pl, added later in this
series, which answers a reconnect's INFO with just "role:slave": an
ASan build aborts there without this change. The change is early in
the series so that it can be taken on its own.
Change-Id: Icdf3d3a23464d861f57406436328d0b166794362
Ref #2211
"set rediscmdtimeout 0" reconnects the Redis connections, but each of
the three calls is guarded by rtpe_redis instead of its own pointer:
- with --redis and no notification connection (no --redis-subscribe and
no subscribed keyspace), redis_reconnect(rtpe_redis_notify) locks a
NULL mutex and rtpengine crashes;
- with --redis-write and no --redis, the write connection is never
reconnected, so its command timeout is never cleared.
Guard each call with its own pointer, as the non-zero branch already
does.
Tested by auto-daemon-tests-redis-errors.pl, added later in this
series ("set rediscmdtimeout 0" over the ng "cli" command with --redis
only): without this change rtpengine crashes and never replies. The
change is first in the series so that it can be taken on its own.
Change-Id: Ia458a6eb0b70a6906ab410fd91087ad5af581f3b
Clear the matching per-destination suppression state when a completed telephone-event is deduplicated at media scope. Otherwise later listeners suppress speech until the timestamp-discrepancy timeout expires.
Add repeated-burst coverage for AMR-WB and Opus listeners in both destination orders, with blocked DTMF, prompt playback, and an event sink.
Closes#2193
Change-Id: Ia73e9324ecc6122b090be36c384b21000ef94609