Closes#2213
With the TLS options, the Sentinel thread still spoke plain TCP to the
Sentinels and for the role check of the master they name, so a Sentinel
set with tls-port and port 0 could not be used.
sentinel_connect() now starts TLS with the same context and options. A
Sentinel's certificate must match its host as given in
--redis-sentinel; a master's must match the address the Sentinels gave,
or --redis-tls-server-name. The handshake happens in the thread, never
under r->lock.
Tested by auto-daemon-tests-redis-tls.pl.
Change-Id: I84daf340c5f464765f234b35255cc76238bc8d13
Closes#2213
There is no way to reach a Redis server that only accepts TLS (managed
services, or tls-port with port 0); asked on the mailing list in 2022:
https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ
With --redis-cafile or --redis-capath, every Redis connection uses TLS,
and --redis-certfile with --redis-keyfile adds a client certificate. As
for MQTT, the options exist only in a build with hiredis_ssl. hiredis
checks only the chain, so the name is checked here, in the
subjectAltName: the host as configured, the IP address, or
--redis-tls-server-name. The handshake is bounded by the command
timeout, or by the connect timeout when there is none.
Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to
Build-Depends).
Change-Id: I74a78be0d40c3755fb1071536deb41443c890817
Closes#2212
Replication is asynchronous, so a master that is replaced takes with it
the writes it acknowledged but had not replicated: a call created then
is missing on the new master, and a call deleted then comes back.
--redis-resync-on-master-change (off by default) keeps the
master_replid of the INFO that redis_connect() already sends. When the
write connection reconnects to another ID, every own call stored at
least once, and the own DELs of the last --redis-resync-delete-window
(at most --redis-resync-delete-max), are queued for the repair thread
of --redis-repair-writes; the calls through the call iterator, one at a
time. No key is scanned. New statistics: redisresyncs, redisresynccalls
and redisresyncdeletes.
Tested by auto-daemon-tests-redis-resync.pl.
Change-Id: I0daac4d2b9ada45a843d27a6de073d8822067a8b
Closes#2212
After a failover the new master refuses writes with NOREPLICAS until it
has a replica. Those writes are counted and logged, but never made
again: a call created then is missing at the next restore, and a call
deleted then comes back.
With --redis-repair-writes (off by default), an own call whose write was
not stored (an error reply, a connection lost before the reply, a
skipped write) is marked in a table keyed by call ID: the call itself,
held, while it is in the call table; its ID and database once it is
deleted, as a deleted call keeps its memory until its last reference
goes. A stored write clears the mark. A marked SET supersedes a marked
DEL of the same ID; if its call was never written, it takes that key
over, so that its own deletion deletes it. A "redis repair" thread
writes the marked calls again, one at a time, through
redis_write_call(), now shared with redis_update_onekey(): a live call
gets its current state, the key of a deleted call a DEL, unless a new
call uses its ID. Signalling writes waiting for r->lock go first, for
100 ms at most. A failed write is retried after a second; at most 100000
calls are kept. New statistic: redisrepairedwrites. A repaired DEL
assumes one writer per database.
Tested by auto-daemon-tests-redis-repair.pl and
auto-daemon-tests-redis-repair-ids.pl.
Change-Id: Ia56a36bba0dd7eed3754694128e2ba6a74827c42
Closes#2212
A subscription made through "sentinel" stayed on the old master after a
failover, and missed the notifications of the new one.
With Sentinel, the notification event loop gets a one-second timer:
when the master changed, it closes the subscription, and
redis_notify_loop() subscribes again on the new master. The
subscription's context also gets a connect callback, since hiredis
frees a context that fails to connect without calling the disconnect
callback. Notifications missed during the switch are not replayed.
Tested by auto-daemon-tests-redis-sentinel-subscribe.pl.
Change-Id: Ia14d48e220bbf98cc48964e76f33ff2a497953e9
Closes#2212
Behind Redis Sentinel, rtpengine stays on the address it was given:
after a failover its writes go to the old master, which refuses them
until it is demoted, then serves them as a replica.
New options --redis-sentinel, --redis-sentinel-master and
--redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in
--redis, --redis-write or --redis-subscribe. A "redis sentinel" thread
asks every Sentinel for the master, takes the highest config epoch
(never an older one than in use), checks the node with ROLE and
publishes it with a generation number. It also runs at once when a
connection meets a replica. redis_check_conn() compares the generation
and reconnects on a change; these connections require the master role.
Without "sentinel", none of this runs.
Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
Change-Id: I7b31b50968fd6a255b9933fbf2c95fc95a8b8408
Closes#2212
--redis-write accepts any role, so after a failover that demotes its
node, rtpengine reconnects to the same address, accepts the replica,
and every later write is refused with READONLY. --redis alone requires
the master at connect, but keeps the connection when its writes are
refused.
The write connection now requires the master role, as --redis alone
does, and a connection that requires it is dropped on a write refused
with READONLY or MASTERDOWN, so that the next operation reconnects and
checks the role. Other refusals keep the connection. The read database
of --redis, when --redis-write is given, still accepts any role.
Tested by auto-daemon-tests-redis-master-only.pl; the READONLY cases
of auto-daemon-tests-redis-errors.pl, which expected the connection to be
kept, now use NOREPLICAS.
Change-Id: Ie164c2c3495c6c8b77553caf9a449e40f8e13699
Ref #2211
A reconnect attempt runs under r->lock, which every ng request that
writes a call waits for (#1865). Against a server that drops packets it
takes the full connect timeout, so signalling stalls for up to that long
at each attempt, for as long as the outage lasts.
Make the attempt without the lock: the thread that reconnects sets
r->reconnecting and owns r->ctx until it clears the flag under the lock.
Other threads wait for it for up to --redis-reconnect-slow, then skip
their operation, as during the pause. A DEL, also the one after a failed
restore, waits for the attempt to end, as it waited for the lock before:
skipped, the key would stay. After the lock was released,
redis_update_onekey() checks again that the call is live.
redis_do_delete() leaves the key to a new call with the same ID, checked
on every DEL, and hands it over to that call if it was never written, so
that its own deletion deletes the key. The CLI and the notification
thread's own connection leave a connection being made alone. A command
on an established connection still holds the lock.
Tested by auto-daemon-tests-redis-reconnect.pl.
Change-Id: Ia98e01c57d4825633e473e3174947daa17afbf13
Ref #2211
On a role mismatch, the reconnect in redis_check_conn() sleeps one
second, asks INFO again and sleeps once more: two seconds under
r->lock, which every Redis update holds before the ng reply (#1865).
After a failover, with --redis on the old master's address, every
update did this, and ng requests timed out.
Make one attempt, with no sleep. After a role failure, or a failure
that took --redis-reconnect-slow (100 ms) or more, skip Redis
operations on that connection for --redis-reconnect-pause (1000 ms),
through restore_tick_us as --redis-allowed-errors does. A quick failure
(connection refused) is retried at once, as before. As with a failed
attempt, a skipped operation is not written; skips are logged once per
pause. The startup connection keeps its eleven attempts.
Tested by auto-daemon-tests-redis-errors.pl.
Change-Id: Ia84288f1c6fe97c4317689fef75d0ff5fc9061f8
Ref #2211
Call updates (SET) and deletions (DEL) are pipelined, and
redis_consume() freed their replies unread: a write refused with
READONLY or NOREPLICAS, as during a Sentinel failover, left no trace in
rtpengine.
Check every reply. An error reply is counted in a new statistic,
rediswriteerrors, and logged at most once a second per connection; the
connection is kept, since a reconnect would get the same answer. A read
error or timeout on the replies drops the connection at once instead of
at the next PING. An encoding failure no longer drops the connection.
Tested by auto-daemon-tests-redis-errors.pl, with a new fake Redis
module, NGCP::Rtpengine::FakeRedis.
Change-Id: I4c49460d7894de0c5eaf5e5c27f5a6460e4115a7
Adds a new config section that allows adjusting behaviour for certain
transcoding scenarios. This only adds the initial support for the config
option.
Change-Id: Ia3d43061adc540fab054e5c99ab804dc1ff53b84
If keyspace notifications are used at all, the respective objects and
threads must be created during startup. This requires at least some
keyspace to be configured. To support usage without any keyspaces
initially (and add them during runtime), add a special case (set
keyspace to -1).
Convert all keyspace variables to signed ints. Ignore negative keyspace
numbers where appropriate. Support Redis endpoint addresses without
database number.
Fixes#1902
Change-Id: I45a3c87bc515f9b14e64ec1ec0906dde271b5f8d