Closes#2213
With the TLS options, the Sentinel thread still spoke plain TCP to the
Sentinels and for the role check of the master they name, so a Sentinel
set with tls-port and port 0 could not be used.
sentinel_connect() now starts TLS with the same context and options. A
Sentinel's certificate must match its host as given in
--redis-sentinel; a master's must match the address the Sentinels gave,
or --redis-tls-server-name. The handshake happens in the thread, never
under r->lock.
Tested by auto-daemon-tests-redis-tls.pl.
Change-Id: I84daf340c5f464765f234b35255cc76238bc8d13
Closes#2213
There is no way to reach a Redis server that only accepts TLS (managed
services, or tls-port with port 0); asked on the mailing list in 2022:
https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ
With --redis-cafile or --redis-capath, every Redis connection uses TLS,
and --redis-certfile with --redis-keyfile adds a client certificate. As
for MQTT, the options exist only in a build with hiredis_ssl. hiredis
checks only the chain, so the name is checked here, in the
subjectAltName: the host as configured, the IP address, or
--redis-tls-server-name. The handshake is bounded by the command
timeout, or by the connect timeout when there is none.
Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to
Build-Depends).
Change-Id: I74a78be0d40c3755fb1071536deb41443c890817
Closes#2212
After a failover the new master refuses writes with NOREPLICAS until it
has a replica. Those writes are counted and logged, but never made
again: a call created then is missing at the next restore, and a call
deleted then comes back.
With --redis-repair-writes (off by default), an own call whose write was
not stored (an error reply, a connection lost before the reply, a
skipped write) is marked in a table keyed by call ID: the call itself,
held, while it is in the call table; its ID and database once it is
deleted, as a deleted call keeps its memory until its last reference
goes. A stored write clears the mark. A marked SET supersedes a marked
DEL of the same ID; if its call was never written, it takes that key
over, so that its own deletion deletes it. A "redis repair" thread
writes the marked calls again, one at a time, through
redis_write_call(), now shared with redis_update_onekey(): a live call
gets its current state, the key of a deleted call a DEL, unless a new
call uses its ID. Signalling writes waiting for r->lock go first, for
100 ms at most. A failed write is retried after a second; at most 100000
calls are kept. New statistic: redisrepairedwrites. A repaired DEL
assumes one writer per database.
Tested by auto-daemon-tests-redis-repair.pl and
auto-daemon-tests-redis-repair-ids.pl.
Change-Id: Ia56a36bba0dd7eed3754694128e2ba6a74827c42
Closes#2212
Behind Redis Sentinel, rtpengine stays on the address it was given:
after a failover its writes go to the old master, which refuses them
until it is demoted, then serves them as a replica.
New options --redis-sentinel, --redis-sentinel-master and
--redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in
--redis, --redis-write or --redis-subscribe. A "redis sentinel" thread
asks every Sentinel for the master, takes the highest config epoch
(never an older one than in use), checks the node with ROLE and
publishes it with a generation number. It also runs at once when a
connection meets a replica. redis_check_conn() compares the generation
and reconnects on a change; these connections require the master role.
Without "sentinel", none of this runs.
Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
Change-Id: I7b31b50968fd6a255b9933fbf2c95fc95a8b8408
Ref #2211
redis, redis-json and redis-subscribe compared raw bytes on their own
sockets, and redis-srtp-index and rollback-redis each carried the same
forked server. They now use NGCP::Rtpengine::FakeRedis, and the forked
server moves into it as redis_serve(). The checks, their names and
their counts are unchanged.
Change-Id: Ieb281e1fe2a772b31a2f928104c741e9c5607297
Ref #2211
The timer that rewrites never-stored SRTP calls can hold a call while it
waits for r->lock. If the call is deleted meanwhile, the deletion,
deciding without the lock that the call was never stored, deletes
nothing, and the timer then writes it: the call is back at the next
restore. Any update racing with a deletion can do the same.
Under r->lock, redis_update_onekey() writes a call only while it is
still in the call table, and redis_do_delete() decides "never stored",
so that a write already past that check is followed by the DEL.
Tested by auto-daemon-tests-redis-destroyed.pl.
Change-Id: I7329a721248b2c8ce64c9e778d2761951db676f9
Ref #2211
Call updates (SET) and deletions (DEL) are pipelined, and
redis_consume() freed their replies unread: a write refused with
READONLY or NOREPLICAS, as during a Sentinel failover, left no trace in
rtpengine.
Check every reply. An error reply is counted in a new statistic,
rediswriteerrors, and logged at most once a second per connection; the
connection is kept, since a reconnect would get the same answer. A read
error or timeout on the replies drops the connection at once instead of
at the next PING. An encoding failure no longer drops the connection.
Tested by auto-daemon-tests-redis-errors.pl, with a new fake Redis
module, NGCP::Rtpengine::FakeRedis.
Change-Id: I4c49460d7894de0c5eaf5e5c27f5a6460e4115a7
Previously, the ports used for a monologue subscription would leak upon re-subscribe.
``__assign_stream_fds`` cleared the stream fd queue without decrementing the ref count. This
prevented ``monologue_destroy`` from releasing the ports on unsubscribe.
Closes#2125
Change-Id: Ib86290d6a35e2388e34098f2847f1e34b3db2c5d
Don't depend on ->socket being set. Instead run the actual ping check
and use that as condition to see whether the daemon is ready or not.
Change-Id: I6d39ca6cb9b3a61a94194fe827768dff28ca0508
We are setting up a global die handler, so we cannot use «local» or the
statement would not have the intended effect.
Warned-by: perlcritic
Fixes: Variables::RequireLocalizedPunctuationVars
Change-Id: Ia0a433b58b4960c3394b25b1f144c0158b30ad1d
A dummy function to ignore the fact
whether RTP packet(s) received or not.
Useful in a combination with rcv_no().
Change-Id: Iaeea881e0a3590808dcc681495095178cb470b94
Support the recvfrom() function. This requires also returning to address
(not just the port) from the tester's rcv() function to be useful, which
in turn requires updating a bunch of tests.
Change-Id: I3c0bcab9b744f2c234662399199abc538876c4fa
New helper function to create a new call without closing previously
opened ports.
Also make it possible to override the automatically inserted call ID in
methods.
Change-Id: Ie9133f78bc7bd5ff4e191966ef26187da02e40ff
Make it possible to run custom code between sending out a command and
expecting and receiving the answer.
Change-Id: I11359b144a27c58de8ec2b136bd6715c7d1b9cce
Don't wait forever for a response from the daemon. Prevents tests
getting stuck in case of broken code.
Change-Id: I83dde4c2d626928fdfb0486e7fb4780e239c83e9
The order between receiving the STUN success and the triggered check is
not guaranteed, therefore resolve possible race condition by expecting
the two packets in either order.
Change-Id: Ibef9907cd4116bc5f3b7d17d936007c8efcabd3b
To be able to ease diagnosing problems in the functional tests, we
should not remove the log files storing stdout and stderr, otherwise
debugging issue from a failed or crashing daemon turn to be rather
brittle.
Change-Id: Id285049da7ad030ef7b041f7e2dde86e97ef6487