Distributed Message Queue Module

Marius Ovidiu Bucur

Charles Chance

   Sipcentric Ltd.

Olle E. Johansson

   Edvina AB

Edited by

Marius Ovidiu Bucur

Charles Chance

   Copyright © 2011 Marius Bucur

   Copyright © 2013 Charles Chance, Sipcentric Ltd.

   Copyright © 2015 Olle E. Johansson, Edvina AB
     __________________________________________________________________

   Table of Contents

   1. Admin Guide

        1. Overview

              1.1. KDMQ Request

        2. Dependencies

              2.1. Kamailio Modules
              2.2. External Libraries or Applications

        3. Parameters

              3.1. server_address(str)
              3.2. server_socket(str)
              3.3. notification_address(str)
              3.4. notification_channel(str)
              3.5. multi_notify(int)
              3.6. num_workers(int)
              3.7. worker_usleep(int)
              3.8. ping_interval(int)
              3.9. remove_inactive(int)
              3.10. fail_count_enabled(int)
              3.11. fail_count_threshold_not_active(int)
              3.12. fail_count_threshold_disabled(int)

        4. Functions

              4.1. dmq_handle_message([continue])
              4.2. dmq_process_message([continue])
              4.3. dmq_send_message(channel, node, body, content_type)
              4.4. dmq_bcast_message(channel, body, content_type)
              4.5. dmq_t_replicate([skip_loop_test])
              4.6. dmq_is_from_node()

        5. RPC Commands

              5.1. dmq.list_nodes
              5.2. dmq.remove

   2. Developer Guide

        1. dmq_load_api(dmq_api_t* api)
        2. register_dmq_peer(dmq_peer_t* peer)
        3. bcast_message(dmq_peer_t* peer, str* body, dmq_node_t* except,
                dmq_resp_cback_t* resp_cback, int max_forwards, str*
                content_type)

        4. send_message(dmq_peer_t* peer, str* body, dmq_node_t* node,
                dmq_resp_cback_t* resp_cback, int max_forwards, str*
                content_type)

   List of Examples

   1.1. KDMQ Request Example
   1.2. Set server_address parameter
   1.3. Set server_socket parameter
   1.4. Set notification_address parameter
   1.5. Set notification_channel parameter
   1.6. Set multi_notify parameter
   1.7. Set num_workers parameter
   1.8. Set worker_usleep parameter
   1.9. Set ping_interval parameter
   1.10. Set remove_inactive parameter
   1.11. Set fail_count_enabled parameter
   1.12. Set fail_count_threshold_not_active parameter
   1.13. Set fail_count_threshold_disabled parameter
   1.14. dmq_handle_message usage
   1.15. dmq_process_message usage
   1.16. dmq_send_message usage
   1.17. dmq_bcast_message usage
   1.18. dmq_t_replicate usage
   1.19. dmq_is_from_node usage
   1.20. dmq.list_nodes usage
   1.21. dmq.remove usage
   2.1. dmq_api_t structure
   2.2. register_dmq_peer usage
   2.3. bcast_message usage
   2.4. send_message usage

Chapter 1. Admin Guide

   Table of Contents

   1. Overview

        1.1. KDMQ Request

   2. Dependencies

        2.1. Kamailio Modules
        2.2. External Libraries or Applications

   3. Parameters

        3.1. server_address(str)
        3.2. server_socket(str)
        3.3. notification_address(str)
        3.4. notification_channel(str)
        3.5. multi_notify(int)
        3.6. num_workers(int)
        3.7. worker_usleep(int)
        3.8. ping_interval(int)
        3.9. remove_inactive(int)
        3.10. fail_count_enabled(int)
        3.11. fail_count_threshold_not_active(int)
        3.12. fail_count_threshold_disabled(int)

   4. Functions

        4.1. dmq_handle_message([continue])
        4.2. dmq_process_message([continue])
        4.3. dmq_send_message(channel, node, body, content_type)
        4.4. dmq_bcast_message(channel, body, content_type)
        4.5. dmq_t_replicate([skip_loop_test])
        4.6. dmq_is_from_node()

   5. RPC Commands

        5.1. dmq.list_nodes
        5.2. dmq.remove

1. Overview

   1.1. KDMQ Request

   The DMQ module implements a distributed message queue on top of
   Kamailio in order to facilitate data propagation and replication
   between multiple instances, referred as "nodes" (or "peers").

   The DMQ data flow between nodes is grouped in a logical entity referred
   as "channel" (or "bus"), many flows (channels) can be active at the
   same time.

   The communication between the nodes is done using SIP messages, after
   all Kamailio's best ability is routing SIP traffic, therefore reusing
   the same protocol empowers DMQ cluster with flexible routing policies,
   advanced authorization and security policies, a.s.o. DMQ sends SIP
   requests using the KDMQ request method (a custom method type specific
   to Kamailio, compliant with SIP specifications RFC3261). Data can be
   sent to all the other active nodes (aka "broadcast") or to a specific
   single node (aka "unicast").

   The nodes can utilize the DMQ channels to pass messages between them to
   distribute data for various services (e.g., cached items, location
   records, active calls, ...). This grouping of messages on channels (per
   service) is similar to the topic concept in a typical pub/sub system.
   The channel name appears in the R-URI username part of KDMQ requests.

   The DMQ cluster transparently deals with node discovery, availability,
   consistency, retransmissions, etc. A new node can join the DMQ cluster
   by sending an availability notification to an existing node in the
   clusters. Nodes that are not responding to KDMQ reguests are removed
   the from local list of active nodes, no longer being considered to be
   part of the KDMQ cluster.

   IMPORTANT: DMQ must be used only between Kamailio instances having same
   major version. Internal structures can be incompatible between
   different major versions and can lead to crashes or unexpected
   behaviour.

   Possible DMQ node statuses:
     * DMQ_NODE_ACTIVE "active" = node is in dmq node list and responded
       with response code 200 OK, to KDMQ message. At startup, all nodes
       are considered active, regardless of their actual state.
     * DMQ_NODE_NOT_ACTIVE "not_active" = node is in dmq node list and did
       not respond to KDMQ message (408 timeout).
     * DMQ_NODE_DISABLED "disabled" = node will be deleted from dmq node
       list.

1.1. KDMQ Request

   KDMQ messages can have different format and content for R-URI username,
   headers and body, being specific to each component that leverages DMQ
   to replicate data, such as htable, dialog or usrloc modules.

   Next request is generated by DMQ module for peers availability
   notifications.

   Example 1.1. KDMQ Request Example
...

KDMQ sip:notification_peer@192.168.40.15:5090 SIP/2.0
Via: SIP/2.0/UDP 192.168.40.15;branch=z9hG4bK55e5.423d95110000
To: <sip:notification_peer@192.168.40.15:5090>
From: <sip:notification_peer@192.168.40.15:5060>;tag=2cdb7a33a7f21abb98fd3a44968
e3ffd-5b01
CSeq: 10 KDMQ
Call-ID: 1fe138e07b5d0a7a-50419@192.168.40.15
Content-Length: 116
User-Agent: kamailio (4.3.0 (x86_64/linneaus))
Max-Forwards: 1
Content-Type: text/plain

sip:192.168.40.16:5060;status=active
sip:192.168.40.15:5060;status=disabled
sip:192.168.40.17:5060;status=not_active

...

2. Dependencies

   2.1. Kamailio Modules
   2.2. External Libraries or Applications

2.1. Kamailio Modules

   The following modules must be loaded before this module:
     * sl.
     * tm.

2.2. External Libraries or Applications

     * The DMQ module itself has no external dependencies. However, each
       peer may need to use its own (de)serialization mechanism, like JSON
       (via jannson module), XML (via xmlops) or string operations with
       transformations. .

3. Parameters

   3.1. server_address(str)
   3.2. server_socket(str)
   3.3. notification_address(str)
   3.4. notification_channel(str)
   3.5. multi_notify(int)
   3.6. num_workers(int)
   3.7. worker_usleep(int)
   3.8. ping_interval(int)
   3.9. remove_inactive(int)
   3.10. fail_count_enabled(int)
   3.11. fail_count_threshold_not_active(int)
   3.12. fail_count_threshold_disabled(int)

3.1. server_address(str)

   The local server address. This is the address with which the local DMQ
   node joins the servers group to send/receive messages.

   Note: if server_socket is not set, then this address has to match a
   local socket, which will be used to send/receive messages.

   Default value is “NULL”.

   Example 1.2. Set server_address parameter
...
modparam("dmq", "server_address", "sip:10.0.0.20:5060")
...
modparam("dmq", "server_address", "sip:10.0.0.20:5061;transport=tls")
...

3.2. server_socket(str)

   The local listen address. This is the interface over which the DMQ
   engine will send/receive messages.

   Note: if this parameter is not set, the socket to send/receive messages
   is built from server_address parameter.

   Default value is “NULL”.

   Example 1.3. Set server_socket parameter
...
modparam("dmq", "server_socket", "udp:10.0.0.20:5060")
...

3.3. notification_address(str)

   The address of another DMQ node from which the local node should
   retrieve initial information about all other nodes. This parameter can
   be specified multiple times in the configuration, to configure multiple
   notification servers. If you configure multiple notification servers,
   the multi_notify parameter needs to be disabled.

   Default value is “NULL”.

   Example 1.4. Set notification_address parameter
...
modparam("dmq", "notification_address", "sip:10.0.0.21:5060")
...
modparam("dmq", "notification_address", "sip:10.0.0.21:5061;transport=tls")
...

3.4. notification_channel(str)

   The name of the channel for notifications about peers availability.

   Default value is “notification_peer”.

   Example 1.5. Set notification_channel parameter
...
modparam("dmq", "notification_channel", "peers")
...

3.5. multi_notify(int)

   Enables the ability to resolve multiple IPv4/IPv6 addresses for a
   single notification address. Please note that this mode is not
   supported if you specify multiple notification address parameter.

   A value of zero resolves to the first IP address found. A non-zero
   value resolves to all IP addresses associated with the host. This
   includes addresses from DNS SRV records, A and AAAA records.

   Default value is “0”.

   Example 1.6. Set multi_notify parameter
...
modparam("dmq", "multi_notify", 1)
...

3.6. num_workers(int)

   The number of worker threads for sending/receiving messages.

   Default value is “2”.

   Example 1.7. Set num_workers parameter
...
modparam("dmq", "num_workers", 4)
...

3.7. worker_usleep(int)

   The default locking/synchronisation mechanism between producer/consumer
   threads is the optimum for most environments. On some systems (e.g.
   FreeBSD) it can cause high CPU load and in such cases, it can be useful
   to disable locking and switch to polling for tasks at set intervals
   instead - putting the thread to sleep in-between and taking it out of
   process during that time.

   A value >0 will disable the default locking and set the polling
   interval (in microseconds), which can be tuned to suit the specific
   environment.

   Default value is 0 (recommended for most systems).

   Example 1.8. Set worker_usleep parameter
...
modparam("dmq", "worker_usleep", 1000)
...

3.8. ping_interval(int)

   The number of seconds between node pings (for checking status of other
   nodes).

   Minimum value is “60” (default).

   Example 1.9. Set ping_interval parameter
...
modparam("dmq", "ping_interval", 90)
...

3.9. remove_inactive(int)

   A value of zero disable removing dmq nodes. Node status will be changed
   to not_active. A non-zero value (default is 1) enable removing nodes.
   For node that is defined as notification address status will be changed
   to not_active. Otherwise it will be marked as disabled and then will be
   removed.

   Default value is “1”.

   Example 1.10. Set remove_inactive parameter
...
modparam("dmq", "remove_inactive", 0)
...

3.10. fail_count_enabled(int)

   Enable per node counting of failed responses.

   Default value is “0” (disabled).

   Example 1.11. Set fail_count_enabled parameter
...
modparam("dmq", "fail_count_enabled", 1)
...

3.11. fail_count_threshold_not_active(int)

   DMQ node status change from "active" to "not_active" if 408 fail counts
   are greater than this threshold value: This threshold can be reached
   both via internal DMQ notification ping mechanism, and other modules
   that use dmq to broadcast messages:
     * Fail counts reset every time a node responds with response code 200
       OK, on response callback of notification ping.
     * Fail counts increase every time a node responds with response code
       408, on response callback of notification ping.
     * Fail counts increase every time a node responds with response code
       408, on response callback of other modules that use dmq, only if
       node is in "active" state.

   Default value is “0”. Move node to "not_active" state on first fail.

   Example 1.12. Set fail_count_threshold_not_active parameter
...
modparam("dmq", "fail_count_threshold_not_active", 100)
...

3.12. fail_count_threshold_disabled(int)

   DMQ node status change from "not_active" to "disabled" if 408 fail
   counts are greater than this threshold value. This threshold can be
   reached only via internal DMQ notification ping mechanism:
     * Fail counts reset every time a node responds with response code 200
       OK, on response callback of notification ping.
     * Fail counts increase every time a node responds with response code
       408, on response callback of notification ping.

   Default value is “1”. Move node to "disabled" state on second fail.

   Example 1.13. Set fail_count_threshold_disabled parameter
...
modparam("dmq", "fail_count_threshold_disabled", 105)
...

4. Functions

   4.1. dmq_handle_message([continue])
   4.2. dmq_process_message([continue])
   4.3. dmq_send_message(channel, node, body, content_type)
   4.4. dmq_bcast_message(channel, body, content_type)
   4.5. dmq_t_replicate([skip_loop_test])
   4.6. dmq_is_from_node()

4.1.  dmq_handle_message([continue])

   Handles a DMQ message by passing it to the appropriate local peer
   (module). The peer is identified by the user part of the To header.

   Meaning of parameters:
     * continue - by default, dmq_handle_message() will end execution of
       routing script. If this optional parameter is set to "1",
       dmq_handle_message() will continue executing the routing script
       after it's been called.

   This function can be used from REQUEST_ROUTE.

   Example 1.14. dmq_handle_message usage
...
    if(is_method("KDMQ")) {
        dmq_handle_message();
    }
...

4.2.  dmq_process_message([continue])

   Similar to dmq_handle_message, but the processing is happening
   immediately, no longer cloning the request in shared memory to be
   passed to a DMQ worker process.

   Meaning of parameters:
     * continue - by default, dmq_process_message() will end execution of
       routing script by returning 0. If this optional parameter is set to
       "1", dmq_process_message() will continue executing the routing
       script after it is been executed, returning 1.

   This function can be used from REQUEST_ROUTE.

   Example 1.15. dmq_process_message usage
...
    if(is_method("KDMQ")) {
        dmq_process_message();
    }
...

4.3.  dmq_send_message(channel, node, body, content_type)

   Sends a DMQ message directly from config file to a single node.

   Meaning of parameters:
     * channel - name of the channel that should handle the message.
     * node - the SIP address of the node to which the message should be
       sent.
     * body - the message body.
     * content_type - the MIME type of the message body.

   This function can be used from any route.

   Example 1.16. dmq_send_message usage
...
        dmq_send_message("channel1", "sip:10.0.0.21:5060", "Message body...", "t
ext/plain");
...

4.4.  dmq_bcast_message(channel, body, content_type)

   Broadcasts a DMQ message from config file to all active nodes (except
   self) on the specific channel.

   Meaning of parameters:
     * channel - name of the channel that should handle the message.
     * body - the message body.
     * content_type - the MIME type of the message body.

   This function can be used from any route.

   Example 1.17. dmq_bcast_message usage
...
    dmq_bcast_message("channel1", "Message body...", "text/plain");
...

4.5.  dmq_t_replicate([skip_loop_test])

   Replicates the current SIP message to all active nodes (except self).
   Useful for replicating REGISTER, PUBLISH etc. in a clustered
   environment.

   Meaning of parameters:
     * skip_loop_test - by default, DMQ checks the source IP of the
       message prior to replication, to ensure it has not been sent by
       another DMQ node (to avoid infinite loops). If this optional
       parameter is set to "1", the loop test is not performed. This makes
       sense, from a performance perspective, if you have already
       performed the necessary checks in the config script (see
       dmq_is_from_node()).

   This function can be used from REQUEST_ROUTE only.

   Example 1.18. dmq_t_replicate usage
...
    dmq_t_replicate();
...

4.6.  dmq_is_from_node()

   Checks whether the current message has been sent by another active DMQ
   node in the cluster.

   This function can be used from REQUEST_ROUTE only.

   Example 1.19. dmq_is_from_node usage
...
        # basic example for REGISTER replication
    if(is_method("REGISTER")) {
        if (dmq_is_from_node()) {
            # coming from a DMQ node - already authenticated there
            # now just save contact, etc...
        } else {
            # coming from end point - authenticate, save contact, etc...
            dmq_t_replicate("1"); # source address checked, skip the loop test
        }
    }
...

5. RPC Commands

   5.1. dmq.list_nodes
   5.2. dmq.remove

5.1. dmq.list_nodes

   List the DMQ nodes. It has no parameters.

   Example 1.20. dmq.list_nodes usage
...
kamctl rpc dmq.list_nodes
...

5.2. dmq.remove

   Remove a DMQ node by address.

   Parameters:
     * address - the SIP address of the DMQ node to remove.

   Example 1.21. dmq.remove usage
...
kamctl rpc dmq.remove sip:1.2.3.4:5060
...

Chapter 2. Developer Guide

   Table of Contents

   1. dmq_load_api(dmq_api_t* api)
   2. register_dmq_peer(dmq_peer_t* peer)
   3. bcast_message(dmq_peer_t* peer, str* body, dmq_node_t* except,
          dmq_resp_cback_t* resp_cback, int max_forwards, str*
          content_type)

   4. send_message(dmq_peer_t* peer, str* body, dmq_node_t* node,
          dmq_resp_cback_t* resp_cback, int max_forwards, str*
          content_type)

   The module provides the following functions that can be used in other
   Kamailio modules.

1.  dmq_load_api(dmq_api_t* api)

   This function binds the DMQ module and fills the structure with the
   exported functions below.

   Example 2.1. dmq_api_t structure
...
typedef struct dmq_api {
        register_dmq_peer_t register_dmq_peer;
        bcast_message_t bcast_message;
        send_message_t send_message;
} dmq_api_t;
...

2.  register_dmq_peer(dmq_peer_t* peer)

   Registers an entity as a DMQ peer which permits receiving/sending
   messages between nodes which support the same peer.

   Example 2.2. register_dmq_peer usage
...
        Example to follow.
...

3.  bcast_message(dmq_peer_t* peer, str* body, dmq_node_t* except,
dmq_resp_cback_t* resp_cback, int max_forwards, str* content_type)

   Broadcast a DMQ message to all nodes in the DMQ bus excluding self,
   inactive nodes and "except" if specified.

   Example 2.3. bcast_message usage
...
        Example to follow.
...

4.  send_message(dmq_peer_t* peer, str* body, dmq_node_t* node,
dmq_resp_cback_t* resp_cback, int max_forwards, str* content_type)

   Send a DMQ message to a single node.

   Example 2.4. send_message usage
...
        Example to follow.
...
