Jan Janak iptelorg GmbH

jan@iptel.org

2005 iptelorg GmbH

Implemented as a module. API independent of transport protocols. Reuse transports available in SER. The possibility to encrypt all communication. The possibility to authenticate clients. Easy integration with existing languages and implementations. Easy and straightforward implementation of management functions in SER modules.

This module implements the XML-RPC transport and encoding interface for ser RPCs. The XML-RPC protocol encodes the name of the method to be called along with its parameter in an XML document which is then conveyed using HTTP (Hyper Text Transfer Protocol) to the server. The server will extract the name of the function to be called along with its parameters from the XML document, execute the function, and encode any data returned by the function into another XML document which is then returned to the client in the body of a 200 OK reply to the HTTP request. XML-RPC is similar to more popular SOAP (Simple Object Access Protocol), which is an XML-based messaging framework used in Web Services developed within the World Wide Web Consortium. Both protocols are using HTTP as the transport protocol for XML documents, but XML-RPC is much simpler and easier to implement than SOAP. Here is an example of single XML-RPC function call to determine current time: currentTime.getCurrentTime ]]> And the response returned by the server: 20011003T08:53:38 ]]> XML-RPC specification spells HTTP as the official transport protocol for XML-RPC documents. SER does not directly support HTTP, it is a SIP server so SIP is the only protocol supported by SER. Because we would like to reuse all transport protocols available in SER, such as TCP and TLS, it would be natural to use modified version of XML-RPC which would run on top of SIP instead of HTTP. XML-RPC documents would be then encoded in the bodies of SIP requests and replies would be sent by the server in the bodies of SIP replies. This way we could reuse all transport protocols (including UDP) and message parsers available in SER. Although this approach seems to be the logical choice, there is one big drawback. No existing XML-RPC implementations support SIP as the transport protocol, and there are many existing implementations available for vast majority of existing languages. See the XML-RPC implementation page for more details. Extending existing implementations with SIP support would not be easy. Because extending available XML-RPC implementation would be too expensive, we could also do it the other way around, keep existing XML-RPC implementations and extend SER to support HTTP. Extending SER with HTTP support is easier than it might seem at a first glance, due to the similarity between SIP requests and HTTP requests. SER already supports TCP, so existing HTTP implementations can send HTTP requests to it. HTTP requests are missing certain mandatory SIP header fields, such as Via, From, and CSeq. The contents of the header fields is mainly used for transaction matching. A SIP server could perform two basic operations when processing an HTTP request: Terminate the request, execute the function and send a reply back. Forward the request to another SIP server. Nothing special is needed on the SIP server terminating the request, except that it has to know where it should send the reply. Parsing of HTTP header field bodies would fail because we do not have parsers for them in SER, but that does not matter anyway because all the information is encoded in the body of the request. HTTP requests contain no Via header fields. Via header fields are used by SIP implementations to determine the destination (IP, transport protocol, and port number) for replies. When processing HTTP requests the SIP server needs to create a fake Via header field based on the source IP address and port number of the TCP connection. The SIP server will use this information when sending a reply back. Forwarding of HTTP requests by SIP proxies is a little bit more complicated and there are several limitations. First of all, we can only use stateless forwarding, no transactional forwarding, because HTTP requests do not contain all the header fields needed for transaction matching. Any attempt to call t_relay on an HTTP requests would fail. HTTP requests always use TCP and thus we could use stateless forwarding on the SIP server, provided that the request will be also forwarded over TCP. Stateless forwarding does not require the mandatory header fields (which are missing here) and it would work. In addition to that, the SIP server would also append fake Via header field to the request and change the contents of the Request-URI. The Request-URI of HTTP requests sent by XML-RPC implementations typically contain something like "/RPC2" and the first SIP server processing the request should rewrite the value with a valid SIP URI. shows a scenario which involves two SIP servers, one performs HTTP request "normalization" and forwarding, and the other terminates the request, executes corresponding function, and generates a reply. Example RPC Scenario Example RPC Scenario Step 1. An HTTP user agent sends an ordinary HTTP request to a SIP server. The user agent can either establish a connection directly to port 5060 or the SIP server can be configured to listen on port 80. The request contains standard HTTP headers and an XML-RPC document in the body: ]]>usrloc.statistics ]]> This particular request calls method "statistics" from from usrloc module of SER. The method has no parameters. The outbound SIP server receives the HTTP request and performs a set of actions called "SIP-normalization". This includes creation of fake Via header field based on the source IP and port of the TCP connection, looking up of the target SIP server that should terminate and process the request, and rewriting of the Request-URI with the SIP URI of the target SIP server. Modified HTTP request will be then forwarded statelessly to the target SIP server. POST sip:proxy01.sip-server.net HTTP/1.0 Via: SIP/2.0/TCP 127.0.0.1:3571 Host: localhost:5060 User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com) Content-Type: text/xml Content-Length: 111 usrloc.statistics ]]> Step 2. "normalized" HTTP request is statelessly forwarded to the target SIP server over TCP. Step 3. The target SIP server receives the HTTP request and executes function called dispatch_rpc from xmlrpc SER module. This function will parse the XML-RPC document in the body of the request and lookup the function to be called among all RPC functions exported by the SER core and modules. Function dispatch_rpc will be called from the configuration file just like any other function: if (method == "POST" || method == "GET") { dispatch_rpc(); break; }; This particular configuration snippet executes the function whenever SER receives GET or POST requests. These two method names indicate HTTP. Step 4. The function dispatch_rpc scans through the list of all exported RPC functions searching for the statistics function of the usrloc module. The SER RPC Module API describes in detail how modules export RPC functions. Step 5. As the RPC function from usrloc module is running and gathering statistics, it calls functions of RPC interface to prepare the result for the caller. Step 6. Once the RPC function finishes, xmlrpc module will build the XML-RPC document from the data received from usrloc module and generate a reply which will be sent to the caller. Steps 7. and 8. HTTP reply is sent back to the caller and the remote procedure call finishes. domain aliases users 0 expired 0 domain location users 0 expired 0 ]]> The scenario described on involves two SIP servers. This is just to demonstrate that in setups containing more SIP servers it is possible to forward HTTP requests from one SIP server to another and use standard SIP routing mechanisms to decide which SIP server should process the request. There is no need to have multiple SIP servers in simple setups, because one SIP server can both add fake Via header field and process the RPC at the same time. Modified configuration file snipped could then look like this: if (method == "POST" || method == "GET") { dispatch_rpc(); # Process the request break; };

The purpose of the functions of this module is to convert XML-RPC document carried in the body of HTTP requests into data returned by the RPC interface and back. The module also contains functions necessary to "normalize" HTTP requests. The module uses xmlrpc-c library to perform XML-RPC related functions. The module always returns 200 OK HTTP reply, it will never return any other HTTP reply. Failures are expressed in XML-RPC documents in the body of the reply. There is basic method introspection support in the module. Currently the module can list all functions exported by the server and for each function it can return the documentation string describing the function.

Requests processed by the module are standard XML-RPC requests encoded in bodies of HTTP requests. ]]>system.listMethods]]> ]]> The name of the method to be called in this example is "listMethods". This is one of the introspection methods. SER will call dispatch_rpc function of xmlrpc module to handle the request. The function will parse the XML-RPC document, lookup listMethods function in the list of all export RPC functions, prepare the context for the function and execute it.

The module will always generate 200 OK. Other response codes and classes are reserved for SER. The status code of the XML-RPC reply, response code, and additional data will be encoded in the body of the reply. Failure replies do not contain any data, just the response code and reason phrase: ]]> faultCode 501 faultString Method Not Implemented ]]> ]]> This particular reply indicates that there is no such RPC method available on the server. Success replies always contain at least one return value. In our case the simplest success replies contain single boolean with value 1: ]]>1]]> ]]> This is exactly how the reply looks like when an RPC function does not add any data to the reply set. If an RPC function adds just a single item (it calls add once with just one character in the formatting string) then the data will be converted to XML-RPC representation according to the rules described in SER RPC Type Conversion and the reply will contain just the single value: ]]>Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))]]> ]]> If an RPC function adds more than one data items to the result set then the module will return an array containing all the data items: ]]> ./ser -f ser.cfg ]]> ]]> This is probably the most common scenario.

The data types of the RPC API are converted to the data types of XML-RPC and vice versa. shows for each RPC API data type corresponding XML-RPC data type. RPC API XML-RPC RPC Example XML-RPC Example Integer ]]> rpc->add("d", 42) 42 ]]> Float ]]> rpc->add("f", -12.214) -12.214 ]]> String ]]> rpc->add("s","Don't panic") Don't panic ]]> Struct ]]> rpc->struct_add(handle,"sd","param1",42,"param2",-12.214) param1 42 param2 -12.214 ]]>

SER xmlrpc modules does not implement all data types allowed in XML-RPC. As well it does not implement arrays and nested structures. This simplification is a feature, not bug. In our case the XML-RPC interface will be used mainly for management purposes and we do not need all the bells and whistles of XML-RPC. Parsing and interpreting nested structures is complex and we try to avoid it.

Due to a bug in Python xmlrpclib there is an interoperability problem with basic clients using it: by default an xmlrpclib client expects the server to immediately close the connection after answering and if the server does not close the connections the xmlrpclib client will wait forever. There are 2 ways to work around this problem: write a "fixed" Transport class and initialize xmlpclib using it (recommended) or make ser close the tcp connection after each request. The examples/xmlrpc_test.py provides a very simple example of using xmlrpclib with a Transport class that works. For the second solution (force closing tcp connections after answering) the XMLRPC route should have a set_reply_close() command before dispatch_rpc(). set_reply_no_connect() is also recommended (avoid trying to open tcp connection to xmlrpc clients that closed it). Alternatively ending the XMLRPC route with return -1, exit -1 or drop -1 can also be used, but note that this will not work for async rpcs (it will close the connection immeidately and not on the async response). Another common problem is CRLF handling. According to the xml spec CR ('\r') must be escaped (to 
) or they will be "normalized" when parsing the xml document. However some xmlrpc clients do not follow this rule (e.g. clients based on the python or php xmlrpclib) and send CRLF unescaped. A possible workaround is to enable automatic LFLF to CRLF conversion (using the double_lf_to_crlf modules parameter) and replace CRLF with LFLF in the client queries.

examples/xmlrpc_test.pl (basic perl application that builds and sends an XMLRPC request from its commandline parameters). examples/xmlrpc_test.py (basic python application that builds and sends an XMLRPC request from its commandline parameters). ser_ctl (complex python application that uses the XML-RPC interface implemented by the xmlrpc module). serweb (php application that can use the XML-RPC interface to call ser functions).