mirror of https://github.com/sipwise/kamailio.git
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
120 lines
4.2 KiB
120 lines
4.2 KiB
Kamailio TCP Tunning/monitoring for lots of open connections
|
|
==============================================================
|
|
|
|
0. Introduction
|
|
----------------
|
|
|
|
This document describes very briefly various settings that should improve
|
|
Kamailio TCP performance for sites handling a lot of TCP traffic (> 1000 open
|
|
connections or very high connection/disconnection rates).
|
|
|
|
For now it deals only with Linux specific optimizations.
|
|
|
|
|
|
1. Useful Linux kernel settings
|
|
-------------------------------
|
|
|
|
1.1 Connection rate/pending connections: by default the connection rate is
|
|
too small
|
|
|
|
net.core.somaxconn - limit of the listen() backlog, default 128
|
|
net.ipv4.tcp_max_syn_backlog - default 1024 or 128
|
|
net.ipv4.tcp_timestamps - default on., should be on (along with
|
|
tcp_tw_recycle and timestamp supporting
|
|
peers allows for fast connections rates)
|
|
|
|
1.2 Connection in close_wait
|
|
|
|
Connection should stay as little as possible
|
|
in close_wait to quickly free the fd/resources for new connections attempts
|
|
WARNING: this could break normal TCP use, use it only if you know what you are
|
|
doing
|
|
|
|
net.ipv4.tcp_max_tw_buckets - maximum number of timewait sockets
|
|
(the default seems to be ok)
|
|
net.ipv4.tcp_tw_recycle - enables fast time wait sockets recycling (default
|
|
off), should be enabled if you have lots of short
|
|
lived connections
|
|
WARNING: see the above warning
|
|
net.ipv4.tcp_tw_reuse - allows reusing of time-wait sockets (default off)
|
|
WARNING: see above
|
|
|
|
net.ipv4.tcp_syncookies - default off, in this case it's probably better to
|
|
keep it off
|
|
|
|
1.3 Port range
|
|
|
|
net.ipv4.ip_local_port_range - should be increased (e.g. 4096-65534)
|
|
|
|
1.4 Open file descriptors
|
|
|
|
fs.file-max - maximum number of fds that will be allocated
|
|
(you probably need to increase it, default
|
|
depends on installed memory)
|
|
|
|
1.5 Buffers size
|
|
|
|
When dealing with large data, check the values for network sockets buffers
|
|
to be sufficient, among them:
|
|
|
|
net.core.rmem_max
|
|
net.core.wmem_max
|
|
net.core.rmem_default
|
|
net.core.wmem_default
|
|
net.ipv4.tcp_mem
|
|
net.ipv4.tcp_rmem
|
|
net.ipv4.tcp_wmem
|
|
|
|
Note: sysctl -w net.ipv4.route.flush=1 - enusre that immediatly subsequent
|
|
connections use the values
|
|
|
|
1.6 Other sysctl that might affect tcp connection rate or the maximum number
|
|
of open connections
|
|
|
|
fs.epoll.max_user_instances - maximum number of devices - per user (2.6.27.8+)
|
|
fs.epoll.max_user_watches - maximum number of "watched" fds - per user
|
|
(2.6.27.8+)
|
|
|
|
net.ipv4.tcp_max_orphans - might be worth a look if things go wrong
|
|
net.core.netdev_max_backlog - maximum device backlog
|
|
|
|
Related applications
|
|
--------------------
|
|
ifconfig <dev> txqueuelen <val> - set device transmission queue len
|
|
|
|
iptables - remove the ip_conntrack module (it limits the maximum tcp
|
|
connections, adds extra overhead (slow)). It's probably better to remove
|
|
all the iptables modules.
|
|
|
|
|
|
2. Monitoring (values to watch for)
|
|
-----------------------------------
|
|
|
|
2.1 File descriptors
|
|
|
|
fs.dentry-state - format: nr. dentries, nr. unused, age_limit, want_pages
|
|
fs.file-nr - format: allocated, unused, max (==fs.file-max)
|
|
fs.inode-state - format: nr. allocated, nr. free, preshrink
|
|
|
|
2.2 TCP
|
|
|
|
/proc/net/netstat - the TW, TWRecycled, TWKilled, PAWPassive, PAWActive,
|
|
PASWEstab fields
|
|
( cat /proc/net/netstat |cut -d" " -f12-17 ;
|
|
cat /proc/net/sockstat)
|
|
/proc/net/sockstat
|
|
|
|
|
|
3. Kamailio settings
|
|
----------------------
|
|
|
|
- Don't forget to increase tcp_max_connections and the amount of shared memory
|
|
- You should increase the number of Kamailio "tcp_children" processes (-N no)
|
|
As a rule of thumb, (maximum simultaneous connections)/2000 should be ok
|
|
- You might have to decrease TCP_BUF_SIZE to a smaller value (e.g 8K)
|
|
- You might want to increase PKG_MEM_POOL_SIZE (for large queues)
|
|
|
|
- You might need to increase the maximum open fds limit before starting Kamailio
|
|
(e.g. ulimit -n 1000000)
|
|
|