NOTE! The information
on this page is for informational purposes only.
SUNET is not responsible for problems occuring when using this information.
The operating systems of today has good support for auto configuration of TCP. Good performance can be acheived just by changing the TCP window size. All operating systems we tested had much too low TCP window size as default to utilize a gigabit etehernet card, even on fairly short distances. We have also found out that only very late releases of the operating systems have good TCP stacks and drivers for the network cards, so upgrading to a fairly rescent OS is vital.
Below is a short description of what we did on the different operating systems we tested. In the table are some values shown with dark grey background - those values are usually the default, and thus not nesseceary to change.
We have used the following conventions:
The values of those parameters varies according to transmission speed and round-trip-time (RTT). Don't blindly set theese values too large, as they consume kernel memory, and too large values may cause degradation in performance.
The [sbmax] parameter should be set to a size larger than the maximum TCP window size.
The theoretical value (in bytes) for [wstd] is bps / 8 * rtt, where bps is bandwidth in bits/second and rtt=round-trip-time. You can use this Perl program that will determine the RTT and do the calculations for You..
Only the NetBSD 2.0 (and later) release support really fast connections, as there has been much work done on the "zero-copy" TCP features and removal of TCP stack linear searches.
|sysctl -w net.inet.tcp.rfc1323=1||Activate window scaling and timestamp options according to RFC 1323.|
|sysctl -w kern.somaxkva=[sbmax]||Set maximum size for all socket buffers together in the system|
|sysctl -w kern.sbmax=[sbmax]||Set maximum size of socket buffer for one TCP flow|
|sysctl -w net.inet.tcp.recvspace=[wstd]||Set default size of TCP receive window.|
|sysctl -w net.inet.tcp.sendspace=[wstd]||Set default size of TCP transmit window.|
|sysctl kern.mbuf.nmbclusters||View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set by recompiling Your kernel - see above!|
You can enter the values to /etc/sysctl.conf to have them set at boot-time, as in this example.
We haven't been able to get as good results with FreeBSD as with NetBSD, which is mostly due to linear searching in buffer chains in the transmit/receive path. FreeBSD 5.2 though gives significantly better performance that the FreeBSD 4 releases.
|sysctl net.inet.tcp.rfc1323=1||Activate window scaling and timestamp options according to RFC 1323.|
|sysctl ipc.maxsockbuf=[sbmax]||Set maximum size of TCP window.|
|sysctl net.inet.tcp.recvspace=[wstd]||Set default size of TCP receive window.|
|sysctl net.inet.tcp.sendspace=[wstd]||Set default size of TCP transmit window.|
|sysctl kern.ipc.nmbclusters||View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set att boot time - see above.|
|sysctl net.inet.tcp.liondmask=7||Used to enable the "netlion" patch above.|
You can enter the values to /etc/sysctl.conf to have them set at boot-time,
as in this example.
The table below shows which parameters that may need to be changed. These are true for both 2.4 and 2.6 kernels. With theese changes, it is possible to get results in the same order as our NetBSD tests, with the exception that Linux will do a data copy in the transmit path so the transmitting machine will be more loaded.
|echo "1" > /proc/sys/net/ipv4/tcp_window_scaling||Activate window scaling according to RFC 1323|
|echo "1" > /proc/sys/net/ipv4/tcp_timestamps||Activate timestamps according to RFC 1323|
|echo [wmax] > /proc/sys/net/core/rmem_max||Set maximum size of TCP receive window.|
|echo [wmax] > /proc/sys/net/core/wmem_max||Set maximum size of TCP transmit window.|
|echo [wmax] > /proc/sys/net/core/rmem_default||Set default size of TCP receive window.|
|echo [wmax] > /proc/sys/net/core/wmem_default||Set default size of TCP transmit window.|
|echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_rmem||Set min, default, max receive window. Used by the autotuning function.|
|echo "[wmin] [wstd] [wmax]" > /proc/sys/net/ipv4/tcp_wmem||Set min, default, max transmit window. Used by the autotuning function.|
|echo "bmin bdef bmax" > /proc/sys/net/ipv4/tcp_mem||Set maximum total TCP buffer-space allocatable. Used by the autotuning function.|
|ifconfig eth? txqueuelen 1000||Define length of transmit queue. Replace "?" with actual interface number.|
Tests performed with Windows 2003 give results similar to the NetBSD results. To achieve this we had to change some variables in the registry.
|HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts=1||Turn on window scaling option|
|HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize=[wmax]||Set maximum size of TCP window|
Further reading on Windows TCP tuning:
Börje Josefsson 2004-06-07