Eilfa - We Teach You Learn - solaris 10 tcp tuningPage 1 of 7 solaris 10 tcp/ip tuning In this solaris 10 tcp/ip tuning article i have collected the most essential and important tcp keys in order to achieve a better throughput, off course according to the situation being examined. Please Note: When mentioned Temporary, it means that after reboot configuration will be erased while when mentioned Permanently, it means that after reboot the configurations will remain intact. Fanout the incoming TCP/IP connections Determines the number of squeues to be used to fanout the incoming TCP/IP connections. The incoming traffic is placed on one of the rings. If the ring is overloaded, packets are dropped. Default - 2 Range - 0 - nCPUs, where nCPUs is the maximum number of CPUs in the system Dynamic? - No. The interface should be plumbed again when changing this parameter. When to Change Consider setting this parameter to a value greater than 2 on systems that have 10 Gbps NICs and many CPUs. Use these commands to set temporarly and permanently the changes: Temporary - ndd -set /dev/tcp ip_soft_rings_cnt Permanently insert into /etc/system - set ip:ip_soft_rings_cnt=16 A value of 0 associates a new TCP/IP connection with the CPU that creates the connection. A value of 1 associates the connection with multiple squeues that belong to different CPUs. The number of squeues that are used to fanout the connection is based upon ip_soft_rings_cnt. Consider setting this parameter to 1 to spread the load across all CPUs in certain situations. For example, when the number of CPUs exceed the number of NICs, and one CPU is not capable of handling the network load of a single NIC, change this parameter to 1. Temporary - ndd -set /dev/tcp ip_squeue_fanout 1 Permanently insert into /etc/system - set ip:ip_squeue_fanout = 1 Remarks In Solaris, the available range of TCP/IP ports is 0 to 65535. However, there are some restrictions that apply: Ports in the range 0 to 1023 are reserved for privileged (root) services, such as telnetd, ftpd, and so on. Ports in the range 1024 to tcp_smallest_anon_port-1 are used for user http://www.eilfa.com/solaris/solaris-10-tcp-tuning.html 12/02/13 and so on. Permanently insert into /etc/system . To limit the range of the port numbers allocated for the general use. failures start to occur. the list will ultimately get full. Default . Solaris allocates anonymous ports above 32768.Eilfa . This leaves the range 32768 to 65535 available for general TCP/IP connections. When the list is full. http://www. The default value of this is 65535. TCP hash table size Check UNDER /etc/system: Controls the hash table size in the TCP module for all TCP connections (default 512).ndd -set /dev/tcp tcp_time_wait_interval 3000 Setting this didnt give a better performance for a lighty web server.eilfa. Temporary . The default value is 32768.solaris 10 tcp tuning Page 2 of 7 services such as NFS server daemon. see tcp_slow_start_initial.html 12/02/13 . kernel sockets The kernel keeps a list of sockets in the TIME_WAIT state.set ipc_tcp_conn_hash_size=8192 congestion window The maximum (Default 4)initial congestion window (cwnd) size in MSS of a TCP connection tcp_rexmit_interval_initial Temporary .We Teach You Learn .com/solaris/solaris-10-tcp-tuning. FONT server. the following two ndd(1M) parameters can be used: tcp_smallest_anon_port: This determines the smallest TCP port number that may be used for an anonymous connection.ndd -set tcp:tcp_conn_hash_size=8192 Controls the hash table size in an IP module for all active (in ESTABLISHED state) TCP connections(default 512). If your server is getting new client connections faster than it can bleed off sockets in the TIME_WAIT state. tcp_largest_anon_port: This is the largest TCP port number that may be used for anonymous connections. Decreasing the timeout increases the bleed-off rate.60000 Temporary .ndd -set /dev/tcp tcp_slow_start_initial 1 ndd -set /dev/tcp tcp_slow_start_after_idle 1 When to Change? For more information. core.ipv4.ipv4. decrease the value.tcp_fin_timeout = 1 in Solaris tcp_time_wait_interval Linux net.tcp_max_orphans = 262144 Linux net.solaris 10 tcp tuning Page 3 of 7 ndd -set /dev/tcp tcp_slow_start_initial 2 ndd -set /dev/tcp tcp_slow_start_initial 1 When to Change? Do not change the value.ip_conntrack_max = 1048576 net.tcp_max_syn_backlog = 262144 in Solaris tcp_conn_req_max_q0 Linux net.core.rmem_max = 16777216 Linux net.ipv4.core. TIME_WAIT ports These ensure that TIME_WAIT ports either get reused or closed fast. If the initial cwnd size causes network congestion under special circumstances.Eilfa .tcp_tw_recycle = 1 TCP memory Linux net.tcp_synack_retries & net.com/solaris/solaris-10-tcp-tuning. Linux net. but these are suitably high for our uses.rmem_default = 16777216 Linux net.ipv4.eilfa.We Teach You Learn .tcp_syncookies = 1 .netfilter. the box doesn't die net. insuring that if conntrack gets turned on.(The "syn cookies" violate the TCP spec thus solaris uses thier own mechanisem).netdev_max_backlog = 262144 in Solaris to tcp_conn_req_max_q Linux tcp_slow_start_after_idle = 262144 SYN cookies protection Linux net.ipv4.ipv4.ipv4.ipv4.html 12/02/13 .tcp_syn_retries = 2 in Solaris tcp_rexmit_interval_min 400 tcp_rexmit_interval_max 60000 tcp_ip_abort_interval 480000 tcp_rexmit_interval_initial 3000 conntrack You shouldn't be using conntrack on a heavily loaded server anyway. insert into /etc/system Linux net.nf_conntrack_max = 1048576 http://www. but Solaris has SYN flood protection enabled by default. If the server stalls.solaris 10 tcp tuning Page 4 of 7 In Solaris use Dtrace script to track connections Dtrace script to track connections TCP/IP connection control blocks Notifies TCP/IP on how long to keep the connection control blocks closed.com/solaris/solaris-10-tcp-tuning. Once enabled. with all of the activities in system processes. When high connection rates occur. Applications must enable this feature with the SO_KEEPALIVE socket option via the setsockopt(3SOCKET) socket call.html 12/02/13 .ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 67500 TCP keepalive TCP keepalive is a feature provided by many TCP implementations.675000 Temporary . Visible delays can occur for up to four minutes. the control blocks are kept for the specified time. the netstat command shows that many of the sockets that are opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state. Default . http://www. during which time the server does not send any responses. The server can stall during peak periods.eilfa.We Teach You Learn . as a way to clean up idle connections in situations like the ones mentioned above.ndd -set /dev/tcp tcp_time_wait_interval 3000 FIN_WAIT_2 state timer interval Specifies the timer interval prohibiting a connection in the FIN_WAIT_2 state to remain in that state. a keepalive probe packet is sent to the other end of the socket provided the connection has remained in the ESTABLISHED state and has been idle for the specified time frame. using the netstat command shows that many of the sockets opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state. including Solaris. Visible delays can occur for up to four minutes. After the applications complete the TCP/IP connection.Eilfa . When high connection rates occur. The server can stall during certain peak periods.60000 Temporary . a large backlog of TCP/IP connections accumulates and can slow server performance. with all of the activity in system processes. Default . during which time the server does not send any responses. but CPU utilization stays high. but CPU utilization stays high. If the server stalls. a large backlog of the TCP/IP connections accumulates and can slow server performance. This time frame is the value specified by the TCP tunable tcp_keepalive_interval. As mentioned before.com/solaris/solaris-10-tcp-tuning.solaris 10 tcp tuning Page 5 of 7 A keepalive probe packet is handled just like any other TCP packet which requires an acknowledgment (ACK) from the other end of the socket connection.ndd -set /dev/tcp tcp_keepalive_interval 300000 Backlog Queue The backlog queue is a large memory structure used to handle incoming packets with the SYN flag set until the moment the three-way handshake process is completed. Using the ndd command we can modify this value. which has no responding peer will therefore be: tcp_keepalive_interval + tcp_ip_abort_interval Default valuses respectively 7200000 480000 The above parameters are global and will affect the entire system. It will be retransmitted per the standard retransmission backoff algorithm. If no response is received by the time specified for the other TCP tunable. The first parameter tcp_conn_req_max_q controls the total number of full connections. the connection is terminated. Additionally.We Teach You Learn . When a maximum number of incoming connections is reached. tcp_ip_abort_interval.html 12/02/13 . However care should be taken to ensure the above parameters remain at a high enough value to avoid unnecessary traffic and other issues such as prematurely closing active connections in situations where a few packets have gone missing. subsequent requests are silently dropped by the operating system. the default value is set to 1024. The backlog queue controls how many half-open connections can be handled by the operating system at the same time. The second tcp_conn_req_max_q0 parameter defines how many halfopen connections are allowed without the dropping of incoming requests. To limit the effects of SYN attacks we should enable some built-in protection mechanisms. as would be the case for any other unacknowledged packet. Keep in mind that TCP keepalive probes have no effect on inactive connections as long as the remote host is still responding to probes. host is probably under a SYN flooding attack. http://www. the source IP addresses of these incoming packets can be spoofed. In Sun Solaris 8. Hence the actual maximum idle time of a connection utilizing TCP keepalive. Moreover. Temporary .eilfa. Run this command to count how many half-open connections are in the backlog queue at the moment netstat -s -P tcp | grep tcpHalfOpenDrop In Sun Solaris there are two parameters which control the maximum number of connections.Eilfa . when we detect a lot of connections in the SYN RECEIVED state. An operating system allocates part of the system memory for every incoming connection. We know that every TCP port can handle a defined number of incoming requests. we can sometimes use techniques such as increasing the backlog queue size and minimizing the total time where a pending connection in kept in allocated memory (in the backlog queue). solaris 10 tcp tuning Page 6 of 7 It is pretty simple really: never change these parameters unless connections are refused because the values are too low. netstat -nP tcp | grep WAIT | wc -l. increase tcp_conn_req_max_q0.ndd -set /dev/tcp tcp_ip_abort_interval 60000 (default is 480000) TCP/IP statistics These set of commands will present some of the TCP/IP statistics you will need in order to follow every change you make in your TCP/IP stack. #######Start of TCP/IP script############# http://www. increase tcp_conn_req_max_q.com/solaris/solaris-10-tcp-tuning. TCP closes this connection. The only way to determine this empirically is to use ‘netstat –s | fgrep –i listendrop’. Temporary .html 12/02/13 .Eilfa .ndd -set /dev/tcp tcp_conn_req_max_q 128 OR 262144 Temporary . If tcpListenDropQ0 is non-zero. When set too high. Temporary . if TCP has been retransmitting for tcp_ip_abort_interval period of time and it has not received any acknowledgment from the other endpoint during this period.netstat -s -P tcp | grep -E "tcpL" netstat -I bnx0 10 iostat -xn 10 TCP/IP script This script will assist you in configuring the TCP/IP parameters on your system. If tcpListenDrop is non-zero. For a given TCP connection.netstat -nP tcp |wc -l.eilfa. establishing outgoing connections to destination servers such as replicas not responding quickly can cause long delays. Temporary .ndd -set /dev/tcp tcp_ip_abort_cinterval 10000 (default is 180000) Specifies the default total retransmission timeout value for a TCP connection.We Teach You Learn .ndd -set /dev/tcp tcp_conn_req_max_q0 1024 OR 30000 Outgoing connection establishe time wait Some systems allow you to configure how long a system waits for an outgoing connection to be established. solaris 10 tcp tuning Page 7 of 7 #!/sbin/sh ndd -set /dev/ip ip_forward_src_routed 0 #(Defalut value alreay set) ndd -set /dev/tcp tcp_rev_src_routes 0 #(Defalut value alreay set) ndd -set /dev/ip ip_forward_directed_broadcasts 0 #(Defalut value alreay set) *ndd -set /dev/tcp tcp_conn_req_max_q0 4096 #(Defalut value 1024) *ndd -set /dev/tcp tcp_conn_req_max_q 1024 #(Defalut value 128) ###Prevent System responding to ICMP timestamp requests ndd -set /dev/ip ip_respond_to_timestamp 0 #(Defalut value alreay set) ###Prevent System responding to ICMP timestamp Broadcast ndd -set /dev/ip ip_respond_to_timestamp_broadcast 0 #(Defalut value alreay set) ndd -set /dev/ip ip_respond_to_address_mask_broadcast 0 #(Defalut value alreay set) ndd -set /dev/ip ip_respond_to_echo_broadcast 0 ndd -set /dev/arp arp_cleanup_interval 60000 ndd -set /dev/ip ip_ire_arp_interval 60000 ndd -set /dev/ip ip_ignore_redirect 1 ndd -set /dev/ip ip_strict_dst_multihoming 1 ndd -set /dev/ip ip_send_redirects 0 ########END of TCP/IP script############### http://www.Eilfa .com/solaris/solaris-10-tcp-tuning.We Teach You Learn .html 12/02/13 .eilfa.