1. Introduction
In this tutorial, we’ll study the cost of many TIME_WAITs on the server side. First, we’ll understand the function of the TIME_WAIT by examining its role in a TCP connection. Then, we’ll explore the effects of too many TIME_WAITs on a server and the associated cost.
2. The Function of TIME_WAIT
TCP (Transmission Control Protocol) provides reliable, ordered, and error-checked transmission of byte streams between hosts on an IP network. As a connection-oriented protocol, TCP requires the client and server hosts to establish a connection before transferring any data. Therefore, any packet transmitted via TCP must be associated with a specific TCP connection.
Each TCP connection is tied to a tuple of four elements: client address, client port, server address, and server port. Thus, multiple simultaneous TCP connections between two hosts using different ports can exist. Also, two hosts can open and close many connections over time and can even use the same address/port pair.
In addition, large-scale networks, such as the Internet, can delay the delivery of packets or generate duplications. Therefore, TCP needs to apply some methods to isolate old connections from new ones. TIME_WAIT is a key component in this connection isolation.
In particular, TCP uses TIME_WAIT to prevent packets from an old (closed) connection from being delivered during a new one:
In this example, a client and server establish a TCP connection (in orange), send three data packets, and close it. Shortly afterward, the same hosts open a new TCP connection (in blue) using the same address/port pair and transfer more data.
However, the network delayed the delivery of the third packet of the first connection. Thus, a packet belonging to the orange connection arrived at the server during the blue connection. This can cause various problems since that packet doesn’t belong to the blue connection.
TCP deals with this issue by defining that every connection that has been closed must wait a certain amount of time before it can be established again. The value set for TCP’s TIME_WAIT defines the total waiting time to allow new connections with the same addresses/ports pair.
RFC793 requires TIME-WAIT to last twice the time of MSL (Maximum Segment Lifetime). Some operating systems allow users to adjust this value. For example, we can change this value in Windows by setting TcpTimedWaitDelay in the registry. On Linux, TIME-WAIT is purposely defined statically in include/net/tcp.h as one minute:
#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
* state, about 60 seconds */
3. The Cost of Many Connections in TIME_WAIT State
As we’ve seen, TCP blocks new connections on the same addresses/ports pair as a recently closed connection until there are no more related packets on the network. For this to happen, one of the endpoints needs to keep a copy of the TCP control block (TCB) indicating that the connection has been terminated recently. We say that such a connection is in the TIME-WAIT state. In most cases, the host that keeps it is the server.
So, a possible effect of TCP’s TIME-WAIT is to prevent a client from establishing a connection with the server. However, this occurs only in two specific cases.
The first case is when both client and server try to connect using the same ports and addresses they used in a connection currently in the TIME-WAIT state. But this usually doesn’t happen because although the server always uses the same port, the client typically uses a different port for each new connection. This means that each new connection a client tries to establish doesn’t match the previous one.
Because of this, the second situation can only occur if the client has used up all their ephemeral ports. In other words, the client would’ve had to establish and close a connection for each of its ephemeral ports in a very short period of time. Again, this isn’t easy to happen, as the client’s ephemeral port range is very large.
For the server, the main cost is related to the additional CPU and memory usage caused by connections in the TIME-WAIT state. However, these connections require far fewer resources than opened connections. Despite this, many connections in a TIME-WAIT state can affect the server’s performance when processing active connections.
It’s unlikely that a single client can cause serious damage to a server by making several connections in a TIME-WAIT state. But certainly, multiple clients can coordinate a DDoS attack by deliberately generating connections in a TIME-WAIT state.
To avoid these situations, we can force the closing or allow the reuse of sockets in a TIME-WAIT state. However, we must be very careful when doing this.
4. Conclusion
In this article, we studied the cost of many TIME_WAIT on the server side. We’ve seen that TIME_WAIT is very useful for preventing an open connection from accepting packets from a recently closed connection. However, extreme cases of too many connections in a TIME_WAIT state on the server can cause problems for both the server and the client.