1. Overview
In this tutorial, we’ll learn how to remove a CLOSE_WAIT socket connection.
First, we’ll learn what the CLOSE_WAIT state is. Then, we’ll see that we can remove the CLOSE_WAIT socket connection by killing the process using it. Alternatively, we’ll learn we can close the file descriptor associated with the CLOSE_WAIT socket to remove it.
2. The CLOSE_WAIT State
When we use the TCP protocol, the socket has a state associated with it. For instance, when a server uses a socket and is waiting for incoming connections on a port, that socket is in the LISTEN state.
We can list all sockets and their states using the ss command. If we add the -ta options, ss lists TCP sockets (the t option) in any state (the a option). Let’s see the result:
$ ss -ta
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:*
LAST-ACK 0 1 192.168.0.5:56466 217.70.190.250:https
ESTAB 0 0 192.168.0.5:51652 31.13.94.52:https
TIME-WAIT 0 0 192.168.0.5:4262 195.201.141.166:https
(...)
We can see the socket state in the first column.
After a connection is established, any side of the connection can end it. If the remote side ends the connection, we can’t send or receive more data. Therefore, we can only close the connection, too.
When one side closes the connection, the socket at the other side changes to the CLOSE_WAIT state. So, the CLOSE_WAIT state means the socket is closed on the remote side, and the system is waiting for the local side to close it.
Then, the only way to remove the CLOSE_WAIT socket connection is to close it. Usually, the process knows when the remote side closes the connection, and the local process closes it. This is normal behavior, so it’s uncommon to see CLOSE_WAIT sockets.
Although, if we see a CLOSE_WAIT socket, it may be because of a software bug. For example, if the process becomes unresponsive and the remote side closed the connection, the socket is never closed. So, if this happens, the socket stays in the CLOSE_WAIT state.
3. Finding the Process With the CLOSE_WAIT Socket
First, we have to find the process with the CLOSE_WAIT socket. We can use the ss command with the -tap flags to see the process associated with each socket. The p flag tells ss to print the process information for each socket.
We can filter the CLOSE_WAIT sockets using grep. However, we should filter by CLOSE-WAIT instead of CLOSE_WAIT. Let’s try it:
$ ss -tap | grep CLOSE-WAIT
CLOSE-WAIT 1 0 127.0.0.1:9999 127.0.0.1:56990 users:(("nc",pid=23117,fd=4)
There is an nc process with PID 23117, and with a socket in the CLOSE_WAIT state.
Alternatively, we can use the state parameter to filter the sockets by the TCP state. So, we can add the parameters state CLOSE-WAIT to list all CLOSE_WAIT sockets. Let’s try it:
$ ss -tap state CLOSE-WAIT
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
1 0 127.0.0.1:9999 127.0.0.1:56990 users:(("nc",pid=23117,fd=4))
We can notice that the ss command listed the same nc process as before.
4. Removing the CLOSE_WAIT Socket Connection
Now that we know the process, we can remove the CLOSE_WAIT socket using one of two methods. One way involves killing the process. However, sometimes, we don’t want to kill the process. So, in those cases, we can use the other method, which involves closing the file descriptor.
4.1. Killing the Process
One way to remove the CLOSE_WAIT socket is to kill the process. When the process terminates, the system also closes all its sockets.
We can use the kill command and the process ID to kill it. We can try to kill it with the default SIGTERM signal. However, the process may be unresponsive, so if the process isn’t killed, we can try again with the SIGKILL signal.
Let’s run kill to terminate the nc process with PID 23117:
$ kill 23117
Now, let’s check if the socket is still in CLOSE_WAIT:
$ ss -tap state CLOSE-WAIT
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
1 0 127.0.0.1:9999 127.0.0.1:56990 users:(("nc",pid=23117,fd=4))
We can see that the socket is still there, possibly because the nc process is unresponsive. So, let’s try with the SIGKILL signal to terminate it forcefully:
$ kill -KILL 23117
Now, let’s check again if there is any CLOSE_WAIT socket:
$ ss -tap state CLOSE-WAIT
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
Finally, we can see there isn’t any CLOSE_WAIT socket.
4.2. Closing the File Descriptor
Another way to remove the CLOSE_WAIT socket is to close the associated file descriptor. To do this, we can use the gdb debugger.
We can use this alternative when we don’t want to kill the process. However, we’ll be manipulating the process execution and changing its internal state. So, after we close the file descriptor, the process may behave erroneously, continue its normal execution, or exit.
In the previous section, we found a process with a CLOSE_WAIT socket and PID 23117. After the PID, ss prints “fd=4”, which means the file descriptor associated with that socket is the number 4.
Now, we can close the file descriptor using gdb. First, we can attach gdb to the process using the -p flag followed by the PID. Then, we can close the file descriptor by executing the print (int)close(FD) command in the gdb prompt. We must replace FD with the file descriptor number associated with the CLOSE_WAIT socket. Finally, we exit from gdb, executing the quit command.
Let’s attach to the process with PID 23117 and close its file descriptor number 4:
$ gdb -p 23117
GNU gdb (GDB) 11.2
Attaching to process 23117
(gdb) print (int)close(4)
$1 = 0
(gdb) quit
Alternatively, we can attach to the process, close the FD, and exit gdb using a one-liner. We add the -batch parameter and the -ex parameter followed by the print (int)close(4) command inside quotes. Let’s try it:
$ gdb -p 23117 -batch -ex 'print (int)close(4)'
0x00007f99d1a0fb84 in select () from /lib64/libc.so.6
$1 = 0
[Inferior 1 (process 23117) detached]
Let’s rerun ss to see if there is any CLOSE_WAIT socket:
$ ss -tap state CLOSE-WAIT
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
We can see there isn’t any CLOSE_WAIT socket now.
5. Conclusion
In this article, we saw how to remove a CLOSE_WAIT socket connection.
First, we saw what the CLOSE_WAIT state means. Then, we learned we can find the process with the CLOSE_WAIT socket and then kill it to remove the CLOSE_WAIT socket connection. Finally, we saw that we can also close the file descriptor associated with the socket using gdb.