1. Overview
In this tutorial, we’ll learn about the UDP socket in Linux. Specifically, we’ll learn how to monitor the socket’s receiving buffer capacity and the command to resize the buffer.
2. UDP Socket and Receive Buffer
In Linux, a socket is a software construct that provides an abstraction for the underlying network hardware. It’s a communication endpoint between two processes, enabling them to exchange data.
The AF_INET socket family supports the stream and datagram sockets, which represent TCP and UDP sockets, respectively. For this family, each socket comes with buffers for outgoing and incoming packets. These buffers are a region of memory that’s used to temporarily store the packet before it’s consumed or transmitted. When there are more packets received than the buffer can fit, the kernel has no choice but to drop all the other packets.
The Linux kernel tracks the statistics of the buffer and dropped packets of each of the sockets in the system and stores it in the /proc/net/udp pseudo-file.
3. Monitoring the UDP Socket Buffer
The /proc/net/udp pseudo-file is the file that tracks the statistics of UDP sockets in the system. Through this file, we can obtain various information and statistics about all the UDP sockets that are currently active. Although we can read the file directly using the cat command to obtain the information, using the ss command as the frontend for the file is preferable as it offers a better output format.
The ss command-line tool reports socket statistics in the system. To obtain the statistics of all the listening UDP sockets, let’s run the ss command with the -ul options:
$ ss -ul
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:mdns 0.0.0.0:*
UNCONN 0 0 0.0.0.0:40564 0.0.0.0:*
...
From the output table, each row represents one listening UDP socket. Then, the Rec-Q and Send-Q columns tell us the number of packets currently in the receiving buffer and send buffer, respectively. When the number of Rec-Q and Send-Q is consistently growing, we can conclude that the process is struggling to process the queue.
To further obtain its buffer memory statistics, we can additionally pass the -m option:
$ ss -ulm
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
UNCONN 0 0 0.0.0.0:mdns 0.0.0.0:*
skmem:(r0,rb212992,t0,tb212992,f4096,w0,o144,bl0,d0)
UNCONN 0 0 0.0.0.0:40564 0.0.0.0:*
skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0)
...
With the -m option, each row contains the skmem data structure that displays information about the buffer. The r0 and rb212992 mean that the receiving buffer has 0 packets in the queue and a 212992 byte of capacity. Then, the subsequent values contain the same information for the sending buffer. At the end of the tuple, the value with the d prefix indicates the number of packets that are dropped. Through this value, we can monitor the statistics of dropped packets on our socket.
4. Resizing the UDP Buffer
The receive buffer sizes in Linux are controlled by two sysctl parameters: net.core.rmem_default and net.core.rmem_max. Firstly, the net.core.rmem_default_value defines the default receive buffer size during socket creation. Therefore, by resizing this value, we can control the default buffer size allocation during socket creation.
Additionally, we can request more buffer sizes when we create a socket through the SO_RCVBUF option. The additional buffer size can only go up to a specific limit. Concretely, the net.core.rmem_max defines the upper cap of the maximum requestable buffer size. Importantly, this value must be greater than net.core.rmem_default.
Let’s use sysctl to get the values for both of the parameters:
$ sysctl net.core.rmem_default
net.core.rmem_default = 212992
$ sysctl net.core.rmem_max
net.core.rmem_max = 212992
In this system, by default, each socket will get a receive buffer with a size of 213 KB. Additionally, the rmem_max value of 213 KB restricts the maximum allowable receive buffer size to 213 KB during socket creation.
To resize either value, we can use the systl -w command. For example, we can resize the rmem_default to 30 KB and the rmem_max value to 800 KB:
$ sudo sysctl -w net.core.rmem_default=10000
net.core.rmem_default = 10000
sudo sysctl -w net.core.rmem_max=800000
net.core.rmem_max = 800000
After we’ve changed the value, we can create a UDP socket and see that it’ll use that value as the default receive buffer size:
$ nc -l -u 9999
$ ss -ulem '( sport = :9999 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:9999 0.0.0.0:* uid:1000 ino:101242 sk:25b <->
skmem:(r0,rb10000,t0,tb212992,f0,w0,o0,bl0,d0)
The first command uses the nc command to create a UDP port and bind it to port 9999. Then, we query for the socket that’s listening to port 9999 and print its statistics. From the skmem output, we can see that the socket has a receiving buffer of size 10 KB, which is the new net.core.rmem_default value we’ve set.
5. Reducing UDP Packet Drops
When we’re facing frequent UDP packet drops, the first instinct we might have is to increase the buffer size. However, depending on the scenario, this might not always be a good approach.
If the frequent UDP packet drops are due to an occasional burst in incoming traffic, and the burst generally does not last for a long period of duration, increasing the buffer might help. This is because, during occasional bursts of traffic, the process might be struggling to catch up. Having a larger buffer size helps smooth out the burst to prevent excessive packet dropping.
On the other hand, if the rate of UDP packet arrival is constantly faster than the rate of processing, increasing the buffer size will only delay the symptoms. The better approach in this scenario is to optimize the processing application to speed up the interval between packet processing. One approach is to parallelize the processing of the packets so that they don’t stay in the buffer for too long. Alternatively, we can optimize the algorithm that processes the packet so that it can be more efficient.
6. Conclusion
In this article, we’ve learned about the UDP packets in general. Then, we’ve seen how Linux sockets can be created to listen to incoming UDP packets. Additionally, we’ve learned that there’s a receiving buffer for the socket that temporarily holds the incoming packet until the process picks up the packet for processing.
Furthermore, we’ve learned how to get the socket buffer statistics using the ss command with the -m option. Besides that, we’ve seen how we can increase the buffer size using the sysctl -w command. Finally, we’ve explained that it’s important to first understand why UDP packets are being dropped before we blindly increase the buffer size.