1. Introduction
As with any other major operating system (OS), under Linux, processes have a process ID (PID). In essence, the PID is a number that identifies the process to the kernel. However, there are scenarios in which this number isn’t immediately obvious or available. In fact, they most often involve network processes.
In this tutorial, we discuss processes that seem to have no PID. First, we briefly describe network processes. Next, we create three types of listening processes and connect to them. After that, we go over ways to get the PID of a network process. Finally, we explore an alternative path for identifying a process.
We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.
2. Network Processes
Network connections require a process:
- TCP
- TCP/IP sockets
- UNIX sockets
- Network File System (NFS)
Yet, sometimes we might not see the PID of that process even with standard tools like netstat and lsof. In fact, network processes form the general reason for an obfuscated link between processes and their PID numbers.
Importantly, a stderr warning precedes the output of both netstat and lsof when running them as a regular user:
$ whoami
baeldung
$ netstat
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
[...]
$ lsof
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
Output information may be incomplete.
After using whoami to verify the shell runs under the regular user baeldung, we run both netstat and lsof to see the warnings.
Now, let’s create different types of connections and see how we can identify the processes behind them.
3. TCP/IP Server
Let’s start with a basic TCP/IP connection. To begin with, we create a server and client. After that, we attempt to identify the processes behind them.
3.1. Create Client and Server as root
First, we create a small server as the root user with netcat (nc):
$ nc -nvlp 6660
listening on [any] 6660 ...
While preventing DNS name resolutions with -n, *we [-l]isten for connections on [-p]ort 6660 of any interface*, ensuring sufficient [-v]erbosity.
After that, from another terminal, we connect to our server as a client:
$ nc localhost 6660
Baeldung.
If we now enter any text and press Return, the listening side would print it out after a line like connect to [127.0.0.1] from (UNKNOWN) [127.0.0.1] 60667. At this point, we have client and server TCP/IP network processes.
3.2. Identify TCP/IP Processes
At this point, we check for our TCP/IP server and client as the baeldung regular user with both netstat and lsof:
$ whoami
baeldung
$ netstat
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 19 0 localhost:60667 localhost:6660 ESTABLISHED -
[...]
$ lsof -i :6660
$
To do so, we leverage the -p flag of netstat to include the PID/Program name column, while the -i flag of lsof filters by port alone.
Notably, as a regular user, we don’t see data in the PID/Program name column of netstat, and the lsof command doesn’t provide any output at all.
Now, let’s try as root:
$ whoami
root
$ netstat -p
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 19 0 localhost:60667 localhost:6660 ESTABLISHED 10666/nc
[...]
$ lsof -i :6660
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nc 10666 root 4u IPv4 4656028 0t0 TCP localhost:6660->localhost:60667 (ESTABLISHED)
nc 10666 root 3u IPv4 4653760 0t0 TCP localhost:60667->localhost:6660 (ESTABLISHED)
In each case, we see the server PID is 10666, but only as a superuser.
4. Named UNIX Socket Connection
The netcat utility has a traditional version nc.traditional, but also a netcat-openbsd (nc.openbsd) version.
First, let’s install the latter with apt:
$ apt install netcat-openbsd
This should ensure we run netcat-openbsd when executing nc or netcat. Alternatively, we can specify the desired binary directly via nc.openbsd or nc.traditional.
4.1. Create Connection
*By utilizing the -U option of nc.openbsd, we can create a named UNIX socket and [-l]isten for connections* without resolving [-n]ames:
$ nc.openbsd -nvlU /home/baeldung/unix.socket
Bound on /home/baeldung/unix.socket
Listening on /home/baeldung/unix.socket
Now, we use the -U option with the same socket path without [-l]istening in another terminal:
$ nc.openbsd -U /home/baeldung/unix.socket
Baeldung.
Once connected, the server side tells us Connection received on /home/baeldung/unix.socket. Again, after typing out some text and pressing Return, we see the message in both the client and server, our new network processes.
4.2. Identify Named UNIX Socket Processes
Once again, we use netstat and lsof:
$ whoami
baeldung
$ netstat -ap
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 7660010 - /home/baeldung/unix.socket
unix 3 [ ] STREAM CONNECTED 7660011 - /home/baeldung/unix.socket
[...]
$ lsof +E | grep unix.socket
$
Since this time we check for named sockets, we add the -a switch to netstat, showing all network connections. On the other hand, +E includes named UNIX sockets in the output of lsof, which we filter through grep.
As with TCP/IP, we either don’t get the socket at all (lsof) or we don’t see the information about any associated processes (netstat).
Yet, checking with root, we can see the information:
$ whoami
root
$ netstat -ap
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 7660010 6671/nc /home/baeldung/unix.socket
unix 3 [ ] STREAM CONNECTED 7660011 6672/nc /home/baeldung/unix.socket
$ lsof +E | grep unix.socket
nc 6667 root 3u unix 0x00000000054403ab 0t0 7660010 /home/baeldung/unix.socket type=STREAM
nc 6668 root 4u unix 0x000000006111cdfc 0t0 7660011 /home/baeldung/unix.socket type=STREAM ->INO=4812980 236459,nc,3u
There is an extra possibility here that the socket or process is owned by the kernel itself.
5. Network FileSystem (NFS) Server
The Network FileSystem (NFS) is an integrated way for a Linux system to seamlessly and securely serve and consume files over a network. As such, NFS is part of the Linux kernel.
Although most Linux distributions ship with the packages preinstalled, we can manually install the server and client with, e.g., apt:
$ apt install nfs-kernel-server nfs-common
Now, we can create an NFS connection.
5.1. Create Client and Server
Similar to the root directory of web servers, let’s now create an export directory and open its permissions with chmod:
$ mkdir /mnt/nfspoint
$ chmod 777 /mnt/nfspoint
Next, we designate the directory as such in the /etc/exports file by adding a line to it:
$ cat /etc/exports
[...]
/home/baeldung/nfspoint *(rw,sync)
In this case, *we allow access to * any client, providing [r]ead and [w]rite permissions and ensuring the data is in sync*.
Finally, we use exportfs to make the share available and restart the nfs-kernel-server:
$ exportfs -a
$ systemctl restart nfs-kernel-server
The -a switch ensures we handle all exports.
At this point, we can mount the filesystem from any machine as long as the firewall allows it. To make things simple, we use localhost, i.e., 127.0.0.1:
$ mkdir /home/baeldung/nfspoint
$ sudo mount localhost:/mnt/nfs /home/baeldung/nfspoint
$ cd /home/baeldung/nfspoint
After these commands, we have the /home/baeldung/nfspoint NFS mount point along with its supporting processes.
5.2. Identify NFS Processes
Notably, as part of the kernel modules, NFS doesn’t require much setup but also avoids the usual process forking, resulting in no actual visible processes for an NFS connection even as root:
$ whoami
root
$ netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:nfs 0.0.0.0:* LISTEN -
[...]
This is important since the kernel doesn’t spawn regular processes for NFS connections. Yet, we might be able to identify another association.
6. inode Data
For some network processes, extracting an associated inode can be helpful. Still, in most cases, we need superuser privileges to get data from the inode.
How we extract process information from its inode can depend on the filesystem.
For example, the find command with its -inum option is fairly universal:
$ find / -inum <INODE_NUMBER>
On the other hand, sometimes this might not get the information we’re after. In such cases, we can use alternative commands:
- btrfs-inspect-internal inode-resolve <INODE_NUMBER>
for a BTRFS filesystem - xfs_db xfs_db -c ‘blockget -n -i <INODE_NUMBER>’ <BLOCK_DEVICE> for an XFS filesystem
- debugfs -R ‘ncheck <INODE_NUMBER>’ <BLOCK_DEVICE> for ext* filesystems
In each case, we might be able to use the inode number to get further information about a given network connection.
7. Summary
In this article, we went over network processes and why they might seem to lack a PID.
In conclusion, while most processes have an ID regardless of different command outputs, some are actually internal kernel structures that don’t need a PID.