1. Introduction
Working with Linux often means using terminals. Unlike a graphical user interface (GUI) environment with clickable windows, switching to and controlling a background process in a command-line interface (CLI) is rarely straightforward.
In this tutorial, we deal with ways to attach a terminal to a process outside the current shell. First, we briefly explore how to process switching works in GUI environments. Next, we look at ways to have a similar mechanism with a CLI. After that, we check exceptions in the face of background jobs. We continue with the basics of attaching to a process, detaching, and identifying detached processes. Next, we delve into a way to confirm where a process is attached. Finally, we follow all steps required to reattach a process with a debugger and a specialized tool.
We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.
2. GUI Process Switching
In desktop environments like GNOME, KDE, and Xfce, controls are available on windows and bars as graphical elements. For example, GNOME’s taskbar (part of the Gnome Panel) allows switching windows and processes with a simple click.
In addition, there may be keyboard shortcuts for common tasks like minimizing, maximizing, and changing the focus of windows and controls.
Of course, any of the above won’t be possible if we have no desktop environment at our disposal. Furthermore, a process can be headless or lack a GUI and window completely.
Even in those cases, we should always at least be able to use a terminal. Let’s explore what we can do there.
3. CLI GUI Emulation
Indeed, it’s common for Linux to impose command-line means of handling input. As a result, we revert to a terminal and shell for our needs.
For instance, there are the screen and tmux commands. They enable a simulated window environment within a CLI.
Once deployed, both commands execute within the terminal but expose a rudimentary interface, allowing users to not only run commands but also:
- create new terminal windows by splitting the main one
- rearrange windows
- switch between windows
- send commands to the different terminals
Effectively, we get the equivalent of multiple nicely-arranged terminal windows in a desktop environment without the latter. Process switching is convenient in such a setting.
For instance, in screen we can use CTRL-a c to create a new window and CTRL-a n to switch to it. Likewise, tmux provides the CTRL-b c and CTRL-b o shortcuts with similar functions.
However, even under these conditions, we may still have processes that aren’t available for direct control.
4. Background Jobs
Linux supports jobs. In short, jobs enable users to run background processes without allowing them to take over the current terminal.
This can be very convenient to allow working in the same shell while waiting for a long task to complete. The easiest way to run a job in the background is by appending an ampersand:
$ sleep 10 &
[1] 666
$
Note, that we get the backgrounded job’s PID: 666. Next, we can check the status of our job along with the initiated command line:
$ jobs
[1]+ Running sleep 10 &
$
How can we get back to the job? By putting it in the foreground with fg:
$ fg
sleep 10
[...]
$
Notably, the mechanism of switching to and between jobs is comparable to attaching our terminal to different subtasks.
5. Attach a Terminal
What does it mean to attach a terminal to a process? Basically, we redirect the current standard streams:
- stdin (0)
- stdout (1)
- stderr (2)
Attaching means ensuring all three are properly assigned to our terminal for a given process. Consequently, any process input or output goes from and to a terminal we control.
In fact, as we saw above, that’s usually what happens by default when we start a process from a given Bash session. Now, let’s explore how we can detach.
6. Detached Processes
A detached process does not belong to any terminal or shell. There are a couple of common methods to isolate a process like this.
6.1. disown
We already covered the disown built-in:
$ sleep 3600 &
[1] 666
$ jobs
[1]+ Running sleep 3600 &
$ disown 666
$ jobs
$
At this point, we don’t have direct access to the background task (PID 666) via the current shell.
Although disown itself is a built-in, it isn’t standardized. This means it can function differently or be completely absent, depending on the shell. Let’s explore the POSIX method.
6.2. nohup
Another way to detach a process is the external but POSIX-compliant nohup tool:
$ nohup sleep 3600 &
[1] 666
nohup: ignoring input and appending output to 'nohup.out'
$ jobs
[1]+ Running nohup sleep 3600 &
Note how the jobs command still shows us the command in the background.
Also, we again get the PID, but also notice our process’ streams. Since nohup is standard, we’ll use it for our needs instead of disown.
7. Identify Detached Processes
We can find detached processes using ps and the PID we received from backgrounding:
$ ps 666
PID TTY STAT TIME COMMAND
666 pts/0 S 0:00 sleep 3600
Notice pts/0 in the TTY column. It means pseudo-terminal 0 is responsible for our background process’ input/output (IO). If we go back to the bash session linked to pts/0, we can confirm the job’s not there:
$ tty
/dev/pts/0
$ jobs
$
Using tty, we ensure the correct terminal is open. We then see that the process is not accessible via jobs.
We’ll call this state of our original process “detached”:
- not part of the jobs list in any terminal
- its original terminal is still open
Let’s exit the terminal, where sleep was started:
$ tty
/dev/pts/0
$ exit
Next, we move to another Bash session and check for the process by PID:
$ tty
/dev/pts/1
$ ps 666
666 ? S 0:00 sleep 3600
Now, we see a question mark (?) in the TTY column. It indicates that the process lacks an attached TTY. In fact, we have to use the x flag of ps to include such processes in the output.
We’ll call this state “fully detached”. Importantly, in this state, any process that awaits input will be terminated, as stdin is already closed.
8. /proc Directory File Descriptors
Although POSIX does not provide a standard for the /proc pseudo-filesystem, most major Linux distributions support it.
In short, it provides a tree-like file structure with information about processes. Of course, part of that is the open file descriptors, which we can get via ls:
$ ls -l /proc/666/fd/
total 0
lrwx------ 1 root root 64 Mar 03 06:56 0 -> /dev/pts/0
lrwx------ 1 root root 64 Mar 03 06:56 1 -> /dev/pts/0
lrwx------ 1 root root 64 Mar 03 06:56 2 -> /dev/pts/0
Notably, these are all linked to our terminal. However, piping data to /dev/pts/0 doesn’t really do much other than fill up our original terminal. Because of this, we can’t interact with our detached process in this way.
How do these descriptors change? Let’s terminate the shell of the original process and find out:
$ tty
/dev/pts/0
$ exit
We see all descriptors are deleted:
$ tty
/dev/pts/1
$
total 0
lrwx------ 1 root root 64 Mar 03 06:56 0 -> '/dev/pts/0 (deleted)'
lrwx------ 1 root root 64 Mar 03 06:56 1 -> '/dev/pts/0 (deleted)'
lrwx------ 1 root root 64 Mar 03 06:56 2 -> '/dev/pts/0 (deleted)'
Indeed, if we were to check the same descriptors for a process started with nohup, the results would be different:
$ nohup sleep 3600 &
[1] 666
nohup: ignoring input and appending output to 'nohup.out'
$ ls -l /proc/666/fd/
total 0
lrwx------ 1 root root 64 Mar 03 16:56 0 -> /dev/null
lrwx------ 1 root root 64 Mar 03 16:56 1 -> /nohup.out
lrwx------ 1 root root 64 Mar 03 16:56 2 -> /nohup.out
Here, input is ignored (/dev/null), while all output goes to a file, as per the notice. Furthermore, there are many ways to check for programs specifically started with nohup, as well as follow their output in another Bash session.
Once we know which process is detached, let’s explore ways to reattach it.
9. Attaching via gdb
Previously, we discussed running process output redirection. There, we used gdb (GNU Project Debugger) to achieve our aims.
Similarly, we can use the GNU Debugger to redirect a process’ input and output:
- Create a named pipe
- Run a process which will expect input
- Open the process in gdb
- Redirect input to a pipe
- Redirect all outputs to files
- Interact with the process via the pipe and files
Let’s see the above in code.
9.1. Pipe and Process Creation
Let’s start by creating a pipe and starting our process:
$ mkfifo /stdin_pipe
$ nohup perl -e '$|++; sleep 60; print STDOUT "Output."; print STDERR "Error."; $s = ; open($f, ">", "/perl_input") or die $!; print $f $s;' & exit
[1] 666
The process itself, a perl one-liner (-e), does the following:
- Forces flushing the buffers after each output
- Sleeps for one minute
- Prints “Output.” to stdout
- Prints “Error.” to stderr
- Assigns a line of stdin input to $s
- Opens (or creates) /perl_input for writing
- Prints $s to /perl_input
The sleep above is for the cases when we’re closing the Bash session before redirecting stdin. In those instances, the process terminates immediately since it would start by waiting for input on a closed handle.
9.2. gdb Modifications
Next, we fire up gdb with our process via its PID:
$ gdb -p 666
[...]
(gdb) call close(0)
$1 = 0
(gdb) call close(1)
$2 = 0
(gdb) call close(2)
$3 = 0
(gdb) call open("/stdin_pipe", 0x180)
$4 = 0
(gdb) call open("/stdout_regular", 0x441, 0x1FF)
$5 = 0
(gdb) call open("/stderr_regular", 0x441, 0x1FF)
$6 = 0
(gdb) quit
A debugging session is active.
Inferior 1 [process 666] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/perl, process 666
[Inferior 1 (process 666) detached]
First, we use call close to remove all associations to the standard streams. Next, we open them in order:
- stdin (0) as the pipe
- stdout (1) as the regular file /stdout_regular
- stderr (2) as the regular file /stderr_regular
Let’s continue by interacting with our process.
9.3. Reattached Process Interaction
Importantly, when opening /stdin_pipe in gdb, we will have to send some input to the pipe from another terminal so we can continue in gdb:
$ echo 'Input.' > /stdin_pipe
What we are writing to /stdin_pipe here will be the actual input sent to our perl snippet and thus written to /perl_input.
Note, the arguments to open() we use for the standard streams are important. In particular, the values of the flags can be seen in the documentation.
After quitting gdb, we can confirm when the process completes via ps 666. Finally, we can ensure all files received the correct data:
$ cat /perl_input
Input.
$ cat /stdout_regular
Output.
$ cat /stderr_regular
Output.
While not a standard method, gdb allows us to handle all file descriptors, including the standard ones, so we can practically attach them to anything.
In theory, this includes terminals, but in practice, we would have to create another tool for that purpose. Its job would be capturing a terminal’s streams as its own and then assigning them to our process the same way we did with GNU Debugger above.
In fact, there is such a utility already.
10. Using reptyr to Attach
Conveniently, reptyr is a tool that attaches terminal streams to a given process. In our case, it shortens the procedure in the previous section considerably:
- Run a process which will expect input.
- Note down the PID of the process.
- Pass the PID to reptyr in a new terminal.
- Interact with the process.
Let’s begin by starting the process as we did previously:
$ nohup perl -e '$|++; sleep 60; print STDOUT "Output."; print STDERR "Error."; $s = ; open($f, ">", "/perl_input") or die $!; print $f $s;' & exit
[1] 666
Next, we start another terminal and run reptyr for its PID:
$ reptyr 666
[-] Timed out waiting for child stop.
The notice can often safely be ignored, as it depends on how a process reacts to the SIGSTOP signal. It should force a transition to the stopped state (T), but not always.
After this, we can send our input line:
$ reptyr 666
[-] Timed out waiting for child stop.
Input.
Finally, after the sleeping is complete, we receive all expected output:
$ reptyr 666
[-] Timed out waiting for child stop.
Input.
Output.Error.
Checking the /perl_input file reveals all is as expected:
$ cat /perl_input
Input.
How does reptyr achieve this? By leveraging ptrace, the base of strace, which we already used for process output following.
11. Summary
In this article, we looked at the general idea of attached and detached processes. Further, we explored how to detach, identify, and reattach processes.
In conclusion, while Linux does provide ways for us to achieve our goals, none of them are strictly standardized.