1. Introduction
When working in pure command-line mode, we’re often subject to the mercy of our terminal. While it’s often possible to avoid being stuck on a single screen, sometimes it’s inevitable. Because of this, we can end up unable to operate due to an unresponsive terminal. It’s not rare that this happens because of a command gone haywire.
In this tutorial, we focus on overburdened terminals and how to deal with them. First, we discuss general ways to overburden a terminal easily. Next, we take the example of a particular command, which often leads to an unusable console. After that, many methods to handle such situations are explored and demonstrated. Finally, we show some prophylactics and conditions to lower the chances of an overburdened terminal halting our work.
We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.
2. Terminal Overload With Runaway Commands
Actually, it’s not hard to overburden a terminal. Here, an overloaded or overburdened terminal will mean we’d have to wait an unreasonable amount of time for the current processes to complete, if it does at all.
This means such a terminal may also ignore termination signals. For example, sending CTRL+C may not stop its current tasks.
Let’s look at some ways in which we can bring a terminal to such a state.
2.1. Paste a Very Long Text
While in a shell, there are basic copy and paste mechanisms. Using these, a user could paste multiple lines, which the shell would interpret as commands. Pasting too many lines to a terminal at once can lead to a stall.
Let’s take a simple example. First, we generate the file noncommands.lst with a valid line of code. The latter will assign the current time to the variable start_time:
$ echo 'start_time=$(date +%s)' > noncommands.lst
After that, we populate the same file with the numbers from 1 to 1000, each on a separate line, prepended with noncommand:
$ for n in {1..1000}; do echo "noncommand$n" >> noncommands.lst; done
We finish the file off with another valid line of code that will calculate the difference in seconds between start_time and the current time:
$ echo 'echo $(( ($(date +%s) - start_time) ))' >> noncommands.lst
$ echo '' >> noncommands.lst
Next, we dump the file in the terminal via cat and copy the content with gdm:
$ cat noncommands.lst
start_time=$(date +%s)
noncommand1
noncommand2
[...]
noncommand999 █
noncommand1000
echo $(( ($(date +%s) - start_time) ))
$
Finally, we paste all of the text back to the terminal:
$ start_time=$(date +%s)
$ nonCommand1
-bash: nonCommand1: command not found
$ nonCommand2
-bash: nonCommand2: command not found
[...]
$ nonCommand1000
-bash: nonCommand1000: command not found
$ echo $(( ($(date +%s) - start_time) ))
167
This means that our terminal will be more or less unusable for 167 seconds. Any attempts to stop or background this stream might just affect one of the pasted commands, but not the entire chain.
On the other hand, sometimes even a single command can present a challenge.
2.2. Runaway Commands
Of course, running commands is what shells are all about. However, miscalculating the time it takes for even a single process to complete means it can overtake our terminal. Indeed, we have options like CTRL+C, but they don’t directly work with all commands.
For example, we can easily trap termination signals in Bash scripts. To prevent interrupting a long sleep with SIGINT, we can use two lines of code in a script:
trap '' SIGINT
sleep 666
Let’s take a typical everyday example.
3. Waiting for the cat Command
For instance, there is the cat command we previously used to dump the noncommands.lst file. When it’s running, the terminal is in canonical input mode. Normally, cat exits on an EOF (End-Of-File), signifying the end of the input.
Uniquely for cat, we can always try to wait for EOF as we did with noncommands.lst earlier.
However, with a very long stream of data, that may not be an option. Still, in those cases, we can just hide the terminal in question, as that precludes redraws, making output much faster.
Yet, the data stream in question can actually be endless, like when reading from the /dev/random pseudo-device:
$ cat /dev/random
[...]
Here, we can’t really wait for an EOF. In cases when waiting isn’t an option, we can turn to more universal methods.
4. Hotkeys to Halt Process Output
There are many ways to regulate or stop output from within an affected terminal. Most of them involve hotkeys. When discussing them, we should also consider the stty terminal settings.
4.1. Software Flow Control
We have already explored software flow control and how it can be a nuisance. Regardless, limiting output is precisely the function of CTRL+S (stop) and CTRL+Q (start).
Simply put, we can press CTRL+S to freeze the current terminal stream of data. After that, we are free to stop the offending process with another hotkey and resume output with CTRL+Q.
4.2. Termination Shortcuts
Alternatively, we can use termination signals locally. Nevertheless, as it’s in canonical mode, cat must first observe a character to interpret its signal.
Conversely, since special characters generated by local hotkeys like CTRL+C and CTRL+S compete with the rest of the output, it may take a while for the process to react. This, of course, also affects software flow control as well.
For this reason, our most viable option could be terminating from another shell.
5. Remotely Signal a Process Stop
At some point, we might need to completely stop runaway commands from outside the terminal. The usual way to achieve this is kill, which should work fine.
However, a bigger issue is identifying the specific instance of the rogue command. If in doubt, we can use killall with a process name. Otherwise, ps is the way to go:
$ ps a
PID TTY STAT TIME COMMAND
6 pts/0 Ss 0:00 -bash
7 pts/1 Ss 0:00 -bash
666 pts/0 S+ 0:05 cat largefish
777 pts/1 R+ 0:00 ps a
Here, we can see process 666 is a cat with the conveniently named largefish file as its argument that has been running for 5 minutes. Thus, it’s a good suspect for our culprit. Another way is to check the TTY matches our frozen terminal, assuming we already know which TTY it was.
Once we know its PID is 666, a simple kill -9 666 should be enough for a SIGKILL to terminate the runaway process.
So, how do we get another terminal to work with and perform these actions? Let’s explore some preventive measures so that we don’t end up with a single broken terminal.
6. Isolating Terminal Screens
Of course, Linux provides multiple accommodations when it comes to using terminals in any shape or form. These range from a graphical user interface (GUI) window of a terminal emulator, through command-line GUI emulation, to simple hotkeys for switching to another TeleTYpewriter (TTY).
6.1. Graphical User Interface
Naturally, terminal emulators in a GUI environment are simply windows or tabs. Due to their nature, we can just as simply handle them like any other such element, ignoring or closing them altogether as we please.
Thus, we can often terminate a problematic process or console with the click of a mouse.
Since a GUI may not be the method of choice for many Linux users, we’ll focus on the command-line interface (CLI).
6.2. Command-Line Interface GUI Emulation
Alternatively, we can employ methods to get the convenience of a GUI in our CLI. To that end, we can separate terminals in pseudo-windows, so any misbehavior is isolated. Let’s see that in action.
First, we check our current terminal device with the tty command:
$ tty
/dev/tty0
Next, we start tmux and split it into separate panes, checking each one with tty:
$ tty |$ tty
/dev/pts/1 |/dev/pts/2
$ |$
|
|
---------------------|---------------------
$ tty |$ tty
/dev/pts/3 |/dev/pts/4
$ |$
|
|
[0] 0:bash* "x" 06:56 06-Jun-22
Here, we see four different pts devices within a single actual TTY. Hence, a misbehaving command in one pts generally does not affect the rest of the panes.
Even when we don’t have an advanced terminal multiplexer like tmux or screen, we should still have options.
6.3. Switching the TTY
Finally, at the lowest level, we have the TTYs themselves. In some distributions, there are many more, but we usually get at least seven. To switch between them, the shortcut is commonly CTRL+ALT+F#, where F# is one of the function keys F1-F7:
$ tty
/dev/tty1
[CTRL+ALT+F5]
$ tty
/dev/tty5
In addition, if we’re using a desktop environment, it’s going to take up one of these terminals. Here, we’re assuming we don’t have a GUI to turn to and are exclusively using the raw CLI, not even a terminal emulator. As already explained, we’d otherwise have a pts, not a tty.
All of this means that, while it may not always be easy, we can usually switch out of a problematic terminal to kill the command that blocked it, as discussed in the previous section.
Importantly, there is no such thing as full isolation. If there was, we would be helpless in the face of simple output overloads, for example. Still, there are many possible setbacks that we should consider when escaping a broken terminal.
7. Remarks
There are some conditions we should consider to fully understand our options when a terminal is overburdened.
7.1. Specific Terminal
Even though it’s rarely irreplaceable, we may want to preserve a particular terminal.
For instance, having set up an environment with certain variables, information, and aliases, we could lose the terminal before saving its settings to a configuration file.
Another reason could be background jobs associated with the terminal we’re trying to preserve. While sometimes we can still restore them, it’s usually when we plan for it.
Even if the current terminal is not so exclusive on its own, we might not have many other options.
7.2. Limited TTYs
Of course, we can have the misfortune of blocking all available TTYs.
Further, there are ways to disable each TTY altogether so we can have only one. It’s as easy as modifying a few files (/etc/init/tty*.conf) or even just a single one (/etc/securetty), depending on the Linux distribution.
7.3. Remapped or Disabled Hotkeys
When all we have is the current terminal and hotkeys within it, we’re dependent on the current stty settings:
$ stty -a
speed 38400 baud; rows 32; columns 131; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S;
susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc
In particular, those for the termination signal key combinations like intr and susp. In fact, all hotkeys can be changed manually via stty, hooked by the Linux distribution, or even by a specific application, i.e., terminal emulator.
7.4. A cat Ruins the Terminal
Finally, there are cases when we do manage to apply one of the solutions to stop a runaway cat, but our terminal is severely affected.
This can easily happen along with a common mistake: using the cat command on a binary file. When we output a binary file in canonical mode, ANSI codes can appear in the output, affecting our shell.
For instance, color change, prompt modification, and input mode codes can make text invisible. To deal with such cases, we might be able to use the tput command to rescue our shell:
$ tput reset
This command-line attempts to reset most settings to their sane or configuration defaults.
8. Summary
In this article, we discussed how to handle overburdened terminals with the primary example of a runaway cat command.
In conclusion, there are multiple ways to overwhelm a terminal, but also many methods to either rescue it or kill the overwhelming process after.