1. Overview
We can use a Secure Shell (SSH) connection to access a remote server and execute commands therein. The usual way to interrupt such a command is by pressing Ctrl+C. However, what happens next, depends on the way we connect.
In this tutorial, we’ll learn about various kinds of SSH sessions and the means to send the interruption signal over the corresponding connection.
2. Interactive Session
Let’s use the ssh command to open a connection to host for user with this syntax:
$ ssh user@host
As a result, we’ll find ourselves in the remote terminal where we can issue commands.
We’ll use the stress-ng performance test command as an example. Despite us connecting to the localhost throughout this tutorial, SSH opens a full-fledged connection:
$ ssh localhost
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-40-generic x86_64)
# ...
Last login: Thu Apr 27 20:45:09 2023 from 127.0.0.1
$ stress-ng --cpu 2 --timeout 60s
stress-ng: info: [8258] setting to a 60 second run per stressor
stress-ng: info: [8258] dispatching hogs: 2 cpu
Now, let’s take a look at the processes tree with the ps command:
$ ps -fax
PID TTY STAT TIME COMMAND
# ...
1025 ? Ss 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
7936 ? Ss 0:00 \_ sshd: joe [priv]
8198 ? S 0:00 \_ sshd: joe@pts/2
8223 pts/2 Ss 0:00 \_ -bash
8258 pts/2 SL+ 0:00 \_ stress-ng --cpu 2 --timeout 60s
8259 pts/2 R+ 0:02 \_ stress-ng-cpu [run]
8260 pts/2 R+ 0:02 \_ stress-ng-cpu [run]
# ...
We see that the sshd server has opened a pseudo-terminal pts/3 for us and started an interactive shell -bash. The command runs inside, and will be interrupted after pressing Ctrl+C:
$ stress-ng --cpu 2 --timeout 60s
stress-ng: info: [11165] setting to a 60 second run per stressor
stress-ng: info: [11165] dispatching hogs: 2 cpu
# Ctrl+C is issued here
^Cstress-ng: info: [11165] successful run completed in 2.45s
$ # in the remote terminal now
It happens because we’re working in the raw mode, so the bytes corresponding to the Ctrl+C keystroke will just be passed over the SSH connection. Then, the remote host interprets them and sends an appropriate interruption signal to the command. The SSH session remains alive.
3. Remote Command in Interactive Pseudo-Terminal
By passing the -t option to ssh, we run the remote command in an interactive pseudo-terminal. The signals are passed through the connection and handled by the remote host. In this way, we can run commands providing textual UI:
$ ssh -t user@host command
Let’s start stress-ng in this way:
$ ssh -t localhost stress-ng --cpu 2 --timeout 60s
We can see that the pseudo-terminal pts/2 is set for the SSH connection on the remote host:
$ ps -fax
PID TTY STAT TIME COMMAND
# ...
1025 ? Ss 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
8602 ? Ss 0:00 \_ sshd: joe [priv]
8638 ? S 0:00 \_ sshd: joe@pts/2
8639 pts/2 SLs+ 0:00 \_ stress-ng --cpu 2 --timeout 60s
8640 pts/2 R+ 0:03 \_ stress-ng-cpu [run]
8641 pts/2 R+ 0:03 \_ stress-ng-cpu [run]
# ...
Now Ctrl+C is signaled to the remote process, which exited. Next, its pseudo-terminal ends, too. Finally, SSH closes the connection:
$ ssh -t localhost stress-ng --cpu 4 --timeout 600s --metrics-brief
stress-ng: info: [11721] setting to a 600 second (10 mins, 0.00 secs) run per stressor
stress-ng: info: [11721] dispatching hogs: 4 cpu
# Ctrl+C is issued here
^Cstress-ng: info: [11721] successful run completed in 3.44s
stress-ng: info: [11721] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
stress-ng: info: [11721] (secs) (secs) (secs) (real time) (usr+sys time)
stress-ng: info: [11721] cpu 18608 3.44 13.57 0.00 5411.91 1371.26
Connection to localhost closed.
4. Remote Command and Non-interactive Session
We can start a command directly on the remote server with this syntax:
$ ssh user@host command
It’s very useful because we can redirect the command’s output to the local machine, so we don’t need to copy files back over the network:
$ ssh user@host command > output
Let’s start stress-ng in this way:
$ ssh localhost stress-ng --cpu 2 --timeout 60s --metrics-brief
Subsequently, let’s study the command’s tree:
$ ps -fax
PID TTY STAT TIME COMMAND
# ...
1025 ? Ss 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
9982 ? Ss 0:00 \_ sshd: joe [priv]
10062 ? S 0:00 \_ sshd: joe@notty
10095 ? SLs 0:00 \_ stress-ng --cpu 2 --timeout 60s
10097 ? R 0:04 \_ stress-ng-cpu [run]
10098 ? R 0:04 \_ stress-ng-cpu [run]
# ...
In this case, we’re running a non-interactive session on the remote host, as notty indicates. Consequently, SSH on our side doesn’t work in the raw mode. Thus**, the Ctrl+C signal is not passed to the remote command but it kills the SSH session instead:**
$ ssh localhost stress-ng --cpu 2 --timeout 60s
stress-ng: info: [10095] setting to a 60 second run per stressor
stress-ng: info: [10095] dispatching hogs: 2 CPU
# Ctrl+C is issued here
^Cjoe@ubuntu:~$ # in the local terminal now
Afterward, let’s examine the remote processes to find the survivors:
$ ps -fax
PID TTY STAT TIME COMMAND
# ...
10095 ? SLs 0:00 stress-ng --cpu 2 --timeout 60s
10097 ? R 0:20 \_ stress-ng-cpu [run]
10098 ? R 0:20 \_ stress-ng-cpu [run]
We end up with a dead connection and an abandoned process on the server.
5. Passing SIGINT to Remote Job
When we run a remote task, closing the connection instead of terminating the process usually isn’t what we expect. As the openSSH implementation doesn’t pass the SIGINT signal to the remote job, we need to find a workaround.
When the SSH connection closes, the related stdin input stream terminates, too. So, let’s detect the stdin termination, and if that happens, let’s kill the task. We can use the cat command, which reads from standard input. Therefore, it runs as long as the input stream is open. We’re going to start our command together with cat:
$ ssh localhost "command < <(cat; kill -INT 0)"
Let’s review this construction piece by piece. First, our command’s input is fed by the output of cat. We achieve this with the process substitution by the <() operator. It opens a subshell for encompassed commands and redirects their output. We can note the space between consecutive < symbols.
Thanks to the semicolon after cat, the successive kill command starts only after the first one finishes. Then, we send the INT signal. The 0 valued argument in kill -INT 0 means that we want to terminate all processes sharing the same group ID (PGID). It’s very useful if the original process forks, as in the case of stress-ng.
Let’s illustrate this construction by starting our CPU stressor:
$ ssh localhost "stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)"
joe@localhost's password:
stress-ng: info: [5916] setting to a 600 second (10 mins, 0.00 secs) run per stressor
stress-ng: info: [5916] dispatching hogs: 4 cpu
On the “remote” machine, let’s see the tree of resulting processes:
$ ps -faxo user,pid,ppid,pgid,stat,args
USER PID PPID PGID STAT COMMAND
# ...
root 976 1 976 Ss sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 5726 976 5726 Ss \_ sshd: joe [priv]
joe 5898 5726 5726 S \_ sshd: joe@notty
joe 5914 5898 5914 Ss \_ bash -c stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)
joe 5916 5914 5914 SL \_ stress-ng --cpu 4 --timeout 600s
joe 5917 5916 5914 S \_ bash -c stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)
joe 5918 5917 5914 S | \_ cat
joe 5919 5916 5914 R \_ stress-ng-cpu [run]
joe 5920 5916 5914 R \_ stress-ng-cpu [run]
joe 5921 5916 5914 R \_ stress-ng-cpu [run]
joe 5922 5916 5914 R \_ stress-ng-cpu [run]
# ...
We can see out that all processes have the same 5914 PGID as cat. Because the kill command runs in the same shell where cat does, it terminates all processes with the same PGID.
6. Conclusion
In this article, we examined ways of executing and terminating commands via SSH. First, we opened the interactive shell on the remote server. Next, we ran the command within an interactive session. In both cases, we learned that the Ctrl+C keystroke terminates the remote command.
Finally, we started the program in a non-interactive way, therefore Ctrl+C closed the ssh connection rather than ending the process. To cope with that, we constructed a watchdog to kill the remote process when SSH terminated.