1. Overview

We can use a Secure Shell (SSH) connection to access a remote server and execute commands therein. The usual way to interrupt such a command is by pressing Ctrl+C. However, what happens next, depends on the way we connect.

In this tutorial, we’ll learn about various kinds of SSH sessions and the means to send the interruption signal over the corresponding connection.

2. Interactive Session

Let’s use the ssh command to open a connection to host for user with this syntax:

$ ssh user@host

As a result, we’ll find ourselves in the remote terminal where we can issue commands.

We’ll use the stress-ng performance test command as an example. Despite us connecting to the localhost throughout this tutorial, SSH opens a full-fledged connection:

$ ssh localhost
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-40-generic x86_64)
# ...
Last login: Thu Apr 27 20:45:09 2023 from 127.0.0.1
$ stress-ng --cpu 2 --timeout 60s
stress-ng: info:  [8258] setting to a 60 second run per stressor
stress-ng: info:  [8258] dispatching hogs: 2 cpu

Now, let’s take a look at the processes tree with the ps command:

$ ps -fax
    PID TTY      STAT   TIME COMMAND
# ...
   1025 ?        Ss     0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
   7936 ?        Ss     0:00  \_ sshd: joe [priv]
   8198 ?        S      0:00      \_ sshd: joe@pts/2
   8223 pts/2    Ss     0:00          \_ -bash
   8258 pts/2    SL+    0:00              \_ stress-ng --cpu 2 --timeout 60s
   8259 pts/2    R+     0:02                  \_ stress-ng-cpu [run]
   8260 pts/2    R+     0:02                  \_ stress-ng-cpu [run]
# ...

We see that the sshd server has opened a pseudo-terminal pts/3 for us and started an interactive shell -bash. The command runs inside, and will be interrupted after pressing Ctrl+C:

$ stress-ng --cpu 2 --timeout 60s
stress-ng: info:  [11165] setting to a 60 second run per stressor
stress-ng: info:  [11165] dispatching hogs: 2 cpu
# Ctrl+C is issued here
^Cstress-ng: info:  [11165] successful run completed in 2.45s
$ # in the remote terminal now

It happens because we’re working in the raw mode, so the bytes corresponding to the Ctrl+C keystroke will just be passed over the SSH connection. Then, the remote host interprets them and sends an appropriate interruption signal to the command. The SSH session remains alive.

3. Remote Command in Interactive Pseudo-Terminal

By passing the -t option to ssh, we run the remote command in an interactive pseudo-terminal. The signals are passed through the connection and handled by the remote host. In this way, we can run commands providing textual UI:

$ ssh -t user@host command

Let’s start stress-ng in this way:

$ ssh -t localhost stress-ng --cpu 2 --timeout 60s

We can see that the pseudo-terminal pts/2 is set for the SSH connection on the remote host:

$ ps -fax
    PID TTY      STAT   TIME COMMAND
# ...
   1025 ?        Ss     0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
   8602 ?        Ss     0:00  \_ sshd: joe [priv]
   8638 ?        S      0:00      \_ sshd: joe@pts/2
   8639 pts/2    SLs+   0:00          \_ stress-ng --cpu 2 --timeout 60s
   8640 pts/2    R+     0:03              \_ stress-ng-cpu [run]
   8641 pts/2    R+     0:03              \_ stress-ng-cpu [run]
# ...

Now Ctrl+C is signaled to the remote process, which exited. Next, its pseudo-terminal ends, too. Finally, SSH closes the connection:

$ ssh -t localhost stress-ng --cpu 4 --timeout 600s --metrics-brief
stress-ng: info:  [11721] setting to a 600 second (10 mins, 0.00 secs) run per stressor
stress-ng: info:  [11721] dispatching hogs: 4 cpu
# Ctrl+C is issued here
^Cstress-ng: info:  [11721] successful run completed in 3.44s
stress-ng: info:  [11721] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: info:  [11721]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [11721] cpu               18608      3.44     13.57      0.00      5411.91        1371.26
Connection to localhost closed.

4. Remote Command and Non-interactive Session

We can start a command directly on the remote server with this syntax:

$ ssh user@host command

It’s very useful because we can redirect the command’s output to the local machine, so we don’t need to copy files back over the network:

$ ssh user@host command > output

Let’s start stress-ng in this way:

$ ssh localhost stress-ng --cpu 2 --timeout 60s --metrics-brief

Subsequently, let’s study the command’s tree:

$ ps -fax
    PID TTY      STAT   TIME COMMAND
# ...
   1025 ?        Ss     0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
   9982 ?        Ss     0:00  \_ sshd: joe [priv]
  10062 ?        S      0:00      \_ sshd: joe@notty
  10095 ?        SLs    0:00          \_ stress-ng --cpu 2 --timeout 60s
  10097 ?        R      0:04              \_ stress-ng-cpu [run]
  10098 ?        R      0:04              \_ stress-ng-cpu [run]
# ...

In this case, we’re running a non-interactive session on the remote host, as notty indicates. Consequently, SSH on our side doesn’t work in the raw mode. Thus**, the Ctrl+C signal is not passed to the remote command but it kills the SSH session instead:**

$ ssh localhost stress-ng --cpu 2 --timeout 60s
stress-ng: info:  [10095] setting to a 60 second run per stressor
stress-ng: info:  [10095] dispatching hogs: 2 CPU
# Ctrl+C is issued here
^Cjoe@ubuntu:~$ # in the local terminal now

Afterward, let’s examine the remote processes to find the survivors:

$ ps -fax
    PID TTY      STAT   TIME COMMAND
# ...
  10095 ?        SLs    0:00 stress-ng --cpu 2 --timeout 60s
  10097 ?        R      0:20  \_ stress-ng-cpu [run]
  10098 ?        R      0:20  \_ stress-ng-cpu [run]

We end up with a dead connection and an abandoned process on the server.

5. Passing SIGINT to Remote Job

When we run a remote task, closing the connection instead of terminating the process usually isn’t what we expect. As the openSSH implementation doesn’t pass the SIGINT signal to the remote job, we need to find a workaround.

When the SSH connection closes, the related stdin input stream terminates, too. So, let’s detect the stdin termination, and if that happens, let’s kill the task. We can use the cat command, which reads from standard input. Therefore, it runs as long as the input stream is open. We’re going to start our command together with cat:

$ ssh localhost "command < <(cat; kill -INT 0)"

Let’s review this construction piece by piece. First, our command’s input is fed by the output of cat. We achieve this with the process substitution by the <() operator. It opens a subshell for encompassed commands and redirects their output. We can note the space between consecutive < symbols.

Thanks to the semicolon after cat, the successive kill command starts only after the first one finishes. Then, we send the INT signal. The 0 valued argument in kill -INT 0 means that we want to terminate all processes sharing the same group ID (PGID). It’s very useful if the original process forks, as in the case of stress-ng.

Let’s illustrate this construction by starting our CPU stressor:

$ ssh localhost "stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)"
joe@localhost's password: 
stress-ng: info:  [5916] setting to a 600 second (10 mins, 0.00 secs) run per stressor
stress-ng: info:  [5916] dispatching hogs: 4 cpu

On the “remote” machine, let’s see the tree of resulting processes:

$ ps -faxo user,pid,ppid,pgid,stat,args
USER         PID    PPID    PGID STAT COMMAND
# ...
root         976       1     976 Ss   sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root        5726     976    5726 Ss    \_ sshd: joe [priv]
joe         5898    5726    5726 S         \_ sshd: joe@notty
joe         5914    5898    5914 Ss            \_ bash -c stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)
joe         5916    5914    5914 SL                \_ stress-ng --cpu 4 --timeout 600s
joe         5917    5916    5914 S                     \_ bash -c stress-ng --cpu 4 --timeout 600s < <(cat; kill -INT 0)
joe         5918    5917    5914 S                     |   \_ cat
joe         5919    5916    5914 R                     \_ stress-ng-cpu [run]
joe         5920    5916    5914 R                     \_ stress-ng-cpu [run]
joe         5921    5916    5914 R                     \_ stress-ng-cpu [run]
joe         5922    5916    5914 R                     \_ stress-ng-cpu [run]
# ...

We can see out that all processes have the same 5914 PGID as cat. Because the kill command runs in the same shell where cat does, it terminates all processes with the same PGID.

6. Conclusion

In this article, we examined ways of executing and terminating commands via SSH. First, we opened the interactive shell on the remote server. Next, we ran the command within an interactive session. In both cases, we learned that the Ctrl+C keystroke terminates the remote command.

Finally, we started the program in a non-interactive way, therefore Ctrl+C closed the ssh connection rather than ending the process. To cope with that, we constructed a watchdog to kill the remote process when SSH terminated.