Tunnelling with SSH – different approaches and tips

29 Jul, 2021

LinkedInTwitter

This blog describes different approaches to SSH tunnelling along with practical examples. Something that is often not understood or remembered – hopefully you find it helpful.

If you would like to know more about how to implement modern data and cloud technologies into your business, we at Digitalis do it all: from the cloud and Kubernetes migrations to fully managed services, we can help you modernize your operations, data, and applications – on-premises, in the cloud and hybrid.

We provide consulting and managed services on a wide variety of technologies. Contact us today for more information or to learn more about each of our services.

Tunnelling with SSH

In the world of server administration, SSH is the de-facto standard tool for securely logging into servers and getting command-line access but with a bit of imagination, there is much more you can do with it. This article assumes that you are already familiar with using SSH to login to machines but there are more features outlined here that you may not be aware of.

Basic port forwarding

One of the most useful features of SSH is that it not only creates a secure connection between you and a remote server but it also allows you to forward TCP connections over the same secure link as your shell access. This can be used to create a secure tunnel for any service that works over TCP, common examples being HTTP services and databases.

This simplest form is to use the -L option when opening an SSH session, for example:

ssh -L 8080:127.0.0.1:8080 [email protected]
Let’s break down what that actually means. The -L option takes one argument split into 3 sections:
local_port:remote_address:remote_port
So the command above says to listen on port 8080 on the local machine where the SSH client is running (e.g. your laptop) and forward any connections on that port to the remote server’s port 8080. The remote address is relative to the remote server so in our case the connection will be forwarded to 127.0.0.1:8080 on the SSH server itself. If the remote machine is running a web server on port 8080 then we can access it by going to http://127.0.0.1:8080 in a browser on our local machine.
Basic port forwarding
Multiple -L options can be supplied on the command line if you need to forward more than one port at a time.

This approach can be extended to forward a port to another machine that lives behind the remote server, for example:

ssh -L 8080:10.0.0.10:8080 [email protected]
In this case SSH still listens on port 8080 on your local machine but instead of forwarding the connections to the remote server’s loopback address they are now forwarded on to another machine on the server’s network. One thing to be aware of is that the connection from the remote SSH server to the target of the port forward (10.0.0.10 in this case) is performed using plain TCP, it is not secured by the SSH tunnel.

By default the local port is opened on the loopback address (127.0.0.1) of your local machine but you can override this behaviour using one of two methods. The -g option tells SSH to listen on all interfaces so other hosts on your network can connect to your forwarded port, or you can specify a local listen address at the start of the -L arguments, for example:

ssh -L 192.168.0.1:8080:10.0.0.10:8080
This tells SSH to listen on the local address 192.168.0.1 instead of the default 127.0.0.1.
ssh tunnel
What if we want to do the same in reverse and expose a local service to something running on the remote server? It’s just as simple, we use the -R option:
ssh -R 8080:127.0.0.1:8080 [email protected]
The -R option looks very similar to the -L option, it also has 3 (optionally 4) sections:
[bind_address:]remote_port:local_address:local_port
The command above tells the remote server to listen for connections on port 8080 and forward them back to the client (your local machine) and connect to 127.0.0.1:8080.
ssh -R
As with local forwarding you can specify a different target address to forward the connection to another machine on your local network, for example:
ssh -R 8080:192.168.0.1:8080 [email protected]
Now if something connects to 127.0.0.1:8080 on the remote server the connection will be forwarded over your SSH session to your local machine and then to port 8080 on 192.168.0.1 in your local network.

Finally you can also tell the remote server to listen on a different address by specifying one of the server’s IP addresses at the start of the arguments:

ssh -R 10.0.0.1:8080:127.0.0.1:8080
This tells the server to listen for connections on 10.0.0.1 instead of 127.0.0.1 so that other machines on the server’s network can connect to the forwarded service. You can also specify * as the address to tell the server to listen on all interfaces. For security reasons this behaviour can be restricted or disabled on the server with the GatewayPorts option in sshd_config.

SSH Over SSH

Based on the port forwarding examples above you have probably realised this could also be used to open an SSH session to another server within the remote network. Let’s say you can connect to a host jump.example.com directly and you want to open an SSH session on a machine behind this host at 10.0.0.2.

You could of course just login to jump.example.com then SSH from there to the machine behind it, for example:

laptop:~$ ssh jump.example.com
jump:~$ ssh 10.0.0.2
10.0.0.2:~$ 
Or using the TCP port forwarding example above you could open a tunnel to get a more direct connection:
ssh -L2222:10.0.0.2:22 jump.example.com
ssh -p2222 127.0.0.1
Note that here we made the local machine listen on port 2222 rather than 22 to avoid any conflicts with a local SSH server.

Both of these methods are cumbersome because they require you to manually open a connection to the jump host and then open another connection to the machine you want to access. Luckily there are ways to simplify the process a little.

A single command like this will open a connection to the jump host and automatically use it to tunnel your connection.

ssh -J jump.example.com 10.0.0.2
Note that when operating in this way the IP address or hostname that you specify will be resolved at the jump server rather than your local machine. This can be useful if your remote network has local DNS, you could specify a hostname like webserver instead of the IP address.

If this is a connection you are likely to use regularly then you can save yourself some typing by adding the jump server to your ~/.ssh/config file so it is used automatically when you connect to this server, for example:

Host 10.0.0.2
  ProxyJump jump.example.com

There are many more things that can be done by editing your ~/.ssh/config file, check out the documentation (https://man.openbsd.org/ssh_config) for more information.

Dynamic Forwarding

So far we have seen how to use SSH to securely tunnel specific TCP ports over the connection but what if we need to forward multiple connections to different hosts on the remote network? And what if we don’t know in advance what ports we will need to forward? Luckily SSH has you covered there too.

SSH has a SOCKS5 proxy built in which can be used by the client to access anything that the server can reach. This is enabled by passing the -D option to the ssh command along with a port number and optional listen address:

ssh -D 127.0.0.1:6000 [email protected]

This command will open a normal SSH shell session and start a SOCKS5 proxy listening at 127.0.0.1:6000 on your local machine. You can now use this with any software that supports SOCKS5 proxies and gain access to services behind the remote server. A common use for this is to connect to web services that are behind the firewall in the remote network, for example internal management tools. Combining this with a browser plugin such as FoxyProxy makes a great solution that in some cases can completely replace a VPN.

ssh proxy
This method can also be used to forward SSH connections to the remote network. Returning to the above example of accessing a machine at 10.0.0.2 via a jump host we can make the same connection using the proxy method:

First start the proxy:

ssh -D 127.0.0.1:6000 jump.example.com
Now use the proxy to connect to the machine behind the SSH server:
ssh "-oProxyCommand /usr/bin/nc -X 5 -x 127.0.0.1:6000 %h %p" 10.0.0.2
As with the ProxyJump option above it is easier to put this in your ~/.ssh/config if you need to use it regularly, for example:
Host 10.0.0.2
  ProxyCommand /usr/bin/nc -X 5 -x 127.0.0.1:6000 %h %p
As with TCP forwarding dynamic forwarding can also be used in reverse. From your SSH command you can tell the server to start a SOCKS proxy which allows it or machines behind it to access anything that you can from the client side. This can be achieved with the -R option:
ssh -R 127.0.0.1:6000 [email protected]
This tells the server to start a SOCKS proxy listening on 127.0.0.1:6000 which it can then use to access anything that your client can reach. As with the other forwarding scenarios this can also be extended to allow access from other devices on the remote network by changing the listen address passed to the -R argument.
socks proxy -r

Forwarding without starting a shell session

All of the commands above start a normal shell on the remote server in addition to the tunnels but sometimes you don’t want that or maybe your user does not have shell access. In this case you can use the -N argument to tell SSH not to start a shell. This can also be combined with the -f argument to put SSH in the background so it doesn’t sit there using up a terminal window.

Putting these together we can do something like this:

ssh -f -N -L 8080:127.0.0.1:8080 [email protected]
This command will open a connection to server.example.com, set up a forward for TCP port 8080, then go into the background. The downside to this is that it is a little more complicated to stop the SSH process if you need to, you have to find its PID and use the kill command to tell it to exit.

When starting a tunnel in the background there is a risk that the process will start, connect to the SSH server and then fail to establish the port forward. With a simple command like above this would leave you with an SSH process in the background but the port forward will not work. To make this easier to manage it is often useful to add the ExitOnForwardFailure option to your SSH command (or your ~/.ssh/config file), which tells the SSH client to disconnect and exit if it cannot establish the requested port forward(s). Adding this to the command above gives us:

ssh -oExitOnForwardFailure=yes -f -N -L 8080:127.0.0.1:8080 [email protected]

Keeping sessions alive

In all of the examples above there is a problem that if your connection to the remote machine is dropped or your IP address changes you will have to manually re-establish the tunnels. This can become very annoying if you are on a spotty WiFi connection or if you are travelling.

Unfortunately this is an area that does not have a built-in solution within the SSH client. There are various third-party tools that can manage SSH tunnels and automatically reconnect when needed, however a simple while loop in bash can do the job well enough in many cases, for example:

while true; do
	ssh -N -oExitOnForwardFailure=yes -L 8080:127.0.0.1:8080 [email protected]
	sleep 1
done
This will just keep repeating the SSH command when it exits. The sleep command is there to prevent it from going into a tight loop if the server is offline or if you have no network connection and the SSH client keeps exiting quickly.

Conclusion

As you can see above there are many different ways you can use SSH to create secure tunnels to remote environments but there are more tunneling features built into SSH that were not covered, for example you can forward UNIX sockets, or you can create an ad-hoc VPN with the -w argument, or one of the most common uses is to forward X11 connections with the -X or -Y arguments.

The examples I have shown are just the basics, in reality they barely scratch the surface of what can be done with a bit of creative thinking.

There are many ways these features can be combined or nested to create tunnels and bypass firewall rules for both beneficial and malicious purposes. It is definitely something to bear in mind when designing your network’s security!

Categories

Archives

Related Articles