Introduction

This post evolved out of an update to this long running Fabric feature request that got too long for a Github comment.

The tl;dr for that issue is: folks want gateway support, the ability to bounce SSH connections off an intermediate host. This is useful for setups where you cannot reach your target node directly, but do have access to some gateway or staging box.

(Side note: this feature’s languished because I personally never needed it: ever since I picked up Fabric I’ve worked in environments savvy enough to have VPNs but small enough they didn’t have multiple networks.)

Recently, I reread most of Fabric #38, and there’s two solutions / SSH features contributors kept arriving at: ProxyCommand and direct-tcpip. I’ll explain what these are (insofar as I understand them myself), then compare/contrast.

Background

Currently, Fabric makes SSH connections to targets as follows:

  1. A Paramiko SSHClient object is created.
  2. Its connect() method is called with parameters like username, host and port.
  3. Within connect(), a socket object is created and attached to the target host/port.

Both of the below methods replace the socket object in step 3 with socket-like objects transparently performing the gateway behavior.

direct-tcpip

This first approach uses an SSH feature called a “direct-tcpip” channel. Unlike normal SSH channels used for executing commands on the server (technically, “session” channels), these tell the remote sshd to connect to a second sshd somewhere else, and forward traffic there.

From Paramiko’s perspective, using direct-tcpip only requires the connect socket parameter change; the rest of the work is done at the Fabric (or other Paramiko client library) level. It works like this:

  • A Fabric setting, e.g. env.gateway, is set to the gateway user/host/port string.
  • At connection time, a preliminary SSHClient object is created and its connect() is called with the gateway’s connection info. The socket parameter isn’t used, so connect() makes a normal connection to the gateway with an internal socket object.
  • That gateway SSHClient’s Transport member is asked to open a channel of type direct-tcpip, parameterized with the host/port of the final target server.
  • The resulting Channel object is passed into another SSHClient.connect() call (on a new SSHClient instance) as the sock parameter. Because Channel implements the file/socket interface, this “just works.”
  • This second SSHClient object is then used throughout the Fabric session like a normal, non-gatewayed client would be.

This is an elegant solution to the core problem – bouncing SSH off an intermediate SSH server. Let’s look at the alternative.

ProxyCommand

Generally

ProxyCommand is an SSH config option telling your local client that traffic for a given server or servers should be proxied through a local subprocess instead of using a normal socket connection. For example, adding this to one’s ~/.ssh/config::

ProxyCommand someapp --gateway mygateway --host %h --port %p

Where someapp is a local program capable of connecting to mygateway and, from there, requesting a connection to the “real” target host and port (which ProxyCommand interpolates via %h and %p.)

If that sounds familiar, it should; it’s exactly what direct-tcpip SSH channels do. In fact, the most common “real world” example of ProxyCommand seems to be this::

ProxyCommand ssh mygateway nc %h %p

Which does the following:

  • Creates a local subprocess invoking the ssh client program
  • …which connects to the gateway host mygateway’s sshd
  • …which opens a subprocess on the gateway invoking the netcat program
  • …which transfers all data it receives to/from the destination host/port.

One of ProxyCommand’s strengths is that it’s not limited to aping how direct-tcpip works, and can use literally any program.

In Paramiko/Fabric

A few solutions have been submitted implementing ProxyCommand support in Paramiko and Fabric, with the general approach being:

  • Use subprocess to spin up the local proxying program.
  • Wrap the resulting Popen object in a file/socket compatible interface.
  • Give that Popen-wrapping instance to the connect method’s socket parameter.

Most patches I’ve seen implementing this simply store a Popen object in memory and ensure its wrapper class terminates it when it’s told to close(). (Others leverage paraproxy which is somewhat less elegant.)

Comparison

These approaches entail tradeoffs which are more or less important depending on one’s use case:

  • ProxyCommand creates subprocesses on the local system and the gateway which requires more programming overhead, resource consumption and potential points of failure.
  • ProxyCommand requires the user have a local ssh program, is on a platform compatible with subprocess, and have e.g. netcat on the gateway.
    • While none of these are terrifically burdensome, there’s no reason to lock out users that can’t meet them, and it conflicts with Paramiko’s “pure Python” design goal.
  • direct-tcpip can only be configured/used by Paramiko & Fabric, whereas ProxyCommand directives stored in ssh_config files can be used by both Python and the regular ssh client.
  • direct-tcpip can only utilize SSH protocol connections; networks filtering SSH traffic require using ProxyCommand and programs that proxy over unfiltered protocols, such as HTTP (httptunnel, Corkscrew) or even DNS (iodine).

Given that these solutions don’t completely overlap, I think it makes sense to implement both. Users who don’t need ProxyCommand’s flexibility are better served by direct-tcip’s more efficient & elegant approach, so the latter will be the “default” solution at the Fabric level.

Moving ahead

My specific plan is as follows (building on existing patches in most situations):

  • Update Paramiko’s API so it performs the low level mechanics:
    • Modify paramiko.client.SSHClient.connect() to accept a socket parameter.
    • Create a new class, e.g. paramiko.proxy.ProxyCommand, for wrapping a subprocess in a socket-like interface.
    • Ensure the ssh_config parser understands ProxyCommand (at a parsing, not implementation, level) if it doesn’t already.
  • Update Fabric so it can make use of the Paramiko-level changes:
    • Add an env.gateway option (and command line flag) which is simply a regular host string.
    • Update fabric.network.connect() so it checks for non-empty env.gateway, creates a gateway connection if so, and uses a direct-tcpip channel off that connection in subsequent connect() calls.
      • Cache these gateway connections in the regular connection cache for maximum flexibility/transparency.
    • Further update fabric.network.connect() so it checks for ProxyCommand directives affecting the target host and uses ProxyCommand objects when necessary.
      • In situations where both options are in effect, ProxyCommand wins (though we may find it prudent to print a warning to the user.)

This approach keeps concerns separated, while allowing high level Fabric users to access either solution as required.