This post evolved out of an update to this long running Fabric feature request that got too long for a Github comment.
The tl;dr for that issue is: folks want gateway support, the ability to bounce SSH connections off an intermediate host. This is useful for setups where you cannot reach your target node directly, but do have access to some gateway or staging box.
(Side note: this feature’s languished because I personally never needed it: ever since I picked up Fabric I’ve worked in environments savvy enough to have VPNs but small enough they didn’t have multiple networks.)
Recently, I reread most of Fabric #38, and there’s two solutions / SSH features contributors kept arriving at: ProxyCommand and direct-tcpip. I’ll explain what these are (insofar as I understand them myself), then compare/contrast.
Currently, Fabric makes SSH connections to targets as follows:
- A Paramiko
SSHClientobject is created.
connect()method is called with parameters like username, host and port.
connect(), a socket object is created and attached to the target host/port.
Both of the below methods replace the socket object in step 3 with socket-like objects transparently performing the gateway behavior.
This first approach uses an SSH feature called a “direct-tcpip” channel. Unlike normal SSH channels used for executing commands on the server (technically, “session” channels), these tell the remote sshd to connect to a second sshd somewhere else, and forward traffic there.
From Paramiko’s perspective, using direct-tcpip only requires the
socket parameter change; the rest of the work is done at the Fabric (or other
Paramiko client library) level. It works like this:
- A Fabric setting, e.g.
env.gateway, is set to the gateway user/host/port string.
- At connection time, a preliminary
SSHClientobject is created and its
connect()is called with the gateway’s connection info. The
socketparameter isn’t used, so
connect()makes a normal connection to the gateway with an internal socket object.
- That gateway
Transportmember is asked to open a channel of type
direct-tcpip, parameterized with the host/port of the final target server.
- The resulting
Channelobject is passed into another
SSHClient.connect()call (on a new
SSHClientinstance) as the
Channelimplements the file/socket interface, this “just works.”
- This second
SSHClientobject is then used throughout the Fabric session like a normal, non-gatewayed client would be.
This is an elegant solution to the core problem – bouncing SSH off an intermediate SSH server. Let’s look at the alternative.
ProxyCommand is an SSH config option telling your local client that traffic
for a given server or servers should be proxied through a local subprocess
instead of using a normal socket connection. For example, adding this to one’s
ProxyCommand someapp --gateway mygateway --host %h --port %p
someapp is a local program capable of connecting to
from there, requesting a connection to the “real” target host and port (which
ProxyCommand interpolates via
If that sounds familiar, it should; it’s exactly what direct-tcpip SSH channels
do. In fact, the most common “real world” example of
ProxyCommand seems to be
ProxyCommand ssh mygateway nc %h %p
Which does the following:
- Creates a local subprocess invoking the
- …which connects to the gateway host
- …which opens a subprocess on the gateway invoking the
- …which transfers all data it receives to/from the destination host/port.
ProxyCommand’s strengths is that it’s not limited to aping how
direct-tcpip works, and can use literally any program.
A few solutions have been submitted implementing
ProxyCommand support in
Paramiko and Fabric, with the general approach being:
subprocessto spin up the local proxying program.
- Wrap the resulting
Popenobject in a file/socket compatible interface.
- Give that
Popen-wrapping instance to the
Most patches I’ve seen implementing this simply store a
Popen object in
memory and ensure its wrapper class terminates it when it’s told to
(Others leverage paraproxy which is somewhat less elegant.)
These approaches entail tradeoffs which are more or less important depending on one’s use case:
ProxyCommandcreates subprocesses on the local system and the gateway which requires more programming overhead, resource consumption and potential points of failure.
ProxyCommandrequires the user have a local
sshprogram, is on a platform compatible with
subprocess, and have e.g.
netcaton the gateway.
- While none of these are terrifically burdensome, there’s no reason to lock out users that can’t meet them, and it conflicts with Paramiko’s “pure Python” design goal.
- direct-tcpip can only be configured/used by Paramiko & Fabric, whereas
ProxyCommanddirectives stored in
ssh_configfiles can be used by both Python and the regular
- direct-tcpip can only utilize SSH protocol connections; networks filtering
SSH traffic require using
ProxyCommandand programs that proxy over unfiltered protocols, such as HTTP (
httptunnel, Corkscrew) or even DNS (iodine).
Given that these solutions don’t completely overlap, I think it makes sense to
implement both. Users who don’t need
ProxyCommand’s flexibility are better
served by direct-tcip’s more efficient & elegant approach, so the latter will
be the “default” solution at the Fabric level.
My specific plan is as follows (building on existing patches in most situations):
- Update Paramiko’s API so it performs the low level mechanics:
paramiko.client.SSHClient.connect()to accept a
- Create a new class, e.g.
paramiko.proxy.ProxyCommand, for wrapping a subprocess in a socket-like interface.
- Ensure the
ProxyCommand(at a parsing, not implementation, level) if it doesn’t already.
- Update Fabric so it can make use of the Paramiko-level changes:
- Add an
env.gatewayoption (and command line flag) which is simply a regular host string.
fabric.network.connect()so it checks for non-empty
env.gateway, creates a gateway connection if so, and uses a
direct-tcpipchannel off that connection in subsequent
- Cache these gateway connections in the regular connection cache for maximum flexibility/transparency.
- Further update
fabric.network.connect()so it checks for
ProxyCommanddirectives affecting the target host and uses
ProxyCommandobjects when necessary.
- In situations where both options are in effect,
ProxyCommandwins (though we may find it prudent to print a warning to the user.)
- In situations where both options are in effect,
- Add an
This approach keeps concerns separated, while allowing high level Fabric users to access either solution as required.