Running SSH commands on multiple servers

Question

At the company I work at, we have no real way to quickly execute shell commands on all of our customers systems, which often results in what we call Tiny Tim tasks, where the list of all of our systems is split between the 5 devs we have, and we simply go through one-by-one, SSHing into each one and executing the necessary commands.

To streamline this process, I created a script in Python3 using Paramiko for SSH. The script takes the path to a private OpenSSH key from command-line args, reads a list of hosts from a file called hosts.txt (lines in this file can be commented by prepending them with --), and a list of shell commands from a file called cmds.txt. Due to the way we set up our customers' systems, each domain has 2 servers, so the script loops over each host in the list twice, prepending disc1. and disc2. respectively to send the commands to both servers.

from paramiko.client import AutoAddPolicy
from paramiko.client import SSHClient
import sys
from sys import argv

if len(argv) < 2:
    print("Invalid args!")
    print("Usage: python3 run.py <private ssh key filepath>")
    exit()

with open('cmds.txt') as c:
    cmds = c.readlines()

with open('hosts.txt') as f:
    lines = f.readlines()

for line in lines:
    for i in range(1, 3):
        line = line.strip()
        host = "disc%s.%s" % (i, line)
        if line.startswith('--'):
            print("Host %s began with --, skipping" % line)
            continue
        client = SSHClient()
        client.set_missing_host_key_policy(AutoAddPolicy)
        client.load_system_host_keys()
        try:
            print("[%s]> Connecting as root on port 22..." % host)
            client.connect(host, 22, 'root', None, None, argv[1])
        except KeyboardInterrupt:
            break
        except Exception as err:
            print("[%s]> Connection failed: %s" % (host, err))
            continue
        for c in cmds:
            print("[%s]> Executing %s..." % (host, c.strip()))
            stdin, stdout, stderr = client.exec_command(c)
            print(stdout.read().decode('utf-8'))
            print(stderr.read().decode('utf-8'))
        client.close()
        print("[%s]> Done" % host)

I ran it through PEP8 and surprisingly didn't get any errors or warnings, but it's still quite messy and I'd appreciate feedback.

Answer 1

Looks like Fabric is what you need.

Fabric is a pretty simple deploy tool (comparing to Ansible/Puppet/etc), that lets you execute simple shell commands over SSH (Paramiko inside).

Features you will get from it:

1. You can specify SSH key path using command line:

   fab command -i /path/to/key.pem
   

   or parse it from command line:

   import sys
   from fabric.api import *
   ...
   env.key_filename = sys.argv[2] or '/path/to/keyfile.pem'
   

2. Describe local/remote tasks in pythonic way: from fabric.api import *

   def local_test():
       local('nosetests .')
   
   def remote_pull():
       run('git pull')
   

   And chain/group/do-anything-you want to them.

3. You can describe host list:

   hosts = [<your_hosts_list>]
   env.hosts = ['disc{}.{}'.format(host, index) for host in hosts for index in xrange(1,3)]
   

   or you can describe roles:

   env.roledefs['group1'] = ['host.name.com']
   env.roledefs['group2'] = ['host1.name.com', 'host2.name.com']
   

   and use this roles/hosts in your tasks configuration then:

   @roles('group1')
   @runs_once
   def add_tag(version):
       run('git tag -f {}'.format(version))
   

And many more features out of the box!

Answer 2

Overall, the code is not complicated and can be understood. I would though focus on improving variable names and modularity:

• remember that the code is much more often read than written - try to name your variables in a meaningful way:
  • for line in lines should be for host in hosts
  • for i in range(1, 3) can be for customer_index in range(1, 3)
  • cmds may be renamed to a more explicit commands
  • c should probably be command
• I would apply the "Extract Method" refactoring method to move all SSH client related logic to a separate function

Other notes:

• SSHClient can be used as a context manager
• remove unused import sys import
• organize import as per PEP8 guidelines
• put your main execution logic to under if __name__ == '__main__': to avoid the code to be executed on import

Improved code:

from sys import argv

from paramiko.client import AutoAddPolicy
from paramiko.client import SSHClient
from paramiko.ssh_exception import SSHException


def apply_commands(host, commands):
    with SSHClient() as client:
        client.set_missing_host_key_policy(AutoAddPolicy)
        client.load_system_host_keys()

        print("[%s]> Connecting as root on port 22..." % host)
        client.connect(host, 22, 'root', None, None, argv[1])

        for command in commands:
            print("[%s]> Executing %s..." % (host, command.strip()))

            stdin, stdout, stderr = client.exec_command(command)

            print(stdout.read().decode('utf-8'))
            print(stderr.read().decode('utf-8'))

    print("[%s]> Done" % host)


if __name__ == '__main__':
    if len(argv) < 2:
        print("Invalid args!")
        print("Usage: python3 run.py <private ssh key filepath>")
        exit()

    with open('cmds.txt') as f:
        commands = f.readlines()

    with open('hosts.txt') as f:
        hosts = f.readlines()

    for host in hosts:
        for customer_index in range(1, 3):
            host = host.strip()
            host = "disc%s.%s" % (customer_index, host)

            if host.startswith('--'):
                print("Host %s began with --, skipping" % host)
                continue

            try:
                apply_commands(host, commands)
            except KeyboardInterrupt:
                break
            except SSHException as err:
                print("[%s]> SSH Error: %s" % (host, err))
                continue