adw0rd.
RUEN
← All posts
·7 min read

Python. Automating Deployment and Routine Tasks with Fabric

DeployDjangoFabricPython

Fabric logo

Fabric (which I’ll simply call “Fabric” from here on) is a Swiss Army knife for deploying Python applications, with a huge number of tools for the job. You no longer have to connect over SSH to update a project, start the web server, or run “compilemessages” or “collectstatic”: now you can describe all this routine work in your Fabric command file.

Fabric also has the concept of roles (essentially groups of servers). For example, you can create “production” or “stage” roles, then specify these roles when running fab, or directly in the file using the “@roles” decorator, and your commands will be applied directly to those groups of servers. Alternatively, you can specify individual hosts instead of a group. More on that below…

This article covers Fabric version 1.4.3.

Installation

To install it, just install the package of the same name, Fabric:

pip install Fabric

Setting Up fabfile

To create your own project deployment commands and helper functions (creating backups, working with version control, running any commands remotely or locally, and so on), you need to create a file called fabfile.py in the root of your Django project, with the following contents, for example:

# coding: utf-8
import os
from fabric.api import run, env, cd, roles

# Списком можно перечислить несколько серверов, которые у вас считаются "продакшеном"
env.roledefs['production'] = ['git@example.org:2244']

def production_env():
    """Окружение для продакшена"""
    env.key_filename = [os.path.join(os.environ['HOME'], '.ssh', 'git_example_org')]  # Локальный путь до файла с ключами
    env.user = 'git'  # На сервере будем работать из под пользователя "git"
    env.project_root = '/home/username/work/project'  # Путь до каталога проекта (на сервере)
    env.shell = '/usr/local/bin/bash -c'  # Используем шелл отличный от умолчательного (на сервере)
    env.python = '/home/username/work/project/venv/bin/python'  # Путь до python (на сервере)


@roles('production')
def deploy():
    production_env()  # Инициализация окружения
    with cd(env.project_root):  # Заходим в директорию с проектом на сервере
        run('git pull origin master')  # Пуляемся из репозитория
        run('find . -name "*.mo" -print -delete')  # Чистим старые скомпиленные файлы gettext'а
        run('{} manage.py compilemessages'.format(env.python))  # Собираем новые файлы gettext'а
        run('{} manage.py collectstatic --noinput'.format(env.python))  # Собираем статику

@roles('production')
def pip_install():
    production_env()
    run('{pip} install --upgrade -r {filepath}'.format(pip=env.pip,
        filepath=os.path.join(env.project_root, 'requirements.txt')))

Now let’s create the “git_example_org” private and public keys for connecting to the server:

$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa): /home/username/.ssh/git_example_org
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/username/.ssh/git_example_org.
Your public key has been saved in /home/username/.ssh/git_example_org.pub.
...

You can supply a password instead of keys, either entering it each time you connect to the server or specifying it directly in the fabfile.py file:

env.password = "some_password"

Usage

Now that we’ve created fabfile.py and configured the SSH connection, we can go to the project directory and run fab deploy (or any other command you’ve defined):

$ cd ~/work/project && source venv/bin/activate
$ fab deploy another_command

You can also specify particular roles or hosts on which to execute a command, such as “deploy”:

$ fab -R production,stage deploy
$ fab -H master.example.org,slave.example.org deploy

# А вот так можно исключить хост из группы во время запуска:
$ fab -R stage -x git@stage4.example.net,git@stage5.example.net deploy

You can run arbitrary commands (for example, “uname -a”):

$ fab -R production -- uname -a
$ fab -H git@example.org:2244 -- uname -a

# Можно передать ssh-ключ:
$ fab -H git@example.org:2244 -i ~/.ssh/git_example_org -- uname -a

# Или пароль:
$ fab -H git@example.org:2244 -p PASSWORD -- uname -a

To display all the commands (you’ve defined) in fabfile.py, run:

$ fab --list

Basically, all the options are described in the documentation.

Passing Arguments

It’s very simple: you define args/kwargs in your command:

def deploy(arg1, arg2, kwarg1='something', kwarg2=True):
    pass

Then you can pass those arguments from the console like this:

fab deploy:'arg1',42,kwarg1=sometext,kwarg2='text with spaces'

More About Fabric’s Basic Functionality

I’ll only cover the functions I use:

cd

Used to specify the directory to change to; “/bin/cd” will be called before every command run within the context. Example:

def something():
    with cd('/var/www'):
        run('ls') # cd /var/www && ls
    with cd('website1'):
        run('ls') # cd /var/www/website1 && ls

lcd

Like “cd”, but local only. Example:

def something():
    with lcd('/tmp'):
        local('ls -la')

run

run(command, shell=True, pty=True, combine_stderr=True)

Runs any command on the server that is available to the user used to connect to the remote machine. Example:

def something():
    run('cat /etc/issue')

$ fab something
>>> [git@example.org:2244] run: cat /etc/issue
>>> [git@example.org:2244] out: Debian GNU/Linux 6.0 \n \l

local

local(command, capture=False)

Runs any command on the local machine. Example:

def something():
    local('cat /etc/issue')

$ fab something
>>> [localhost] local: cat ssue
>>> Ubuntu 12.04 LTS \n \l

sudo

sudo(command, shell=True, pty=True, combine_stderr=True, user=None)

Runs a command under sudo. Example:

def something():
    sudo('service mysql restart', user="mysql")

open_shell

Opens a shell on the remote host. Example:

def something():
    # Что-то делаем...
    open_shell()  # Запускаем shell, а выходим по "exit" или "Ctrl+D"
    # После выхода из шелла продолжаем!

abort

Aborts command execution. Example:

from fabric.api import abort
from fabric.contrib.files import exists

def something():
    if not exists('/tmp/super.sock'):
        abort('Alarm!')

warn

Displays a warning message without aborting command execution. Example:

def something():
    warn("Something wrong!")

puts

Displays a message, much like print(), but works through Fabric’s output. Example:

def something():
    puts("Don't worry, be happy!")

env

An environment variable to which you can add the attributes you need and use them as necessary. There was already an example of its use above:

...
def production_env():
    """Окружение для продакшена"""
    env.key_filename = [os.path.join(os.environ['HOME'], '.ssh', 'git_example_org')]  # Локальный путь до файла с ключами
    env.user = 'git'  # Будем работать из под пользователя "git"
    env.project_root = '/home/username/work/project'  # Путь до каталога проекта
    env.shell = '/usr/local/bin/bash -c'  # Используем шелл отличный от умолчательного
    env.python = '/home/username/work/project/venv/bin/python'  # Путь до python

@roles('production')
def something():
    production_env()  # Инициализация окружения
    # Используем проинициализированный "env"
    run('{python} -V'.format(python=env.python))

$ fab something
>>> [git@example.org:2244] run: /home/username/work/project/venv/bin/python -V
>>> [git@example.org:2244] out: Python 2.8

settings

You can use settings to execute functions with the specified settings. In other words, you could set “env.warn_only=True” through env, but that would apply globally, whereas this applies only within the “with” context. Example:

def something():
    with settings(warn_only=True):
        run('test -e {}'.format(path))  # Даже если произойдет ошибка, warn_only не дас прерваться фабрике

@with_settings

A decorator that works like “settings”, but affects the entire command. Example:

@with_settings(warn_only=True)
def something():
    run('test -e {}'.format(path))

@roles

A decorator that lets you specify the group of hosts on which your command will run. Example:

from fabric.api import env, roles, run

env.roledefs['production'] = ['git@example.org:2244']
env.roledefs['stage'] = ['git@198.51.100.42', 'git@example.net', 'git@stage4.example.net', 'git@stage5.example.net']

@roles('stage')
def something():
    run('ls /var/log')

@hosts

A decorator that lets you specify a list of hosts on which your command will run. Example:

from fabric.api import hosts, run

@hosts('git@example.org:2244', 'git@example.com')
def something():
    run('ls /var/log')

@task

A decorator indicating that a function is a command. If you don’t use this decorator at all in fabfile.py, all functions will be commands. If you specify it for even one function, you’ll have to specify it for all commands.

It accepts a number of parameters, such as “alias”, for giving commands shorter names:

from fabric.api import task

@task(alias='boom')
def deploy_with_migrations():
    pass

get

get(filepath)

Retrieves (downloads) a file from a remote host. Example:

def something():
    get('/path/to/somefile.jpg')

$ fab something
>>> [git@example.org:2244] download: /home/username/git@example.org-2244/somefile.jpg <- /path/to/somefile.jpg

put

put(filepath)

Uploads a file to a remote host. Example:

def something():
    put('/path/to/somefile.txt', '/tmp/file.txt')
    # или так
    with cd('/tmp'):
        put('/path/to/somefile.txt', 'file.txt')

@parallel

A decorator for running a command in parallel. Example:

@parallel(pool_size=10)
def something():
    pass

@serial

A decorator that does the opposite. Example:

@serial
def something():
    pass

prefix

prefix(command)

The command passed as “command” will be executed for all run/sudo calls.
For example, for virtualenv:

def something():
    with prefix("source /path/to/venv/bin/activate"):
        run('./manage.py syncdb')

The result will be:

$ source /path/to/venv/bin/activate && ./manage.py syncdb

prompt

prompt(sometext)

Similar to Python’s raw_input(): it prompts the user for input, after which you can add logic based on the user’s response. Example:

def something():
    response = prompt('You are sure?')
    if response == "Yes":
        run('rm -rf /')

reboot

Reboots the machines for the specified hosts. Example:

def something():
    reboot(wait=120)

Fabric comes with batteries included:

  • fabric.contrib.console
  • fabric.contrib.django
  • fabric.contrib.files
  • fabric.contrib.project

For example, the last one includes “rsync_project”, which you can use like this:

from fabric.contrib.project import rsync_project

env.roledefs['production'] = ['git@example.org:2244']

def production_env():
    env.directory = '/home/username/work/project/'
    env.rsync_excludes = ['*.pyc' , '*.db', '*~', ...]

@roles('production')
def rsync():
    production_env()
    rsync_project(env.directory, 'files', exclude=env.rsync_excludes)

Recommendations

To extend the standard API, I recommend taking a look at the https://github.com/ronnix/fabtools project. There you’ll find functionality for working with Supervisor, PostgreSQL, MySQL, Deb, Crontab, VirtualEnv, Nginx, Redis, Postfix, and so on.

I also recommend checking out Chef, Puppet, and Buildout.

Comments are closed. Read the original discussion in Russian →