FreeBSD. Installing and Configuring supervisor

Supervisor is a system for managing processes in an operating system. For example, you may have programs that need to be restarted according to certain rules. Rather than writing a management subsystem for them (rc scripts, a monitoring and restart system), you can use supervisor.

It launches processes as its subprocesses, so it has full control over them and knows their exact state.

supervisorctl provides system and web interfaces for monitoring and managing processes. You can give users access to specific programs, and they, in turn, will be able to see the state of those programs and perform actions on them (start, stop, restart). There is also an XML-RPC interface that you can use to write your own extensions and applications that need access to supervisord.

You can group your programs and perform actions on them collectively (for example, restarting all programs). You can also specify a priority for each program, thereby establishing the restart order.

This wonderful utility is written in Python and runs on Linux, Mac OS X, Solaris, and FreeBSD.
This article covers version 3.0b1.

Installation

To install it, simply run:

pip install supervisor

FreeBSD has a port, sysutils/py-supervisor, but the version of supervisor there is older than the one in PyPI. If you use the port anyway, you do not need to run the command echo_supervisord_conf, since the port will copy supervisord.conf to the right place itself.

The port also includes an initialization script, sysutils/py-supervisor/files/supervisord.in. It is less functional than my fork (which I will try to add to the port itself soon), so I recommend using my fork right away:

mkdir /var/run/supervisor
fetch -o /usr/local/etc/rc.d/supervisord https://raw.github.com/gist/3809678/supervisord
chmod +x /usr/local/etc/rc.d/supervisord

Having an initialization script will let you start supervisor automatically when the operating system starts. Add the following to /etc/rc.conf:

# Обязательно:
supervisord_enable="YES"
# По желанию:
supervisord_conffile="/usr/local/etc/supervisord.conf" # путь до конфигурационного файла
supervisord_pidfile="/var/run/supervisor/supervisord.pid" # путь до pid-файла
supervisord_user="supervisor"
supervisord_group="supervisor"
supervisord_logdir="/var/log/supervisord"  # скрипт создаст каталог для логов, если его нет

Configuration

First, we need to generate a configuration file and set the correct paths to the support files:

mkdir /var/run/supervisor
echo_supervisord_conf | sed -r \
-e "s#/tmp/supervisord.log#/var/log/supervisord.log#" \
-e "s#/tmp/supervisor(d*)\.(.*)#/var/run/supervisor/supervisor\1.\2#" \
> /usr/local/etc/supervisord.conf

Next, open the configuration file we generated, /usr/local/etc/supervisord.conf, in an editor and edit the unix_http_server section:

[unix_http_server]
; Путь до сокета
file=/var/run/supervisor/supervisor.sock
; Установите нужные права для файла сокета, иначе другие пользователи не смогут использовать supervisor
chmod=0770
; Аналогично предыдущему, например "supervisor:supervisor"
chown=<uid>:<gid>
; Авторизация в supervisord укажите имя пользователя
username=<username>
; Авторизация в supervisordl: укажите пароль
password=<password>

An example for Ubuntu Linux:

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0770                       ; sockef file mode (default 0700)
chown=root:supervisor

Don’t forget to create the group and add the user we need to it:

sudo groupadd supervisor
sudo usermod -a -G supervisor <USERNAME>

If you specified the user and group supervisor:supervisor in unix_http_server.chown, you should add them to the system:

pw groupadd supervisor
pw useradd supervisor -G supervisor

If you want to work with supervisorctl as another user, you should add that user to the supervisor group, for example like this:

pw usermod <username> -G wheel,supervisor

since we have allowed the group to access the socket; see unix_http_server.chmod.

If you do not want to enter a username and password when logging into supervisorctl, specify them directly in the supervisorctl section:

[supervisorctl]
; Путь до сокета
serverurl=unix:///var/run/supervisor/supervisor.sock
; Имя пользователя из unix_http_server.username
username=<username>
; Пароль из unix_http_server.password
password=<password>

Now add instructions for including additional configuration files at the very end of the file /usr/local/etc/supervisord.conf:

[include]
; Маска для подключаемых конфигурационных файлов
files = /usr/local/etc/supervisor/*.conf

Example Program Configuration

Create a file, /usr/local/etc/supervisor/<project>.conf, that will contain sections for the programs in your project.

As an example, let’s create one section with the program <project>_uwsgi, containing instructions for your uWSGI:

[program:<project>_uwsgi]
; Имя программы в supervisor, например будет выводится в supervisorctl
process_name=%(program_name)s
; Вы можете указать сколько таких процессов надо запустить, по умолчанию 1
numprocs=1
; Путь до проекта (chdir)
directory=/<project>
; Команда для запуска программы
command=/<project>/venv/bin/uwsgi /<project>/uwsgi/production.ini
; Из под какого пользователя запускать программу
user=<username>
; При загрузке самого supervisor запускать программу
autostart=true
; Если программа аварийно завершилась, то перезапускать её
autorestart=true
; Перенаправляет пришедший STDERR в ответ supervisor'у в STDOUT (эквивалент /the/program 2>&1)
redirect_stderr=true
; Таймаут в секундах, после которого supervisor пошлет SIGKILL процессу,
; которому до этого посылал SIGCHLD
stopwaitsecs=60
; Какой сигнал посылать для остановки программы
stopsignal=INT
; Путь до error-лога
stderr_logfile=/var/log/<project>/wsgi_err.log
; Путь до output-лога
stdout_logfile=/var/log/<project>/wsgi_out.log
; Максимальный размер файла output-лога, после чего будет "rotate"
stdout_logfile_maxbytes=100MB
; Количество файлов output-лога
stdout_logfile_backups=30
; Размер буфера для output-лога
stdout_capture_maxbytes=1MB

For additional options in this section, see here. You can find example sections here and here.

To avoid writing out the full paths to python/pip/uwsgi, you can set an environment variable:

[program:adw0rd_uwsgi]
environment=PATH="/path/to/project/venv/bin"
directory=/path/to/project
command=uwsgi uwsgi/production.ini
...

The thing is that source venv/bin/activate updates $PATH with the absolute path to venv/bin, while we change $PATH ourselves. This is described here. Taken from here: http://stackoverflow.com/questions/6532486/supervising-virutalenv-django-app-via-supervisor

Usage

To start the supervisord daemon, run:

/usr/local/etc/rc.d/supervisord start

Now run the command supervisorctl in the console, and you will see the following:

supervisor> status
<project>_uwsgi                   RUNNING    pid 54020, uptime 0:01:11

where <project>_uwsgi is the name of your program, and RUNNING is its current state.

Alternatively, you can invoke supervisorctl like this:

supervisorctl status <project>_uwsgi
supervisorctl restart <project>_uwsgi

The following program process states are possible:

• STOPPED — the process was stopped normally, i.e., it did not crash;
• STARTING — the process is starting;
• RUNNING — the process has started/is running;
• BACKOFF — the process was starting but failed to start (supervisor restarts such processes. If it cannot start the process after some time, the program’s state is set to FATAL. Read about the startretries option on this page);
• STOPPING — the process is stopping;
• EXITED — the process has exited (it will be started automatically if autorestart=true is set);
• FATAL — the process could not start (most likely there are errors in the program’s configuration file);
• UNKNOWN — an unknown process state (“supervisord programming error”).

Possible process state transitions:

Next, enter help, and you will see a list of available commands:

supervisor> help

default commands (type help ):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update 
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

I put together a small table of all the commands:

Events in supervisor

supervisor also has an event mechanism that lets you catch events and notify someone about them. An example:

[eventlistener:memmon]
command=memmon -a 200MB -m bob@example.com
events=TICK_60

Read more here.

Third-Party Applications and Plugins for supervisor

On the official PLUGINS.rst page, you can find interesting plugins as well as external applications that integrate with supervisor.

Here are some of them:

• To monitor and control processes, for example their memory usage, you can use superlance.
• supervisor-wildcards can help you with name completion and running process commands in parallel.
• For integration with Nginx, you may find ngx_supervisord useful.
• If you use Nagios, you may need Supervisord-Nagios-Plugin.
• If you use Django, you will find it convenient to install the django-supervisor application, which lets you retrieve information about and manage processes in supervisor.
• If you use Fabric, you may be interested in trying fabtools/supervisor.py.