Debian. The Jenkins Continuous Integration Server for Continuous Testing of Django Projects · English

Jenkins logo

Today I’ll explain how to implement continuous testing for a Django project using Jenkins CI. We’ll also collect and display various test reports; to do that, we’ll install and configure some additional plugins. The next articles will cover how to deploy Jenkins nodes on FreeBSD and how to run BDD tests with Selenium WebDriver support through Lettuce and splinter.

We’ll use git to supply the code under test, with ssh as its transport (I use gitolite for this), and the server address will be git.example.org. The address of the service running Jenkins will be http://jenkins.example.org/.

I’ve already mentioned Jenkins before, along with Atlassian Bamboo and Hudson, but for my own projects I chose Jenkins for its simplicity, functionality, number of useful plugins, and success stories among Django projects. As for Hudson, Jenkins is a fork of it; you can read about the reasons for the split in an article on Habr.

By the way, SaaS fans can skip installing Jenkins and use ShiningPanda instead.

Terminology

In Jenkins terminology, Job and Project mean the same thing; we’ll simply call them “jobs.”

The term Build refers to the result of running a job. It can end in one of three states:

SUCCESS (a successful build, with no errors throughout the job’s execution);
FAILED (some error occurred);
ABORTED (the build was interrupted for some reason, usually manually).

The term View (specifically, “List View”) refers to a page listing jobs. You configure it either by selecting jobs using checkboxes or by matching job names against a specified pattern.

As you read this article, you’ll encounter these environment variables:

$JENKINS_HOME — the path to the Jenkins home directory, most likely “/var/lib/jenkins”. This is where all Jenkins configuration files, jobs, builds, and so on will be stored—in short, everything related to Jenkins.
$WORKSPACE — the path to the directory in which the job runs;

Installing Jenkins on Debian

Add the key and repository:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'

Update and install:

sudo apt-get update
sudo apt-get install openjdk-8-jre jenkins

These instructions were taken from here.

Configuring Jenkins

Open /etc/default/jenkins for editing and add the following directives:

# Указываем нужный порт,
# на который далее мы будем проксировать через Nginx
PORT=8090

# Устанавливаем нужную нам кодировку,
# чтобы текст, который попал в stdout (например из тестов) выводился корректно.
# А также устанавливаем нужную таймзону, у меня это UTC
JAVA_ARGS="-Dfile.encoding=UTF8 -Duser.timezone=UTC"

Now let’s start Jenkins:

sudo service jenkins start

After starting Jenkins, you can see the current encoding and time zone in the "Manage Jenkins" > "System Information" section (direct address: http://jenkins.example.com:8090/systemInfo ).

Now let’s configure the basic Jenkins settings. Go to "Manage Jenkins" > "Configure System":

In the Jenkins Location section, specify the following:
- In the Jenkins URL field, enter the address https://jenkins.example.org/;
- In the System Admin E-mail Address field, enter the email address that Jenkins emails will come from, for example “jenkins@example.org”.
Next, we need to configure email notifications. Go to the E-mail Notification section and configure SMTP:
- In the SMTP server field, enter “localhost” or another mail server serving your server. If you’re using a local sendmail, install it.
In the Git plugin section:
- In the Global Config user.name Value field, enter “Jenkins”;
- In the Global Config user.email Value field, enter “jenkins@example.org”.

Setting Up Proxying Through Nginx

Here’s a sample Nginx configuration. We’ll use a self-signed SSL certificate and add base-auth:

server {
    listen 443 ssl;
    server_name jenkins.example.org;

    ssl_protocols SSLv3 TLSv1;
    ssl_certificate /etc/nginx/certs/jenkins.pem;
    ssl_certificate_key /etc/nginx/certs/jenkins.key;

    location / {
        auth_basic "Password, please!";
        auth_basic_user_file /etc/nginx/auth-file;
        proxy_read_timeout 300;
        proxy_send_timeout 300;
        proxy_pass http://localhost:8090;
    }
}
server {
    listen 80;
    server_name jenkins.example.org;
    return 301 https://jenkins.example.org$request_uri;
}

Let’s generate a username/password pair for “base-auth” authentication:

htpasswd -cb /etc/nginx/auth-file jenkins MY_SECRET_PASSWORD

Let’s generate a self-signed SSL certificate to encrypt traffic:

mkdir /etc/nginx/certs
cd /etc/nginx/certs
openssl req -new -x509 -nodes -newkey rsa:1024 -days 3650 -keyout jenkins.key -out jenkins.pem \
-subj "/C=RU/ST=/L=/O=/OU=/CN=jenkins.example.org/emailAddress=jenkins@example.org"

Restart Nginx to apply the settings:

nginx -s reload

Don’t forget to hide port 8090 behind the firewall:

iptables -t filter -A INPUT -i lo -p tcp --dport 8090 -j ACCEPT
iptables -t filter -A INPUT -i eth0 -p tcp --dport 8090 -j DROP

Here, “eth0” is the external network interface.

Now let’s save our rules to “/etc/iptables.rules”: iptables-save > /etc/iptables.rulesAnd add them to startup:
echo "pre-up iptables-restore < /etc/iptables.rules" >> /etc/network/interfaces
To be more precise, you need to add pre-up to your interface’s section, but since the interface we need is usually defined at the end of the file, the recommendation above will be correct. A more detailed example:
auto eth0
iface eth0 inet static
        hwaddress aa:bb:cc:dd:ee:ff
        address   203.0.113.42
        netmask   255.255.240.0
        gateway   203.0.113.1
        pre-up iptables-restore < /etc/iptables.rules

Installing Plugins

Go to "Manage Jenkins" > "Manage Plugins" > "Available" and select:

GIT plugin, which will also install the Git Client Plugin—since we’re using git, we obviously need a plugin for working with this SCM;

If you’re having problems with GIT plugin, and its version is 1.3.0 while the GIT client plugin version is 1.0.4, read the comments on bug JENKINS-17204. There’s a comment of mine there that will definitely help you. In short, there’s a bug in the new version of the plugin, and you just need to roll back one version.

AnsiColor Plugin (a fork and the more up-to-date original on GitHub)—lets you view the colored logs generated by your program (for example, harvest has a verbose mode with colored output, and so on);

In the job settings, check Color ANSI Console Output and select the appropriate ANSI color map; mine is “xterm”.

Timestamper—will display the current time on every line of console log output;

In the job settings, check Add timestamps to the Console Output.

xUnit Plugin—if your tests can generate reports in xUnit format (and Lettuce and django-jenkins can), you’ll be able to display test results in a human-friendly form (graphs and tables), like this, for example:

Test execution graph

Build-timeout Plugin—will abort hung jobs in Jenkins;
Cron Column Plugin—displays information about configured job schedules in your job list.

Optionally, you can add the following plugins:

Cobertura Plugin—for displaying a test coverage report (code coverage);
Violations—for displaying reports from pylint, pyflakes, and pep8.

Violations

Configuring the git Plugin

sudo su - jenkins
mkdir ~/.ssh && cd ~/.ssh
emacs config

Put the following lines in the file config:

Host git.example.org
    HostName       git.example.org
    User           git
    IdentityFile   ~/.ssh/git_example_org
    IdentitiesOnly yes

Now let’s create authentication keys:

ssh-keygen -t rsa

In gitolite (or GitHub, Assembla, etc., depending on what you use), add the generated public key ~/.ssh/git_example_org.pub.

While we’re on the server, it wouldn’t hurt to specify a user for git:

git config --global user.email "jenkins@example.org"
git config --global user.name "Jenkins"

Check that the ssh keys and settings work:

git clone git@git.example.org:repo-name.git /tmp/test

Installing and Configuring django-jenkins

This application simplifies integration with Jenkins CI; we’ll use it to generate the reports we need. Create a separate “settings” file for it in the project, for example project/settings/jenkins.py:

import settings

# Добавим это приложение в INSTALLED_APPS
INSTALLED_APPS = settings.INSTALLED_APPS + (
    'django_jenkins',
)

# Укажем нужные нам ранеры
JENKINS_TASKS = (
    # 'django_jenkins.tasks.with_coverage',
    # 'django_jenkins.tasks.django_tests',
    'django_jenkins.tasks.dir_tests',
    'django_jenkins.tasks.run_pep8',
    'django_jenkins.tasks.run_pyflakes',
    'django_jenkins.tasks.run_pylint',
    # 'django_jenkins.tasks.run_jslint',
    # 'django_jenkins.tasks.run_csslint',
    # 'django_jenkins.tasks.run_sloccount',
    # 'django_jenkins.tasks.lettuce_tests',
)

# И какие приложения нужно тестировать
PROJECT_APPS = (
    'app1',
    'app2',
)

Also, when calling the manage.py jenkins command, you can pass additional options to the runners specified in JENKINS_TASKS; see manage.py jenkins --help for this.

Now let’s install django-jenkins in the project, along with the dependencies we specified:

echo "
django-jenkins
pep8
pyflakes
pylint
" >> requirements/tests.txt
pip install --upgrade -r requirements/tests.txt

The original project repository is on GitHub, and you can install it from PyPI. I also recommend reading the tutorial.

Adding and Configuring a Job/Project

To add a job, click “New Job” and select “Build a free-style software project”.

In the “Source Code Management” section, check “Git Repositories”. Enter git@git.example.org:repo-name.git in “Repository URL”, and specify the branch you need, for example “master”, in “Branches to build”.

Now let’s configure "Poll SCM" > "Schedule" (we’ll check the repository for changes every 2 minutes; if there are changes, a new build will start):

*/2 * * * *

Add a section for running tests, "Build" > "Add build step" > "Execute shell":

# Создаем каталог виртуального окружения
virtualenv --no-site-packages --distribute venv
# Сделаем symlink на файл настроек для django_jenkins
ln -fs $WORKSPACE/project/settings/jenkins.py $WORKSPACE/project/local_settings.py
# Устанавливаем зависимости для проекта и тестов
venv/bin/pip install --upgrade -r requirements/base.txt
venv/bin/pip install --upgrade -r requirements/tests.txt
# Запускаем тесты с использованием django_jenkins
venv/bin/python ./manage.py jenkins --verbosity=2 --output-dir=reports

I usually put all of this into an sh script and add it to the repository, then simply specify that script to run.

You can now run the job manually by clicking “Build Now”. Alternatively, add a commit to the repository, and the job will start automatically in 2 minutes.

After running the tests, you should have a $WORKSPACE/reports directory containing various reports (xUnit, pep8, pylint, etc.).

Backing Up Jenkins

I honestly tried using the thinBackup backup plugin, but it has a huge problem: it stops Jenkins to make a backup.

In the end, I gave up on it and wrote a simple script that backs up only the job configurations; I don’t need anything else.

cd /var/lib/jenkins/jobs
for JOB in *; do
    BACKUP_JOB_DIR="/var/lib/jenkins/Backups/configs/$JOB"
    mkdir -p $BACKUP_JOB_DIR
    cp "$JOB/config.xml" "$BACKUP_JOB_DIR/config.xml";
    # git commit =am "Autocommit (`hostname`): `date`"
done

Just create another job that will periodically back up the configurations.

Finally

Below the search bar is the ENABLE AUTO REFRESH link. Click it to refresh the page automatically and make Jenkins more interactive.

I recommend reading:

Django and Continuous Integration—discusses Jenkins CI, Shining Panda, and Travis CI.
Setting Up Jenkins for a Django Project from Scratch — Habrahabr
HowTo: Continuous Integration for Django in Jenkins Using Selenium — Habrahabr
A django-jenkins tutorial