Django. Getting to Know ElasticSearch · English

I needed to install an FTS service for one of my projects. I’d been using SphinxSearch for quite a long time, but decided to ask the community (here and here) which tool was currently more popular and met the following requirements:

Russian morphology and stemming
Faceted search (attributes)
A convenient API for searching and indexing
Easy scaling
Preferably real-time indexes
Preferably an admin interface for analyzing and managing indexes

Most of my colleagues use ElasticSearch. After reading a few articles about it and watching a few presentation videos, I decided to try version two, specifically ElasticSearch 2.1. I also remembered that LogStash had become part of elastic, and I’d been eyeing it for a long time as a replacement for Sentry.

In short, ElasticSearch, like Solr, is a sort of wrapper around Lucene. But that wrapper is precisely where all the goodies of this wonderful search engine live: facets, an HTTP-JSON API, multitenancy (the ability to customize the search index for different groups of users), scaling, and a convenient dashboard.

Installing ElasticSearch 2.1

The Ubuntu 15.10 packages contain the old version, 1.6.2. I didn’t bother adding a PPA; for testing, I decided to simply download the archive from the official website:

curl -L -O https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.0/elasticsearch-2.1.0.tar.gz
tar -xzf elasticsearch-2.1.0.tar.gz
cd elasticsearch-2.1.0/

Let’s install two useful plugins right away:

# Anticipate Issues & Scale Faster
# https://www.elastic.co/products/marvel
./bin/plugin install elasticsearch/marvel/latest

# Monitoring and Management of instances and clusters
# https://github.com/royrusso/elasticsearch-HQ
./bin/plugin install royrusso/elasticsearch-HQ

If you’re running Linux under VirtualBox, you need to specify the following in the conf/elasticsearch.yml settings: network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: ["10.0.2.2", "127.0.0.1", "[::1]"] where 10.0.2.2 is the host machine’s address. You can find it in the VirtualBox network settings, or check where connections to the guest machine are coming from using netstat -an| grep SOMEPORT.

Disable the Marvel agent in conf/elasticsearch.yml, since we don’t need it right now:

marvel.agent.enabled: false

Start ElasticSearch and check that the service is running:

./bin/elasticsearch

# Должен вернуться JSON
curl http://localhost:9200/

# Проверяем что работает веб-интерфейс Elastic HQ
curl http://localhost:9200/_plugin/hq/

If something isn’t working, check the STDOUT of the running elasticsearch instance.

Using It with Django

While Googling for a ready-made solution for conveniently indexing models in Django, I came across
django-elasticsearch. Let’s install it:

pip install elasticsearch
pip install git+https://github.com/liberation/django_elasticsearch.git

Next, following the documentation, modify our model in README.md:

from django.db import models
from django_elasticsearch.models import EsIndexable


class Tag(EsIndexable, models.Model):
    name = models.CharField(max_length=64)
    [...]

Specify the index name in settings.py:

ELASTICSEARCH_DEFAULT_INDEX = '<project_name>'

Run the indexing:

 ./manage.py shell
 >>> from products.models import Tag
 >>> Tag.es.create_index()
 >>> Tag.es.reindex_all()

It’s handy to keep an eye on Elastic HQ while doing this and see our index fill up.

Next, let’s try searching:

>>> Tag.es.search('something')
[{u'description': u'', u'name': u'Something', u'id': 1}, {u'description': u'', u'name': u'SomeThing2', u'id': 2}, ...]

>>> Tag.es.search('something').deserialize()
[<Tag: 1 Something>, <Tag: 2 SomeThing2>]

Hooking into Signals and Updating the Index in Real Time

To do this, we need to specify the following in settings.py:

ELASTICSEARCH_AUTO_INDEX = True

Now we’ll use the “post_syncdb”, “post_save”, and “post_delete” signals to notify the search engine of new changes to the model. There’s nothing else to do: any change to or deletion of an object in a model connected through the EsIndexable mixin will immediately be applied in ElasticSearch.

Installing ElasticSearch 2.1

Using It with Django

Hooking into Signals and Updating the Index in Real Time

Further Reading