adw0rd.
RUEN
← All posts
·6 min read

Sphinx. Installing, configuring, and using the search engine

FreeBSDPHPSphinxSearchWindowsWordpressSearch engines

sphinx

Not long ago, I wrote an article called “FreeBSD. Installing and configuring Yandex.Server” and promised in the comments to talk about the Sphinx search engine. This is one of three planned articles about this wonderful search engine; in the next articles, I’ll cover the PHP extension and the MySQL extension.

The operating system we’ll be installing and configuring it on is FreeBSD, but I’ll also give tips for Windows users.

As an example, we’ll develop a search feature for my Wordpress blog. For the most impatient readers, here’s a demonstration of the search engine in action: http://demo.adw0rd.ru/sphinxsearch/.

Installation

Installation is quite straightforward


# cd /usr/ports/textproc/sphinxsearch
# make config

Select:

  • “mysql” if you use this DBMS and want to work with its data directly from Sphinx
  • “iconv” for support for different encodings

# make install

That’s it, installation is complete!

You can download the source code and Windows versions here: http://sphinxsearch.com/downloads.html. After downloading the Windows version, simply extract the archive and add the full path to the \sphinx\bin directory to the “Path” system variable.

Configuration

Before you can search, you need to index your information. Let’s create our configuration file and specify what to index and how, as well as how to serve search results.

Let’s create a configuration file for our blog search at /home/sphinx/sphinx.conf:



source adw0rd_wp
{
        # Параметры подключения к БД
        type = mysql
        sql_host = localhost
        sql_user = user
        sql_pass = password
        sql_db = database_name
        sql_port = 3306

        # Установим кодировку для работы с БД
        sql_query_pre = SET NAMES utf8
        sql_query_pre = SET CHARACTER SET utf8

        # Запрос выборки данных для индексации
        sql_query = SELECT ID as post_id, post_title, post_content FROM wp_posts WHERE post_type = 'post'

        # Запрос доп. информации для вывода результата (используется утилитой "search")
        sql_query_info = SELECT * FROM wp_posts WHERE ID = $id

        # Время простоя (sleep) перед посылкой запросов серверу (предназначен для разгрузки сервера БД)
        # Если установите "= 1000", то засыпание будет длится 1 секунду
        sql_ranged_throttle = 0
}

index adw0rd_wp
{
        # Использовать соответствующий source-блок настроек при индексации
        source = adw0rd_wp

        # Путь до файлов индекса
        path = /home/sphinx/data/adw0rd_wp

        # Способ хранения индекса (none, inline, extern)
        # Подробнее http://www.sphinxsearch.com/docs/manual-0.9.8.html#conf-docinfo
        docinfo = extern

        # Memory lock (http://www.sphinxsearch.com/docs/manual-0.9.8.html#conf-mlock)
        mlock = 0

        # Использование английского и русского стемминга
        morphology = stem_enru

        # Минимальная длина индексируемого слова
        min_word_len = 2

        # Установка используемой кодировки
        charset_type = utf-8

        # Таблица символов (http://www.sphinxsearch.com/docs/manual-0.9.8.html#conf-charset-table)
        charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F

        # Минимальная длина инфикса (префикс в том числе)
        min_infix_len = 2

        # Использовать оператор усечения "*" (http://www.sphinxsearch.com/docs/manual-0.9.8.html#conf-enable-star)
        enable_star = 1
}

indexer
{
        # Максимальный лимит используемой памяти RAM
        mem_limit = 32M
}

searchd
{
        # Адрес сервера
        address = 127.0.0.1

        # Порт
        port = 3312

        # Лог
        log = /home/sphinx/log/searchd.log

        # Лог запросов
        query_log = /home/sphinx/log/query.log

        # Таймаут на соединение с сервером (в секундах). При истечении времени происходит обрыв
        read_timeout = 5

        # Максимальное кол-во потомков от процесса
        max_children = 30

        # Путь до pid-файла
        pid_file = /home/sphinx/log/searchd.pid

        # Максимальное кол-во результатов выдачи
        max_matches = 1000
}

In newer versions of SphinxSearch, you need to use:

listen = 127.0.0.1:3312

Instead of

# Адрес сервера
address = 127.0.0.1

# Порт
port = 3312

Full documentation: http://www.sphinxsearch.com/docs/.

Let’s go over the SQL queries we’re using:


# Запрос выборки данных для индексации
sql_query = SELECT post_title, post_content FROM wp_posts WHERE post_type = 'post'

Here we select article titles and content from the blog’s posts table, specifying that the post type must be “post” (this is the version of the post considered published and current, since there are also revision versions of posts).


# Запрос доп. информации для вывода результата (используется утилитой "search")
sql_query_info = SELECT * FROM `wp_posts` WHERE `ID` = $id

This query is used to display additional search result data in the “search” command-line utility.

Indexing

Let’s specify the configuration file we created as the config:


# indexer --config /home/sphinx/sphinx.conf --all

To update the current index, use “–rotate”; it will add new data to the index we created and update any changed data accordingly.


# indexer --config /home/sphinx/sphinx.conf --rotate

P.S. If indexing fails, leave a comment on the article, and we’ll see what happened… :)

For searching from the console, there’s the “search” utility:


# search --config /home/sphinx/sphinx.conf искомая комбинация

Starting the search server

To work with the search server, we need to start it:


# searchd --config /home/sphinx/sphinx.conf

After that, we’ll be able to use the Sphinx API.

Using the Sphinx API for PHP

To work with the Sphinx API, you need to include the sphinxapi.php file; you can find this library bundled with the distribution. For example, when installing from FreeBSD ports, the path is "/usr/ports/textproc/sphinxsearch/work/sphinx-0.9.8.1/api/sphinxapi.php", and for Windows, it’s “sphinx\api\sphinxapi.php”.

Now let’s write a script to work with the search engine’s indexed data—that is, to query for search words and display the search results!


<html>
<head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
    <title>Sphinx search for Wordpress</title>
</head>
<body>

<form action="" method="get">
    <input name="s" size="40" value="<?php echo @$_GET['s'];?>" />
    <input type="submit" value="Искать!" />
</form>
<?php

if(isset($_GET['s']) and strlen($_GET['s']) > 2) {

    // Подключаем sphinx-api
    require_once ("sphinxapi.php");

    // Искомая комбинация
    $string = $_GET['s'];

    // Создаем объект клиента для Sphinx API
    $sphinx = new SphinxClient();

    // Подсоединяемся к Sphinx-серверу
    $sphinx->SetServer('localhost', 3312);

    // Совпадение по любому слову
    $sphinx->SetMatchMode(SPH_MATCH_ANY);

    // Результаты сортировать по релевантности
    $sphinx->SetSortMode(SPH_SORT_RELEVANCE);

    // Задаем полям веса (для подсчета релевантности)
    $sphinx->SetFieldWeights(array ('post_title' => 20, 'post_content' => 10));

    // Результат по запросу (* - использование всех индексов)
    $result = $sphinx->Query($string, '*');

    // Если есть результаты поиска, то
    if ($result && isset($result['matches']))
    {

        // Соединяемся с БД
        mysql_connect('localhost', 'user', 'password');
        mysql_select_db('adw0rd_wp');

        // Устанавливаем кодировки
        mysql_query('SET NAMES utf8');
        mysql_query('SET CHARACTER SET utf8');

        // Получаем массив ID постов блога
        $ids = array_keys($result['matches']);

        // Выводим посты отсортированные по релевантности
        $id_list = implode(',', $ids);
        $sql = '
            SELECT `ID`, `post_title`, `post_content`
                FROM `wp_posts`
                WHERE `ID` IN ('.$id_list.')
                ORDER BY FIELD(`ID`, '.$id_list.')';

        $resource = mysql_query($sql);

        // Выводим результаты поиска
        echo '<ol>';
        while ($result = mysql_fetch_assoc($resource)) {
            echo '<li><span><a href="/?p='.$result['ID'].'">'.$result['post_title'].'</a></span><div>'.mb_substr(htmlspecialchars($result['post_content']), 0, 400).'</div>';
        }
        echo '</ol>';
    }
}
?>
</body>
</html>

Documentation for the Sphinx API for PHP: http://www.sphinxsearch.com/wiki/doku.php?id=php_api_docs

Summary

In a short time, we’ve managed to deploy a blazing-fast “do-it-yourself search service” with full-text search capabilities. A demonstration in action: http://demo.adw0rd.ru/sphinxsearch/.

If you have any questions about this article, leave them in the comments.

P.S. If you’re unfamiliar with terms such as “stemming”, “infix”, “word form”, and so on, there’s a project called Wikipedia, where you can easily learn about them!

Comments are closed. Read the original discussion in Russian →