sphinx

Centos Linux下安装coreseek/sphinx 全过程

为了完成亿枝客服务器顺利转到Linux上,特作了这个测试工作,以下是测试成功的全过程。

环境准备:

centos5.2

xampp1.7.1 (Apache 2.2.11, MySQL 5.1.33, PHP 5.2.9)

coreseek3.1b3 / Sphinx 0.9.9-rc1

一、Centos安装

这步就省略了,有点要注意,在安装过程中一定全部选择服务器里的开发工具与开发库,如果忘记选择的话,就要在安装之前执行以下命令:

yum install gcc

yum install gcc-c++

yum install python

yum install python-devel

yum install gtk+

yum install libtool

yum install automake
yum install autoconf

yum install mysql-devel

二、xampp安装

参考http://www.apachefriends.org/zh_cn/xampp-linux.html#1677

三、coreseek安装

cd /opt/software

下载mmseg

wget http://www.coreseek.com/uploads/sources/mmseg3_0b3.tar.gz

下载coreseek
wget http://www.coreseek.com/uploads/sources/csft3.1b3.tar.gz

解压缩两个文件包
tar -xzvf mmseg3_0b3.tar.gz
tar -xzvf csft3.1b3.tar.gz

编译mmseg
cd /opt/software/mmseg.3_0b3
./configure –prefix=/usr/local/mmseg
make
make install

注意:如果在这一步出错,且出错提示为:css/UnigramCorpusReader.cpp:89: error: ’strncmp’ was not declared in this scope
则需手工编辑.src/css目录下UnigramCorpusReader.cpp 文件,在其第一行加上

#include <string.h>
然后执行make clean 再重新 make,make install即可通过

在这一步安装完了后,将会在 /usr/local 下产生 mmseg目录,
手工修改 /usr/local/mmseg/include/mmseg/freelist.h
vi   /usr/local/mmseg/include/mmseg/freelist.h
在上面添加
#include <string.h>

编译coreseek
cd  /opt/software/csft3.1b3/
到这里都一切正常;下面就开始可能有问题出现
第一configure

./configure –prefix=/usr/local/coreseek –with-python –with-mysql –with-mmseg-includes=/usr/local/mmseg/include/mmseg –with-mmseg-libs=/usr/local/mmseg/lib/

make

make install

可能的错误一:

pydatasource.cpp:742: 错误:从类型 ‘const char*’ 到类型 ‘char*’ 的转换无效
pydatasource.cpp:742: 错误:  初始化实参 2,属于 ‘PyObject* PyObject_GetAttrString(PyObject*, char*)’
make[2]: *** [pydatasource.o] 错误 1
make[2]: Leaving directory `/opt/csft3.1b3/src’
make[1]: *** [all] 错误 2
make[1]: Leaving directory `/opt/csft3.1b3/src’
make: *** [all-recursive] 错误 1

解决办法:yum install python-devel  或者 去了–with-python

可能错误二:

sphinxutils.cpp:793: error: cannot convert ‘int*’ to ‘Py_ssize_t*’ for argument ‘2’ to ‘int PyDict_Next(PyObject*, Py_ssize_t*, PyObject**, PyObject**)’
sphinxutils.cpp:802: warning: unused variable ‘nRet’
make[2]: *** [sphinxutils.o] 错误 1
make[2]:正在离开目录 `/home/syu/sphinx/csft3_0b4/src’
make[1]: *** [all] 错误 2
make[1]:正在离开目录 `/home/syu/sphinx/csft3_0b4/src’
make: *** [all-recursive] 错误 1

解决办法:手工打开 src目录下的 sphinxutils.cpp 修改第789行左右int pos = 0; 修改为   Py_ssize_t pos = 0;

执行make clean 重新make & make install 即可

此步安装完成后,将在/usr/local/下生成 coreseek 目录

四、重新编译xampp中MySql,支持sphinxse

应用coreseek/sphinx,有两种办法,一是通过sphinx提供的API reference,二是通过MySQL storage engine (SphinxSE)

我们采用的是第二种办法,也是我喜欢的方法,比较简单,不用修改太多程序。

由于我们sphinxse安装需要重新编译mysql,但是我们的环境是xampp安装包,这个编译有点麻烦,参考重新编译/构建Xampp中的MYSQL

五、sphinx配置

创建目录

mkdir /usr/local/coreseek/log

mkdir /usr/local/coreseek/pid

mkdir /usr/local/coreseek/data/dict

编辑sphinx.conf

cp /usr/local/coreseek/etc/sphinx.conf.dist  /usr/local/coreseek/etc/sphinx.conf

vi /usr/local/coreseek/etc/sphinx.conf

编辑修改如下内容

source yicike_search
{
type                                    = mysql
sql_host                                = localhost
sql_user                                =
sql_pass                                =
sql_db                                  = mainyicike
sql_port                                = 3306  # optional, default is 3306
sql_query_pre                   = set names utf8
sql_query_pre                   = SET SESSION query_cache_type=OFF
sql_query                               = \
select id,title,category_id  from pricecomparison_product
sql_attr_uint        = category_id
sql_ranged_throttle     = 0
}

index yicike_search
{
source                  = yicike_search
path                    = /usr/local/coreseek/data/yicike_search
docinfo                 = extern
mlock                   = 0
morphology              = none
stopwords               = /usr/local/coreseek/data/dict/stopwords.txt
min_word_len            = 1
charset_type            = zh_cn.utf-8
charset_dictpath        = /usr/local/coreseek/data/dict
min_prefix_len          = 0
min_infix_len           = 0
ngram_len               = 1
ngram_chars = U+4E00..U+9FBF, U+3400..U+4DBF, U+20000..U+2A6DF, U+F900..U+FAFF,\
U+2F800..U+2FA1F, U+2E80..U+2EFF, U+2F00..U+2FDF, U+3100..U+312F, U+31A0..U+31BF,\
U+3040..U+309F, U+30A0..U+30FF, U+31F0..U+31FF, U+AC00..U+D7AF, U+1100..U+11FF,\
U+3130..U+318F, U+A000..U+A48F, U+A490..U+A4CF
html_strip              = 0

}

source yicike_search_ctitle:yicike_search
{
sql_query                               = \
select id,title,PAGE_KEYWORDS from pricecomparison_category
sql_ranged_throttle     = 0
}

index yicike_search_ctitle
{
source                  = yicike_search_ctitle
path                    = /usr/local/coreseek/data/yicike_search_ctitle
docinfo                 = extern
mlock                   = 0
morphology              = none
stopwords               = /usr/local/coreseek/data/dict/stopwords.txt
min_word_len            = 1
charset_type            = zh_cn.utf-8
charset_dictpath        = /usr/local/coreseek/data/dict
min_prefix_len  = 0
min_infix_len           = 0
ngram_len               = 1
ngram_chars = U+4E00..U+9FBF, U+3400..U+4DBF, U+20000..U+2A6DF, U+F900..U+FAFF,\
U+2F800..U+2FA1F, U+2E80..U+2EFF, U+2F00..U+2FDF, U+3100..U+312F, U+31A0..U+31BF,\
U+3040..U+309F, U+30A0..U+30FF, U+31F0..U+31FF, U+AC00..U+D7AF, U+1100..U+11FF,\
U+3130..U+318F, U+A000..U+A48F, U+A490..U+A4CF
html_strip              = 0

}

indexer
{
mem_limit                       = 320M
# max_iops                      = 40
# max_iosize            = 1048576
}

searchd
{

listen                                = 3312
log =/usr/local/coreseek/log/searchd.log
query_log=/usr/local/coreseek/log/query.log
read_timeout=5
max_children=30
pid_file=/usr/local/coreseek/pid/searchd.pid
max_matches=1000000
seamless_rotate =1
preopen_indexes=0
unlink_old  =1
}

创建sphinxse数据表

CREATE TABLE IF NOT EXISTS `sphinx` (
`id` int(11) NOT NULL,
`weight` int(11) NOT NULL,
`query` varchar(255) NOT NULL,
`category_id` int(11) NOT NULL,
KEY `Query` (`query`)
) ENGINE=SPHINX DEFAULT CHARSET=utf8 CONNECTION=’sphinx://localhost:3312/yicike_search’;

CREATE TABLE IF NOT EXISTS `sphinxc` (
`id` int(11) NOT NULL,
`weight` int(11) NOT NULL,
`query` varchar(255) NOT NULL,
KEY `Query` (`query`)
) ENGINE=SPHINX DEFAULT CHARSET=utf8 CONNECTION=’sphinx://localhost:3312/yicike_search_ctitle’;

参考文档:

http://blog.csdn.net/syu/archive/2009/01/11/3754818.aspx

http://www.sphinxsearch.com/docs/current.html#sphinxse

http://www.coreseek.com/forum/index.php?action=vthread&forum=2&topic=165

http://blog.tom.com/benge_zhao/article/5052.html

作者: 独思客
原载: 亿枝客比较导购网
版权所有。转载时必须以链接形式注明作者和原始出处及本声明。

原创文章,转载请注明: 转载自亿赐客比较购物搜索网

本文链接地址: Centos Linux下安装coreseek/sphinx 全过程

By LEO on 2009年05月16日 | 技术整理 | A comment?
标签:, , , , , ,

重新编译/构建Xampp中的MYSQL并支持SPHINXSE

为了配合亿枝客服务器系统转到Linux上,将sphinxSE编译到xampp的MYSQL里的测试工作,以下是测试成功全过程记录。

软件准备

xampp1.7.1 (mysql-5.1.33)

Development package xampp1.7.1

mysql-5.1.33源码

sphinx-0.9.9-rc1

yum install openssl

xampp及开发套件安装,并取出mysql编译/构建脚本

mkdir /opt/software

 

cd /opt/software

 

$ wget 'http://www.apachefriends.org/download.php?xampp-linux-1.7.1.tar.gz'
$ wget 'http://www.apachefriends.org/download.php?xampp-linux-devel-1.7.1.tar.gz'

 

$ tar xzf xampp-linux-1.7.1.tar.gz -C /opt
$ tar xzf xampp-linux-devel-1.7.1.tar.gz -C /opt

 

这步完成后,我们就会在/opt/lampp/share/lampp/configures.tar.gz

 

找到xampp的重新编译/构建的全部脚本。

 

cd /opt/lampp/share/lampp/

 

mkdir configrures

 

tar xzfv /opt/lampp/share/lampp/configures.tar.gz –C /configures

 

这步完成后,我们就会发现/opt/lampp/share/lampp/configures/mysql目录下,

 

有一个configure-oswald文件,

 

这就是重新编译与构建mysql的脚本.

 

 

下载并解压mysql5.1.33源码

cd /opt/software

wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.33.tar.gz/from/ftp://mirror.csclub.uwaterloo.ca/mysql/

   tar -zxvf mysql-5.1.33.tar.gz  -C /opt

  cd /opt/mysql-5.1.33

 

sphinx下载安装

cd /opt/software

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.9-rc1.tar.gz
tar zxvf sphinx-0.9.9-rc1.tar.gz  -C  /opt
cd /opt/sphinx-0.9.9-rc1
cp -R /opt/sphinx-0.9.9-rc1/mysqlse /opt/mysql-5.1.33/storage/sphinx

cd /opt/mysql-5.1.33

sh BUILD/autorun.sh

 

开始编译与构建XAMPP中的MYSQL

重新开一个窗口

vi /opt/lampp/share/lampp/configures/mysql/configure-oswald

就会看到以下脚本

手动一条一条执行以下命令,注意执行之前关闭lampp。

perl -pi -e ’s/”\/etc\/”/”\/opt\/lampp\/etc\/”/g’ mysys/default.c
perl -pi -e ’s/”\/etc\/mysql\/”/”\/etc\/xampp\/”/g’ mysys/default.c

perl -pi -e “s/DBUG_PRINT/\/\/DBUG_PRINT/g” mysys/my_sync.c
perl -pi -e “s/DBUG_ENTER/\/\/DBUG_ENTER/g” mysys/my_sync.c

env LD_RUN_PATH=/opt/lampp/lib LD_LIBRARY_PATH=”/opt/lampp/lib” CFLAGS=”-O3 -mpentiumpro -I/opt/lampp/include -L/opt/lampp/lib -Wl,–rpath -Wl,/opt/lampp/lib -felide-constructors -fno-exceptions -fno-rtti” CXX=”gcc” CXXFLAGS=”-O3 -mpentiumpro -I/opt/lampp/include -L/opt/lampp/lib -Wl,–rpath -Wl,/opt/lampp/lib -felide-constructors -fno-exceptions -fno-rtti”

./configure –prefix=/opt/lampp –enable-assembler –enable-local-infile –with-mysqld-user=nobody –with-unix-socket-path=/opt/lampp/var/mysql/mysql.sock –with-extra-charsets=complex –libexecdir=/opt/lampp/sbin –sysconfdir=/opt/lampp/etc –datadir=/opt/lampp/share –localstatedir=/opt/lampp/var/mysql –infodir=/opt/lampp/info –includedir=/opt/lampp/include –mandir=/opt/lampp/man –with-ssl=/opt/lampp  –enable-thread-safe-client  –with-plugins=max-no-ndb  –with-plugins=sphinx

make

执行到这里可能有错,然后执行make clean,去了–with-ssl=/opt/lampp  ,重新执行一篇。

make install

mv /opt/lampp/sql-bench /opt/lampp/share/mysql/sql-bench

rm -rf /opt/lampp/share/mysql/sql-bench

 

检查是否安装好

/opt/lampp/lampp start

这里如果出现XAMPP: Couldn’t start MySQL! 那么就去查看一下mysql错误日志,

错误日志位置在my.cnf里配置为log-error=/opt/lampp/logs/mysql_log

根据日志提示的问题,再去google上寻找解决答案,基本上都能解决。

/opt/lampp/lampp/bin/mysql –uroot

show engines;

我遇到的几个启动不成功的原因

1、/opt/lampp/sbin/mysqld: unknown variable ‘innodb_data_home_dir=/opt/lampp/var/mysql/’

解决办法:

vi /opt/lampp/etc/my.cnf

注释以下所有

#skip-innodb
innodb_data_home_dir = /opt/lampp/var/mysql/
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /opt/lampp/var/mysql/
# You can set .._buffer_pool_size up to 50 – 80 %
# of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

然后启动MYSQL

/opt/lampp/lampp startmysql

/opt/lampp/bin/mysql –uroot

mysql>INSTALL PLUGIN innodb  SONAME ‘ha_innodb.so’
mysql> show plugin;

+————+——–+—————-+————–+———+
| Name       | Status | Type           | Library      | License |
+————+——–+—————-+————–+———+
| binlog     | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| CSV        | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| MEMORY     | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| MyISAM     | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| SPHINX     | ACTIVE | STORAGE ENGINE | NULL         | GPL     |
| PBXT       | ACTIVE | STORAGE ENGINE | libpbxt.so   | GPL     |
| InnoDB     | ACTIVE | STORAGE ENGINE | ha_innodb.so | GPL     |
+————+——–+—————-+————–+———+

这时再把my.cnf里注释取了,再重新启动。

参考资料

http://www.ibm.com/developerworks/cn/opensource/os-php-fastapps3/

http://www.apachefriends.org/zh_cn/xampp-linux.html

原创文章,转载请注明: 转载自亿赐客比较购物搜索网

本文链接地址: 重新编译/构建Xampp中的MYSQL并支持SPHINXSE