write down,forget

[独家]Elastic Search,偶然发现的强悍的搜索引擎

<Category: Lucene, 小道消息> 查看评论



Search Engine for the Cloud



h1. ElasticSearch

h2. A Distributed Search Engine

h3. “http://www.elasticsearch.com”:http://www.elasticsearch.com

ElasticSearch is a distributed RESTful search engine built for the cloud. Features include:

* Distributed and Highly Available Search Engine. 分布式、高可用的搜索引擎

** Each index is fully sharded with a configurable number of shards.支持sharding,并且灵活可配

** Each shard can have one or more replicas. 每个shard节点可以有一个或多个备份库

** Read / Search operations performed on either one of the replica shard.有个问题,怎么对这些节点进行数据同步的?

* Multi Tenant with Multi Types. 多类型

** Support for more than one index. 支持一个或多个索引

** Support for more than one type per index. 支持每个类型多个索引

** Index level configuration (number of shards, index storage, …).索引基本的配置,如shard和索引存储

* Various set of APIs 丰富的api

** HTTP RESTful API 支持RESTful方式的api

** Native Java API.原生java支持的api

** All APIs perform automatic node operation rerouting.所有api支持自动节点路由操作

* Document oriented 文档还比较全

** No need for upfront schema definition.不需要对模型进行预定义

** Schema can be defined per type for customization of the indexing process.模型可以根据类型来定制

* Reliable, Asynchronous Write Behind for long term persistency.

* (Near) Real Time Search.

* Built on top of

** Each shard is a fully functional Lucene index

** All the power of Lucene easily exposed through simple configuration / plugins.

* Per operation consistency

** Single document level operations are atomic, consistent, isolated and durable.

* under Apache 2 License. 开源的,Apache2协议

h2. Getting Started

Fist of all, DON’T PANIC. It will take 5 minutes to get the gist of what ElasticSearch is all about.

h3. Installation

* Download and unzip the ElasticSearch installation.

* Run @bin/elasticsearch -f@ on unix, or @bin/elasticsearch.bat@ on windows. Windows下直接运行bin下面的bat批处理就行了

* Run @curl -X GET http://localhost:9200/@.

* Start more servers …

h3. Indexing

Lets try and index some twitter like information. First, lets create a twitter user, and add some tweets (the @twitter@ index will be created automatically):


Now, lets see if the information was added by GETting it:


h3. Searching

Mmm search…, shouldn’t it be elastic?

Lets find all the tweets that @kimchy@ posted:

We can also use the query language ElasticSearch provides instead of a query string:

Just for kicks, lets get all the documents stored (we should see the user as well):


We can also do range search (the @postDate@ was automatically identified as date)

There are many more options to perform search, after all, its a search product no? All the familiar Lucene queries are available through the JSON query language, or through the query parser.

h3. Multi Tenant – Indices and Types

Maan, that twitter index might get big (in this case, index size == valuation). Lets see if we can structure our twitter system a bit differently in order to support such large amount of data.

ElasticSearch support multiple indices, as well as multiple types per index. In the previous example we used an index called @twitter@, with two types, @user@ and @tweet@.

Another way to define our simple twitter system is to have a different index per user. Here is the indexing curl’s in this case:

The above index information into the @kimchy@ index, with two types, @info@ and @tweet@. Each user will get his own special index.

Complete control on the index level is allowed. As an example, in the above case, we would want to change from the default 5 shards with 1 replica per index, to only 1 shard with 1 replica per index (== per twitter user). Here is how this can be done (the configuration can be in yaml as well):

Search (and similar operations) are multi index aware. This means that we can easily search on more than one

index (twitter user), for example:

Or on all the indices:

{One liner teaser}: And the cool part about that? You can easily search on multiple twitter users (indices), with different boost levels per user (index), making social search so much simpler (results from my friends rank higher than results from my friends friends).

h3. Distributed, Highly Available, and Write Behind

Lets face it, things will fail….

ElasticSearch is a highly available and distributed search engine. Each index is broken down into shards, and each shard can have one or more replica. By default, an index is created with 5 shards and 1 replica per shard (5/1)(恩,可以借鉴,牛叉). There are many topologies that can be used, including 1/10 (improve search performance), or 20/1 (improve indexing performance, with search executed in a map reduce fashion across shards).

In order to play with distributed nature, simply bring more nodes up and shut down nodes. The system will continue to serve requests (make sure you use the correct http port) with the latest data indexed.

If the whole cluster is brought down, all the indexed data will be lost (each shard local storage is temporal). For long term persistency, write behind should be enabled. This is as simple as configuring the @elasticsearch.yml@ configuration file (which effectively enables write behind to file system for all indices created unless configured otherwise when creating the index):

Alternatively, elastic search can be started with the following command line:

@elasticsearch -f -Des.gateway.type=fs@.

The above configuration will persist the indices create on ElasticSearch to a file system (path can be configured) in an asynchronous, reliable fashion. Other gateways implementations can be easily implemented and more will be provided out of the box in later versions (did someone say AmazonS3/Hadoop/Cassandra?).酷~~

h3. Where to go from here?

We have just covered a very small portion of what ElasticSearch is all about. For more information, please refer to: .

h3. Building from Source

ElasticSearch uses Gradle:http://www.gradle.org for its build system. In order to create a distribution, simply run @gradlew@, the distribution will be created under @build/distributions@.

h1. License

本文来自: [独家]Elastic Search,偶然发现的强悍的搜索引擎