es 的一个特点,版本更新快,且各个大版本 不兼容,并且 查询语法变化也特别多
并且需要注意的分词器要和es的版本完全一致才可以 ,否则报错
es 最新版本为 7,安装版本与公司一致 6.7.1
mac 安装 使用 6.3.0
brew install elasticsearch
然后修改配置文件
network.host: 0.0.0.0 ##千万不可以是127.0.0.1 localhost 否则无法被其他电脑访问
http.port: 9200
bootstrap.memory_lock: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#cluster.name: my-application
然后安装 中文分词器插件
./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.3.0/elasticsearch-analysis-ik-6.3.0.zip
最后运行即可
centos7安装 es 6.7.1
下载地址
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.7.1.rpm
使用yum 离线安装
yum install ./elasticsearch-6.7.1.rpm
修改配置文件
cd /etc/elasticsearch/
vi elasticsearch.yml
network.host: 0.0.0.0
http.port: 9200
bootstrap.memory_lock: false
#path.data: /var/lib/elasticsearch
#path.logs: /var/log/elasticsearch
#cluster.name: my-application
安装中文分词器,特别要注意版本必须保持一致
cd /usr/share/elasticsearch/bin
./elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.7.1/elasticsearch-analysis-ik-6.7.1.zip
centos7 安装 kibana
wget [https://artifacts.elastic.co/downloads/kibana/kibana-6.7.1-x86_64.rpm](https://artifacts.elastic.co/downloads/kibana/kibana-6.7.1-x86_64.rpm)
Kibana 需要修改自己的 public host 和 es host 和port,需要和你实际es 配置的保持一致
cd /etc/kibana/
vi kibana.yml
elasticsearch.hosts: ["http://0.0.0.0:9200"]
server.host: "0.0.0.0"
server.port: 5601
es kibana 目录都加入到 环境变量
vi /etc/profile
export ELASTICSEARCH_HOME=/usr/share/elasticsearch
export KIBANA_HOME=/usr/share/kibana
export PATH=$PATH:$ELASTICSEARCH_HOME/bin:$KIBANA_HOME/bin
分别启动他们 先前台 启动观察 ,最后后台启动
切换用户 su elasticsearch 这里会报一个小错【This account is currently not available] ,
修改 /etc/passwd 文件 即可
安装完elasticsearch之后系统会自动创建一个elasticsearch用户,在启动elasticsearch的时候需要切换该用户启动,但是在我切换到elasticsearch用户的时候显示This account is currently not available。
查资料发现是因为该用户的shell不可用,查看/etc/passwd文件发现此用户的shell是/sbin/nologin需修改成/bin/bash
cd /usr/share/elasticsearch/bin
./elasticsearch 观察
./elasticsearch -d
/usr/share/kibana/bin
./kibana 观察
#后台启动 不带日志
nohup /usr/share/kibana/bin/kibana > /dev/null 2>&1 &
#后台启动 带错误日志
nohup /usr/share/kibana/bin/kibana > /dev/null 2>log &
修改完后就可以切换到elasticsearch用户了。
/etc/elasticsearch/
编辑 /etc/security/limits.conf,追加以下内容;
* soft nofile 65536
* hard nofile 65536
此文件修改后需要重新登录用户,才会生效
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决办法:
编辑 /etc/sysctl.conf,追加以下内容:
vm.max_map_count=655360
保存后,执行:
sysctl -p
java 调用 es
http://www.cnblogs.com/xyddm/p/10417371.html
程序中遇到 【# [ElasticsearchStatusException contains unrecognized parameter: [ccs_minimize_roundtrips]]]](https://stackoverflow.com/questions/55602377/elasticsearchstatusexception-contains-unrecognized-parameter-ccs-minimize-roun)
这个主要是是 es jar 版本的问题 ,切换和服务器相同的版本即可
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>6.7.1</version>
</dependency>
package com.hhz.es.searchhhz;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.termvectors.TermVectorsFilter;
import org.elasticsearch.client.*;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.index.query.TermQueryBuilder;
import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig.Builder;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.elasticsearch.client.RestClientBuilder.HttpClientConfigCallback;
import org.elasticsearch.client.RestClientBuilder.RequestConfigCallback;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.script.ScriptType;
import org.elasticsearch.script.mustache.SearchTemplateRequestBuilder;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.elasticsearch.index.search.MultiMatchQuery;
import org.springframework.data.elasticsearch.core.ElasticsearchTemplate;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQuery;
public class searchadpt {
private static TransportClient client;
private ElasticsearchClient esclient ;
private RestHighLevelClient esrestclient;
// public Client getClient(){
//
// esclient =
// }
private static String hosts = "192.168.199.102";// "0.0.0.0";//"172.16.8.3"; // 集群地址,多个用,隔开
private static int port = 9200; // 使用的端口号
private static String schema = "http"; // 使用的协议
private static ArrayList<HttpHost> hostList = null;
private static int connectTimeOut = 1000; // 连接超时时间
private static int socketTimeOut = 30000; // 连接超时时间
private static int connectionRequestTimeOut = 500; // 获取连接的超时时间
private static int maxConnectNum = 100; // 最大连接数
private static int maxConnectPerRoute = 100; // 最大路由连接数
static {
hostList = new ArrayList<>();
String[] hostStrs = hosts.split(",");
for (String host : hostStrs) {
hostList.add(new HttpHost(host, port, schema));
}
}
public RestHighLevelClient initRestClient() {
RestClientBuilder builder = RestClient.builder(hostList.toArray(new HttpHost[0]));
// 异步httpclient连接延时配置
builder.setRequestConfigCallback(new RequestConfigCallback() {
@Override
public Builder customizeRequestConfig(Builder requestConfigBuilder) {
requestConfigBuilder.setConnectTimeout(connectTimeOut);
requestConfigBuilder.setSocketTimeout(socketTimeOut);
requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
return requestConfigBuilder;
}
});
// 异步httpclient连接数配置
builder.setHttpClientConfigCallback(new HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
httpClientBuilder.setMaxConnTotal(maxConnectNum);
httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute);
return httpClientBuilder;
}
});
// RestClient client = builder.build(); // new RestClient(builder);
RestHighLevelClient client =new RestHighLevelClient(builder);
if(client != null)
{
// client.performRequest()
System.out.println("连接成功!"+client.cluster());
}
return client;
}
public Client initESClient() {
// 配置你的es,如果你的集群名称不是默认的elasticsearch,需要以下这步
Settings settings = Settings.builder().put("cluster.name","elasticsearch_apio")
.put("client.transport.ignore_cluster_name",false)
.put("client.transport.sniff",true)
.build();
// 这里可以同时连接集群的服务器,可以多个,并且连接服务是可访问的 172.16.8.3
try {
// 创建client
client = new PreBuiltTransportClient(settings).addTransportAddress(new TransportAddress(InetAddress.getByName("172.16.8.3"),9200));
} catch (Exception e) {
e.printStackTrace();
}finally
{
if(client != null)
{
System.out.println("连接成功!");
}
}
return client;
}
public void closeESClient() {
client.close();
System.out.println("连接关闭!");
}
// private TermVectorsFilter tf=null;
// private ElasticsearchClient client =null;
// private ElasticsearchTemplate et= null;
//
// SearchQuery searchQuery = new NativeSearchQueryBuilder().withFilter(must()).build();
// TermQueryBuilder termQueryBuilderfilter=TermQueryBuilder.fromXContent(null);
// termQueryBuilderfilter.
// BoolQueryBuilder bbf
// MultiMatchQuery.QueryBuilder
public void search(String index, String type, String name) throws IOException {
BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
boolBuilder.must(QueryBuilders.matchQuery("admin_tag", name)); // 这里可以根据字段进行搜索,must表示符合条件的,相反的mustnot表示不符合条件的
// boolBuilder.must(QueryBuilders.matchQuery("id", tests.getId().toString()));
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(boolBuilder);
sourceBuilder.from(0);
sourceBuilder.size(100); // 获取记录数,默认10
sourceBuilder.fetchSource(new String[] { "id", "name" }, new String[] {}); // 第一个是获取字段,第二个是过滤的字段,默认获取全部
SearchRequest searchRequest = new SearchRequest(index);
searchRequest.types(type);
searchRequest.source(sourceBuilder);
SearchResponse response = esrestclient.search(searchRequest, RequestOptions.DEFAULT);
// System.out.println("search: " + JSON.toJSONString(response));
SearchHits hits = response.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
System.out.println("search -> " + hit.getSourceAsString());
}
}
public List<Map<String,Object>> searchZd(String query, String msg) {
Map<String, Object> template_params = new HashMap<String, Object>();
template_params.put("param_gender", msg);
template_params.put("param_query", query);
final List<Map<String, Object>> list = new ArrayList<Map<String, Object>>();
//模板查询
SearchResponse sr = new SearchTemplateRequestBuilder(client)
.setScript("{\n" +
" \"query\" : {\n" +
" \"match\" : {\n" +
" \"{{param_query}}\" : \"{{param_gender}}\"\n" +
" }\n" +
" }\n" +
"}")
.setScriptType(ScriptType.INLINE)
.setScriptParams(template_params)
.setRequest(new SearchRequest())
.get()
.getResponse();
long totalHits1 = sr.getHits().totalHits; //命中个数
for (SearchHit searchHit : sr.getHits().getHits()) {
final Map<String, Object> sourceAsMap = searchHit.getSourceAsMap();
list.add(sourceAsMap);
}
System.out.println(totalHits1 + "---" + sr.toString());
return list;
}
public static void main(String[] args)throws IOException{
searchadpt se=new searchadpt();
se.esrestclient= se.initRestClient();
se.search("goods","goods_type","装修记录");
// se.initESClient();
// List<Map<String,Object>> result=se.searchZd("admin_tag","装修记录");
//
// se.closeESClient();
}
}
scala
import org.apache.http.HttpHost
import org.apache.http.client.config.RequestConfig.Builder
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder
import org.elasticsearch.action.search.SearchRequest
import org.elasticsearch.action.search.SearchResponse
import org.elasticsearch.client._
import org.elasticsearch.client.RestClientBuilder.HttpClientConfigCallback
import org.elasticsearch.client.RestClientBuilder.RequestConfigCallback
import org.elasticsearch.client.transport.TransportClient
import org.elasticsearch.common.settings.Settings
import org.elasticsearch.common.transport.TransportAddress
import org.elasticsearch.index.query.BoolQueryBuilder
import org.elasticsearch.index.query.QueryBuilders
import org.elasticsearch.script.ScriptType
import org.elasticsearch.script.mustache.SearchTemplateRequestBuilder
import org.elasticsearch.search.SearchHit
import org.elasticsearch.search.SearchHits
import org.elasticsearch.search.builder.SearchSourceBuilder
import org.elasticsearch.transport.client.PreBuiltTransportClient
import java.io.IOException
import java.net.InetAddress
import java.util.ArrayList
import java.util.HashMap
import java.util.List
import java.util.Map
import org.apache.http.client.config.RequestConfig
import org.apache.http.impl.client.BasicCredentialsProvider
import org.elasticsearch.action.index.IndexRequest
object searchadpt {
private var client = null
// public Client getClient(){
//
// esclient =
// }
// private TermVectorsFilter tf=null;
// private ElasticsearchClient client =null;
// private ElasticsearchTemplate et= null;
// SearchQuery searchQuery = new NativeSearchQueryBuilder().withFilter(must()).build();
// TermQueryBuilder termQueryBuilderfilter=TermQueryBuilder.fromXContent(null);
// termQueryBuilderfilter.
// BoolQueryBuilder bbf
// MultiMatchQuery.QueryBuilder
@throws[IOException]
def main(args: Array[String]): Unit = {
val eshosts = "0.0.0.0"//"172.16.8.3"; // 集群地址,多个用,隔开
val esport = 9200// 使用的端口号
val schema = "http"// 使用的协议
var hostList = null
val connectTimeOut = 1000// 连接超时时间
val socketTimeOut = 30000
val connectionRequestTimeOut = 500// 获取连接的超时时间
val maxConnectNum = 100// 最大连接数
val maxConnectPerRoute = 100// 最大路由连接数
val builder = RestClient.builder(new HttpHost(eshosts, esport, "http"))
// 异步httpclient连接延时配置
builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
override def customizeRequestConfig(requestConfigBuilder: RequestConfig.Builder): RequestConfig.Builder = {
requestConfigBuilder.setConnectTimeout(connectTimeOut)
requestConfigBuilder.setSocketTimeout(socketTimeOut)
requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut)
requestConfigBuilder
}
})
// 异步httpclient连接数配置
builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
httpClientBuilder.setMaxConnTotal(maxConnectNum)
httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute)
httpClientBuilder
}
})
// RestClient client = builder.build(); // new RestClient(builder);
val client = new RestHighLevelClient(builder)
if (client != null) { // client.performRequest()
System.out.println("连接成功!" + client.cluster)
}
val index= "goods"
val `type` ="goods_type"
val name ="装修记录"
val indexreq= new IndexRequest(index,`type`,name)
val res = client.index(indexreq,RequestOptions.DEFAULT)
print(res)
// .index(index=self.index_name, doc_type=self.index_type, body=doc)
val boolBuilder = QueryBuilders.boolQuery
boolBuilder.must(QueryBuilders.matchQuery("admin_tag", name))// 这里可以根据字段进行搜索,must表示符合条件的,相反的mustnot表示不符合条件的
// boolBuilder.must(QueryBuilders.matchQuery("id", tests.getId().toString()));
val sourceBuilder = new SearchSourceBuilder
sourceBuilder.query(boolBuilder)
sourceBuilder.from(0)
sourceBuilder.size(100)// 获取记录数,默认10
sourceBuilder.fetchSource(Array[String]("id", "name"), Array[String]())// 第一个是获取字段,第二个是过滤的字段,默认获取全部
val searchRequest = new SearchRequest(index)
searchRequest.types(`type`)
searchRequest.source(sourceBuilder)
// searchRequest.setCcsMinimizeRoundtrips(false)
val response = client.search(searchRequest, RequestOptions.DEFAULT)
// System.out.println("search: " + JSON.toJSONString(response));
val hits = response.getHits
val searchHits = hits.getHits
for (hit <- searchHits) {
System.out.println("search -> " + hit.getSourceAsString)
}
// val credentialsProvider = new BasicCredentialsProvider()
// val srcClient :RestHighLevelClient= new RestHighLevelClient(RestClient.builder(new HttpHost(eshosts, esport, "http")).setHttpClientConfigCallback((httpClientBuilder: HttpAsyncClientBuilder) => httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)))
// val se = new searchadpt
// se.esrestclient = se.srcClient
// se.search("topic_activity", "doc", "装修记录")
// se.initESClient();
// List<Map<String,Object>> result=se.searchZd("admin_tag","装修记录");
// se.closeESClient();
}
// try hostList = new ArrayList[HttpHost]
// val hostStrs = hosts.split(",")
// for (host <- hostStrs) {
// hostList.add(new HttpHost(host, port, schema))
// }
}
class searchadpt {
private val esclient = null
private var esrestclient = null
private val eshosts = "0.0.0.0"//"172.16.8.3"; // 集群地址,多个用,隔开
private val esport = 9200// 使用的端口号
private val schema = "http"// 使用的协议
private var hostList = null
private val connectTimeOut = 1000// 连接超时时间
private val socketTimeOut = 30000
private val connectionRequestTimeOut = 500// 获取连接的超时时间
private val maxConnectNum = 100// 最大连接数
private val maxConnectPerRoute = 100// 最大路由连接数
val credentialsProvider = new BasicCredentialsProvider()
val srcClient :RestHighLevelClient= new RestHighLevelClient(RestClient.builder(new HttpHost(eshosts, esport, "http")).setHttpClientConfigCallback((httpClientBuilder: HttpAsyncClientBuilder) => httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)))
def initRestClient: RestHighLevelClient = {
val builder = RestClient.builder(new HttpHost(eshosts, esport, "http"))
// 异步httpclient连接延时配置
builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
override def customizeRequestConfig(requestConfigBuilder: RequestConfig.Builder): RequestConfig.Builder = {
requestConfigBuilder.setConnectTimeout(connectTimeOut)
requestConfigBuilder.setSocketTimeout(socketTimeOut)
requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut)
requestConfigBuilder
}
})
// 异步httpclient连接数配置
builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
httpClientBuilder.setMaxConnTotal(maxConnectNum)
httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute)
httpClientBuilder
}
})
// RestClient client = builder.build(); // new RestClient(builder);
val client = new RestHighLevelClient(builder)
if (client != null) { // client.performRequest()
System.out.println("连接成功!" + client.cluster)
}
return client
}
// def initESClient: Client = { // 配置你的es,如果你的集群名称不是默认的elasticsearch,需要以下这步
// val settings = Settings.builder.put("cluster.name", "elasticsearch_apio").put("client.transport.ignore_cluster_name", false).put("client.transport.sniff", true).build
//// 这里可以同时连接集群的服务器,可以多个,并且连接服务是可访问的 172.16.8.3
// try // 创建client
// searchadpt.client = new PreBuiltTransportClient(settings).addTransportAddress(new TransportAddress(InetAddress.getByName("172.16.8.3"), 9200))
// catch {
// case e: Exception =>
// e.printStackTrace()
// } finally if (searchadpt.client != null) System.out.println("连接成功!")
// searchadpt.client
// }
//
// def closeESClient(): Unit = {
// searchadpt.client.close()
// System.out.println("连接关闭!")
// }
@throws[IOException]
def search(index: String, `type`: String, name: String): Unit = {
val boolBuilder = QueryBuilders.boolQuery
boolBuilder.must(QueryBuilders.matchQuery("admin_tag", name))// 这里可以根据字段进行搜索,must表示符合条件的,相反的mustnot表示不符合条件的
// boolBuilder.must(QueryBuilders.matchQuery("id", tests.getId().toString()));
val sourceBuilder = new SearchSourceBuilder
sourceBuilder.query(boolBuilder)
sourceBuilder.from(0)
sourceBuilder.size(100)// 获取记录数,默认10
sourceBuilder.fetchSource(Array[String]("id", "name"), Array[String]())// 第一个是获取字段,第二个是过滤的字段,默认获取全部
val searchRequest = new SearchRequest(index)
searchRequest.types(`type`)
searchRequest.source(sourceBuilder)
val response = srcClient.search(searchRequest, RequestOptions.DEFAULT)
// System.out.println("search: " + JSON.toJSONString(response));
val hits = response.getHits
val searchHits = hits.getHits
for (hit <- searchHits) {
System.out.println("search -> " + hit.getSourceAsString)
}
}
// def searchZd(query: String, msg: String): List[Map[String, AnyRef]] = {
// val template_params = new HashMap[String, AnyRef]
// template_params.put("param_gender", msg)
// template_params.put("param_query", query)
// val list = new ArrayList[Map[String, AnyRef]]
// //模板查询
// val sr = new SearchTemplateRequestBuilder(searchadpt.client).setScript("{\n" + " \"query\" : {\n" + " \"match\" : {\n" + " \"{{param_query}}\" : \"{{param_gender}}\"\n" + " }\n" + " }\n" + "}").setScriptType(ScriptType.INLINE).setScriptParams(template_params).setRequest(new SearchRequest).get.getResponse
// val totalHits1 = sr.getHits.totalHits//命中个数
// for (searchHit <- sr.getHits.getHits) {
// val sourceAsMap = searchHit.getSourceAsMap
// list.add(sourceAsMap)
// }
// System.out.println(totalHits1 + "---" + sr.toString)
// list
// }
}
python
pip install elasticsearch
from elasticsearch import Elasticsearch
ip=['0.0.0.0']
port =9200
es=Elasticsearch(ip,port=port)
es.cluster.__dict__
_index_mappings = {
"mappings": {
'goods_type': {
"properties": {
"title": {
"type": "text",
"index": True,
"analyzer": "ik_max_word",
"search_analyzer": "ik_max_word"
},
"date": {
"type": "text",
"index": True
},
"keyword": {
"type": "text",
"index": False
},
"source": {
"type": "text",
"index": False
},
"link": {
"type": "text",
"index": False
}
}
}
}
}
es.indices.create(index='goods', body=_index_mappings)