环境准备
java版本：无需准备（es7.x内置JDK12）
操作系统：CentOS7
安装包：https://www.elastic.co/cn/downloads/past-releases#elasticsearch
我这里选择的是7.10.1版本的ElasticSearch
ElasticSearch单机搭建
ElasticSearch下载
## 下载
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.2-linux-x86_64.tar.gz
## 解压
tar -zxvf elasticsearch-7.10.2-linux-x86_64.tar.gz -C /
## 软链接
ln -s /elasticsearch-7.10.2/ es/
ElasticSearch建立用户组用户
ElasticSearch 无法使用root权限安装所以需要创建用户组用户账号
## 创建es用户组
groupadd es
## 创建用户 esdm 并设置密码
useradd esdm
passwd esdm
## 用户esdm添加到es用户组
usermod -G es esdm
## 赋权
chown -R esdm /es
chown -R esdm /elasticsearch-7.10.2
chown -R esdm /es
修改配置文件
mkdir /data/es
vim /es/config/elasticsearch.yml
elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /data/es
path.logs: /es/log
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.186.131"]
cluster.initial_master_nodes: ["node-1"]
http.cors.enabled: true
http.cors.allow‐origin: "*"
bootstrap.system_call_filter: false
bootstrap.memory_lock: false
## xpack配置加密
#xpack.security.enabled: true
#xpack.license.self_generated.type: basic
#xpack.security.transport.ssl.enabled: true
机器内存较小可以调整jvm.options内存大小
修改启动文件使其使用自带JDK11
if [ ! -z "$JAVA_HOME" ]; then
JAVA="$JAVA_HOME/bin/java"
JAVA_TYPE="JAVA_HOME"
else
if [ "$(uname -s)" = "Darwin" ]; then
# macOS has a different structure
JAVA="$ES_HOME/jdk.app/Contents/Home/bin/java"
else
JAVA="$ES_HOME/jdk/bin/java"
fi
JAVA_TYPE="bundled jdk"
fi
改为
if [ "$(uname -s)" = "Darwin" ]; then
# macOS has a different structure
JAVA="$ES_HOME/jdk.app/Contents/Home/bin/java"
else
JAVA="$ES_HOME/jdk/bin/java"
fi
JAVA_TYPE="bundled jdk"
启动es实例
su esdm
nohup /es/bin/elasticsearch 2>&1 &
报错修改
普通用户打开文件的最大数限制
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
su root
vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
普通用户启动线程数限制
max number of threads [1024] for user [es] likely too low, increase to at least [4096]
su root
vim /etc/security/limits.d/20-nproc.conf
普通用户虚拟内存修改
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
su root
vim /etc/sysctl.conf
sysctl -p
vm.max_map_count=262144
ik分词器
ik下载
## 创建目录
mkdir -p /es/plugins/ik
cd /es/plugins/ik
## 下载
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.10.2/elasticsearch-analysis-ik-7.10.2.zip
## 解压
unzip elasticsearch-analysis-ik-7.10.2.zip
重启es,ik配置成功
Kibana搭建
Kibana下载
## 下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.10.2-linux-x86_64.tar.gz
## 解压
tar -zxvf kibana-7.10.2-linux-x86_64.tar.gz -C /
## 软链接
ln -s /kibana-7.10.2-linux-x86_64/ kibana
修改配置文件
vim /kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.186.131"
elasticsearch.hosts: ["http://localhost:9200"]
赋权
su root
chown -R esdm /kibana-7.10.2-linux-x86_64/
启动实例
su esdm
/kibana/bin/kibana
Logstash 部署
我这里使用的是ELKF架构，所以Logstash监听的是filebeat
请注意Logstash版本与ES版本保持一致
Logstash7.x版本的没有安装，可能与这里有稍许出入,这里安装的是Losgstash6.x版本
Logstash下载
## 下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.3.tar.gz
## 解压
tar -zxvf logstash-6.4.3.tar.gz -C /
修改配置文件
input {
beats {
port => 5044
}
}
filter {
if [fields][index_name] == "mq" and [fields][index_name] == "java" {
grok {
match => [
"message", "%{IPORHOST:http_host} %{IPORHOST:user_ip} - - \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion:float})?|%{DATA:rawrequest})\" %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) %{QS:referrer} %{QS:useragent} (?:%{NUMBER:request_time:float}|-) (?:%{NUMBER:upstream_time:float}|-)"
]
}
geoip {
source => "192.168.186.131"
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
useragent {
target => "ua"
source => "useragent"
}
}
}
output{
if[fields][index_name] == "mq" {
elasticsearch {
hosts => ["http://192.168.186.131:9200"]
index => "mq-%{+YYYY.MM.dd}"
}
}
if[fields][index_name] == "java" {
elasticsearch {
hosts => ["http://192.168.186.131:9200"]
index => "mq-java-zd-%{+YYYY}"
}
}
}
索引信息修改
logstash默认使用的是logstash模板访问地址http://localhost:9200/_template/logstash 可以看到具体的模板信息
我们要想修改有2种办法，1是修改默认模板，2是output中指定模板信息
修改默认模板
在kibana中执行
就是将上面链接的模板信息copy下来修改
- 删掉最外层命名
- 修改index_patterns，为你索引匹配路由，由上面的output中index
- 修改分片副本数在settings.index下加入number_of_shards和number_of_replicas2个配置
PUT /_template/logstash { "order": 0, "version": 60001, "index_patterns": ["mq-*"], "settings": { "index": { "number_of_shards": 1, "number_of_replicas": 0, "refresh_interval": "5s" } }, "mappings": { "_default_": { "dynamic_templates": [ { "message_field": { "path_match": "message", "match_mapping_type": "string", "mapping": { "type": "text", "norms": false } } }, { "string_fields": { "match": "*", "match_mapping_type": "string", "mapping": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } ], "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "keyword" }, "geoip": { "dynamic": true, "properties": { "ip": { "type": "ip" }, "location": { "type": "geo_point" }, "latitude": { "type": "half_float" }, "longitude": { "type": "half_float" } } } } } }, "aliases": {} }output中指定模板信息
同样也是编写上面的模板json文件，放入指定目录，然后在output中指定模板文件位置即可
elasticsearch { hosts => ["http://192.168.186.131:9200"] index => "mq-%{+YYYY.MM.dd}" template => "/path/to/mytemplate" template_name => "myname" }
filebeat 部署
filebeat 下载
## 下载
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.3-linux-x86_64.tar.gz
## 解压
tar -zxvf filebeat-6.4.3-linux-x86_64.tar.gz -C /
修改配置文件
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
## 指定tag方便logstash区别
fields:
index_name: mq
## 监控文件目录
paths:
- /logs/server.log.*
- type: log
enabled: true
fields:
index_name: java
paths:
- /logs/java/web*
## 多行文本合一 非xxxx-xx-xx为前缀的合并到前一行
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
## 输出到logstash
output.logstash:
hosts: ["localhost:5044"]
然后启动即可
