一、需求及基础：
场景：
1、开发人员不能登录线上服务器查看详细日志
2、各个系统都有日志，日志数据分散难以查找
3、日志数据量大，查询速度慢，或者数据不够实时
4、一个调用会涉及到多个系统，难以在这些协调中快速定位数据
二、ELS的概念：
elasticsearch：搜索引擎，提供索引，搜索功能
Logstash：接收，处理，转发日志
Kibana：独立的、美观的图形数据web界面
三、安装及配置
1、 elasticsearch安装
安装java环境，1.8.20或以上的版本
1、安装Java环境 tar xf jdk-8u201-linux-x64.tar.gz vim /etc/profile export JAVA_HOME=/root/jdk export PATH=$JAVA_HOME/bin:$PATH source /etc/profile 2、安装elasticsearch wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.0.tar.gz tar xf elasticsearch-1.7.0.tar.gz ln -s elasticsearch-1.7.0 /usr/local/elasticsearch 3、修改配置文件 grep -n ‘^[a-z]‘ /usr/local/elasticsearch/config/elasticsearch.yml 32:cluster.name: elasticsearch #必须修改 40:node.name: "stu02" #必须修改 47:node.master: true 51:node.data: true 107:index.number_of_shards: 5 111:index.number_of_replicas: 1 145:path.conf: /usr/local/elasticsearch/config 149:path.data: /usr/local/elasticsearch/data 159:path.work: /usr/local/elasticsearch/work 163:path.logs: /usr/local/elasticsearch/logs 167:path.plugins: /usr/local/elasticsearch/plugins 184:bootstrap.mlockall: true 4、启动 /usr/local/elasticsearch/bin/elasticsearch -d #（可加入参数，-Xms512m -Xmx512m） netstat -lnt | egrep ‘9200|9300‘ jps -lvm
测试：
curl 127.0.0.1:9200 curl -i -XGET http://192.168.4.16:9200
安装ELS监控管理插件
1、离线安装插件 wget http://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip /usr/local/elasticsearch/bin/plugin -i marvel -u file:///tmp/marvel-latest.zip 2、在线安装插件 /usr/local/elasticsearch/bin/plugin -i elasticsearch/marvel/latest/ /usr/local/elasticsearch/bin/plugin install mobz/elasticsearch-head 3、卸载插件 /usr/local/elasticsearch/bin/plugin -r marvel
生产可能需要配置已下：
max_file_descriptors: 64000 /etc/sysctl.conf sysctl -w vm_max_count=262144
2、 logstash安装
1、安装Java环境 tar xf jdk-8u201-linux-x64.tar.gz vim /etc/profile export JAVA_HOME=/root/jdk export PATH=$JAVA_HOME/bin:$PATH source /etc/profile 2、安装logstash wget https://download.elastic.co/logstash/logstash/logstash-1.5.3.tar.gz tar xf logstash-1.5.3.tar.gz ln -s /root/logstash-1.5.3/ /usr/local/logstash
标准输入和标准输出
/usr/local/logstash/bin/logstash -e ‘input { stdin{} } output { stdout{} }‘
使用ruby进行更详细的输出
/usr/local/logstash/bin/logstash -e ‘input { stdin{} } output { stdout{codec => rubydebug}}‘
输出到elasticsearch
/usr/local/logstash/bin/logstash -e ‘input { stdin{} } output { elasticsearch
{ host => "192.168.4.16" protocol => "http"} }‘
读取日志并输出到/tmp下
vim /etc/logstash/conf.d/logstash.conf
input { file { path => "/tmp/messages" } } output { file { path => "/tmp/log-%{+YYYY-MM-dd}messages.gz" gzip => true } }
测试配置文件
/usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -t
启动服务
/usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
测试
cat /var/log/messages >> /tmp/messages
ll /tmp/log-2019-02-11messages.gz
把输出直接传输到elasticsearch
vim /etc/logstash/conf.d/logstash.conf input { file { path => "/tmp/messages" } } output { file { path => "/tmp/log-%{+YYYY-MM-dd}.messages.gz" gzip => true } elasticsearch { host => ["192.168.4.16"] protocol => "http" index => "system-message-%{+YYYY.MM.dd}" } }
在集群管理平台查看结果
打开浏览器输入：http://192.168.4.16:9200/_plugin/head/
将logstash输出给redis
一台logstash的配置文件
input { file { path =>"/tmp/messages" } } output { redis { data_type => "list" key => "system-message" host => "192.168.4.17" port => "6379" db => "0" } }
另一台logstash的配置文件
input { redis { data_type => "list" key => "system-message" host => "192.168.4.17" port => "6379" db => "0" } } output { elasticsearch { host => ["192.168.4.16"] protocol => "http" index => "redis-message-%{+YYYY.MM.dd}" } }
原文：https://www.cnblogs.com/wuhg/p/10375727.html