环境介绍：
服务端：192.168.0.248 ：（es ,logstash,kibana,redis）
客户端：192.168.0.151 192.168.0.219
一：服务器端配置 192.168.0.248：
1）安装es:
https://www.cnblogs.com/yangxiaochu/p/9894536.html
2)安装redis
1 cd /opt 2 wget http://download.redis.io/releases/redis-4.0.1.tar.gz 3 tar xzf redis-4.0.1.tar.gz 4 cd redis-4.0.1 5 make
1 [root@elk redis-4.0.1]# ls 2 00-RELEASENOTES BUGS CONTRIBUTING COPYING deps INSTALL Makefile MANIFESTO README.md redis.conf runtest runtest-cluster runtest-sentinel sentinel.conf src tests utils 3 [root@elk redis-4.0.1]# cd src/ 4 [root@elk src]# make install PREFIX=/usr/local/redis 5 CC Makefile.dep 6 7 Hint: It‘s a good idea to run ‘make test‘ ;) 8 9 INSTALL install 10 INSTALL install 11 INSTALL install 12 INSTALL install 13 INSTALL install 14 [root@elk redis-4.0.1]# cp redis.conf /usr/local/redis/ 15 [root@elk redis-4.0.1]# cd /usr/local/redis/ 16 [root@elk redis]# ls 17 bin redis.conf 18 [root@elk redis]# /usr/local/redis/bin/redis-se 19 redis-sentinel redis-server 20 [root@elk redis]# /usr/local/redis/bin/redis-server /usr/local/redis/redis.conf 21 6929:C 02 Nov 10:12:41.884 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 22 6929:C 02 Nov 10:12:41.885 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=6929, just started 23 6929:C 02 Nov 10:12:41.885 # Configuration loaded 24 _._ 25 _.-``__ ‘‘-._ 26 _.-`` `. `_. ‘‘-._ Redis 4.0.1 (00000000/0) 64 bit 27 .-`` .-```. ```\/ _.,_ ‘‘-._ 28 ( ‘ , .-` | `, ) Running in standalone mode 29 |`-._`-...-` __...-.``-._|‘` _.-‘| Port: 6379 30 | `-._ `._ / _.-‘ | PID: 6929 31 `-._ `-._ `-./ _.-‘ _.-‘ 32 |`-._`-._ `-.__.-‘ _.-‘_.-‘| 33 | `-._`-._ _.-‘_.-‘ | http://redis.io 34 `-._ `-._`-.__.-‘_.-‘ _.-‘ 35 |`-._`-._ `-.__.-‘ _.-‘_.-‘| 36 | `-._`-._ _.-‘_.-‘ | 37 `-._ `-._`-.__.-‘_.-‘ _.-‘ 38 `-._ `-.__.-‘ _.-‘ 39 `-._ _.-‘ 40 `-.__.-‘ 41 42 6929:M 02 Nov 10:12:41.887 # Server initialized 43 6929:M 02 Nov 10:12:41.887 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run 44 the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled‘ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.6929:M 02 Nov 10:12:41.887 * Ready to accept connections 45 46 47 ^C6929:signal-handler (1541124774) Received SIGINT scheduling shutdown... 48 6929:M 02 Nov 10:12:54.552 # User requested shutdown... 49 6929:M 02 Nov 10:12:54.553 * Saving the final RDB snapshot before exiting. 50 6929:M 02 Nov 10:12:54.554 * DB saved on disk 51 6929:M 02 Nov 10:12:54.554 * Removing the pid file. 52 6929:M 02 Nov 10:12:54.554 # Redis is now ready to exit, bye bye...
编辑配置文件
1 [root@elk redis]# vim redis.conf 2 # 将daemonize的值改为yes 默认情况，Redis不是在后台运行，我们需要把redis放在后台运行 3 # By default Redis does not run as a daemon. Use ‘yes‘ if you need it. 4 # Note that Redis will write a pid file in /var/run/redis.pid when daemonized. 5 daemonize yes 6 bind 0.0.0.0
重新启动redis
1 [root@elk redis]# /usr/local/redis/bin/redis-server /usr/local/redis/redis.conf 2 6935:C 02 Nov 10:14:19.229 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 3 6935:C 02 Nov 10:14:19.229 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=6935, just started 4 6935:C 02 Nov 10:14:19.229 # Configuration loaded
测试redis
1 [root@elk redis]# /usr/local/redis/bin/redis-cli 2 127.0.0.1:6379> set aa aavlue 3 OK 4 127.0.0.1:6379> get aa 5 "aavlue" 6 127.0.0.1:6379> exit
3)安装logstash
1 cd /opt 2 wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.2.tar.gz 3 tar -xzf logstash-5.5.2.tar.gz
测试一下：
测试logstash 是否正常运行
1 /opt/logstash-5.5.2/bin/logstash -e ‘input{stdin{}}output{stdout{codec=>rubydebug}}‘
敲入Hello World,回车
输出结果：
Hello World { "@timestamp" => 2017-08-30T17:24:25.553Z, "@version" => "1", "host" => "node4", "message" => "Hello World" }
开启logstash indexer
1 cd /opt/logstash-5.5.2 2 mkdir etc 3 vim /opt/logstash-5.5.2/etc/logstash_indexer.conf
1 input { 2 redis { 3 host => "192.168.0.248" 4 data_type => "list" 5 port => "6379" 6 key => "logstash:redis" 7 type => "redis-input" 8 } 9 } 10 11 output { 12 elasticsearch { 13 hosts => ["192.168.0.248:9200"] 14 index => "logstash-%{type}-%{+YYYY.MM.dd}" 15 document_type => "%{type}" 16 flush_size => 20000 17 idle_flush_time => 10 18 sniffing => true 19 template_overwrite => true 20 } 21 }
input {}解释
读取Redis key logstash:redis 的数据
output {}解释
批量发送Elasticsearch，本插件的 flush_size 和 idle_flush_time 两个参数共同控制 Logstash 向 Elasticsearch 发送批量数据的行为。以上面示例来说：Logstash 会努力攒到 20000 条数据一次性发送出去，但是如果 10 秒钟内也没攒够 20000 条，Logstash 还是会以当前攒到的数据量发一次。 默认情况下，flush_size 是 500 条，idle_flush_time 是 1 秒。这也是很多人改大了 flush_size 也没能提高写入 ES 性能的原因——Logstash 还是 1 秒钟发送一次。
启动logstash
cd /opt/logstash-5.5.2/logs/ mkdir log1 log2
启动 logstash indexer logstash 读redis日志发送到es
1 nohup /opt/logstash-5.5.2/bin/logstash -f /opt/logstash-5.5.2/etc/logstash_indexer.conf --path.data=/opt/logstash-5.5.2/logs/log2 > /dev/null 2>&1 &
4)安装kibana
1 root@elk opt]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz 2 --2018-11-02 10:35:24-- https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz 3 Resolving artifacts.elastic.co (artifacts.elastic.co)... 107.21.202.15, 107.21.237.188, 107.21.237.95, ... 4 Connecting to artifacts.elastic.co (artifacts.elastic.co)|107.21.202.15|:443... connected. 5 HTTP request sent, awaiting response... 200 OK 6 Length: 51073441 (49M) [binary/octet-stream] 7 Saving to: ‘kibana-5.5.2-linux-x86_64.tar.gz’ 8 9 100%[===================================================================================================================================================>] 51,073,441 274KB/s in 3m 41s 10 11 2018-11-02 10:39:06 (226 KB/s) - ‘kibana-5.5.2-linux-x86_64.tar.gz’ saved [51073441/51073441] 12 13 [root@elk opt]# ls 14 elasticsearch-5.5.2 jdk1.8.0_102 kibana-5.5.2-linux-x86_64.tar.gz logstash-5.5.2.tar.gz redis-4.0.1.tar.gz 15 elasticsearch-5.5.2.tar.gz jdk-8u102-linux-x64.tar.gz logstash-5.5.2 redis-4.0.1 16 [root@elk opt]# tar xf kibana-5.5.2-linux-x86_64.tar.gz 17 [root@elk opt]# ls 18 elasticsearch-5.5.2 jdk1.8.0_102 kibana-5.5.2-linux-x86_64 logstash-5.5.2 redis-4.0.1 19 elasticsearch-5.5.2.tar.gz jdk-8u102-linux-x64.tar.gz kibana-5.5.2-linux-x86_64.tar.gz logstash-5.5.2.tar.gz redis-4.0.1.tar.gz 20 [root@elk opt]# cd kibana-5.5.2-linux-x86_64 21 [root@elk kibana-5.5.2-linux-x86_64]# ls 22 bin config data LICENSE.txt node node_modules NOTICE.txt optimize package.json plugins README.txt src ui_framework webpackShims 23 [root@elk kibana-5.5.2-linux-x86_64]# vim config/kibana.yml 24 [root@elk kibana-5.5.2-linux-x86_64]# cd bin/ 25 [root@elk bin]# ls 26 kibana kibana-plugin 27 [root@elk bin]# ./kibana 28 29 log [02:41:35.783] [info][status][plugin:kibana@5.5.2] Status changed from uninitialized to green - Ready 30 log [02:41:35.873] [info][status][plugin:elasticsearch@5.5.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch 31 log [02:41:35.908] [info][status][plugin:console@5.5.2] Status changed from uninitialized to green - Ready 32 log [02:41:35.945] [info][status][plugin:metrics@5.5.2] Status changed from uninitialized to green - Ready 33 log [02:41:36.137] [info][status][plugin:timelion@5.5.2] Status changed from uninitialized to green - Ready 34 log [02:41:36.142] [info][listening] Server running at http://0.0.0.0:5601 35 log [02:41:36.143] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow 36 log [02:41:40.958] [info][status][plugin:elasticsearch@5.5.2] Status changed from yellow to yellow - No existing Kibana index found 37 log [02:41:41.453] [info][status][plugin:elasticsearch@5.5.2] Status changed from yellow to green - Kibana index ready 38 log [02:41:41.456] [info][status][ui settings] Status changed from yellow to green - Ready
编辑配置文件修改
server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://192.168.0.248:9200"
客户端配置
1）安装logstash
安装过程同上
编辑logstash收集日志发送给redis的配置文件
input {
file {
type => "nginx_access_log"
path => "/usr/local/nginx/logs/host.access.log"
}
}
output {
redis {
host => "192.168.0.248"
data_type => "list"
port => "6379"
key => "logstash:redis"
}
}
启动logstash agent
[root@web1 log1]# nohup /opt/logstash-5.5.2/bin/logstash -f /opt/logstash-5.5.2/etc/logstash_agent_nginx.conf --path.data=/opt/logstash-5.5.2/logs/log1 > /dev/null 2>&1
kibana页面配置
原文：https://www.cnblogs.com/yangxiaochu/p/9897699.html