日志集中管理服务

 linux系统日志

在 centos7 中，系统日志消息由两个服务负责处理：systemd-journald和rsyslog

/var/log目录由 rsyslog 维护，里面存放一些特定系统和服务的日志文件

                                                                                                                                  日志文件作用

# 清空日志方法
方法1:仅清空文件内容，不改变inode号(建议使用)
[root@server ~]# echo "" > /var/log/btmp
方法2：删除再创建文件，inode号改变，要重启相应的服务。
[root@server ~]# rm -f /var/log/btmp && touch /var/log/btmp

# rsyslog服务配置文件

# 通过该配置文件，可以看到各类日志及其日志文件存放位置

[root@server ~]# cat /etc/rsyslog.conf
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514　　　　　　　　是否允许514端口接收使用UDP协议转发过来的日志

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514  　　　　　 是否允许514端口接收使用TCP协议转发过来的日志 

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don‘t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog                                # -表示先存在内存，存到一定量再一次性写到硬盘中

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

# 日志输入的规则

.级别名 高于这个级别的日志都记录

.=级别名 仅记录该级别的日志

.!级别名 记录除了该级别的日志

类别名.none 不记录某个类别

自定义ssh服务的日志类型和存储位置

# 在rsyslog服务的配置文件中添加ssh的日志级别和日志存放位置

[root@server ~]# vi /etc/rsyslog.conf

74 local0.*                                                /var/log/sshd.log

# 修改sshd的配置文件，设置日志级别与rsyslog中的日志级别相同

[root@server ~]# vi /etc/ssh/sshd_config

32 SyslogFacility local0

#重启rsyslog服务

[root@server ~]# systemctl restart rsyslog

#重启ssh服务
[root@server ~]# systemctl restart sshd

# 查看sshd服务的日志存储位置是否有日志产生,可以看到有日志产生

[root@server ~]# cat /var/log/sshd.log
Jul 20 17:52:03 server sshd[19021]: Server listening on 0.0.0.0 port 22.
Jul 20 17:52:03 server sshd[19021]: Server listening on :: port 22.

# 日志防止删除

# 可以使用隐藏属性进行设置，+a,只可以追加内容，不可删除文件

[root@server ~]# chattr +a /var/log/sshd.log
[root@server ~]# lsattr /var/log/sshd.log
-----a---------- /var/log/sshd.log

[root@server ~]# rm -f /var/log/sshd.log
rm: cannot remove ‘/var/log/sshd.log’: Operation not permitted

[root@server ~]# chattr -a /var/log/sshd.log

[root@server ~]# rm -f /var/log/sshd.log

日志切割、日志轮替(logrotate)

# linux下的日志分为动态增长和静态增长
# 都可以使用split工具进行切割
# logrotate支持按时间和大小来自动切割,以防止日志文件太大。
# logrotate(轮替、轮循、轮转)：当日志达到某个特定的大小或时间,我们将日志按大小、按时间切割,之前的日志(归档日志、历史日志)保留一个备份,再创建一个同名的文件保存新的日志。
# logrotate的配置文件
# /etc/logrotate.d/ 存放指定服务日志切割规则的配置文件

[root@server ~]# cat /etc/logrotate.conf
# see "man logrotate" for details

#全局配置日志切割规则
# rotate log files weekly
weekly　　# 每周切割一次

# keep 4 weeks worth of backlogs
rotate 4      #保留至今的4份历史数据，切割走的数据就是历史数据

# create new (empty) log files after rotating old ones
create　　# 指定权限和所有者、所属主

# use date as a suffix of the rotated file
dateext　　# 以日期为文件后缀名

# uncomment this if you want your log files compressed
#compress　　# 切割后的历史数据，使用gzip压缩

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we‘ll rotate them here
/var/log/wtmp {　　# 单独配置日志切割规则
monthly
create 0664 root utmp
minsize 1M　　# 文件大小达到这个值就进行切割
rotate 1
}

/var/log/btmp {　　# 单独配置日志切割规则
missingok　　# 文件丢失不报错
monthly　　# 每月切割一次
create 0600 root utmp　　# 设置文件的权限和所有者、所属组
rotate 1
}

# system-specific logs may be also be configured here.

# 其他的日志切割规则
# delaycompress # 延迟压缩，上周的切割日志不压缩，上上周之前的切割日志压缩
# notifempty # 日志文件为空，不进行轮循
# -----------------------------------------
# postrotate # 启动标志
#### 执行脚本
# endscript # 结束标志
# /var/lib/logrotate/logrotate.status

# 默认记录logrotate上次轮换日志文件的时间

# 即可以看到 当前的日志文件的内容 仅包含从上次轮换日志文件的时间 至今的内容。

[root@server ~]# ll /var/lib/logrotate/logrotate.status
-rw-r--r--. 1 root root 814 Jul 20 03:31 /var/lib/logrotate/logrotate.status

[root@server ~]# cat /var/lib/logrotate/logrotate.status
logrotate state -- version 2
"/var/log/yum.log" 2020-6-25-0:0:0
"/var/log/cups/page_log" 2020-6-25-0:0:0
"/var/log/cups/error_log" 2020-6-25-0:0:0
"/var/log/boot.log" 2020-7-16-3:29:1
"/var/log/cups/access_log" 2020-7-6-10:32:1
"/var/log/wtmp" 2020-6-25-0:0:0
"/var/log/chrony/*.log" 2020-6-25-0:0:0
"/var/log/spooler" 2020-7-19-3:44:12
"/var/log/btmp" 2020-7-1-9:10:1
"/var/log/iscsiuio.log" 2020-6-25-0:0:0
"/var/log/maillog" 2020-7-19-3:44:12
"/var/log/libvirt/libvirtd.log" 2020-6-25-0:0:0
"/var/log/libvirt/qemu/*.log" 2020-6-25-0:0:0
"/var/log/wpa_supplicant.log" 2020-6-25-0:0:0
"/var/log/secure" 2020-7-19-3:44:12
"/var/log/numad.log" 2020-6-25-0:0:0
"/var/log/ppp/connect-errors" 2020-6-25-0:0:0
"/var/log/messages" 2020-7-19-3:44:12
"/var/log/cron" 2020-7-19-3:44:12
"/var/account/pacct" 2020-6-25-0:0:0

# ssh服务日志切割

# 编辑ssh服务日志轮替规则

# 日志路径 { 轮替规则 }

[root@server ~]# vi /etc/logrotate.d/sshd
/var/log/sshd.log{
missingok
weekly
create 0600 root root
minsize 1M
rotate 3
}

# 重启rsyslog服务，因为/var/log/sshd.log日志是由rsyslog进行管理的，要切割也要rsyslog知道才行

[root@server ~]# systemctl restart rsyslog

[root@server ~]# logrotate -d /etc/logrotate.d/sshd　　# 预加载配置文件，看会不会报错

# 提示目前日志不需要轮询
reading config file /etc/logrotate.d/sshd
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/sshd.log weekly (3 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/sshd.log

[root@server ~]# logrotate -vf /etc/logrotate.d/sshd　　　　# 强制轮询
reading config file /etc/logrotate.d/sshd
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/sshd.log forced from command line (3 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/sshd.log
log needs rotating
rotating log /var/log/sshd.log, log->rotateCount is 3
dateext suffix ‘-20200720‘
glob pattern ‘-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]‘
renaming /var/log/sshd.log.3 to /var/log/sshd.log.4 (rotatecount 3, logstart 1, i 3),
old log /var/log/sshd.log.3 does not exist　　# 日志3重命名为日志4
renaming /var/log/sshd.log.2 to /var/log/sshd.log.3 (rotatecount 3, logstart 1, i 2),
old log /var/log/sshd.log.2 does not exist　　# 日志2重命名为日志3
renaming /var/log/sshd.log.1 to /var/log/sshd.log.2 (rotatecount 3, logstart 1, i 1),
old log /var/log/sshd.log.1 does not exist　　# 日志1重命名为日志2
renaming /var/log/sshd.log.0 to /var/log/sshd.log.1 (rotatecount 3, logstart 1, i 0),
old log /var/log/sshd.log.0 does not exist　　# 日志0重命名为日志1
log /var/log/sshd.log.4 doesn‘t exist -- won‘t try to dispose of it　　# 日志4不存在，故不能处理它

fscreate context set to system_u:object_r:var_log_t:s0

renaming /var/log/sshd.log to /var/log/sshd.log.1　　# 重命名当前日志为日志1
creating new /var/log/sshd.log mode = 0600 uid = 0 gid = 0　　# 创建一个同名当前日志文件
set default create context

# 查看日志轮替情况

# 可以看到sshd.log的大小为0，即是刚新建的当前日志文件。

[root@server ~]# ll /var/log/sshd.log*
-rw-------. 1 root root 0 Jul 20 19:55 /var/log/sshd.log
-rw-------. 1 root root 141 Jul 20 17:52 /var/log/sshd.log.1

# nginx服务日志切割

# 编辑nginx服务日志轮替规则

[root@server ~]# vi /etc/logrotate.d/nginx
/usr/local/nginx/logs/*.log{
daily
rotate 5
sharedscripts　　#所有的日志文件都轮转完毕后统一执行一次切割脚本，适用多个日志文件同时执行。

# 满足以下条件，才会进行轮替日志
postrotate　　#执行命令的开始标志
if [ -f /usr/local/nginx/logs/nginx.pid ];then　　#判断nginx是否启动
/usr/local/nginx/sbin/nginx -s reload

#让nginx重新加载配置文件，生成新的日志文件，如果nginx没启动不做操作
fi
endscript　　#执行命令结束标志
}

# 重启rsyslog服务，因为/var/log/nginx日志是由rsyslog进行管理的，要切割也要rsyslog知道才行。

[root@server ~]# systemctl restart rsyslog
[root@server ~]# logrotate -d /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx
error: /etc/logrotate.d/nginx:4 unknown option ‘sharescripts‘ -- ignoring line
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /usr/local/nginx/logs/*.log after 1 days (5 rotations)
empty log files are rotated, old logs are removed
considering log /usr/local/nginx/logs/access.log
log does not need rotating (log has been already rotated)considering log /usr/local/nginx/logs/error.log
log does not need rotating (log has been already rotated)

[root@server ~]# logrotate -vf /etc/logrotate.d/nginx　　# 强制轮询
reading config file /etc/logrotate.d/nginx
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /usr/local/nginx/logs/*.log forced from command line (5 rotati ons)
empty log files are rotated, old logs are removed
considering log /usr/local/nginx/logs/access.log　　#访问日志
log needs rotating
considering log /usr/local/nginx/logs/error.log　　#错误日志
log needs rotating
rotating log /usr/local/nginx/logs/access.log, log->rotateCount is 5　　#保留5次归档日志
dateext suffix ‘-20200720‘
glob pattern ‘-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]‘
renaming /usr/local/nginx/logs/access.log.5 to /usr/local/nginx/logs/access.log. 6 (rotatecount 5, logstart 1, i 5),
old log /usr/local/nginx/logs/access.log.5 does not exist
renaming /usr/local/nginx/logs/access.log.4 to /usr/local/nginx/logs/access.log. 5 (rotatecount 5, logstart 1, i 4),
old log /usr/local/nginx/logs/access.log.4 does not exist
renaming /usr/local/nginx/logs/access.log.3 to /usr/local/nginx/logs/access.log. 4 (rotatecount 5, logstart 1, i 3),
old log /usr/local/nginx/logs/access.log.3 does not exist
renaming /usr/local/nginx/logs/access.log.2 to /usr/local/nginx/logs/access.log. 3 (rotatecount 5, logstart 1, i 2),
old log /usr/local/nginx/logs/access.log.2 does not exist
renaming /usr/local/nginx/logs/access.log.1 to /usr/local/nginx/logs/access.log. 2 (rotatecount 5, logstart 1, i 1),
renaming /usr/local/nginx/logs/access.log.0 to /usr/local/nginx/logs/access.log. 1 (rotatecount 5, logstart 1, i 0),
old log /usr/local/nginx/logs/access.log.0 does not exist
log /usr/local/nginx/logs/access.log.6 doesn‘t exist -- won‘t try to dispose of it
rotating log /usr/local/nginx/logs/error.log, log->rotateCount is 5
dateext suffix ‘-20200720‘
glob pattern ‘-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]‘
renaming /usr/local/nginx/logs/error.log.5 to /usr/local/nginx/logs/error.log.6 (rotatecount 5, logstart 1, i 5),
old log /usr/local/nginx/logs/error.log.5 does not exist
renaming /usr/local/nginx/logs/error.log.4 to /usr/local/nginx/logs/error.log.5 (rotatecount 5, logstart 1, i 4),
old log /usr/local/nginx/logs/error.log.4 does not exist
renaming /usr/local/nginx/logs/error.log.3 to /usr/local/nginx/logs/error.log.4 (rotatecount 5, logstart 1, i 3),
old log /usr/local/nginx/logs/error.log.3 does not exist
renaming /usr/local/nginx/logs/error.log.2 to /usr/local/nginx/logs/error.log.3 (rotatecount 5, logstart 1, i 2),
old log /usr/local/nginx/logs/error.log.2 does not exist
renaming /usr/local/nginx/logs/error.log.1 to /usr/local/nginx/logs/error.log.2 (rotatecount 5, logstart 1, i 1),
renaming /usr/local/nginx/logs/error.log.0 to /usr/local/nginx/logs/error.log.1 (rotatecount 5, logstart 1, i 0),
old log /usr/local/nginx/logs/error.log.0 does not exist
log /usr/local/nginx/logs/error.log.6 doesn‘t exist -- won‘t try to dispose of i t
fscreate context set to system_u:object_r:usr_t:s0
renaming /usr/local/nginx/logs/access.log to /usr/local/nginx/logs/access.log.1　　#当前的访问日志重命名为访问日志1
fscreate context set to unconfined_u:object_r:usr_t:s0
renaming /usr/local/nginx/logs/error.log to /usr/local/nginx/logs/error.log.1　　#当前的错误日志重命名为错误日志1
running postrotate script
set default create context

# 查看日志轮询情况

# 可以看到已经轮询一次日志了。

[root@server ~]# ll /usr/local/nginx/logs/*log*
-rw-r--r--. 1 root root 0 Jul 20 20:28 /usr/local/nginx/logs/access.log
-rw-r--r--. 1 root root 4737185 Jul 19 11:17 /usr/local/nginx/logs/access.log.1
-rw-r--r--. 1 root root 61 Jul 20 20:28 /usr/local/nginx/logs/error.log
-rw-r--r--. 1 root root 3007 Jul 20 20:26 /usr/local/nginx/logs/error.log.1

搭建日志收集服务器

日志收集服务器：服务器作为日志接收端，客户端作为日志发送端，所有的客户端上的日志都通过514/tcp端口号发送到服务器上进行管理。

运行原理： 服务端开放514端口，允许客户端通过该端口将指定的日志远程传输到服务端的/var/log/messages文件中。

服务端IP：172.20.77.201

客户端IP：172.20.77.202

# 服务端配置，使得rsyslog服务支持使用tcp协议传输日志
# tcp协议收集日志-可靠完整
# udp协议收集日志-速度快-不保证数据完整
# 编辑rsyslog的配置文件，启用tcp协议收集日志

[root@server ~]# vi /etc/rsyslog.conf

19 $ModLoad imtcp
20 $InputTCPServerRun 514

# 重启rsyslog服务

[root@server ~]# systemctl restart rsyslog

#查看514端口监听情况

[root@server ~]# netstat -tlnp|grep 514
tcp　　 0　　 0 0.0.0.0:514 　　 0.0.0.0:* 　　 LISTEN 　　 40191/rsyslogd
tcp6 　 0 　　 0 :::514　　　　　  :::* 　　　　 LISTEN　　 40191/rsyslogd

#防火墙放行TCP 514端口

[root@server ~]# firewall-cmd --permanent --add-port=514/tcp

[root@server ~]# firewall-cmd --reload

# 客户端配置，收集客户端上的所有类型和等级的日志，并发送到指定的日志服务器。

[root@client ~]# vi /etc/rsyslog.conf

 91 *.* @@172.20.77.201:514

#重启rsyslog服务
[root@client ~]# systemctl restart rsyslog

# 服务器使用 udp 协议，客户端使用的配置文件中这一行只能有一个@

# *.* @172.20.77.201:514

# 服务器使用 tcp 协议，客户端使用的配置文件中这一行必须有两个@@

# *.* @@172.20.77.201:514

# 客户端上手动生成日志

# 可以看到，客户端上是有日志生成的

[root@client ~]# logger "this is test log"
[root@client ~]# tail -1 /var/log/messages
Jul 20 20:59:26 client root: this is test log

#查服务端日志接收情况

[root@server ~]# tail -f /var/log/messages|grep client

Jul 20 20:59:26 client root: this is test log

# 仅收集客户端上的sshd服务日志

# 在客户端上自定义ssh服务的日志类型和存储位置，这里类型设置为local0，存储位置为/var/log/sshd.log

[root@client ~]# vi /etc/rsyslog.conf

local0.* @@172.20.77.201:514

#重启rsyslog服务
[root@client ~]# systemctl restart rsyslog

#测试一下服务端接收到客户端日志情况

 通过以上实验，可以通过一台日志服务器，集中管理多台远程主机的日志了

日志集中管理服务

原文：https://www.cnblogs.com/happysnowy/p/13339927.html