Linux集群利用rsyslog记录所有节点的操作指令_rsyslog记录所有命令日志

需求

想实现这么一个需求，假设集群中有3个节点，其中有一个master节点，使用者可以在3个节点上输入任意指令，每有一个操作，都将其操作指令实时发送到master节点，并将操作信息记录在某个特定文件中。

环境

操作系统安装完成之后，已自带rsyslog服务

[root@master profile.d]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-01-13 15:55:07 CST; 3 days ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 631 (rsyslogd)
   Memory: 480.0K
   CGroup: /system.slice/rsyslog.service
           └─631 /usr/sbin/rsyslogd -n

Jan 13 15:55:07 master.node systemd[1]: Starting System Logging Service...
Jan 13 15:55:07 master.node rsyslogd[631]:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="631" x-info="http://www.rsyslog.com"] start
Jan 13 15:55:07 master.node systemd[1]: Started System Logging Service.
1
2
3
4
5
6
7
8
9
10
11
12
13
14

查看rsyslog版本

[root@master profile.d]# rsyslogd -version
rsyslogd 8.24.0, compiled with:
	PLATFORM:				x86_64-redhat-linux-gnu
	PLATFORM (lsb_release -d):		
	FEATURE_REGEXP:				Yes
	GSSAPI Kerberos 5 support:		Yes
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	memory allocator:			system default
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes
	Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

配置日志服务

安装RELP协议包

3个节点均安装RELP包

yum install -y rsyslog-relp
1

配置日志服务器

master节点充当日志服务器
修改配置文件/etc/rsyslog.conf，增加有关RELP两行

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
# $ModLoad imtcp
# $InputTCPServerRun 514

$ModLoad imrelp  #加载RELP模块
$InputRELPServerRun 2514  #传输端口
1
2
3
4
5
6
7
8
9
10

传输端口监听
修改配置文件/etc/sysconfig/rsyslog

# Options for rsyslogd
# Syslogd options are deprecated since rsyslog v3.
# If you want to use them, switch to compatibility mode 2 by "-c 2"
# See rsyslogd(8) for more details
SYSLOGD_OPTIONS="-r2514 -c2"
1
2
3
4
5

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
1
2

重启日志服务

systemctl restart rsyslog
1

查看服务端口状态

[root@master profile.d]# netstat -antp | grep 2514
tcp        0      0 0.0.0.0:2514            0.0.0.0:*               LISTEN      14602/rsyslogd      
tcp6       0      0 :::2514                 :::*                    LISTEN      14602/rsyslogd 
1
2
3

配置日志客户机端

以worker2节点为例
修改配置文件 /etc/rsyslog.conf

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
# $ModLoad imtcp
# $InputTCPServerRun 514

$ModLoad omrelp  #加载RELP模块
*.* :omrelp:10.XX.XX.52:2514  #目标主机端口
1
2
3
4
5
6
7
8
9
10

如果是master节点本身，则不需要再次配置目标主机端口，只需要配置后文中的收集脚本就行了

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
1
2

重启日志服务

systemctl restart rsyslog
1

查看服务端口状态

[root@worker2 opt]# netstat -antp | grep 2514
tcp       90      0 10.XX.XX.50:36516     10.XX.XX.52:2514      ESTABLISHED 1064/rsyslogd 
1
2

测试

在服务端（master节点）查看/var/log/messages日志文件

[root@master profile.d]# tail -f /var/log/messages 
1

在客户机输入测试指令

[root@worker2 opt]# logger -t kern -p error "test log nihao"
1

观察服务端日志信息

Jan 17 09:22:26 master rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="631" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 17 09:22:26 master systemd: Starting System Logging Service...
Jan 17 09:22:26 master rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="14602" x-info="http://www.rsyslog.com"] start
Jan 17 09:22:26 master systemd: Started System Logging Service.
Jan 17 09:23:07 master systemd: Reloading.
Jan 17 09:30:45 worker2 systemd: Stopping System Logging Service...
Jan 17 09:30:45 worker2 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="31376" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 17 09:30:45 worker2 systemd: Starting System Logging Service...
Jan 17 09:30:45 worker2 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1064" x-info="http://www.rsyslog.com"] start
Jan 17 09:30:45 worker2 systemd: Started System Logging Service.
Jan 17 09:34:16 worker2 kern: test log nihao
1
2
3
4
5
6
7
8
9
10
11

测试成功。

操作指令记录

指令信息

日志产生时的年月日时间

[root@worker2 opt]# date +%Y-%m-%d
2020-01-17
1
2

当前会话登陆的用户

[root@worker2 opt]# who am i
root     pts/6        2020-01-16 17:09 (10.180.201.20)
1
2

当前操作用户

[root@worker2 opt]# whoami
root
1
2

注意两者区别：
假设以root用户登陆的xshell会话连接，操作完su user1之后
会话登陆用户是root，操作用户就变为user1了

当前最新的操作指令

[root@worker2 opt]# history 1 | { read x cmd; echo "$cmd"; }
history 1 | { read x cmd; echo "$cmd"; }
1
2

自动收集脚本

以worker2节点为例
设置环境变量，进入/etc/profile.d/目录下，新建脚本

[root@worker2 ~]# cd /etc/profile.d/
[root@worker2 profile.d]# touch operate.sh
1
2

编辑脚本内容

# export HISTORY_FILE=/var/log/operate/operate.log
# datetmp=`date +%Y-%m-%d`
whotmp=`who am i |awk '{print $1 " " $2 " " $5}'` #根据需要取
# whotmp2=`whoami`
# commandtmp=`history 1 | { read x cmd; echo "$cmd"; }`
export  PROMPT_COMMAND='logger -t kern -p err "${whotmp} $(whoami) :  $(history 1 | { read x cmd; echo "$cmd"; })"'
1
2
3
4
5
6

保存退出，使环境变量生效

[root@worker2 profile.d]# source /etc/profile
1

在master节点查看收集的信息

[root@master ~]# tail -f /var/log/messages
1

在客户节点worker2当前会话窗口测试指令
同节点其它会话窗口需要执行source /etc/profile或者重新打开会话窗口

[root@worker2 profile.d]# nihao
-bash: nihao: command not found
[root@worker2 profile.d]# date
Fri Jan 17 11:14:46 CST 2020
1
2
3
4

master节点查看

......
Jan 17 11:14:45 worker2 kern: root pts/4 (10.XX.XX.20) root :  nihao
Jan 17 11:14:46 worker2 kern: root pts/4 (10.XX.XX.20) root :  date
1
2
3

指令保存文件设置

# 新建操作日志保存位置
[root@master profile.d]# mkdir -p /var/log/operate
[root@master profile.d]# cd /var/log/operate/
# 新建操作日志保存文件
[root@master operate]# touch operate.log
# 将文件赋予一个最低权限的用户
[root@master operate]# chown nobody:nobody /var/log/operate/operate.log 
# 该文件赋予所有人的写权限
[root@master operate]# chmod 002 /var/log/operate/operate.log 
# 设置文件权限,使所有用户对该文件只有追加权限
[root@master operate]# chattr +a /var/log/operate/operate.log
1
2
3
4
5
6
7
8
9
10
11

文件状态

[root@master operate]# ll /var/log/operate/
total 0
--------w- 1 nobody nobody 0 Jan 17 10:18 operate.log
1
2
3

自定义保存

master节点修改配置文件/etc/rsyslog.conf

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

# --Save operate log --   新增指定日志文件
local3.info                                             /var/log/operate/operate.log
1
2
3
4
5
6
7
8

重启rsyslog服务

[root@master ~]# systemctl restart rsyslog
1

修改收集脚本

# export HISTORY_FILE=/var/log/operate/operate.log
# datetmp=`date +%Y-%m-%d`
whotmp=`who am i |awk '{print $1 " " $2 " " $5}'`
# whotmp2=`whoami`
# commandtmp=`history 1 | { read x cmd; echo "$cmd"; }`
export  PROMPT_COMMAND='logger -p local3.info "${whotmp} $(whoami) :  $(history 1 | { read x cmd; echo "$cmd"; })"'
1
2
3
4
5
6

使环境变量生效

[root@worker2 profile.d]# source /etc/profile
1

在客户机worker2上测试输入指令

[root@worker2 profile.d]# haha
-bash: haha: command not found
[root@worker2 profile.d]# date
Fri Jan 17 11:32:34 CST 2020
1
2
3
4

在master节点查看收集到的信息

[root@master ~]# tail -f /var/log/operate/operate.log 
...
Jan 17 11:32:30 worker2 root: root pts/4 (10.180.201.20) root :  haha
Jan 17 11:32:34 worker2 root: root pts/4 (10.180.201.20) root :  date
1
2
3
4

各节点单独记录日志

如果把操作日志保存在各个节点上，不需要集中保存在master节点
那么只需要在各个节点 上准备好目标日志文件
然后配置环境变量中的脚本就行了

export HISTORY_FILE=/var/log/monitor/monitor.log
export PROMPT_COMMAND='{ date "+%y-%m-%d %T $(who am i |awk "{print \$1\" \"\$2\" \"\$5}") $(whoami) $(history 1 | { read x cmd; echo "$cmd"; })"; } >>$HISTORY_FILE'
1
2

或根据主机名保存

export MY_HISTORY_PATH=/var/log/monitor
if [ ! -d "$MY_HISTORY_PATH" ];then
  mkdir -p "$MY_HISTORY_PATH"
fi
export HISTORY_FILE=$MY_HISTORY_PATH/$(hostname).log
if [ ! -f "$HISTORY_FILE" ];then
  touch "$HISTORY_FILE"
  # chmod 777 $HISTORY_FILE
fi
echo $HISTORY_FILE
export PROMPT_COMMAND='{ date "+%y-%m-%d %T $(hostname) $(who am i |awk "{print \$1\" \"\$2\" \"\$5}") $(whoami) : $(history 1 | { read x cmd; echo "$cmd"; })"; } >>$HISTORY_FILE'
1
2
3
4
5
6
7
8
9
10
11

关于日志的格式

默认的日志格式时间看起来不爽
修改master节点配置文件/etc/rsyslog.conf

# --Save operate log --   新增指定日志文件
$template myformat,"%$NOW% %TIMESTAMP:8:15% %hostname% %msg%\n"
local3.info              /var/log/operate/operate.log;myformat
1
2
3

修改完并重启rsyslog服务后日志显示效果如下

# 日期 时间 日志产生的主机名 登陆用户 会话形式 哪个IP在会话 操作用户 ： 内容
...
2020-01-19 09:51:57 worker2  root pts/0 (10.XX.XX.20) root :  date
2020-01-19 09:51:59 worker2  root pts/0 (10.XX.XX.20) root :  nihao
1
2
3
4

日志的转储
进入/etc/logrotate.d目录，新建operate文件

/var/log/operate/operate.log
{
    weekly     #每周转储一次
    minsize 10M    #超过10M的时候才真正转储
    rotate 10      #共有10个文件
    prerotate      #转储前执行
       chattr -a /var/log/operate/operate.log #取消权限
    endscript
    postrotate     #转储后执行
       chattr +a /var/log/operate/operate.log #加上权限
       /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true   #重启rsyslog
       /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true   #重启rsyslog
    endscript
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14

手动强制转储

[root@master logrotate.d]# /usr/sbin/logrotate -vf /etc/logrotate.conf 
1

效果如下

[root@master operate]# ll
total 20
--------w- 1 nobody nobody     0 Jan 19 11:17 operate.log
--------w- 1 nobody nobody 17924 Jan 19 11:20 operate.log-20200119
1
2
3
4

参考

日志服务：
https://blog.csdn.net/wwl012345/article/details/90408772
本地收集：
http://www.manongjc.com/article/61750.html
服务配置：
https://blog.51cto.com/11555417/2163289?source=dra
https://www.cnblogs.com/leo2li/p/8418798.html
自定义格式：
https://blog.csdn.net/a9254778/article/details/79234132
日志转储：
https://www.jianshu.com/p/71a4c61384f1
https://blog.csdn.net/taiyang1987912/article/details/47829895
https://www.cnblogs.com/fyy-hhzzj/p/9593362.html