经排查发现本地的测试环境磁盘空间满了,造成数据库不能写入数据.
最终的罪魁祸首是日志文件.
由于我们使用的是lvs负载均衡,前端的Keepalived启动之后不能初始化ipvs协议,所以日志在不停的刷一下的内容,速度之快令人乍舌,日志文件的大小在不停的疯长,最终导致空间不足,杯具的事情就这样发生了.
错误内容
查看系统日志文件内容
tail -f /var/log/messages
你会看见很多如下的信息在不停的跳动.
May 16 08:35:22 linux-node1 Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.144 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.146 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink reflector May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink command channel May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'. May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Configuration is using : 13073 Bytes May 16 08:35:22 linux-node1 Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.144 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.146 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink reflector May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink command channel May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'. May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Configuration is using : 13073 Bytes
看这句"IPVS: Can't initialize ipvs: Protocol not available",已经很是明显的指出了不能初始化ipvs协议,看看解决办法吧
解决办法
首先你必须得先安装ipvsadm
yum install ipvsadm
要是你已经安装过了,这句就直接忽略掉吧,之后在执行一下命令
ipvsadm
这一个小小的命令就能够给你解决上边的问题了,现在你再看看你的message里边还有没有一直在刷新,哈哈~~~
[root@linux-node1 keepalived]# ipvsadm IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.20.23.146:http wrr persistent 300 -> web02:0 Route 1 0 0 -> 10.20.23.250:0 Route 1 0 0
