赞
踩
watchdog_timer:
expire time: 4s = (watchdog_threshold * 2) / 5(watchdog_threshold 由/proc/sys/kernel/watchdog_thresh配置,系统默认是10,下同)
function:updating hrtimer_interrupts when watchdog_timer expires
/|\
0 4 8 | 12 14 18
|-------|-------|-------|-------|-------|----
|
|
|
|
0 10 20
|-------------------|----------------------|
nmi_check
nmi:
expire time: 10s = watchdog_thresh perf event的counter值也是由watchdog_thresh来确定;
function:nmi check
if(hrtimer_interrupts==hrtimer_interrupts_save)--->hard_lockup--->warning or crash
else hrtimer_interrupts_save = hrtimer_interrupts
从上面可以看,理论上nmi check之前,watchdog会进行2-3次的喂狗操作,造成hardlockup的原因基本可以总结以下二点:
1:中断被关闭,长时间未打开,导致watchdog定时器中断不被响应,无法每4s一次喂狗,从而被nmi到时函数断定为hardlockup
2:由于nmi基于硬件cpu频率计时,如果频率不稳定或 Turbo-Mode被使能,突增变大,就可能会造成nmi检测提前,误报hardlockup。
针对2,开源合入了相应补丁,得到了很好的解释:
https://access.redhat.com/labs/psb/versions/kernel-3.10.0-862.el7/patches/kernel-watchdog-Prevent-false-positives-with-turbo-modes
补丁的思想:
由于喂狗操作会每4s进行一次,为了确保方式上述2情况的发生,此补丁引入了过滤器变量watchdog_hrtimer_sample_threshold=8s,此过滤器保证了,
至少有一次喂狗操作,然后利用这个变量在hardlockup检测前进行判断;
+static bool watchdog_check_timestamp(void) +{ + s64 delta, now = ktime_get_mono_fast_ns(); + + delta = now - __this_cpu_read(last_timestamp);//获取二个nmi检测的时间差 + if (delta < watchdog_hrtimer_sample_threshold) {//如果时间差小于8s,说明了nmi时钟快了,防止误报,则不做检查! + /* + * If ktime is jiffies based, a stalled timer would prevent + * jiffies from being incremented and the filter would look + * at a stale timestamp and never trigger. + */ + if (__this_cpu_inc_return(nmi_rearmed) < 10) + return false;//nmi检测直接返回,不做检查 + } + __this_cpu_write(nmi_rearmed, 0); + __this_cpu_write(last_timestamp, now); + return true; +}
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。