Linux内存踩踏问题分析方法_address between user and kernel address ranges

unable to handle kernel paging request at virtual address之类的问题有可能是內存越界导致该指针所在内存被破坏了或者可能踩到use after free 的内存。可以使用打开kasan以及slub的一些宏去分析。

202<1>.(1)[2801:kworker/u8:0]Unable to handle kernel paging request at virtual address 4ca519894b81866e
345687.217558] <1>.(1)[2801:kworker/u8:0]Mem abort info:
[345687.218222] <1>.(1)[2801:kworker/u8:0] ESR = 0x96000004
[2022-08-02_11:46:10.607][345687.218916] <1>.(1)[2801:kworker/u8:0] Exception class = DABT (current EL), IL = 32 bits
[2022-08-02_11:46:10.607][345687.219973] <1>.(1)[2801:kworker/u8:0] SET = 0, FnV = 0
[2022-08-02_11:46:10.608][345687.220667] <1>.(1)[2801:kworker/u8:0] EA = 0, S1PTW = 0
[2022-08-02_11:46:10.610][345687.221425] <1>.(1)[2801:kworker/u8:0]Data abort info:
[2022-08-02_11:46:10.611][345687.221495] <1>-(3)[0:swapper/3][name:bc&]tick broadcast enter counter cpu: 595, 61, 420, 396, success counter cpu: 67, 15, 94, 78, fail counter cpu: 0, 0, 0, 0, interrupt counter cpu: 129, 29, 186, 156, o: , p: , f: , t: 345687220000000, 345687220000000, 345687220000000, 345687220000000,
[2022-08-02_11:46:10.615][345687.225359] <1>.(1)[2801:kworker/u8:0] ISV = 0, ISS = 0x00000004
[2022-08-02_11:46:10.615][345687.226151] <1>.(1)[2801:kworker/u8:0] CM = 0, WnR = 0
[2022-08-02_11:46:10.636][345687.226834] <1>.(1)[2801:kworker/u8:0][4ca519894b81866e] address between user and kernel address ranges
[2022-08-02_11:46:10.637][345687.228047] <1>-(1)[2801:kworker/u8:0]Internal error: Oops: 96000004 [#1] SMP
[2022-08-02_11:46:11.626][345688.238547] <1>-(1)[2801:kworker/u8:0]Kernel Offset: 0x0 from 0xffffff8008000000
[2022-08-02_11:46:11.629][345688.239522] <1>-(1)[2801:kworker/u8:0]PHYS_OFFSET: 0x40000000
...
[2022-08-02_11:46:11.736][345688.345248] <1>-(1)[2801:kworker/u8:0]CPU: 1 PID: 2801 Comm: kworker/u8:0 Tainted: G O 4.19.205 #0
[2022-08-02_11:46:11.739][345688.346569] <1>-(1)[2801:kworker/u8:0]Hardware name: 
[2022-08-02_11:46:11.739][345688.347626] <1>-(1)[2801:kworker/u8:0]Workqueue: uether process_rx_w
[2022-08-02_11:46:11.739][345688.348456] <1>-(1)[2801:kworker/u8:0]pstate: 20c00005 (nzCv daif +PAN +UAO)
[2022-08-02_11:46:11.739][345688.349374] <1>-(1)[2801:kworker/u8:0]pc : skb_release_data+0x68/0x128
[2022-08-02_11:46:11.739][345688.350223] <1>-(1)[2801:kworker/u8:0]lr : skb_release_data+0x1c/0x128
[2022-08-02_11:46:11.739][345688.351069] <1>-(1)[2801:kworker/u8:0]sp : ffffff800ad6b270
[2022-08-02_11:46:11.739][345688.351796] <1>-(1)[2801:kworker/u8:0]x29: ffffff800ad6b270 x28: 0000000000480020
[2022-08-02_11:46:11.742][345688.352778] <1>-(1)[2801:kworker/u8:0]x27: ffffff800ad6bb00 x26: 0000000000000000
[2022-08-02_11:46:11.742][345688.353760] <1>-(1)[2801:kworker/u8:0]x25: ffffffc00b6f807c x24: ffffffc01f89682c
[2022-08-02_11:46:11.742][345688.354743] <1>-(1)[2801:kworker/u8:0]x23: ffffff8009703300 x22: ffffffffffffffea
[2022-08-02_11:46:11.744][345688.355725] <1>-(1)[2801:kworker/u8:0]x21: 0000000000000000 x20: ffffffc00b6fbec0
[2022-08-02_11:46:11.744][345688.356707] <1>-(1)[2801:kworker/u8:0]x19: ffffffc01f896800 x18: 0000000000000000
[2022-08-02_11:46:11.744][345688.357688] <1>-(1)[2801:kworker/u8:0]x17: 0000000000000000 x16: 0000000000000000
[2022-08-02_11:46:11.747][345688.358670] <1>-(1)[2801:kworker/u8:0]x15: 0000000000000000 x14: 8107f391955730d5
[2022-08-02_11:46:11.749][345688.359652] <1>-(1)[2801:kworker/u8:0]x13: 7c1328f75da48bbe x12: ffffffffffffffff
[2022-08-02_11:46:11.749][345688.360634] <1>-(1)[2801:kworker/u8:0]x11: 0000000000000040 x10: 0000000000000001
[2022-08-02_11:46:11.749][345688.361616] <1>-(1)[2801:kworker/u8:0]x9 : 0000000000000001 x8 : 00000000000000e5
[2022-08-02_11:46:11.753][345688.362597] <1>-(1)[2801:kworker/u8:0]x7 : 000000003d445918 x6 : 000000004ca25abc
[2022-08-02_11:46:11.753][345688.363579] <1>-(1)[2801:kworker/u8:0]x5 : ffffffc0161c1900 x4 : ffffffc03fc79f60
[2022-08-02_11:46:11.753][345688.364561] <1>-(1)[2801:kworker/u8:0]x3 : 000000004ca25abd x2 : 4ca519894b818666
[2022-08-02_11:46:11.756][345688.365543] <1>-(1)[2801:kworker/u8:0]x1 : 0000000000000000 x0 : ffffffc00b6fbec0
[2022-08-02_11:46:11.756][345688.366528] <1>-(1)[2801:kworker/u8:0]CPU: 1 PID: 2801 Comm: kworker/u8:0 Tainted: G O 4.19.205 #0
[2022-08-02_11:46:11.756][345688.367847] <1>-(1)[2801:kworker/u8:0]Hardware name: 
[2022-08-02_11:46:11.756][345688.368896] <1>-(1)[2801:kworker/u8:0]Workqueue: uether process_rx_w
[2022-08-02_11:46:11.756][345688.369722] <1>-(1)[2801:kworker/u8:0]Call trace:
[2022-08-02_11:46:11.759][345688.370345] <1>-(1)[2801:kworker/u8:0] dump_backtrace+0x0/0x154
[2022-08-02_11:46:11.761][345688.371118] <1>-(1)[2801:kworker/u8:0] show_stack+0x24/0x30
[2022-08-02_11:46:11.761][345688.371849] <1>-(1)[2801:kworker/u8:0] dump_stack+0x90/0xb8
[2022-08-02_11:46:11.761][345688.372579] <1>-(1)[2801:kworker/u8:0] mrdump_common_die+0x220/0x250
[2022-08-02_11:46:11.761][345688.373406] <1>-(1)[2801:kworker/u8:0] ipanic_die+0x38/0x44
[2022-08-02_11:46:11.764][345688.374136] <1>-(1)[2801:kworker/u8:0] notifier_call_chain+0x70/0x90
[2022-08-02_11:46:11.764][345688.374962] <1>-(1)[2801:kworker/u8:0] atomic_notifier_call_chain+0x3c/0x4c
[2022-08-02_11:46:11.764][345688.375866] <1>-(1)[2801:kworker/u8:0] notify_die+0x40/0x60
[2022-08-02_11:46:11.764][345688.376594] <1>-(1)[2801:kworker/u8:0] die+0xc8/0x1e8
[2022-08-02_11:46:11.766][345688.377256] <1>-(1)[2801:kworker/u8:0] die_kernel_fault+0x68/0x78
[2022-08-02_11:46:11.766][345688.378052] <1>-(1)[2801:kworker/u8:0] __do_kernel_fault+0xd8/0x100
[2022-08-02_11:46:11.766][345688.378867] <1>-(1)[2801:kworker/u8:0] do_translation_fault+0x94/0x98
[2022-08-02_11:46:11.769][345688.379705] <1>-(1)[2801:kworker/u8:0] do_mem_abort+0x6c/0x100
[2022-08-02_11:46:11.769][345688.380465] <1>-(1)[2801:kworker/u8:0] el1_da+0x1c/0xac
[2022-08-02_11:46:11.769][345688.381148] <1>-(1)[2801:kworker/u8:0] skb_release_data+0x68/0x128
[2022-08-02_11:46:11.769][345688.381954] <1>-(1)[2801:kworker/u8:0] skb_release_all+0x30/0x3c
[2022-08-02_11:46:11.772][345688.382738] <1>-(1)[2801:kworker/u8:0] __kfree_skb+0x30/0x44
[2022-08-02_11:46:11.772][345688.383477] <1>-(1)[2801:kworker/u8:0] kfree_skb+0xc0/0xc4
[2022-08-02_11:46:11.772][345688.384195] <1>-(1)[2801:kworker/u8:0] ip_finish_output+0x158/0x1f0
[2022-08-02_11:46:11.772][345688.385010] <1>-(1)[2801:kworker/u8:0] ip_output+0xb4/0x128
[2022-08-02_11:46:11.774][345688.385737] <1>-(1)[2801:kworker/u8:0] ip_forward_finish+0x8c/0x9c
[2022-08-02_11:46:11.774][345688.386541] <1>-(1)[2801:kworker/u8:0] ip_forward+0x394/0x448
[2022-08-02_11:46:11.774][345688.387291] <1>-(1)[2801:kworker/u8:0] ip_rcv_finish+0x4c/0x60
[2022-08-02_11:46:11.774][345688.388053] <1>-(1)[2801:kworker/u8:0] ip_sabotage_in+0x48/0x60
[2022-08-02_11:46:11.777][345688.388826] <1>-(1)[2801:kworker/u8:0] nf_hook_slow+0x58/0xc8
[2022-08-02_11:46:11.777][345688.389575] <1>-(1)[2801:kworker/u8:0] ip_rcv+0xa0/0xd8
[2022-08-02_11:46:11.777][345688.390259] <1>-(1)[2801:kworker/u8:0] __netif_receive_skb_one_core+0x6c/0x94
[2022-08-02_11:46:11.780][345688.391185] <1>-(1)[2801:kworker/u8:0] __netif_receive_skb+0x1d0/0x1e8
[2022-08-02_11:46:11.780][345688.392033] <1>-(1)[2801:kworker/u8:0] netif_receive_skb_internal+0xb4/0xc4
[2022-08-02_11:46:11.782][345688.392935] <1>-(1)[2801:kworker/u8:0] netif_receive_skb+0x58/0x90
[2022-08-02_11:46:11.782][345688.393741] <1>-(1)[2801:kworker/u8:0] br_netif_receive_skb+0x40/0x4c
[2022-08-02_11:46:11.782][345688.394579] <1>-(1)[2801:kworker/u8:0] br_pass_frame_up+0xe8/0x164
[2022-08-02_11:46:11.785][345688.395384] <1>-(1)[2801:kworker/u8:0] br_handle_frame_finish+0x128/0x3d0
[2022-08-02_11:46:11.785][345688.396265] <1>-(1)[2801:kworker/u8:0] br_nf_hook_thresh+0x70/0x108
[2022-08-02_11:46:11.785][345688.397080] <1>-(1)[2801:kworker/u8:0] br_nf_pre_routing_finish+0x188/0x2e0
[2022-08-02_11:46:11.785][345688.397982] <1>-(1)[2801:kworker/u8:0] br_nf_pre_routing+0x2f0/0x358
[2022-08-02_11:46:11.788][345688.398809] <1>-(1)[2801:kworker/u8:0] nf_hook_slow+0x58/0xc8
[2022-08-02_11:46:11.788][345688.399558] <1>-(1)[2801:kworker/u8:0] br_handle_frame+0x338/0x35c
[2022-08-02_11:46:11.788][345688.400363] <1>-(1)[2801:kworker/u8:0] __netif_receive_skb_core+0x318/0x964
[2022-08-02_11:46:11.790][345688.401266] <1>-(1)[2801:kworker/u8:0] __netif_receive_skb_one_core+0x50/0x94
[2022-08-02_11:46:11.790][345688.402192] <1>-(1)[2801:kworker/u8:0] __netif_receive_skb+0x1d0/0x1e8
[2022-08-02_11:46:11.790][345688.403040] <1>-(1)[2801:kworker/u8:0] process_backlog+0x9c/0x174
[2022-08-02_11:46:11.791][345688.403833] <1>-(1)[2801:kworker/u8:0] net_rx_action+0x164/0x354
[2022-08-02_11:46:11.794][345688.404615] <1>-(1)[2801:kworker/u8:0] __do_softirq+0x220/0x2c4
[2022-08-02_11:46:11.794][345688.405388] <1>-(1)[2801:kworker/u8:0] do_softirq+0x54/0x64
[2022-08-02_11:46:11.794][345688.406117] <1>-(1)[2801:kworker/u8:0] netif_rx_ni+0x9c/0xac
[2022-08-02_11:46:11.797][345688.406857] <1>-(1)[2801:kworker/u8:0] process_rx_w+0x298/0x2a0
[2022-08-02_11:46:11.797][345688.407631] <1>-(1)[2801:kworker/u8:0] process_one_work+0x208/0x34c
 

kasan介绍：

The Kernel Address Sanitizer (KASAN) is a dynamic memory safety error detector designed to find out-of-bound and use-after-free.

相关文档链接地址：

Kernel Address Sanitizer (KASAN) — The Linux Kernel documentation

kasan配置：

CONFIG_KASAN=y

CONFIG_KASAN_GENERIC=y

CONFIG_KASAN_INLINE=y

CONFIG_KASAN_STACK=1

CONFIG_KASAN_OUTLINE can not set sometimes

CONFIG_FRAME_WARN=0

CONFIG_KASAN_GENERIC

CONFIG_KASAN_EXTRA

但是kasan占用内存可能比较高，建议内存大于2GB ram再使用。对于其以下内存可使用slub debug 调试。kasan并不一定能探测到内存踩坏，有可能是dma的一些硬件控制器将其损坏。

[内核内存] linux内核态内存检测技术_padding overwritten-CSDN博客

SLUB DEBUG原理

slab/slub调试内存可参考内核官网的一些介绍：

• https://www.kernel.org/doc/Documentation/slab.txt

• https://www.kernel.org/doc/Documentation/slabinfo.txt

https://www.kernel.org/doc/html/latest/vm/slub.html

SLUB DEBUG原理

CONFIG_SLAB

CONFIG_SLUB_DEBUG=y

CONFIG_DEBUG_SLAB_LEAK

CONFIG_SLAB_FREELIST_DEBUG

- CONFIG_SLUB=y

- CONFIG_SLUB_DEBUG=y

- CONFIG_SLUB_DEBUG_ON=y

cat /proc/slabinfo 查看slab info.

tools/vm/slabinfo.c编译生成slabinfo，然后执行如下：

- watch -n 5 slabinfo -v

CONFIG_LOG_BUF_SHIFT 增加kernel log buf //防止打印太多，不够缓存
cat /proc/slabinfo
发现 kmalloc -128的num_objs如果有明显的增长，那可能是通过 kmalloc-128 申请内存后存在泄漏的情况。

cat /sys/kernel/slab/kmalloc-128/alloc_calls
echo 1 >/sys/kernel/slab/kmalloc-128/trace;cat /proc/kmsg > /tmp/slab_trace.txt
也可以是如下trace:
echo 1 >/sys/kernel/slab/kmalloc-
kmalloc-128/
kmalloc-256/
kmalloc-4k/
kmalloc-8k/
kmalloc-rcl-1k/
kmalloc-rcl-2k/
kmalloc-rcl-512/
kmalloc-1k/
kmalloc-2k/
kmalloc-512/
kmalloc-rcl-128/ kmalloc-rcl-256/ kmalloc-rcl-4k/ kmalloc-rcl-8k/

trace打印如下：

gdb资料网址：

- https://www.gnu.org/software/gdb/documentation/

- https://linux.die.net/man/1/gdb

- https://gist.github.com/rkubik/b96c23bd8ed58333de37f2b8cd052c30

decode_stacktrace: make stack dump output useful again [LWN.net]

Bug hunting — The Linux Kernel documentation

kasan资料：

KASAN实现原理

The Kernel Address Sanitizer (KASAN) — The Linux Kernel documentation

kasan开启判断：

kernel log:

1、swapper: kasan: [name:kasan_init&]KernelAddressSanitizer initialized

2、cat proc/meminfo | grep MemTotal

3、aarch64-linux-gnu-objdump -Sd vmlinux | grep __asan_

4、zcat proc/config.gz 检查上述config是否打开