Kernel BUG in apm module.

[EvgeniiMekhanik]

To Reproduce
sudo venv/bin/python3.12 ./run_tests.py -n tests.fault_injection.test_fault_injection_base.TestSchedConfig.test_init_modules_tfw_kvmalloc_node

I see this BUG two times.
It's very hard to reproduce. Previously I see it on master branch on my local vm (but I though that my be it was problem in my build, because I can't reproduce it).
I catch it again on CI on the new PR (but I am sure that it has nothing to do with the changes from the PR). Also as I catch it on master a long time ago).

Version or commit hash
Tempesta FW 2a35da6
Tempesta test fbcd279ea65454d5f54dc16e7f1ef0055444c498

Stacktrace or debug log

May  5 12:38:11 192.168.122.82 [ 3503.705579] [tdb] Shutdown Tempesta DB
May  5 12:38:11 192.168.122.82 [ 3504.235153] [tempesta sc] OS configuration or hardware is not suitable/ (Total 1 errors)
May  5 12:38:11 192.168.122.82 [ 3504.235313] [tempesta sc] 	1. The minimal amount of RAM required by Tempesta is 8192 MB, but you have 7428 MB
May  5 12:38:11 192.168.122.82 [ 3504.249492] [tempesta sc] Kernel config params are missing or have different values. Please, check configuration of `/boot/config-6.12.12.tfw-b158a7a`/ (Total 2 errors)
May  5 12:38:11 192.168.122.82 [ 3504.251252] [tempesta sc] 	1. `CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE={'1'}` is not defined.
May  5 12:38:11 192.168.122.82 [ 3504.253736] [tempesta sc] 	2. `CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE={'1'}` is not defined.
May  5 12:38:11 192.168.122.82 [ 3504.255360] [tempesta sc] Verification passed with errors!
May  5 12:38:12 192.168.122.82 [ 3504.256855] [tempesta sc] Skipped Netconsole configuration and Support Server connection
May  5 12:38:12 192.168.122.82 [ 3504.416788] [tdb] Start Tempesta DB
May  5 12:38:12 192.168.122.82 [ 3504.456572] [tempesta fw] Initializing Tempesta FW kernel module...
May  5 12:38:12 192.168.122.82 [ 3504.693535] [tempesta fw] Warning: Vhost default doesn't have certificate with matching SAN/CN.
[ 3504.693535]     Maybe that's fine, but it's worth checking the
[ 3504.693535]     config - if there is no relations between the
[ 3504.693535]     names, then host name confusion attack is possible.
May  5 12:38:12 192.168.122.82 [ 3504.702005] [tempesta fw] Configuration processing is completed.
May  5 12:38:12 192.168.122.82 [ 3504.744111] [tdb] Opened table /opt/tempesta/db/filter0.tdb: size=16777216 rec_size=32 base=0000000001c94e4e
May  5 12:38:12 192.168.122.82 [ 3504.857431] BUG: unable to handle page fault for address: ffff970960a600c8
May  5 12:38:12 192.168.122.82 [ 3504.858183] #PF: supervisor read access in kernel mode
May  5 12:38:12 192.168.122.82 [ 3504.858706] #PF: error_code(0x0000) - not-present page
May  5 12:38:12 192.168.122.82 [ 3504.859236] PGD 0 P4D 0 
May  5 12:38:12 192.168.122.82 [ 3504.859502] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
May  5 12:38:12 192.168.122.82 [ 3504.860001] CPU: 0 UID: 0 PID: 327253 Comm: tempesta.sh Tainted: G           OE      6.12.12.tfw-b158a7a #1
May  5 12:38:12 192.168.122.82 [ 3504.860969] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May  5 12:38:12 192.168.122.82 [ 3504.861514] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
May  5 12:38:12 192.168.122.82 [ 3504.862444] RIP: 0010:tfw_apm_prcntl_tmfn+0x157/0x600 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.863106] Code: c5 00 48 8b 02 48 83 f8 ff 74 e1 49 89 c4 89 c0 48 c7 02 ff ff ff ff 48 8d 1c 80 48 8b 04 24 49 c1 ec 20 48 c1 e3 07 48 01 c3 <44> 39 63 20 0f 83 3f ff ff ff 8b 73 28 45 89 e7 44 39 63 2c 0f 82
May  5 12:38:12 192.168.122.82 [ 3504.864953] RSP: 0018:ffffb1a0c0005e48 EFLAGS: 00010282
May  5 12:38:12 192.168.122.82 [ 3504.865470] RAX: ffff94edca06e0a8 RBX: ffff970960a600a8 RCX: 0000000000000003
May  5 12:38:12 192.168.122.82 [ 3504.866191] RDX: ffff94edc504d900 RSI: 0000000000000004 RDI: ffffffffbb925f20
May  5 12:38:12 192.168.122.82 [ 3504.866914] RBP: ffff94edc504d780 R08: 0000000000000003 R09: ffffb1a0c0005f10
May  5 12:38:12 192.168.122.82 [ 3504.867620] R10: 0000000000000201 R11: 0000000000000bfa R12: 00000000ffff94ed
May  5 12:38:12 192.168.122.82 [ 3504.868345] R13: 0000000000000030 R14: ffffd1a0bfd877c8 R15: ffff94edc504d780
May  5 12:38:12 192.168.122.82 [ 3504.869066] FS:  00007ff138d85740(0000) GS:ffff94ef37c00000(0000) knlGS:0000000000000000
May  5 12:38:12 192.168.122.82 [ 3504.869880] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  5 12:38:12 192.168.122.82 [ 3504.870458] CR2: ffff970960a600c8 CR3: 000000012a636004 CR4: 0000000000772ef0
May  5 12:38:12 192.168.122.82 [ 3504.871180] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  5 12:38:12 192.168.122.82 [ 3504.871896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May  5 12:38:12 192.168.122.82 [ 3504.872613] PKRU: 55555554
May  5 12:38:12 192.168.122.82 [ 3504.872899] Call Trace:
May  5 12:38:12 192.168.122.82 [ 3504.873170]  <IRQ>
May  5 12:38:12 192.168.122.82 [ 3504.873396]  ? __die+0x24/0x80
May  5 12:38:12 192.168.122.82 [ 3504.873733]  ? page_fault_oops+0x158/0x500
May  5 12:38:12 192.168.122.82 [ 3504.874194]  ? search_module_extables+0x3e/0x80
May  5 12:38:12 192.168.122.82 [ 3504.874667]  ? search_bpf_extables+0x63/0xa0
May  5 12:38:12 192.168.122.82 [ 3504.875146]  ? exc_page_fault+0xf1/0x1a0
May  5 12:38:12 192.168.122.82 [ 3504.875548]  ? asm_exc_page_fault+0x26/0x40
May  5 12:38:12 192.168.122.82 [ 3504.875985]  ? tfw_apm_prcntl_tmfn+0x157/0x600 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.876584]  ? tfw_apm_prcntl_tmfn+0x47/0x600 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.877158]  ? task_tick_fair+0x40/0x2a0
May  5 12:38:12 192.168.122.82 [ 3504.877576]  ? sched_clock+0x10/0x40
May  5 12:38:12 192.168.122.82 [ 3504.877962]  ? __pfx_tfw_apm_prcntl_tmfn+0x20/0x20 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.878585]  ? __pfx_tfw_apm_prcntl_tmfn+0x20/0x20 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.879199]  call_timer_fn+0x27/0x120
May  5 12:38:12 192.168.122.82 [ 3504.879598]  __run_timers+0x1c9/0x280
May  5 12:38:12 192.168.122.82 [ 3504.879982]  ? enqueue_hrtimer+0x35/0xa0
May  5 12:38:12 192.168.122.82 [ 3504.880378]  ? __hrtimer_run_queues+0x141/0x2a0
May  5 12:38:12 192.168.122.82 [ 3504.880852]  ? kvm_clock_get_cycles+0x18/0x40
May  5 12:38:12 192.168.122.82 [ 3504.881311]  run_timer_softirq+0x76/0xe0
May  5 12:38:12 192.168.122.82 [ 3504.881720]  handle_softirqs+0xcd/0x2c0
May  5 12:38:12 192.168.122.82 [ 3504.882143]  __irq_exit_rcu+0x63/0x80
May  5 12:38:12 192.168.122.82 [ 3504.882531]  sysvec_apic_timer_interrupt+0x75/0xa0
May  5 12:38:12 192.168.122.82 [ 3504.883041]  </IRQ>
May  5 12:38:12 192.168.122.82 [ 3504.883268]  <TASK>
May  5 12:38:12 192.168.122.82 [ 3504.883494]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
May  5 12:38:12 192.168.122.82 [ 3504.884072] RIP: 0010:folio_mark_accessed+0x18/0xe0
May  5 12:38:12 192.168.122.82 [ 3504.884578] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 53 48 8b 07 48 89 fb a8 04 75 19 f0 80 0f 04 <48> 8b 03 a9 00 00 40 00 74 05 f0 80 63 02 bf 5b c3 cc cc cc cc 48
May  5 12:38:12 192.168.122.82 [ 3504.886411] RSP: 0018:ffffb1a0c0caba18 EFLAGS: 00000202
May  5 12:38:12 192.168.122.82 [ 3504.886950] RAX: 0017ffffc0000028 RBX: ffffe04c48e1f1c0 RCX: ffffb1a0c0cabbb8
May  5 12:38:12 192.168.122.82 [ 3504.887650] RDX: 000000000387a000 RSI: 0000000000000000 RDI: ffffe04c48e1f1c0
May  5 12:38:12 192.168.122.82 [ 3504.888373] RBP: 0000000010000000 R08: ffffb1a0c0cabba8 R09: 000000000c786000
May  5 12:38:12 192.168.122.82 [ 3504.889087] R10: 0000000000000000 R11: 000000000c786000 R12: 000000000c786000
May  5 12:38:12 192.168.122.82 [ 3504.889794] R13: ffffb1a0c0cabbb8 R14: 0000000000001000 R15: 000000000000000b
May  5 12:38:12 192.168.122.82 [ 3504.890541]  filemap_read+0x201/0x340
May  5 12:38:12 192.168.122.82 [ 3504.890937]  __kernel_read+0x161/0x300
May  5 12:38:12 192.168.122.82 [ 3504.891368]  tdb_file_open+0x25f/0x500 [tempesta_db]
May  5 12:38:12 192.168.122.82 [ 3504.891891]  tdb_open+0x12d/0x2e0 [tempesta_db]
May  5 12:38:12 192.168.122.82 [ 3504.892364]  tfw_cache_start+0x1c5/0x340 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.892914]  tfw_ctlfn_state_io+0x32c/0x580 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.893494]  ? __pfx_tfw_ctlfn_state_io+0x20/0x20 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.894126]  proc_sys_call_handler+0x1bb/0x2e0
May  5 12:38:12 192.168.122.82 [ 3504.894596]  vfs_write+0x301/0x420
May  5 12:38:12 192.168.122.82 [ 3504.894996]  ksys_write+0x63/0xe0
May  5 12:38:12 192.168.122.82 [ 3504.895349]  do_syscall_64+0x51/0x120
May  5 12:38:12 192.168.122.82 [ 3504.895727]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
May  5 12:38:12 192.168.122.82 [ 3504.896254] RIP: 0033:0x7ff138b1c5a4
May  5 12:38:12 192.168.122.82 [ 3504.896634] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 ea 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
May  5 12:38:12 192.168.122.82 [ 3504.898484] RSP: 002b:00007fff7276aff8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
May  5 12:38:12 192.168.122.82 [ 3504.899250] RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007ff138b1c5a4
May  5 12:38:12 192.168.122.82 [ 3504.899963] RDX: 0000000000000006 RSI: 000055cb4145b6b0 RDI: 0000000000000001
May  5 12:38:12 192.168.122.82 [ 3504.900681] RBP: 00007fff7276b020 R08: 0000000000000073 R09: 0000000000000000
May  5 12:38:12 192.168.122.82 [ 3504.901403] R10: 00000000ffffffff R11: 0000000000000202 R12: 0000000000000006
May  5 12:38:12 192.168.122.82 [ 3504.902121] R13: 000055cb4145b6b0 R14: 00007ff138c045c0 R15: 00007ff138c01ee0
May  5 12:38:12 192.168.122.82 [ 3504.902853]  </TASK>
May  5 12:38:12 192.168.122.82 [ 3504.903091] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) tcp_diag inet_diag nf_conntrack_netlink xt_nat xt_tcpudp veth ib_core tls xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables overlay cfg80211 binfmt_misc intel_rapl_msr intel_rapl_common skx_edac_common nfit kvm_intel kvm rapl i2c_i801 i2c_mux i2c_smbus lpc_ich joydev input_leds serio_raw mac_hid sch_fq_codel dm_multipath netconsole msr efi_pstore drm nfnetlink dmi_sysfs qemu_fw_cfg ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 crct10dif_pclmul crc32_pclmul psmouse ghash_clmulni_intel sha512_ssse3 virtio_blk sha256_ssse3 virtio_net ahci net_failover libahci virtio_rng failover aesni_intel crypto_simd cryptd [last unloaded: tempesta_lib(OE)]
May  5 12:38:12 192.168.122.82 [ 3504.911286] CR2: ffff970960a600c8
May  5 12:38:12 192.168.122.82 [ 3504.911630] ---[ end trace 0000000000000000 ]---
May  5 12:38:12 192.168.122.82 [ 3504.912121] RIP: 0010:tfw_apm_prcntl_tmfn+0x157/0x600 [tempesta_fw]
May  5 12:38:12 192.168.122.82 [ 3504.912770] Code: c5 00 48 8b 02 48 83 f8 ff 74 e1 49 89 c4 89 c0 48 c7 02 ff ff ff ff 48 8d 1c 80 48 8b 04 24 49 c1 ec 20 48 c1 e3 07 48 01 c3 <44> 39 63 20 0f 83 3f ff ff ff 8b 73 28 45 89 e7 44 39 63 2c 0f 82
May  5 12:38:12 192.168.122.82 [ 3504.914629] RSP: 0018:ffffb1a0c0005e48 EFLAGS: 00010282
May  5 12:38:12 192.168.122.82 [ 3504.915166] RAX: ffff94edca06e0a8 RBX: ffff970960a600a8 RCX: 0000000000000003
May  5 12:38:12 192.168.122.82 [ 3504.915881] RDX: ffff94edc504d900 RSI: 0000000000000004 RDI: ffffffffbb925f20
May  5 12:38:12 192.168.122.82 [ 3504.916605] RBP: ffff94edc504d780 R08: 0000000000000003 R09: ffffb1a0c0005f10
May  5 12:38:12 192.168.122.82 [ 3504.917327] R10: 0000000000000201 R11: 0000000000000bfa R12: 00000000ffff94ed
May  5 12:38:12 192.168.122.82 [ 3504.918038] R13: 0000000000000030 R14: ffffd1a0bfd877c8 R15: ffff94edc504d780
May  5 12:38:12 192.168.122.82 [ 3504.918763] FS:  00007ff138d85740(0000) GS:ffff94ef37c00000(0000) knlGS:0000000000000000
May  5 12:38:12 192.168.122.82 [ 3504.919575] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  5 12:38:12 192.168.122.82 [ 3504.920162] CR2: ffff970960a600c8 CR3: 000000012a636004 CR4: 0000000000772ef0
May  5 12:38:12 192.168.122.82 [ 3504.920893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  5 12:38:12 192.168.122.82 [ 3504.921616] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May  5 12:38:12 192.168.122.82 [ 3504.922335] PKRU: 55555554
May  5 12:38:12 192.168.122.82 [ 3504.922634] Kernel panic - not syncing: Fatal exception in interrupt
May  5 12:38:12 192.168.122.82 [ 3504.923660] Kernel Offset: 0x38400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
May  5 12:38:12 192.168.122.82 [ 3504.924731] Rebooting in 1 seconds..

[EvgeniiMekhanik]

During running this test in the loop I also catch

May 24 17:11:37 192.168.122.226 [308706.743308] [tempesta sc] OS configuration or hardware is not suitable/ (Total 3 errors)
May 24 17:11:37 192.168.122.226 [308706.743480] [tempesta sc]   1. Current loaded kernel is not suitable with TFW: 6.12.12+
May 24 17:11:37 192.168.122.226 [308706.745333] [tempesta sc]   2. Current loaded kernel has not suitable version: 6.12.12+, supported versions 6.12.12+
May 24 17:11:37 192.168.122.226 [308706.746987] [tempesta sc]   3. The minimal amount of RAM required by Tempesta is 8192 MB, but you have 7433 MB
May 24 17:11:37 192.168.122.226 [308706.756323] [tempesta sc] Kernel config params are missing or have different values. Please, check configuration of `/boot/config-6.12.12+`/ (Total>
May 24 17:11:37 192.168.122.226 [308706.757445] [tempesta sc]   1. `CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE={'1'}` is not defined.
May 24 17:11:37 192.168.122.226 [308706.758953] [tempesta sc]   2. `CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE={'1'}` is not defined.
May 24 17:11:37 192.168.122.226 [308706.760055] [tempesta sc] Sysctl config params are missing or have different values. Please, check configuration of `/etc/sysctl.conf`/ (Total 1 er>
May 24 17:11:37 192.168.122.226 [308706.761036] [tempesta sc]   1. `kernel.softlockup_panic={'1'}` is not defined.
May 24 17:11:37 192.168.122.226 [308706.762506] [tempesta sc] Verification passed with errors!
May 24 17:12:37 192.168.122.226 [308706.763354] [tempesta sc] Skipped Netconsole configuration and Support Server connection
May 24 17:12:37 192.168.122.226 [308766.653711] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
May 24 17:12:37 192.168.122.226 [308766.658717] rcu:    2-...!: (0 ticks this GP) idle=a50c/1/0x4000000000000000 softirq=28430817/28430817 fqs=498
May 24 17:12:37 192.168.122.226 [308766.661802] rcu:    (detected by 1, t=15004 jiffies, g=46940177, q=2870 ncpus=4)
May 24 17:12:37 192.168.122.226 [308766.663813] Sending NMI from CPU 1 to CPUs 2:
May 24 17:12:37 192.168.122.226 [308766.663890] NMI backtrace for cpu 2
May 24 17:12:37 192.168.122.226 [308766.663920] CPU: 2 UID: 0 PID: 0 Comm: swapper/2 Tainted: G           OE      6.12.12+ #117
May 24 17:12:37 192.168.122.226 [308766.663926] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May 24 17:12:37 192.168.122.226 [308766.663930] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
May 24 17:12:37 192.168.122.226 [308766.663931] RIP: 0010:mm_cid_get+0x8f/0x200
May 24 17:12:37 192.168.122.226 [308766.663980] Code: 01 f0 f0 4c 0f b3 38 8b 05 ce 46 83 02 85 c0 75 7f 8b 35 64 d4 de 01 8d 5e 3f c1 eb 03 81 e3 f8 ff ff 1f 4c 01 f3 eb 08 f3 90 <8b>
May 24 17:12:37 192.168.122.226 [308766.663982] RSP: 0018:ffffbe45800c7e30 EFLAGS: 00000046
May 24 17:12:37 192.168.122.226 [308766.663988] RAX: 0000000000000004 RBX: ffff97b98006f888 RCX: 0000000000000004
May 24 17:12:37 192.168.122.226 [308766.663990] RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffff97b98006f888
May 24 17:12:37 192.168.122.226 [308766.663991] RBP: 0000468a88016560 R08: 0000000000000001 R09: 0000000000000001
May 24 17:12:37 192.168.122.226 [308766.663992] R10: 0000000000000001 R11: 0000000000000000 R12: ffff97b98006f380
May 24 17:12:37 192.168.122.226 [308766.663993] R13: ffff97baf7d36440 R14: ffff97b98006f880 R15: 00000000ffffffff
May 24 17:12:37 192.168.122.226 [308766.663994] FS:  0000000000000000(0000) GS:ffff97baf7d00000(0000) knlGS:0000000000000000
May 24 17:12:37 192.168.122.226 [308766.663998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 24 17:12:37 192.168.122.226 [308766.663999] CR2: 00007f0aa0eee200 CR3: 00000001084b4004 CR4: 0000000000772ef0
May 24 17:12:37 192.168.122.226 [308766.664002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 24 17:12:37 192.168.122.226 [308766.664003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 24 17:12:37 192.168.122.226 [308766.664004] PKRU: 55555554
May 24 17:12:37 192.168.122.226 [308766.664007] Call Trace:
May 24 17:12:37 192.168.122.226 [308766.664027]  <NMI>
May 24 17:12:37 192.168.122.226 [308766.664055]  ? nmi_cpu_backtrace+0x99/0x120
May 24 17:12:37 192.168.122.226 [308766.664076]  ? nmi_cpu_backtrace_handler+0x11/0x20
May 24 17:12:37 192.168.122.226 [308766.664086]  ? nmi_handle+0x5e/0x160
May 24 17:12:37 192.168.122.226 [308766.664099]  ? default_do_nmi+0x42/0x100
May 24 17:12:37 192.168.122.226 [308766.664109]  ? exc_nmi+0x122/0x1a0
May 24 17:12:37 192.168.122.226 [308766.664110]  ? end_repeat_nmi+0xf/0x53
May 24 17:12:37 192.168.122.226 [308766.664132]  ? mm_cid_get+0x8f/0x200
May 24 17:12:37 192.168.122.226 [308766.664134]  ? mm_cid_get+0x8f/0x200
May 24 17:12:37 192.168.122.226 [308766.664136]  ? mm_cid_get+0x8f/0x200
May 24 17:12:37 192.168.122.226 [308766.664138]  </NMI>
May 24 17:12:37 192.168.122.226 [308766.664138]  <TASK>
May 24 17:12:37 192.168.122.226 [308766.664139]  __schedule+0x783/0xae0
May 24 17:12:37 192.168.122.226 [308766.664152]  schedule_idle+0x23/0x40
May 24 17:12:37 192.168.122.226 [308766.664154]  do_idle+0x148/0x240
May 24 17:12:37 192.168.122.226 [308766.664175]  cpu_startup_entry+0x29/0x40
May 24 17:12:37 192.168.122.226 [308766.664177]  start_secondary+0x11e/0x140
May 24 17:12:37 192.168.122.226 [308766.664188]  common_startup_64+0x13e/0x141
May 24 17:12:37 192.168.122.226 [308766.664205]  </TASK>
May 24 17:12:37 192.168.122.226 [308766.664834] rcu: rcu_preempt kthread timer wakeup didn't happen for 12511 jiffies! g46940177 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
May 24 17:12:37 192.168.122.226 [308766.689350] rcu:    Possible timer handling issue on cpu=0 timer-softirq=9427549
May 24 17:12:37 192.168.122.226 [308766.690089] rcu: rcu_preempt kthread starved for 12512 jiffies! g46940177 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
May 24 17:12:37 192.168.122.226 [308766.691161] rcu:    Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
May 24 17:12:37 192.168.122.226 [308766.692081] rcu: RCU grace-period kthread stack dump:
May 24 17:12:37 192.168.122.226 [308766.692606] task:rcu_preempt     state:I stack:0     pid:17    tgid:17    ppid:2      flags:0x00004000
May 24 17:12:37 192.168.122.226 [308766.693541] Call Trace:
May 24 17:12:37 192.168.122.226 [308766.693834]  <TASK>
May 24 17:12:37 192.168.122.226 [308766.694068]  ? __pfx_rcu_gp_kthread+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.694531]  __schedule+0x3dc/0xae0
May 24 17:12:37 192.168.122.226 [308766.694929]  ? _raw_spin_unlock_irqrestore+0xe/0x40
May 24 17:12:37 192.168.122.226 [308766.695431]  ? __pfx_rcu_gp_kthread+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.695889]  schedule+0x27/0x100
May 24 17:12:37 192.168.122.226 [308766.696246]  schedule_timeout+0x91/0x160
May 24 17:12:37 192.168.122.226 [308766.696661]  ? __pfx_process_timeout+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.697142]  rcu_gp_fqs_loop+0xf9/0x520
May 24 17:12:37 192.168.122.226 [308766.697548]  rcu_gp_kthread+0xd2/0x160
May 24 17:12:37 192.168.122.226 [308766.697950]  kthread+0xcf/0x100
May 24 17:12:37 192.168.122.226 [308766.698307]  ? __pfx_kthread+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.698701]  ret_from_fork+0x31/0x60
May 24 17:12:37 192.168.122.226 [308766.699106]  ? __pfx_kthread+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.699493]  ret_from_fork_asm+0x22/0x60
May 24 17:12:37 192.168.122.226 [308766.699925]  </TASK>
May 24 17:12:37 192.168.122.226 [308766.700186] rcu: Stack dump where RCU GP kthread last ran:
May 24 17:12:37 192.168.122.226 [308766.700739] Sending NMI from CPU 1 to CPUs 0:
May 24 17:12:37 192.168.122.226 [308766.701253] NMI backtrace for cpu 0
May 24 17:12:37 192.168.122.226 [308766.701258] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G           OE      6.12.12+ #117
May 24 17:12:37 192.168.122.226 [308766.701262] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May 24 17:12:37 192.168.122.226 [308766.701263] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
May 24 17:12:37 192.168.122.226 [308766.701264] RIP: 0010:mm_cid_get+0x95/0x200
May 24 17:12:37 192.168.122.226 [308766.701269] Code: 38 8b 05 ce 46 83 02 85 c0 75 7f 8b 35 64 d4 de 01 8d 5e 3f c1 eb 03 81 e3 f8 ff ff 1f 4c 01 f3 eb 08 f3 90 8b 35 4b d4 de 01 <48>
May 24 17:12:37 192.168.122.226 [308766.701273] RSP: 0018:ffffffff9f803de0 EFLAGS: 00000046
May 24 17:12:37 192.168.122.226 [308766.701275] RAX: 0000000000000004 RBX: ffff97b98006f888 RCX: 0000000000000004
May 24 17:12:37 192.168.122.226 [308766.701276] RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffff97b98006f888
May 24 17:12:37 192.168.122.226 [308766.701278] RBP: 0000468a88016560 R08: 0000000000000001 R09: 0000000000000001
May 24 17:12:37 192.168.122.226 [308766.701279] R10: 0000000000000001 R11: 0000000000000000 R12: ffff97b98006f380
May 24 17:12:37 192.168.122.226 [308766.701280] R13: ffff97baf7c36440 R14: ffff97b98006f880 R15: 00000000ffffffff
May 24 17:12:37 192.168.122.226 [308766.701281] FS:  0000000000000000(0000) GS:ffff97baf7c00000(0000) knlGS:0000000000000000
May 24 17:12:37 192.168.122.226 [308766.701283] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 24 17:12:37 192.168.122.226 [308766.701284] CR2: 00007f52fb946b48 CR3: 00000001084b4002 CR4: 0000000000772ef0
May 24 17:12:37 192.168.122.226 [308766.701286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 24 17:12:37 192.168.122.226 [308766.701287] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 24 17:12:37 192.168.122.226 [308766.701288] PKRU: 55555554
May 24 17:12:37 192.168.122.226 [308766.701289] Call Trace:
May 24 17:12:37 192.168.122.226 [308766.701290]  <NMI>
May 24 17:12:37 192.168.122.226 [308766.701292]  ? nmi_cpu_backtrace+0x99/0x120
May 24 17:12:37 192.168.122.226 [308766.701294]  ? nmi_cpu_backtrace_handler+0x11/0x20
May 24 17:12:37 192.168.122.226 [308766.701297]  ? nmi_handle+0x5e/0x160
May 24 17:12:37 192.168.122.226 [308766.701299]  ? default_do_nmi+0x42/0x100
May 24 17:12:37 192.168.122.226 [308766.701301]  ? exc_nmi+0x122/0x1a0
May 24 17:12:37 192.168.122.226 [308766.701303]  ? end_repeat_nmi+0xf/0x53
May 24 17:12:37 192.168.122.226 [308766.701307]  ? mm_cid_get+0x95/0x200
May 24 17:12:37 192.168.122.226 [308766.701309]  ? mm_cid_get+0x95/0x200
May 24 17:12:37 192.168.122.226 [308766.701311]  ? mm_cid_get+0x95/0x200
May 24 17:12:37 192.168.122.226 [308766.701313]  </NMI>
May 24 17:12:37 192.168.122.226 [308766.701313]  <TASK>
May 24 17:12:37 192.168.122.226 [308766.701314]  __schedule+0x783/0xae0
May 24 17:12:37 192.168.122.226 [308766.701317]  schedule_idle+0x23/0x40
May 24 17:12:37 192.168.122.226 [308766.701319]  do_idle+0x148/0x240
May 24 17:12:37 192.168.122.226 [308766.701322]  cpu_startup_entry+0x29/0x40
May 24 17:12:37 192.168.122.226 [308766.701324]  rest_init+0xcc/0xe0
May 24 17:12:37 192.168.122.226 [308766.701330]  start_kernel+0x60d/0x880
May 24 17:12:37 192.168.122.226 [308766.701366]  x86_64_start_reservations+0x18/0x40
May 24 17:12:37 192.168.122.226 [308766.701380]  x86_64_start_kernel+0x7a/0x80
May 24 17:12:37 192.168.122.226 [308766.701382]  common_startup_64+0x13e/0x141
May 24 17:12:37 192.168.122.226 [308766.701386]  </TASK>
May 24 17:12:37 192.168.122.226 [308766.702195] Kernel panic - not syncing: RCU Stall
May 24 17:12:37 192.168.122.226 [308766.724031] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G           OE      6.12.12+ #117
May 24 17:12:37 192.168.122.226 [308766.724885] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May 24 17:12:37 192.168.122.226 [308766.725416] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
May 24 17:12:37 192.168.122.226 [308766.726346] Call Trace:
May 24 17:12:37 192.168.122.226 [308766.726643]  <IRQ>
May 24 17:12:37 192.168.122.226 [308766.726872]  panic+0x336/0x3a0
May 24 17:12:37 192.168.122.226 [308766.727212]  ? try_to_wake_up+0x2f2/0x660
May 24 17:12:37 192.168.122.226 [308766.727639]  rcu_sched_clock_irq+0x1111/0x1120
May 24 17:12:37 192.168.122.226 [308766.728109]  ? timekeeping_update+0xdd/0x140
May 24 17:12:37 192.168.122.226 [308766.728555]  ? _raw_spin_unlock_irqrestore+0xe/0x40
May 24 17:12:37 192.168.122.226 [308766.729076]  ? timekeeping_advance+0x37a/0x580
May 24 17:12:37 192.168.122.226 [308766.729529]  ? perf_event_task_tick+0x46/0x80
May 24 17:12:37 192.168.122.226 [308766.730021]  update_process_times+0x65/0xa0
May 24 17:12:37 192.168.122.226 [308766.730460]  tick_nohz_handler+0x8f/0x140
May 24 17:12:37 192.168.122.226 [308766.730905]  ? __pfx_tick_nohz_handler+0x20/0x20
May 24 17:12:37 192.168.122.226 [308766.731375]  __hrtimer_run_queues+0x10f/0x2a0
May 24 17:12:37 192.168.122.226 [308766.731827]  hrtimer_interrupt+0xfa/0x240
May 24 17:12:37 192.168.122.226 [308766.732258]  __sysvec_apic_timer_interrupt+0x55/0x120
May 24 17:12:37 192.168.122.226 [308766.732780]  sysvec_apic_timer_interrupt+0x6c/0xa0
May 24 17:12:37 192.168.122.226 [308766.733287]  </IRQ>
May 24 17:12:37 192.168.122.226 [308766.733519]  <TASK>
May 24 17:12:37 192.168.122.226 [308766.733756]  asm_sysvec_apic_timer_interrupt+0x1a/0x20