Arc collapse in 2.3.0?

[IvanVolosyuk]

I noticed strange behavior change in ARC size. It looks like often arc size significantly drops for no obvious reason in ZFS 2.3.0. I still trying to figure in what circumstances it happens, so for now looking for evidence from other people and suggestions on how to debug this.

It seems if rollback to 2.2.7 the arc usage is way more stable, even if I set 'zfs_arc_shrinker_limit=0', which became default on 2.3.0.
ARC size seem to go down without any major changes to pool, no major spikes in RAM usage. With 128G of system ram (8GB apps, 16GB VMs) the ARC can comfortably sit at 90G, but sometimes when a new app requests just 4-5GB - ARC can drop 40-50G giving me ~70G free ram and arc usage dropping to almost arc_min_size (16G).
I don't know if it bad interaction with zram (which tops at 8G max used system ram), swappiness=175 or huge pages.
VM uses 16 of 1GB huge pages preallocated at boot, kernel cmdline: "hugepagesz=1G hugepages=16"
It looks like kernel uses transparent huge pages quite aggressively as well, with 2 1-GB pages and 90GB in 2MB pages.
Kernel: 6.12.16
ZFS: 2.3.0

Did anyone-else experienced anything like that? Any hints on how to debug this / where to look for more information?

MemTotal:       131012228 kB
MemFree:        71117912 kB
MemAvailable:   71337516 kB
Buffers:               0 kB
Cached:           425136 kB
SwapCached:        10504 kB
Active:          9376028 kB
Inactive:        1034576 kB
Active(anon):    9220948 kB
Inactive(anon):   912504 kB
Active(file):     155080 kB
Inactive(file):   122072 kB
Unevictable:      784372 kB
Mlocked:          783920 kB
SwapTotal:      83886076 kB
SwapFree:       75491180 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:      10676536 kB
Mapped:           346412 kB
Shmem:             78412 kB
KReclaimable:    1064372 kB
Slab:            4114120 kB
SReclaimable:    1064372 kB
SUnreclaim:      3049748 kB
KernelStack:       44568 kB
PageTables:        86840 kB
SecPageTables:      1356 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    141003580 kB
Committed_AS:   48505808 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     3543388 kB
VmallocChunk:          0 kB
Percpu:            22016 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2269184 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        16777216 kB
DirectMap4k:    23278316 kB
DirectMap2M:    92209152 kB
DirectMap1G:    18874368 kB

[amotin]

Some complaining about collapse, others about not evicting enough... It is good to see equilibrium in universe. ;)

If serious, there were changes in 2.3.0 aside of zfs_arc_shrinker_limit=0 to make ZFS more obedient and timely to kernel memory pressure requests. I have no reason to think that ZFS itself evicts unreasonably. So may be the question should be addressed to the kernel. Previously I already reported obvious memory pressure bugs to Linux kernel developers and had them fixed. It may be a time for more to be traced and reported.