lnav is slow if too many matched search lines

[kcwu]

lnav version
git master (lnav 0.12.4-171-gfe17260)

Describe the bug
I have a text file with 5M lines and 4M of them containing KEYWORD. When searching the KEYWORD (via UI "/" key), it takes 700+ seconds to finish the search. In comparison, the search can finish in 5 seconds if the matching count is small. The search speed also depends on the cursor position.

To Reproduce
Steps to reproduce the behavior:

With this test file (base64 -d a.txt.zst.b64.txt | zstd -d -c > a.txt)
case 1

1. lnav a.txt
2. use /1aaa to search keyword "1aaa"
3. lnav shows the search progress Hit 1 of ..... for "1aaa". The number increases slowly, roughly 5k/sec

case 2

1. lnav a.txt
2. use /1aaa to search keyword "1aaa"
3. while lnav searching the keyword, use UP/DOWN key to move the cursor. When the cursor moves to above line 15, the hit count increases faster significantly.
4. If the cursor stays above line 15, the search could finish in less than 5 seconds.
   for case 2, I also attached my screen recording. Use the following command to replay:
   $ scriptreplay timing.txt typescript.txt

[tstack]

Thanks for nagging me to fix this, I've ignored it for too long.

I've pushed a change that tries to address the performance. On my M2 Mac, searching a file with 3M lines for a pattern that matches 2M of those lines takes 1.6s. That's a good improvement from v0.12.4 where it took 6.64s when doing the search from the start of the file and .. an unbearably long time when starting from a midpoint.

[kcwu]

Is this issue fully addressed? I still feel slow with the change for my test file.

[tstack]

Is this issue fully addressed? I still feel slow with the change for my test file.

I've pushed a change that should help. The problem with "a.txt" is that only the first 16 lines are treated as long lines. So, line 16 is treated as a log message with 5.8M lines, which was aggravating another part of the code that operated on the full message.

[kcwu]

Ah, when I redacted my log file, I replaced space characters with "a" and make lots of lines not look like log messages. Sorry. I think you can close this bug since my real issue is already resolved.

[tstack]

I think this is working better now, going to close.