range-diff: add configurable memory limit for cost matrix

[pcasaretto]

Problem Description

When git range-diff is given extremely large ranges, it can result in either:

1. Segmentation fault due to integer overflow in array index calculations
2. Excessive memory consumption leading to system hangs or OOM kills
3. Poor user experience with the command appearing to hang for minutes

Reproduction Case

In a Shopify's large monorepo a range-diff command like this crashes after several minutes with a SIGBUS error

$ git range-diff 4430f36511..316c1276c6 cb5240b6a8..2bbd292091

Range statistics:

• First range: 256,783 commits
• Second range: 1 commit
• Total: 256,784 commits
• Memory required for cost matrix: n² × 4 bytes = ~260GB

Stack Trace (Segmentation Fault)

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x6e000ae3b8)
  * frame #0: 0x000000010029a284 git`get_correspondences(a=0x000000016fde6188, b=0x000000016fde6160, creation_factor=60) at range-diff.c:356:20
    frame #1: 0x0000000100299310 git`show_range_diff(range1="4430f36511cbacf5c517c6851e2b8508a72dfd30..316c1276c63f55ad9413fa18bf3b6483564a9cf4", range2="cb5240b6a8ba59b4a1f282559ee0742721b0cafc..2bbd292091e376d177ce62e264dae8872ca6be5a", range_diff_opts=0x000000016fde6308) at range-diff.c:593:3
    frame #2: 0x00000001000c719c git`cmd_range_diff(argc=2, argv=0x0000600000bcd8c0, prefix="areas/core/shopify/", repo=0x0000000100468b20) at range-diff.c:167:8
    frame #3: 0x000000010000277c git`run_builtin(p=0x00000001004408d8, argc=3, argv=0x0000600000bcd8c0, repo=0x0000000100468b20) at git.c:480:11
    frame #4: 0x0000000100001020 git`handle_builtin(args=0x000000016fde6c70) at git.c:746:9
    frame #5: 0x0000000100002074 git`run_argv(args=0x000000016fde6c70) at git.c:813:4
    frame #6: 0x0000000100000d3c git`cmd_main(argc=3, argv=0x000000016fde7350) at git.c:953:19
    frame #7: 0x000000010012750c git`main(argc=4, argv=0x000000016fde7348) at common-main.c:9:11
    frame #8: 0x000000018573ab98 dyld`start + 6076

Root Cause Analysis

The crash occurs in get_correspondences() at line 356:

static void get_correspondences(struct string_list *a, struct string_list *b, ...)
{
    int n = a->nr + b->nr;  // Integer overflow: 256,784 fits in int
    ...
    ALLOC_ARRAY(cost, st_mult(n, n));  // Would allocate ~260GB
    ...
    cost[i + n * j] = c;  // Line 356: Invalid memory access
}

Problems:

1. Integer overflow: While n=256,784 fits in an int, n*n overflows
2. Memory allocation: Even with proper types, allocating 260GB is impractical
3. Poor UX: read_patches() takes minutes to process 256k commits before any error

Solution

This PR adds an early size check that runs before attempting to read patches:

Key Changes

1. New function count_range_commits(): Uses git rev-list --count to quickly count commits
2. Early validation in show_range_diff(): Checks total commit count before expensive operations
3. Configurable limit: MAX_RANGE_DIFF_TOTAL_PATCHES = 10000 (allows ~400MB matrix)

Performance Comparison

Before (current master):

• Large ranges: Hangs for 5+ seconds in read_patches(), then crashes or exhausts memory
• User experience: Command appears frozen with no feedback

After (this PR):

$ time git range-diff 4430f36511..316c1276c6 cb5240b6a8..2bbd292091
error: ranges too large for range-diff: 256784 total commits (max: 10000)

real    0m1.16s
user    0m1.16s
sys     0m0.12s

• Immediate error message in ~1 second
• Clear explanation of the problem
• No memory exhaustion or crashes

Why 10,000 Limit?

• Generous for real use: Even large refactoring series rarely exceed 1,000 commits
• Memory bounded: 10,000 commits = ~400MB cost matrix (reasonable for most systems)
• Performance: Ensures range-diff completes in reasonable time

Testing

Tested with:

• Large ranges (256k commits): Returns error in ~1 second ✅
• Normal ranges (5-100 commits): Works as before ✅
• Edge case (exactly 10,000): Works correctly ✅
• Invalid ranges: Proper error handling ✅

Defense in Depth

This PR keeps the existing check in get_correspondences() as a backup, providing two layers of protection:

1. Early check prevents wasted work and provides good UX
2. Later check ensures safety even if counts are bypassed

Alternative Approaches Considered

1. Only fix integer overflow: Would prevent crash but still allow impractical operations
2. Dynamic memory limit: Too complex and platform-specific
3. Streaming approach: Would require major refactoring of the algorithm

[dscho]

Thank you for this patch! It is reasonable, I just have one suggestion how to improve it.

Please note that there has been a highly over-engineered attempt at addressing this problem before: https://lore.kernel.org/git/[email protected]/t/#me423268c4f14a0d37c0ac3e83dc7d5e9cea3661a. You probably want to mention this in the "cover letter" (i.e. in the initial PR comment that will be sent), even though that patch series' contributor seems to be AWOL for years already.

[pcasaretto]

Thank you @dscho for the thoughtful review!

I attempted to implement your suggestion of checking content size within read_patches(), but discovered an issue:

read_patches() currently buffers the entire output of git log -p into memory before processing:

  if (strbuf_read(&contents, cp.out, 0) < 0) {  // Line 87 - reads ALL output
      error_errno(_("could not read `log` output"));
      ...
  }
  // Only AFTER reading everything do we process line by line
  for (; size > 0; size -= len, line += len) {
      // Check limits here is too late - memory already consumed
  }

For the test case with 256k commits, this means ~6GB is read into the contents strbuf before any limits can be checked. By the time we could check content size or commit count in the loop, the memory is already exhausted.

To properly implement early exit as you suggested, we would need to:

1. Refactor read_patches() to process the git log output in a streaming fashion
2. Read and process line-by-line from the pipe instead of buffering everything
3. Check limits during streaming

Would you prefer:

1. Keep this simpler fix that at least prevents the crash (two passes but prevents the memory issue)
2. Attempt the more complex streaming refactor

I'll also reference the previous RFC attempt as you suggested.

[dscho]

@pcasaretto wow, thorough work! Personally, I would prefer the streaming approach, but I could understand if it is unreasonable to ask for such a huge refactor just to get the bug fix in. Your choice!

[pcasaretto]

After pairing with @thehcma, we've updated the approach to address the memory exhaustion issue more directly.

Instead of pre-counting commits, we now check the actual memory requirements of the cost matrix just before allocation in get_correspondences(). This approach:

1. Checks actual memory usage: We calculate n² × sizeof(int) and compare it against a configurable limit, which is more precise than just counting commits.

2. Adds a --max-memory option: Users can specify memory limits using human-readable sizes (e.g., --max-memory=1G, --max-memory=500M). The option accepts standard suffixes (k/m/g) that Git users are familiar with from other commands.

3. Defaults to 4GB: This allows comparing ranges of approximately 32,000 commits, which should be generous for real-world use cases while preventing impractical operations that would exhaust memory.

4. Provides clear error messages: When the limit is exceeded, users see both the required and available memory in human-readable format, for example:

   fatal: range-diff: unable to compute the range-diff, since it exceeds
   the maximum memory for the cost matrix: 256 GiB (274877906944 bytes)
   needed, 4.00 GiB (4294967296 bytes) available
   

This solution avoids the performance overhead of spawning additional processes while still preventing the crashes. Worth noting, that the process still takes a while to process and takes up around 10GB for the particular command that triggered the crash. As you noted, integrating this into read_patches() would be ideal, but that would require significant refactoring since it currently buffers all output before processing. Although I'm interested in the attempt, I think this is a good starting point.

What do you think about this approach, particularly:

• The choice of 4GB as the default limit?
• The --max-memory option name and syntax?

[gitgitgadget]

There are issues in commit dc9c6a6:
range-diff: add configurable memory limit for cost matrix
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

[pcasaretto]

Update: 4GB was too much for 32bit systems. Made the limit 2GB in those cases.