UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

Akshay-Venkatesh · 2025-01-06T21:29:01Z

Why/What?

Export handles are used for VA recycling check today but this is not guaranteed to be unique for newer memory allocators like VMM, and CUDA memory pools. We need to switch to using buffer_id attribute to perform this check.
We call MempoolDestroy when invalidating import entries which can hurt performance for mempool caching logic as we would be forced to reimport mempools again. This PR frees the suballocation instead of freeing the imported mempool altogether.

Akshay-Venkatesh · 2025-01-06T21:30:37Z

@yosefe , @tvegas1 We got reports of VA recycling not being detected internally. This PR aims to address the issue for newer allocators. If the changes look good, is there an option to backport this to 1.18 at all?

src/uct/cuda/cuda_ipc/cuda_ipc_md.h

iyastreb · 2025-01-07T06:54:11Z

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

+    arg1     = (const void *)&key->ph;
+    arg2     = (const void *)&region->key.ph;
+#endif
+        if (memcmp(arg1, arg2, cmp_size) == 0) {


Does it makes sense to update the test to cover this behavior?

yosefe · 2025-01-07T08:45:02Z

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

+#if HAVE_CUDA_FABRIC
+    cmp_size = sizeof(key->ph.buffer_id);
+    arg1     = (const void *)&key->ph.buffer_id;
+    arg2     = (const void *)&region->key.ph.buffer_id;
+#else
+    cmp_size = sizeof(key->ph);
+    arg1     = (const void *)&key->ph;
+    arg2     = (const void *)&region->key.ph;
+#endif


need to fix indent

any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?

any reason to not compare buffer_id also for legacy, regardless of HAVE_CUDA_FABRIC?

I had this to begin with but key->ph is of type CUipcMemHandle when HAVE_CUDA_FABRIC is not declared. So we would need to change the type if we want to also include buffer_id.

src/uct/cuda/cuda_ipc/cuda_ipc_md.h

tvegas1 · 2025-01-07T15:25:38Z

Export handles are used for VA recycling check today but this is not guaranteed to be unique for newer memory allocators like VMM, and CUDA memory pools. We need to switch to using buffer_id attribute to perform this check.

This PR solves the case where cuda reused pointer value (alloc/free/alloc), and failed to detect that it's actually not same allocation instance, right?

tvegas1 · 2025-01-13T15:48:05Z

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

@@ -124,7 +124,7 @@ static ucs_status_t uct_cuda_ipc_close_memhandle(uct_cuda_ipc_cache_region_t *re
                        (CUdeviceptr)region->mapped_addr, region->key.b_len));
        }
    } else if (region->key.ph.handle_type == UCT_CUDA_IPC_KEY_HANDLE_TYPE_MEMPOOL) {
-        return UCT_CUDADRV_FUNC_LOG_WARN(cuMemPoolDestroy(region->key.ph.pool));
+        return UCT_CUDADRV_FUNC_LOG_WARN(cuMemFree((CUdeviceptr)region->mapped_addr));


i did not understand why this change? previously we were erroneously freeing mem pool?

@tvegas1 It not erroneous but excessive. If a suballocation from the same mempool gets sent over in the future, then we would have benefited from using the same imported mempool but closememhandle occurs when there is VA recycling detected. This change addresses the problem by deferring all imported mempool releases to the point of UCP context destruction. The drawback of this approach as with previous IPC handling is that if the exporter explicitly destroys the mempool, there is a reference on the importer side until context destruction. This needs to be separately handled for legacy memory and newer allocations like VMM and Mallocasync in a different PR.

brminich · 2025-01-28T09:55:58Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

+                goto found;
+            } else {
+                /* VA recycling case. Remove entry. A given pointer should only
+                 * belong to one region so need to look through rest of the


do you mean no need?

src/uct/cuda/cuda_ipc/cuda_ipc_md.h

Akshay-Venkatesh · 2025-02-25T22:47:56Z

@yosefe @brminich can you also provide a review again if you don't mind?

src/uct/cuda/cuda_ipc/cuda_ipc_md.h

brminich · 2025-02-26T08:17:51Z

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

+        if (memcmp((const void*)&key->ph.buffer_id,
+                   (const void*)&region->key.ph.buffer_id,
+                   sizeof(key->ph.buffer_id)) == 0) {
            /*cache hit */


can we just compare buffer_id without using memcmp?

brminich · 2025-02-26T08:22:29Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

-    status = UCT_CUDADRV_FUNC(cuIpcGetMemHandle(legacy_handle, (CUdeviceptr)addr),
-                              UCS_LOG_LEVEL_ERROR);
+    key->ph.handle_type = UCT_CUDA_IPC_KEY_HANDLE_TYPE_LEGACY;
+    status = UCT_CUDADRV_FUNC(cuIpcGetMemHandle(&key->ph.handle.legacy,


you can use UCT_CUDADRV_FUNC_LOG_ERR instead

brminich · 2025-02-26T08:24:22Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

+                 * items in the linked list. Skip to export phase. */
+                ucs_trace("VA recycling detected for (%p ... %p) (%llu, %llu)",
+                          address, address + length, buffer_id, key->ph.buffer_id);
+                ucs_list_del(&key->link);


seems need to use ucs_list_for_each_safe now

yosefe

add unit test (gtest) to reproduce the issue that same fabric handle represents a different buffer

yosefe · 2025-02-26T17:20:28Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

+        goto err;
+    }
+
+    key->ph.handle_type = UCT_CUDA_IPC_KEY_HANDLE_TYPE_ERROR;


move after allocation

yosefe · 2025-02-26T17:21:34Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

+    ucs_trace("exporting handle for %p: base %p b_len %lu buffer_id %llu", addr,
+              (void*)key->d_bptr, key->b_len, key->ph.buffer_id);


remove this log and print buffer id in 233-234 log message - no need two log msgs

yosefe · 2025-02-26T17:25:31Z

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

-            goto found;
+            ucs_trace("found range (%p ... %p) in local cache", address,
+                      address + length);
+            if (buffer_id == key->ph.buffer_id) {


can remove this logic since rcache evicts memh if memory is released (using memory hooks invalidate logic)

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

src/uct/cuda/cuda_ipc/cuda_ipc_md.h

src/uct/cuda/cuda_ipc/cuda_ipc_md.c

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c

…10405)

Akshay-Venkatesh requested review from brminich, yosefe and tvegas1 January 6, 2025 21:29

iyastreb reviewed Jan 7, 2025

View reviewed changes

yosefe reviewed Jan 7, 2025

View reviewed changes

tvegas1 reviewed Jan 13, 2025

View reviewed changes

brminich reviewed Jan 28, 2025

View reviewed changes

rakhmets reviewed Jan 28, 2025

View reviewed changes

src/uct/cuda/cuda_ipc/cuda_ipc_md.h Outdated Show resolved Hide resolved

brminich reviewed Feb 26, 2025

View reviewed changes

brminich previously approved these changes Feb 26, 2025

View reviewed changes

yosefe reviewed Feb 26, 2025

View reviewed changes

Akshay-Venkatesh dismissed brminich’s stale review via c488a4d February 26, 2025 17:48

rakhmets reviewed Mar 1, 2025

View reviewed changes

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c Outdated Show resolved Hide resolved

src/uct/cuda/cuda_ipc/cuda_ipc_md.c Outdated Show resolved Hide resolved

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c Outdated Show resolved Hide resolved

src/uct/cuda/cuda_ipc/cuda_ipc_md.h Show resolved Hide resolved

brminich previously approved these changes Mar 2, 2025

View reviewed changes

Akshay-Venkatesh dismissed brminich’s stale review via 9a0bef0 March 3, 2025 17:18

yosefe reviewed Mar 4, 2025

View reviewed changes

src/uct/cuda/cuda_ipc/cuda_ipc_md.c Outdated Show resolved Hide resolved

src/uct/cuda/cuda_ipc/cuda_ipc_cache.c Outdated Show resolved Hide resolved

Akshay-Venkatesh added a commit to Akshay-Venkatesh/ucx that referenced this pull request Mar 6, 2025

UCT/CUDA_IPC: Use buffer_id to detect VA recycling (backport openucx#…

43917f2

…10405)

Akshay-Venkatesh mentioned this pull request Mar 6, 2025

UCT/CUDA_IPC: Use buffer_id to detect VA recycling - v1.18.x #10536

Merged

yosefe approved these changes Mar 7, 2025

View reviewed changes

rakhmets approved these changes Mar 7, 2025

View reviewed changes

UCT/CUDA_IPC: Use buffer id to detect VA recylcing

21b4c78

Akshay-Venkatesh force-pushed the topic/cuda-ipc-buffer-id-va-recycle branch from 9f48a14 to 21b4c78 Compare March 11, 2025 16:19

yosefe merged commit 853475e into openucx:master Mar 13, 2025
151 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

Akshay-Venkatesh commented Jan 6, 2025 •

edited

Loading

Akshay-Venkatesh commented Jan 6, 2025

iyastreb Jan 7, 2025

yosefe Jan 7, 2025

Akshay-Venkatesh Jan 7, 2025

tvegas1 commented Jan 7, 2025

tvegas1 Jan 13, 2025

Akshay-Venkatesh Jan 13, 2025

brminich Jan 28, 2025

Akshay-Venkatesh commented Feb 25, 2025

brminich Feb 26, 2025

brminich Feb 26, 2025

brminich Feb 26, 2025

yosefe left a comment

yosefe Feb 26, 2025

yosefe Feb 26, 2025

yosefe Feb 26, 2025

		ucs_trace("exporting handle for %p: base %p b_len %lu buffer_id %llu", addr,
		(void*)key->d_bptr, key->b_len, key->ph.buffer_id);

UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

UCT/CUDA_IPC: Use buffer id to detect VA recylcing #10405

Conversation

Akshay-Venkatesh commented Jan 6, 2025 • edited Loading

Why/What?

Akshay-Venkatesh commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvegas1 commented Jan 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Akshay-Venkatesh commented Feb 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosefe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Akshay-Venkatesh commented Jan 6, 2025 •

edited

Loading