[shortfin] Implement async alloc/dealloc of buffers. #507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benvanik
approved these changes
Nov 14, 2024
stellaraccident
force-pushed
the
shortfin_async_alloc
branch
2 times, most recently
from
f79a1ae to
172f914
Compare
commit 3902491d7789cd7a6f0ef2bb1572abefff1073cf Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 20:56:02 2025 -0800 Flip flag off. commit 1880f4962b14071c7897a1c18770c246243784b7 Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 20:48:51 2025 -0800 Disable logging. commit b29b3c5 Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 20:30:40 2025 -0800 Allow no alias override. commit e5514c3 Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 19:04:14 2025 -0800 Do whole inv result tracking. commit 878ed0c Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 12:41:43 2025 -0800 Fix header commit 4f2f6b8 Merge: c527839 8b806bf Author: Stella Laurenzo <[email protected]> Date: Tue Feb 18 12:35:35 2025 -0800 Merge branch 'main' of github.com:nod-ai/sharktank into shortfin_async_alloc commit c527839 Merge: e3f1eac 51cf2f4 Author: Stella Laurenzo <[email protected]> Date: Wed Nov 13 17:41:51 2024 -0800 Merge branch 'main' into shortfin_async_alloc commit e3f1eac Author: Stella Laurenzo <[email protected]> Date: Wed Nov 13 17:09:43 2024 -0800 [shortfin] Implement async alloc/dealloc of buffers. This has been a todo since day one. For device buffers, this now properly stream orders the alloc/dealloc. Fix npe Enable logging Disable async device alloc. Fix dealloc barrier. Put dealloca on timeline Properly queue order dealloca.
stellaraccident
force-pushed
the
shortfin_async_alloc
branch
from
b57912c to
141f7ba
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SyncImportTimelineResourcewill be emitted per result that cannot be async deallocated.ProgramInvocation.assume_no_aliasinstance boolean to disable the assumption which allows async deallocation to work.ProgramIncovation.global_no_aliasproperty to control process-wide.This is a very fiddly optimization which requires (esp in multi-device cases) a number of things to line up. Tested on amdgpu and CPU with a number of sample workloads (with logging enabled and visually confirmed).
See #980 for detailed analysis and further work required.