-
Notifications
You must be signed in to change notification settings - Fork 1k
[POC] Using fasteners to control parallel execution of Conan #18253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
uilianries
wants to merge
21
commits into
conan-io:develop2
Choose a base branch
from
uilianries:feature/lock-fasteners
base: develop2
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
6177a69
Add validation for concurrent cache in Conan
uilianries 86e341d
Add test cases for config and export
uilianries e3b04a7
Remove assert for exceptions
uilianries 30f59a4
Add Winerror for Windows
uilianries 568a5c8
Add missing comma for error list
uilianries 70a1608
Make sure there will be execeptions
uilianries 8080114
Remove doubled variable
uilianries 046241c
Merge branch 'develop2' of https://github.com/conan-io/conan into tes…
uilianries f2a4c6c
Add lock feature using fastener
uilianries 66c51c6
Add decorator for write and read
uilianries 998759c
Add tests for write-read concurrent
uilianries 3220eb6
Use global config for timeout
uilianries 0a5a0e9
Use context manager for locking
uilianries 4fd0164
Update fasteners required version
uilianries d68b8fa
Remove timeout for fasteners
uilianries c9b6096
Use Context manager for lock
uilianries d86b39c
Use regular name for filelock folder name
uilianries 52641b1
Revert set_core_confs
uilianries b513786
Avoid locking when installing conan pacakge
uilianries 70b4282
Move parallel cache test to integration folder
uilianries 49872e4
Use thread + subprocess for paralle cache test
uilianries File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
""" The semaphore module provides inter-process locking mechanisms to ensure Conan commands can | ||
run concurrently without conflicts. | ||
|
||
It uses the fasteners library to create and manage locks across multiple processes. Thus, this | ||
module is a proxy in case the project need to use a different library in the future. | ||
""" | ||
import os | ||
from datetime import datetime | ||
from typing import Any | ||
|
||
import fasteners | ||
|
||
from conan.errors import ConanException | ||
from conan.api.output import ConanOutput | ||
from contextlib import contextmanager | ||
from conan.internal.cache.cache import PkgCache | ||
|
||
|
||
CONAN_SEMAPHORE_FILELOCK = "conan_semaphore.lock" | ||
|
||
|
||
def _filelock_path(conan_api: Any) -> str: | ||
""" Get the path to the interprocess file lock. | ||
|
||
:param cache_folder: ConanAPI cache folder path | ||
:return: Path to the file lock in Conan cache temporary directory | ||
""" | ||
cache = PkgCache(conan_api.cache_folder, conan_api.config.global_conf) | ||
return os.path.join(cache.filelock_folder, CONAN_SEMAPHORE_FILELOCK) | ||
|
||
|
||
@contextmanager | ||
def interprocess_lock(conan_api: Any) -> None: | ||
""" Context manager to acquire an interprocess lock. | ||
|
||
This method uses the fasteners library to create an interprocess lock, and serves as a proxy | ||
for the library. The lock is acquired using the InterProcessLock class, which allows multiple | ||
processes to safely access shared resources. The lock is released automatically when the | ||
context manager exits. | ||
|
||
:param conan_api: ConanAPI instance | ||
:return: None | ||
""" | ||
filelock_path = _filelock_path(conan_api) | ||
lock = fasteners.InterProcessLock(filelock_path) | ||
pid = os.getpid() | ||
try: | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Acquiring semaphore lock.") | ||
lock.acquire() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore has been locked.") | ||
yield | ||
except Exception as error: | ||
raise ConanException(f"Failed to acquire interprocess lock: {error}") | ||
finally: | ||
lock.release() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore has been released.") | ||
|
||
|
||
@contextmanager | ||
def interprocess_write_lock(conan_api: Any) -> None: | ||
""" Context manager to acquire an interprocess write lock. | ||
|
||
This method uses the fasteners library to create an interprocess write lock, and serves as a | ||
proxy for the library. The lock is acquired using the InterProcessReaderWriterLock class, | ||
which allows multiple processes to safely access shared resources. The lock is released | ||
automatically when the context manager exits. | ||
|
||
:param conan_api: ConanAPI instance | ||
:return: None | ||
""" | ||
filelock_path = _filelock_path(conan_api) | ||
lock = fasteners.InterProcessReaderWriterLock(filelock_path) | ||
pid = os.getpid() | ||
try: | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Acquiring semaphore write lock.") | ||
lock.acquire_write_lock() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore write has been locked.") | ||
yield | ||
except Exception as error: | ||
raise ConanException(f"Failed to acquire interprocess write lock: {error}") | ||
finally: | ||
lock.release_write_lock() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore write has been released.") | ||
|
||
|
||
@contextmanager | ||
def interprocess_read_lock(conan_api: Any) -> None: | ||
""" Context manager to acquire an interprocess read lock. | ||
|
||
This method uses the fasteners library to create an interprocess read lock, and serves as a | ||
proxy for the library. The lock is acquired using the InterProcessReaderWriterLock class, | ||
which allows multiple processes to safely access shared resources. The lock is released | ||
automatically when the context manager exits. | ||
|
||
:param conan_api: ConanAPI instance | ||
:return: None | ||
""" | ||
filelock_path = _filelock_path(conan_api) | ||
lock = fasteners.InterProcessReaderWriterLock(filelock_path) | ||
pid = os.getpid() | ||
try: | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Acquiring semaphore read lock.") | ||
lock.acquire_read_lock() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore read has been locked.") | ||
yield | ||
except Exception as error: | ||
raise ConanException(f"Failed to acquire interprocess read lock: {error}") | ||
finally: | ||
lock.release_read_lock() | ||
ConanOutput().debug(f"{datetime.now()} [{pid}]: Semaphore read has been released.") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
import os | ||
import subprocess | ||
import threading | ||
|
||
from conan.test.utils.test_files import temp_folder | ||
from conan.test.utils.tools import TestClient | ||
|
||
|
||
def _run_config_install(cmd, env, cwd, results, index) -> None: | ||
""" | ||
Run the command in a subprocess and store the return code in the results list. | ||
""" | ||
completed = subprocess.run( | ||
cmd, | ||
env=env, | ||
cwd=cwd, | ||
stdout=subprocess.PIPE, | ||
stderr=subprocess.PIPE, | ||
text=True, | ||
) | ||
results[index] = completed.returncode | ||
if completed.returncode: | ||
print(f"[worker {index}] stderr:\n{completed.stderr}") | ||
|
||
|
||
def test_parallel_config_subprocess(): | ||
"""Validate that subprocesses can run concurrently without issues. | ||
|
||
This test starts 30 separate subprocesses, each running the `conan config install` command. | ||
No command should fail, and the cache should be updated correctly. | ||
""" | ||
workers = 30 | ||
|
||
extra_folder = temp_folder(path_with_spaces=False) | ||
cache_folder = temp_folder(path_with_spaces=False) | ||
env = os.environ.copy() | ||
env["CONAN_HOME"] = cache_folder | ||
|
||
test_client = TestClient(cache_folder=cache_folder) | ||
test_client.run("profile detect --force") | ||
test_client.save({os.path.join(extra_folder, "profiles", "foobar"): "include(default)"}) | ||
cmd = ["conan", "config", "install", "-vvv", extra_folder, "--type=dir"] | ||
|
||
threads = [] | ||
return_codes = [None] * workers | ||
|
||
for index in range(workers): | ||
thread = threading.Thread( | ||
target=_run_config_install, | ||
args=(cmd, env, os.getcwd(), return_codes, index), | ||
daemon=True, # dies with the main program | ||
) | ||
thread.start() | ||
threads.append(thread) | ||
|
||
for thread in threads: | ||
thread.join() | ||
|
||
assert all(rc == 0 for rc in return_codes), f"Some subprocesses failed: {return_codes}" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fastener has context manager support as well, but we would expose it for any part in the code, instead of using this proxy.