Skip to content

Conversation

@Mantisus
Copy link
Collaborator

@Mantisus Mantisus commented Nov 4, 2025

Use ApifyStorageClient(request_queue_access='shared') for test_concurrent_processing_simulation. This is the expected usage for concurrent access.

Issues

Closes: #529

@Mantisus Mantisus requested a review from Pijukatel November 4, 2025 15:47
@Mantisus Mantisus self-assigned this Nov 4, 2025
Copy link
Contributor

@Pijukatel Pijukatel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a feeling that this test was flaky for the shared client even before the single client was introduced. Could you stress test it a little locally since the CI is now not running integration tests on fork PRs?

@Mantisus
Copy link
Collaborator Author

Mantisus commented Nov 6, 2025

Could you stress test it a little locally since the CI is now not running integration tests on fork PRs?

Out of about 70 runs, this test failed once due to an actor timeout (execution time exceeding 10 minutes).

But there really is a problem. Normally, it takes less than 10 seconds to execute, but in some runs, it takes up to 5 minutes.

UPD: I will try to find the reasons for this.

@Pijukatel Pijukatel self-requested a review November 6, 2025 12:00
@Mantisus Mantisus marked this pull request as draft November 6, 2025 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve Apify RQ storage client resilience to concurrent processing

2 participants