⚡ Bolt: Offload blocking I/O calls in FastAPI endpoints to worker threads#273
Open
⚡ Bolt: Offload blocking I/O calls in FastAPI endpoints to worker threads#273
Conversation
- Wraps blocking function calls (e.g., generator.generate, file read/write) with `anyio.to_thread.run_sync` inside FastAPI async routes. - Resolves event loop starvation caused by CPU/IO heavy blocking operations. Co-authored-by: anchapin <[email protected]>
Contributor
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
Reviewer's GuideOffloads blocking file and generator operations in FastAPI async endpoints to worker threads using anyio.to_thread.run_sync with functools.partial, and updates internal Bolt documentation to capture this async I/O offloading pattern. Sequence diagram for FastAPI route offloading blocking I/O to worker threadssequenceDiagram
actor Client
participant FastAPIApp
participant AnyioThreadWorker
participant Generator
participant FileSystem
Client->>FastAPIApp: HTTP POST render_pdf
activate FastAPIApp
FastAPIApp->>AnyioThreadWorker: anyio.to_thread.run_sync(generator.generate)
deactivate FastAPIApp
activate AnyioThreadWorker
AnyioThreadWorker->>Generator: generate(variant, output_format, output_path)
Generator->>FileSystem: write PDF to output_path
FileSystem-->>Generator: PDF written
Generator-->>AnyioThreadWorker: return
deactivate AnyioThreadWorker
activate FastAPIApp
FastAPIApp->>AnyioThreadWorker: anyio.to_thread.run_sync(output_pdf.exists)
deactivate FastAPIApp
activate AnyioThreadWorker
AnyioThreadWorker->>FileSystem: check file exists
FileSystem-->>AnyioThreadWorker: exists bool
AnyioThreadWorker-->>FastAPIApp: exists bool
deactivate AnyioThreadWorker
activate FastAPIApp
FastAPIApp->>AnyioThreadWorker: anyio.to_thread.run_sync(output_pdf.read_bytes)
deactivate FastAPIApp
activate AnyioThreadWorker
AnyioThreadWorker->>FileSystem: read_bytes
FileSystem-->>AnyioThreadWorker: PDF bytes
AnyioThreadWorker-->>FastAPIApp: PDF bytes
deactivate AnyioThreadWorker
activate FastAPIApp
FastAPIApp-->>Client: HTTP 200 PDF response
deactivate FastAPIApp
Flow diagram for using anyio.to_thread.run_sync with functools.partialflowchart TD
A[Start async FastAPI route] --> B[Identify blocking sync function call]
B --> C[Prepare callable using functools.partial]
C --> D[Call anyio.to_thread.run_sync with partial]
D --> E[Blocking work runs in worker thread]
E --> F[Worker thread performs file I O or generator call]
F --> G[Result returned to event loop]
G --> H[Async route awaits result without blocking event loop]
H --> I[Compose HTTP response]
I --> J[Return response to client]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- In
render_resume_pdf,yaml.dumpis still executed on the event loop and can be relatively heavy; consider offloading theyaml.dumpcall itself intoanyio.to_thread.run_syncrather than just thewrite_textto fully avoid blocking. - The same
anyio.to_thread.run_sync(functools.partial(...))pattern is repeated in several endpoints; consider introducing a small helper utility (e.g.,run_in_thread(func, **kwargs)) to centralize this and reduce duplication. - The update to
.jules/bolt.mdreplaces previous performance notes rather than appending a new entry; double-check whether those historical notes are meant to be preserved alongside the new async I/O guidance.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `render_resume_pdf`, `yaml.dump` is still executed on the event loop and can be relatively heavy; consider offloading the `yaml.dump` call itself into `anyio.to_thread.run_sync` rather than just the `write_text` to fully avoid blocking.
- The same `anyio.to_thread.run_sync(functools.partial(...))` pattern is repeated in several endpoints; consider introducing a small helper utility (e.g., `run_in_thread(func, **kwargs)`) to centralize this and reduce duplication.
- The update to `.jules/bolt.md` replaces previous performance notes rather than appending a new entry; double-check whether those historical notes are meant to be preserved alongside the new async I/O guidance.
## Individual Comments
### Comment 1
<location path="api/main.py" line_range="572-578" />
<code_context>
- with open(resume_yaml_path, "w", encoding="utf-8") as f:
- yaml.dump(yaml_data, f, default_flow_style=False)
+ yaml_str = yaml.dump(yaml_data, default_flow_style=False)
+ await anyio.to_thread.run_sync(
+ functools.partial(resume_yaml_path.write_text, yaml_str, encoding="utf-8")
</code_context>
<issue_to_address>
**suggestion (performance):** `yaml.dump` itself may be the heaviest part of this operation and is still running on the event loop thread.
The file write is now offloaded, but YAML serialization for larger resumes can still block the event loop. If you expect large payloads or high concurrency, consider running `yaml.dump` inside `to_thread.run_sync` as well (e.g., perform both dump and write in the same worker-thread function).
```suggestion
converter = JSONResumeConverter()
yaml_data = converter.json_resume_to_yaml(_resume_storage[resume_id]["json_resume"])
await anyio.to_thread.run_sync(
lambda: resume_yaml_path.write_text(
yaml.dump(yaml_data, default_flow_style=False),
encoding="utf-8",
)
)
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
572
to
+578
| converter = JSONResumeConverter() | ||
| yaml_data = converter.json_resume_to_yaml(_resume_storage[resume_id]["json_resume"]) | ||
|
|
||
| with open(resume_yaml_path, "w", encoding="utf-8") as f: | ||
| yaml.dump(yaml_data, f, default_flow_style=False) | ||
| yaml_str = yaml.dump(yaml_data, default_flow_style=False) | ||
| await anyio.to_thread.run_sync( | ||
| functools.partial(resume_yaml_path.write_text, yaml_str, encoding="utf-8") | ||
| ) |
There was a problem hiding this comment.
suggestion (performance): yaml.dump itself may be the heaviest part of this operation and is still running on the event loop thread.
The file write is now offloaded, but YAML serialization for larger resumes can still block the event loop. If you expect large payloads or high concurrency, consider running yaml.dump inside to_thread.run_sync as well (e.g., perform both dump and write in the same worker-thread function).
Suggested change
| converter = JSONResumeConverter() | |
| yaml_data = converter.json_resume_to_yaml(_resume_storage[resume_id]["json_resume"]) | |
| with open(resume_yaml_path, "w", encoding="utf-8") as f: | |
| yaml.dump(yaml_data, f, default_flow_style=False) | |
| yaml_str = yaml.dump(yaml_data, default_flow_style=False) | |
| await anyio.to_thread.run_sync( | |
| functools.partial(resume_yaml_path.write_text, yaml_str, encoding="utf-8") | |
| ) | |
| converter = JSONResumeConverter() | |
| yaml_data = converter.json_resume_to_yaml(_resume_storage[resume_id]["json_resume"]) | |
| await anyio.to_thread.run_sync( | |
| lambda: resume_yaml_path.write_text( | |
| yaml.dump(yaml_data, default_flow_style=False), | |
| encoding="utf-8", | |
| ) | |
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 What: Wrapped blocking operations like
generator.generate,pdf_path.read_bytes, andresume_yaml_path.write_textin FastAPI async endpoints usinganyio.to_thread.run_synccombined withfunctools.partial.🎯 Why: Synchronous operations inside an
async defFastAPI route block the ASGI event loop, preventing concurrent handling of incoming HTTP requests and causing severe performance degradation under load.📊 Impact: Eliminates event loop blocking during heavy operations like PDF rendering and AI tailing. Expected to drastically improve throughput and reduce latency spikes for concurrent API requests.
🔬 Measurement: Run a load test against the API with parallel requests (e.g., using
wrkor Locust). Verify that response times for fast endpoints remain low even while slow endpoints (like PDF rendering) are running concurrently.PR created automatically by Jules for task 2772379728622250420 started by @anchapin
Summary by Sourcery
Offload blocking resume and cover letter generation work in FastAPI endpoints to background worker threads to avoid blocking the event loop and improve concurrency.
Enhancements:
Documentation: