Async Saving Traces (JUD-1289)

[Mandolaro]

📝 Summary

• 1. Upserting Trace Metadata now is defer into a Async Queue

🎥 Demo of Changes

✅ Checklist

• Tagged Linear ticket in PR title. Ie. PR Title (JUD-XXXX)
• Video demo of changes
• Reviewers assigned
• Docs updated (if necessary)
• Cookbooks updated (if necessary)

[linear]

JUD-1289 Make the trace saving async

[gemini-code-assist]

Summary of Changes

Hello @Mandolaro, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors how trace metadata is saved by introducing an asynchronous processing model. Instead of blocking on upsert_trace calls, these operations are now queued and handled by a dedicated background worker thread. This enhancement is designed to boost application performance and responsiveness by decoupling trace saving from the main application logic, while also ensuring data integrity through comprehensive flushing and shutdown procedures.

Highlights

• Asynchronous Trace Saving: The core save operation for trace metadata (upsert_trace) is now performed asynchronously in a dedicated background thread. This change aims to improve the responsiveness of the main application flow by offloading potentially blocking I/O operations.
• Dedicated Background Worker: A new JudgmentTraceWorker thread has been introduced within otel_span_processor to manage and process trace upsert requests from a thread-safe queue. This ensures that trace saving operations do not block the main execution path.
• Graceful Shutdown and Flushing: Robust mechanisms have been added to ensure that all pending trace upserts are processed and flushed before application shutdown or when force_flush is explicitly called, minimizing the risk of data loss for background operations.
• Synchronous Fallback: In scenarios where the asynchronous trace queue becomes full, the system is designed to gracefully fall back to performing the trace upsert synchronously. This ensures that trace data is still saved, even if the benefits of asynchronous processing are temporarily lost.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors trace saving to be asynchronous using a background worker thread, which is a great step for improving performance. The implementation of the async queue is solid, but I've identified a couple of high-severity issues related to robustness and thread safety that should be addressed. Specifically, there's a risk of data loss on transient network errors and a race condition during the shutdown process. I've also included a medium-severity suggestion to simplify the code by removing a redundant parameter.