Shared langfuse_context causes race condition when handling concurrent requests

[avirajsingh7]

Describe the bug
We are using a shared global langfuse_context instance across multiple requests in the /threads route. This route is asynchronous, but it schedules background jobs that call a synchronous function process_run_core.

Scenario:

• Request 1 sets the langfuse_context, returns a 200 OK, and schedules Job 1.
• Request 2 quickly follows, overwrites the global langfuse_context, returns 200 OK, and schedules Job 2.
• Job 1 starts executing after Job 2 has already overwritten the global context.
• This causes Job 1 to incorrectly associate the Langfuse trace/logs with the context from Request 2.
• This issue is amplified with more concurrent requests, leading to unpredictable observability data and incorrect project/session attribution in Langfuse.
• The problem arises when multiple requests are processed concurrently

Expected behavior
Each job should maintain its own isolated Langfuse context, ensuring logs and traces are correctly attributed to the originating request/project/thread.

Current behavior
The global langfuse_context is shared and mutable, causing race conditions and cross-contamination of logs/traces in concurrent execution.