Fix 4 audio WebSocket bugs: debug leak, timestamp drift, format config, error handling#67
Open
skywinder wants to merge 75 commits into
Open
Fix 4 audio WebSocket bugs: debug leak, timestamp drift, format config, error handling#67skywinder wants to merge 75 commits into
skywinder wants to merge 75 commits into
Conversation
# Conflicts: # .env.example # backend/app/routes/api.chat.ts
Applied PR feedback from @Bakuutin on #51: - Replaced getRootDB() with getMongoResource(auth) - Converted all direct database calls to use mongo resource abstraction - Ensures proper access logging and traceability - Maintains compatibility with future releases All database operations now go through the resource layer: - db.collection().countDocuments() → mongo({ action: "count", ... }) - db.collection().find().toArray() → mongo({ action: "find", ... }) This provides: - Access control and audit logging - Centralized permission enforcement - Consistent security checks Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Introduced a comprehensive plan for speaker diarization, addressing issues with anonymous speaker labels and lack of cross-session linking. - Enabled existing enrollment and speaker management code, previously unused, to enhance functionality. - Established a modular architecture with a feature flag for speaker identification, allowing for independent toggling of features. - Detailed infrastructure setup for GPU deployment and fallback options, ensuring efficient processing. - Organized files according to project conventions, enhancing maintainability and clarity.
- Updated DEVELOPMENT.md with detailed instructions for local GPU inference, speaker identification, and database migrations. - Added new migrations for speaker profiles and diarization matched speaker fields to support voice enrollment and identification. - Introduced new worker scripts for voice enrollment and speaker matching, enabling efficient processing of audio data. - Implemented a new Voice Profiles page in the frontend for managing speaker profiles, including recording and uploading audio. - Enhanced the GPU services setup in docker-compose.yml to include a diarization service with health checks and environment configurations. - Updated the feature flags in the server configuration to support speaker identification, allowing for dynamic feature management.
- Updated .env.example to include new environment variables for GPU services, including DIARIZATION_SERVER_URL and HF_TOKEN for local development. - Removed unused indexes in the diarization migration to streamline database operations. - Added python-dotenv as a dependency in pyproject.toml for loading environment variables. - Implemented dotenv loading in the service API to support local development configurations. - Expanded documentation with a new guide on speaker identification, detailing setup, usage, and troubleshooting steps.
…ocumentation - Added DIARIZATION_SERVER_URL and SPEAKER_SIMILARITY_THRESHOLD environment variables to docker-compose.yml for improved configuration. - Introduced a new comprehensive guide on testing the speaker identification feature, detailing setup, enrollment, and troubleshooting steps in SPEAKER_IDENTIFICATION_TESTING.md. - Refactored API calls in frontend components to utilize the new callResource function for better resource management.
- Updated the audio upload handler to allow dynamic bucket selection for file uploads, defaulting to "uploads". - Introduced a new audio utility module for WAV creation and audio processing, facilitating better audio handling in the frontend. - Enhanced the Voice Profiles page to support saving and selecting recorded audio samples, integrating with GridFS for storage. - Added functionality to delete saved voice samples from GridFS, improving user experience in managing audio data. - Updated documentation to reflect changes in audio backend requirements and usage instructions for Mac and GPU servers.
…od schema expects "doc" - causing a 400 Bad Request error
Incorrect result access pattern: The code used result.get("data", []) expecting wrapped responses, but the mongo resource returns data directly:
find returns [...] (array directly)
findOne returns the document directly (or None)
This caused 'NoneType' object has no attribute 'get' errors
- Updated the Zod schema for speaker matching to include descriptive comments for parameters, improving clarity on their usage. - Modified the VoiceProfilesPage to correctly handle the result from the query, ensuring it processes an array directly. - Improved the UI for saved voice samples, making the list always visible when samples exist and adding a tooltip for user guidance. - Introduced a new Diarization job in the Python backend, enabling speaker diarization on audio chunks within specified time ranges.
- Added logging for audio processing in the worker and Python backend to improve debugging and monitoring. - Updated the JobsResource to include detailed logging for job enqueueing, success, and error handling. - Enhanced the VoiceProfilesPage to support attaching voice samples to profiles and improved sample management. - Introduced a new WaveformPlayer component in the frontend for visualizing audio waveforms, enhancing user experience in audio playback. - Updated the enrollment job to link voice samples to profiles, improving data integrity and management.
- Removed extensive logging from the JobsResource in the worker, streamlining the enqueue method for better performance. - Simplified the audio retrieval function in the Python backend by eliminating debug logging, enhancing code clarity and maintainability. - Ensured that the job enqueueing process returns success with the job ID directly, improving response efficiency.
- Updated the WaveformPlayer to fetch audio using the apiClient, enhancing security with authentication headers. - Created a blob URL for audio playback, improving the handling of audio data. - Added cleanup for blob URLs to prevent memory leaks, ensuring efficient resource management during playback.
- Deleted the APIKeysPage component to streamline API key management. - Updated the router to remove the API Keys route and adjusted the SettingsLayout to reflect this change. - Modified the description in the SettingsLayout for the API section to clarify its purpose. - Enhanced the APISettingsPage to incorporate API key management functionalities directly, improving user experience and accessibility.
- Updated transcription output schema to include additional metadata fields: processed, hasMore, transcriptionId, audioDuration, wordCount, segmentCount, and textPreview. - Implemented word count calculation and added metadata to transcription documents. - Updated JobDetailPage to fetch and display transcription details, including audio duration, word count, and segments. - Enhanced JobsPage to show transcription-specific information when applicable, including text preview and duration. These changes improve the handling and presentation of transcription data across the application.
- Updated DEVELOPMENT.md to include new environment variables for development mode and their effects. - Added a new script (sync-env.sh) to synchronize .env with .env.example, allowing for easy management of environment variables.
- Add scripts/setup.sh for streamlined first-run experience - Fix token-create to output both MYCELIA_CLIENT_ID and MYCELIA_TOKEN - Improve sync-env.sh to detect first-run and suggest setup.sh - Update README Quick Start to use setup.sh --start - Add legend to .env.example ([AUTO], [OPTIONAL], [CLI-ONLY]) Co-authored-by: Cursor <cursoragent@cursor.com>
- Updated the schema processing in RunJobDialog to exclude the $schema property, improving compatibility with AJV8. - Modified CreateJobPage to similarly strip the $schema from the input schema, ensuring consistent handling across components and enhancing overall code clarity.
…ance ObjectsPage starred objects fetching - Added TimelinePlayerControls component to the TimelineHeader for improved playback functionality. - Updated layout in TimelineHeader to accommodate new player controls alongside existing elements. - Modified fetchStarredObjects in ObjectsPage to always retrieve all starred items, ensuring favorites are consistently visible regardless of search filters.
…ation - Added server connection testing functionality that automatically validates the endpoint and provides feedback on server status. - Implemented dynamic token name generation based on browser information and current date for improved user experience. - Updated UI to reflect suggested token names and adjusted error handling for server responses.
…h script - Expanded the dev-environment documentation to include server endpoints, port references, hot reload capabilities, and first-time setup instructions for better clarity. - Updated the git-commit-to-branch script to support additional options for committing changes, including staged-only commits, branch creation, and improved error handling. - Enhanced user interaction with clearer prompts and help messages for better usability.
…ation - Added server connection testing functionality that automatically validates the endpoint and provides feedback on server status. - Implemented dynamic token name generation based on browser information and current date for improved user experience. - Updated UI to reflect suggested token names and adjusted error handling for server responses.
…erver is listening - Moved worker initialization to occur after the HTTP server has started, ensuring that workers depend on the HTTP API are properly set up. - Simplified the server listening logic by using a promise to handle the asynchronous operation of starting the server.
… and progress indicators - Added a new section in JobDetailPage for displaying detailed information about conversation extractor and transcription sequence creator jobs, including processing time, conversations created, chunks processed, and error reporting. - Updated JobsPage to show result summaries for completed jobs, including conversation extractor and transcription sequence creator, with appropriate badges for empty results and progress indicators for ongoing jobs.
…d progress tracking - Introduced a new MetricCell component in JobDetailPage for consistent rendering of job metrics such as audio duration, word count, segments, and processing time. - Enhanced JobsPage with a JobDateRange component to display date ranges for job timelines, improving clarity on job progress and results. - Updated job progress indicators to provide detailed feedback for various job types, including transcription, VAD, and conversation extraction.
…nd progress indicators - Added functionality to pre-select job type in CreateJobPage based on URL query parameters, enhancing user experience. - Updated JobsPage to include additional progress indicators and detailed feedback for job results, including handling of 'has more' states and error messages for various job types.
- Added a new function `parseJobError` to convert job failure reasons into user-friendly error labels and details. - Updated JobDetailPage to display parsed error information for failed jobs, enhancing clarity for users. - Enhanced JobsPage to show parsed error details for failed jobs, improving the overall user experience with better error feedback.
…roved filtering - Implemented a new aggregation pipeline in JobsResource to count jobs by status, providing a comprehensive overview of job states. - Updated the JobsPage to display aggregated job counts, including totals for each status, and adjusted job filtering based on selected types. - Enhanced UI components to reflect the new job status data, improving user experience and clarity in job management.
- Added aggregation for job states (active, waiting, delayed) in JobsResource to provide a comprehensive overview of job statuses. - Updated JobsPage to display new job status metrics and adjusted filtering logic to respect selected job types. - Improved UI components to reflect the enhanced job status data, enhancing user experience and clarity in job management.
- Updated .env.example to include optional backup settings for Percona Backup for MongoDB and GPU service configurations for remote inference. - Added new service definition for pbm-agent in docker-compose.yml to facilitate MongoDB backups. - Enhanced README.md to reference new database backup and recovery documentation in DEVELOPMENT.md.
…cessary loading state calculation
…ogic. Added error listeners for WebSocket connections and improved PCM audio byte calculation. Removed unnecessary debug file writing in audio chunk creation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
createAudioChunkunconditionally wrote raw audio todebug.${format}on every chunk flush — unnecessary disk I/O in production and audio data leakcalculateTimeFromBytesused Opus frame estimate (16KB/s) but buffer contains decoded PCM (32KB/s for 16kHz mono), causing 2x cumulative timestamp errorflush()passed onlyformatConfig.format(string) tocreateAudioChunk, discardingsampleRateandchannels— ffmpeg would use wrong parameters for non-16kHz audio/ws/audioand/ws_omiendpoints lacked immediatews.on("error")before async auth, risking uncaught exception crash from broken pipe during that windowTest plan
/ws/audio— verify nodebug.*files created in working directoryMade with Cursor