Implement search-based data loader with configurable keyword and semantic matching #62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add keyword_search and semantic_search data sources to visualization system - Create search interface in FileUpload component with configurable search types - Generate synthetic repository data based on search terms with relevance scores - Update visualization utilities to handle search relevance metrics - Support using search as data sources for node color, size, and edge properties Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/1ee31c31-03ad-46c8-aedb-75ebe0b07426 Co-authored-by: jakethekoenig <[email protected]>
- Add searchTerm field to VisualizationConfig interface - Create search term input in UnifiedVisualizationControls when search data sources are active - Update computeNodeMetrics to dynamically calculate search relevance based on current search term - Add calculateKeywordRelevance and calculateSemanticRelevance functions to visualization utils - Re-compute node metrics when search term changes to provide real-time search visualization - Fix issue where search data sources had no way to configure the search term Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/ee39b139-d1b1-4c9c-9dd0-3d3f5a6c55c7 Co-authored-by: jakethekoenig <[email protected]>
- Remove duplicate calculateKeywordRelevance and calculateSemanticRelevance functions from FileUpload.tsx - Import search relevance functions from visualizationUtils to eliminate ~40 lines of code duplication - Add useEffect to keep local search term in sync with config changes in UnifiedVisualizationControls - Verify all references to keyword_search and semantic_search data sources are properly handled Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/4923143d-4d99-471c-9a02-075ed6a47ac8 Co-authored-by: jakethekoenig <[email protected]>
- Create Express.js backend server with content API endpoints - Add /api/file-content endpoint to read individual files - Add /api/search-content endpoint for bulk keyword/semantic search - Implement enhanced search algorithms that work on actual file content - Add API client utility for frontend-backend communication - Update computeNodeMetrics to use content-based search when backend available - Create start_project_with_api.sh script to run both servers - Add proper error handling and fallback to path-based search - Include security measures (path validation, file size limits, excluded files) The search now analyzes actual file content rather than just filenames, providing much more meaningful relevance scores for repository exploration. Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/bfbc40ad-edee-40a5-9178-88a46d0c29e8 Co-authored-by: jakethekoenig <[email protected]>
- Add missing `applicableTo: 'node'` field to keyword_search and semantic_search data sources - This fixes issue where search data sources weren't appearing in the "+ Add Data Source" dropdown - Search data sources now properly integrate with the refactored node/edge feature system from main branch - Users can now select keyword and semantic search as node features for visualization Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/9a0bb939-6ab8-4f2d-a7a2-debea3f042fa Co-authored-by: jakethekoenig <[email protected]>
- Add console logs to track search API calls and responses - Log search term, number of files being searched, and relevance results - Help identify if the issue is in API calls or visualization application - Temporary debugging code to troubleshoot why search isn't affecting node sizes Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/1071f137-05b2-4498-b9ac-0045514db814 Co-authored-by: jakethekoenig <[email protected]>
- Reinstall frontend npm packages to resolve dependency conflicts from merge - Update package-lock.json with clean dependency tree - Fix React useState hook errors that were preventing app from loading - Servers are now running successfully with clean dependency state Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/c9b864e7-4d0c-4185-8bd5-dd6212361ea1 Co-authored-by: jakethekoenig <[email protected]>
- Add Vite server proxy to forward /api/* requests to backend on localhost:3001 - Update apiClient to use relative URLs instead of direct localhost:3001 - Eliminates browser CORS and connection issues when accessing backend API - Both health check and search-content endpoints now work through proxy - Frontend can now successfully reach backend for content-based search functionality Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/d9cf2c73-bb9f-4591-9214-067b42dd47ac Co-authored-by: jakethekoenig <[email protected]>
- Add try/catch block to health check endpoint to log any errors - Help diagnose 500 errors when frontend tries to reach backend - Temporary debugging code to troubleshoot backend API issues Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/e4fcbd4c-5e11-4132-8bbf-d245daade2cd Co-authored-by: jakethekoenig <[email protected]>
- Increase Express.js request size limit to 50mb - Fixes 403 Forbidden errors when frontend sends large file lists for search - Backend can now handle large search requests from frontend - Search functionality should now work properly without being blocked Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/4a16f08f-9ee6-4ce1-8942-7cb0fcc15e77 Co-authored-by: jakethekoenig <[email protected]>
- Add console log to show the exact request body received by the search endpoint - Help diagnose 403 errors when frontend sends search requests - Temporary debugging code to troubleshoot backend API issues Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/02fe509d-89e7-40b1-bcb6-6308fa446af0 Co-authored-by: jakethekoenig <[email protected]>
- Add rewrite to Vite proxy to correctly forward API requests - Update apiClient to use correct URLs for all API calls - Fixes issue where frontend was not sending requests to backend - Search functionality should now work properly with Vite proxy Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/54e54add-d93a-4cae-9a9a-3975b53257ac Co-authored-by: jakethekoenig <[email protected]>
- Remove /api prefix from all backend endpoints - Fixes 404 errors when frontend tries to reach backend - Backend endpoints now correctly match the Vite proxy rewrite - Search functionality should now work properly with Vite proxy Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/f759724b-ee85-49ea-9020-0235b4d11ba9 Co-authored-by: jakethekoenig <[email protected]>
Created a new script `.mentat/preview/backend.sh` to automatically start the backend server in preview environments. This script performs a clean installation of npm dependencies to prevent corruption issues and then starts the Node.js server. This ensures the backend API is reliably available for the frontend to connect to. Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/repo-visualizer/log/0f9b3b4d-b057-42a7-944d-f7941fd109fb Co-authored-by: jakethekoenig <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds search functionality as a new data loader option, allowing users to generate repository visualizations based on search relevance rather than just uploading existing data files.
Key Features
Search Data Loader Interface
New Data Sources
keyword_searchandsemantic_searchdata sources to the visualization systemDynamic Search Configuration
Code Quality Improvements
Usage
The search generates realistic file structures and relationships, with relevance scores determining how files appear in the visualization when search data sources are used.
🤖 This PR was created with Mentat. See my steps and cost here ✨