Add integration tests for new API endpoints, update testing pattern

Author: dacharyc

server/express/package.json

@@ -31,9 +31,11 @@
     "@types/express": "^4.17.21",
     "@types/jest": "^29.5.14",
     "@types/node": "^20.10.5",
+    "@types/supertest": "^6.0.3",
     "@types/swagger-jsdoc": "^6.0.4",
     "@types/swagger-ui-express": "^4.1.8",
     "jest": "^29.7.0",
+    "supertest": "^7.1.4",
     "ts-jest": "^29.1.1",
     "ts-node": "^10.9.2",
     "typescript": "^5.3.3"

server/express/src/app.ts

@@ -161,5 +161,10 @@ process.on("SIGTERM", () => {
   process.exit(0);
 });
 
-// Start the server
-startServer();
+// Export the app for testing
+export { app };
+
+// Only start the server if this file is run directly (not imported for testing)
+if (require.main === module) {
+  startServer();
+}

server/express/src/utils/errorHandler.ts

@@ -38,13 +38,16 @@ export function errorHandler(
 ): void {
   // Log the error for debugging purposes
   // In production, we recommend using a logging service
-  console.error("Error occurred:", {
-    message: err.message,
-    stack: err.stack,
-    url: req.url,
-    method: req.method,
-    timestamp: new Date().toISOString(),
-  });
+  // Suppress error logging during tests to keep test output clean
+  if (process.env.NODE_ENV !== "test") {
+    console.error("Error occurred:", {
+      message: err.message,
+      stack: err.stack,
+      url: req.url,
+      method: req.method,
+      timestamp: new Date().toISOString(),
+    });
+  }
 
   // Determine the appropriate HTTP status code and error message
   const errorDetails = parseErrorDetails(err);

server/express/tests/integration/README.md

@@ -1,40 +1,67 @@
 # Integration Tests
 
-This directory contains integration tests for the Express MongoDB application. Unlike unit tests, these tests connect to
-a real MongoDB instance to verify end-to-end functionality.
+This directory contains integration tests for the Express MongoDB application. Unlike unit tests, these tests make actual HTTP requests to the Express app using `supertest` and connect to a real MongoDB instance to verify end-to-end functionality.
 
 ## Overview
 
-The integration tests are organized into two main categories:
+The integration tests are organized into three main categories:
 
 1. **Movie CRUD Integration Tests** (`movie.integration.test.ts`)
-   - Tests basic database operations (Create, Read, Update, Delete)
-   - Verifies MongoDB driver functionality with a real database
-   - Tests pagination, filtering, and sorting
+   - Tests basic CRUD API endpoints using `supertest`
+   - Makes actual HTTP requests to the Express app
+   - Tests pagination, filtering, and sorting via API
    - Automatically skipped if `MONGODB_URI` is not set
 
 2. **MongoDB Search Integration Tests** (`mongodbSearch.integration.test.ts`)
-   - Tests MongoDB Search functionality
-   - Works with both local MongoDB and Atlas instances
-   - Tests search index creation, polling, and search queries
+   - Tests the MongoDB Atlas Search API endpoint using `supertest`
+   - Makes actual HTTP requests to test the `/api/movies/search` endpoint
+   - Tests search by plot, directors, cast, pagination, and search operators
    - Automatically skipped if `ENABLE_SEARCH_TESTS` is not set
 
-**Note:** Tests use `describeIntegration` and `describeSearch` wrappers (from `setup.ts`) that automatically skip entire
-test suites when the required environment variables are not set.
+3. **Advanced Endpoints Integration Tests** (`advancedEndpoints.integration.test.ts`)
+   - Tests advanced API endpoints using `supertest`
+   - Makes actual HTTP requests to test:
+     - **Aggregation endpoints**: Movies with comments, statistics by year, directors with most movies
+     - **Atlas Search endpoint**: Compound queries, phrase matching, fuzzy matching
+     - **Vector Search endpoint**: Semantic similarity search using embeddings
+   - Aggregation tests automatically run if `MONGODB_URI` is set
+   - Atlas Search tests require `ENABLE_SEARCH_TESTS=true`
+   - Vector Search tests require `VOYAGE_API_KEY` to be set
+
+**Note:** Tests use `describeIntegration`, `describeSearch`, and `describeVectorSearch` wrappers (from `setup.ts` and test files) that automatically skip entire test suites when the required environment variables are not set.
+
+## Testing Approach
+
+These integration tests use **supertest** to make actual HTTP requests to the Express application, testing the complete request/response cycle including:
+- Routing
+- Request parsing
+- Controller logic
+- Database operations
+- Response formatting
+- Error handling
+
+This approach ensures that the API endpoints work correctly from the client's perspective.
 
 ## Requirements
 
-### Basic Integration Tests
+### Basic Integration Tests (CRUD and Aggregations)
 
 - **MONGODB_URI** environment variable must be set
 - MongoDB instance must be accessible (can be local MongoDB or Atlas)
 
-### Search Tests
+### Atlas Search Tests
 
 - **MongoDB instance** with Search enabled (local MongoDB or Atlas)
 - **MONGODB_URI** environment variable
 - **ENABLE_SEARCH_TESTS=true** environment variable to enable tests
 
+### Vector Search Tests
+
+- **MONGODB_URI** environment variable must be set
+- **VOYAGE_API_KEY** environment variable must be set with a valid Voyage AI API key
+- MongoDB instance must have the `embedded_movies` collection with vector embeddings
+- Vector search index must be configured on the `embedded_movies` collection
+
 ## Running the Tests
 
 ### Run All Integration Tests
@@ -51,6 +78,9 @@ npx jest --config jest.integration.config.json tests/integration/movie.integrati
 
 # Run only Search tests (requires Search-enabled MongoDB)
 npx jest --config jest.integration.config.json tests/integration/mongodbSearch.integration.test.ts
+
+# Run only Advanced Endpoints tests (aggregations, search, vector search)
+npx jest --config jest.integration.config.json tests/integration/advancedEndpoints.integration.test.ts
 ```
 
 ### Set Environment Variables
@@ -66,15 +96,27 @@ MONGODB_URI=mongodb+srv://username:[email protected]/sample_mflix?ret
 
 # Optional: Enable Search tests (requires Search-enabled MongoDB)
 ENABLE_SEARCH_TESTS=true
+
+# Optional: Enable Vector Search tests (requires Voyage AI API key)
+VOYAGE_API_KEY=your-voyage-ai-api-key
 ```
 
 ## Test Structure
 
 ### Setup and Teardown
 
+All integration tests follow an **idempotent pattern** to ensure they can be run multiple times without side effects:
+
 - **Global Setup** (`beforeAll` in `setup.ts`): Connects to MongoDB once before all tests
 - **Global Teardown** (`afterAll` in `setup.ts`): Closes MongoDB connection after all tests
-- **Test Cleanup** (`afterEach` in test files): Removes test data after each test to ensure isolation
+- **Pre-Test Cleanup** (`beforeAll` in test files): Removes any orphaned test data from previous runs
+- **Post-Test Cleanup** (`afterEach` in test files): Removes test data created during each test
+
+This ensures:
+- Tests can be run multiple times without conflicts
+- The dataset is left in an untouched state after test execution
+- Tests are isolated from each other
+- Failed test runs don't affect subsequent runs
 
 ## Troubleshooting
 
@@ -94,6 +136,15 @@ If you see "⚠️  Skipping MongoDB Search tests - ENABLE_SEARCH_TESTS not set"
 2. Verify your MongoDB instance has Search enabled (available in both local MongoDB and Atlas)
 3. For local MongoDB, ensure you're running a version that supports Search
 
+### Vector Search Tests are Skipped
+
+If you see "⚠️  Vector Search tests skipped - VOYAGE_API_KEY not set":
+
+1. Set `VOYAGE_API_KEY` environment variable with your Voyage AI API key
+2. Verify your MongoDB instance has the `embedded_movies` collection
+3. Ensure the vector search index is configured on the `embedded_movies` collection
+4. The `embedded_movies` collection should have documents with `plot_embedding_voyage_3_large` field
+
 ### Connection Errors
 
 If tests fail with connection errors:
@@ -132,9 +183,10 @@ If tests timeout, you may need to:
 
 | Aspect | Unit Tests | Integration Tests |
 |--------|-----------|-------------------|
+| HTTP Requests | Mocked with Jest | Real HTTP requests via supertest |
 | Database | Mocked with Jest | Real MongoDB instance |
 | Speed | Fast (milliseconds) | Slower (seconds) |
 | Dependencies | None | Requires MongoDB |
 | Isolation | Complete | Requires cleanup |
 | CI/CD | Always run | Conditional (requires MONGODB_URI) |
-| Purpose | Test business logic | Test database operations |
+| Purpose | Test business logic | Test end-to-end API functionality |