Standardized formats, locations, and data contracts

[wesleytodd]

A primary goal I have is to be able to compare/track results over time. To do this we need repeatable/comparable data, ideally in a standardized format for tooling needs.

For example, one common thing folks need to include when they run benchmarks and other performance tests is the system stats they ran on. I think we should have a standard way to collect these and a standard format to include them in our saved results.

Additionally I think we want to include some "build metadata" like the repo and git ref from the repo where the tests were run, any relevant tool settings (ex. autocannon parallelization), and relevant information about the purpose of the run.

Because we are distributed across orgs and repos I think we need a strategy for where these things will live. I propose the following:

1. Tools and shared code (ex. common autocannon request groups) for perf testing should live in the perf-wg repo
2. Load tests for express and all express dependencies should live in the express repo
3. Load tests for middleware and non-express deps should live in their repos
4. Benchmark tests should live in the repo they test
5. Historical results should live in a directory adjacent to the test code

We could go so far as to standardize where in each repo these live. For example, it would be easier to have tooling that works across all the repos if we agreed on where some things lived. With that in mind, I propose the following:

perf
├── bench
│   └── example
│       ├── index.mjs
│       ├── package.json
│       └── results
|          ├── result-1749656765670.json
└── load
    └── example
        ├── index.mjs
        ├── package.json
        └── results
            ├── result-1749614997462.json

In addition to where things are kept, I think we would benefit from a standard schema for storing the results. As shown above, I propose a JSON file stored like this results/result-<timestamp>.json. The schema can probably be defined as a JSON schema, but here is a rough idea I had for a load test result to get us started:

{
  "serverMetadata": {
    "cpus": [...],
    "arch": "..."
  },
  "clientMetadata": {
    "cpus": [...],
    "arch": "..."
  },
  "serverResults": {
    "metrics": { ... }
  },
  "clientResults": {
    "latency": { ... }
  },
}

[GroophyLifefor]

The current implementation for performance testing, demonstrated using the cors repository, aligns well with the previously discussed strategy for standardized data collection and format.

perf-wg repository

Firstly I created an common collectMetadata file within the perf-wg/shared/ directory. This module is designed to capture essential system information, including:

• Architecture: os.arch()
• Platform: os.platform()
• CPU Details: Model and speed for each CPU core
• Memory: Total system memory in MB

This shared utility ensures consistent metadata collection across different performance tests.

perf-wg/shared/collectMetadata.mjs

import os from 'os';

export function collectMetadata() {
  return {
    arch: os.arch(),
    platform: os.platform(),
    cpus: os.cpus().map(cpu => ({
      model: cpu.model,
      speedMHz: cpu.speed
    })),
    memoryMB: Math.round(os.totalmem() / (1024 * 1024))
  };
}

cors repository

The cors repository has been used to demonstrate the initial folder structure and test execution.

File Structure:
The established structure within the cors repository is as follows:

cors/
└── perf/
    └── load/
        └── cors-get/
            ├── index.mjs
            ├── package.json
            └── results/
                └── result-<timestamp>.json

cors/perf/load/cors-get/index.mjs

// Server
import express from 'express';
import cors from '../../../lib/index.js';

// Benchmarking
import autocannon from 'autocannon';
import { writeFileSync } from 'fs';
import { collectMetadata } from '../../../../perf-wg/shared/collectMetadata.mjs';

// Constants
const port = 3000;
const url = "http://localhost:" + port;

/// / / / / / / / /
/// Server Start
const app = express();
let isStarted = false;

app.use(cors());

app.get('/', (req, res) => {
  res.json({ message: 'Hello from CORS test server!' });
});

const server = app.listen(port, () => {
  console.log('Test server running on ' + url);
  isStarted = true;
});
/// Server End

while (!isStarted) {
  await new Promise(resolve => setTimeout(resolve, 100));
}
await new Promise(resolve => setTimeout(resolve, 100));
console.log('Server is ready for benchmarking');

const result = await autocannon({
  url,
  connections: 100,
  duration: 10, // 10 seconds duration
});

const data = {
  schemaVersion: '1.0.0', // it's may not necessary, but it's a good practice to have a versioning system

  timestamp: Date.now(),
  runMetadata: {
    repo: 'https://github.com/expressjs/cors',
    gitRef: 'refs/heads/main',
    toolSettings: {
      autocannonConnections: 100,
      autocannonDuration: 10,
    },
  },
  serverMetadata: collectMetadata(),
  clientMetadata: collectMetadata(),
  serverResults: { metrics: { throughput: result.requests.average } },
  clientResults: { latency: { averageMs: result.latency.average } },
};

const filename = `perf/load/cors-get/results/result-${data.timestamp}.json`;
writeFileSync(filename, JSON.stringify(data, null, 2));
console.log(`Saved result to ${filename}`);

// Cleanup
server.close(() => {
  console.log('Server closed');
});

cors/perf/load/cors-get/package.json

{
  "name": "cors-load-test",
  "type": "module",
  "dependencies": {
    "autocannon": "^8.0.0",
    "express": "^5.1.0"
  }
}

Example Result File: An example output file, cors\perf\load\cors-get\results\result-1749804645953.json, demonstrates the proposed schema. It includes:

• schemaVersion: 1.0.0
• timestamp: The time of the test run
• runMetadata:
  • repo: https://github.com/expressjs/cors
  • gitRef: refs/heads/main
  • toolSettings: Details about autocannon connections and duration
• serverMetadata: Collected system information for the server (architecture, platform, CPU details, memory).
• clientMetadata: Collected system information for the client (architecture, platform, CPU details, memory).
• serverResults: Contains metrics, specifically throughput from the autocannon run.
• clientResults: Contains latency metrics, specifically averageMs.

This initial implementation successfully demonstrates the integration of shared metadata collection, standardized test execution, and structured result storage, forming a solid foundation for comparable performance tracking.

cors\perf\load\cors-get\results\result-1749804645953.json

{
  "schemaVersion": "1.0.0",
  "timestamp": 1749804645953,
  "runMetadata": {
    "repo": "https://github.com/expressjs/cors",
    "gitRef": "refs/heads/main",
    "toolSettings": {
      "autocannonConnections": 100,
      "autocannonDuration": 10
    }
  },
  "serverMetadata": {
    "arch": "x64",
    "platform": "win32",
    "cpus": [
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      }
    ],
    "memoryMB": 24317
  },
  "clientMetadata": {
    "arch": "x64",
    "platform": "win32",
    "cpus": [
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      },
      {
        "model": "11th Gen Intel(R) Core(TM) i5-11400H @ 2.70GHz",
        "speedMHz": 2688
      }
    ],
    "memoryMB": 24317
  },
  "serverResults": {
    "metrics": {
      "throughput": 7731.6
    }
  },
  "clientResults": {
    "latency": {
      "averageMs": 12.48
    }
  }
}

Explorer changes

Example Terminal

perf:load script in cors default package.json
"perf:load": "node perf/load/cors-get/index.mjs"

$ npm run perf:load                                                                                                                            

> cors@2.8.5 perf:load
> node perf/load/cors-get/index.mjs

Test server running on http://localhost:3000
Server is ready for benchmarking
Saved result to perf/load/cors-get/results/result-1749804561951.json
Server closed