Research paper: TypePatrol

Requirements

• Python 3.12+ (run.py uses f-strings with nested quotes, which is a syntax error on older versions) with PyYAML installed (pip install pyyaml).
• Node.js and npm (npm link is used to expose the local instrumentation packages).
• Network access: the scripts clone the packages to be tested from GitHub.

Instructions to run experiments

• Follow the steps mentioned in the setup section to set up the local npm packages that are used in running tests
• run.py: This script is used to run all the required experiments of this project related to the 15 JavaScript packages.

python3 run.py [MUTATION_PERCENTAGE]

• MUTATION_PERCENTAGE (optional): An integer specifying the percentage of mutable runtime locations that are mutated in each testing iteration. It is forwarded to the instrumentation packages as the LIMIT environment variable, which computes maxMutations = floor(totalLocations * LIMIT / 100). Defaults to 3 (i.e. 3% of the locations) if not specified.

• Examples:

  python3 run.py          # Mutates 3% of the locations (default)
  python3 run.py 10       # Mutates 10% of the locations
  python3 run.py 50       # Mutates 50% of the locations

• The 3_percent_results/, 10_percent_results/, 25_percent_results/ and 50_percent_results/ directories hold the results reported for the corresponding values.

• run_ts.py: This script is used to run all the required experiments of this project related to the 15 TypeScript packages. It takes the same optional MUTATION_PERCENTAGE argument with the same meaning and the same default of 3.

python3 run_ts.py [MUTATION_PERCENTAGE]

• Examples:

  python3 run_ts.py       # Mutates 3% of the locations (default)
  python3 run_ts.py 10    # Mutates 10% of the locations
  python3 run_ts.py 50    # Mutates 50% of the locations

• What the scripts do for every package: clone the package into two directories (see benchmarkDir / benchmarkIterDir below), run npm i and link the local instrumentation package, create the runtime/ log directory, instrument the sources, run the test suite once to collect the base runtime information, run the static-vs-dynamic type analysis, instrument the test files for mutation, and finally hand over to iterTesting.py / tsiterTesting.py for the mutation iterations.

• packageConfig.yaml and packageConfigTs.yaml: These files contain the configuration for the packages to be tested. Top-level keys:

  • benchmarkDir: directory into which each package is cloned for the base (unmutated) run. Created automatically.
  • benchmarkIterDir: directory into which a second copy of each package is cloned; this copy has its tests instrumented for mutation and is used by the iteration scripts.
  • packages: one entry per package, with the fields below.

  Per-package fields:

  • skip: set to false for the packages you want to test, true to skip a package. This is the supported way to run a single package.
  • url, branchName: the fork and branch that are cloned (git clone --branch <branchName> --single-branch <url>).
  • commit_hash: documentation only — it records the commit on which the reported results were obtained. The scripts do not check it out (the checkout is commented out in run.py / run_ts.py); the branch tip is used as-is.
  • language, moduleFormat (JS only: CommonJS or ESM, needed to emit the right import syntax for the instrumentation package).
  • srcLoc: source file/directory to instrument. testLoc: list of test files/directories to instrument for mutation.
  • typesLoc (JS only): location of the static type declarations used by packages/typematching.js.
  • excludeTs (TS only): excludes .ts files during test mutation.
  • runTestsCommand: the npm script used to run the tests. It must not clean previously generated coverage/log data.
  • specialManualSteps: false, or a description of manual edits. When set, the script pauses and asks for confirmation before continuing.

  Note that a few TypeScript packages are special-cased directly in run_ts.py (e.g. node-tar builds with npm run prepare and is re-linked, chokidar is rebuilt after test instrumentation, the MQTT.js symlink is removed), so not everything is expressible in the YAML.

• Tested versions of the libraries: The versions of the libraries tested are forked into a github account (https://github.com/typepatrol/), from where the scripts clone the packages to be tested. The forked branch name and the commit hash are specified in the packageConfig.yaml and packageConfigTs.yaml files. The scripts clone the specified branch; the commit hash is recorded for reference only.

• Outputs:

  • results_run.json / results_runTs.json (in this directory): the aggregated results per package, appended by iterTesting.py / tsiterTesting.py.
  • <benchmarkDir>/<package>/output.txt and <benchmarkIterDir>/<package>/output.txt: the console output of the test run and of the analysis/iteration stages.
  • <benchmarkDir>/<package>/runtime/: the logs of the base run; <benchmarkIterDir>/<package>/runtime/iterations/<i>/: the logs of each mutation iteration.

• mutation_coverage.py — run this manually after the experiments: run.py / run_ts.py do not call it. Once the runs have finished (and while benchmark*/ and benchmark*_iter/ are still on disk), run

python3 mutation_coverage.py

from this directory. It reads the base logs of every package together with all 10 iteration folders and adds/overwrites mutation_coverage, new_covered_locations, total_entries and unique_locations in results_run.json and results_runTs.json.

• It expects every package listed in the results files to have its benchmark*/<pkg>/runtime/ logs and all ten runtime/iterations/<i>/ folders present, and raises FileNotFoundError otherwise — so run it only after a full run, not after testing a single package.

• Because iterTesting.py / tsiterTesting.py update the per-package entry key by key instead of replacing it, keys that the current run does not produce (notably mutation_coverage and new_covered_locations) survive from the previous run. A results file can therefore mix numbers from different runs; re-run mutation_coverage.py after every set of experiments, or delete the package entry first, if the values must belong to a single run.

• Environment variables used by the instrumentation at runtime (set by the scripts, but useful when debugging a package by hand):

  • PACKAGE_DIR (JavaScript) / PACKAGE_PATH (TypeScript): the package directory whose runtime/ folder receives the logs.
  • LIMIT: the mutation percentage described above. SEED: the seed of the mutation RNG (iterTesting.py / tsiterTesting.py derive a fresh seed per iteration from a fixed base seed of 42, so runs are reproducible).

Local packages used

• instrumentation (JavaScript packages):

  • instrumentationpackage (Directory) : local npm package used in logging runtime types, runtime test files mutation
  • instrument.js : file to instrument JavaScript files/directories containing JS files. Usage: node instrumentation/instrument.js <CommonJS|ESM> <path>
  • instrumentType3.js : Instruments javascript code to log runtime common method invocations and "===", "!==" checks. Usage: node instrumentation/instrumentType3.js <CommonJS|ESM> <path>
  • testMutation.js : Instruments test files of JS npm packages to enable runtime mutation in the testing. Usage: node instrumentation/testMutation.js <CommonJS|ESM> <path>
  • typeCheck.js : helper used for type checking of the collected runtime values.

• tsinstrumentation (TypeScript packages):

  • tsinstrument.ts : TypeScript file where the instrumentation logic is implemented
  • tsinstrument.js : compiled from tsinstrument.ts, used for instrumenting TS package source files. run_ts.py recompiles it automatically (npx tsc tsinstrument.ts) before every run and overwrites this file. Note that this compilation currently reports type errors (and tsinstrument.js still contains a call to the undefined visitFunctionLike, on the export-assignment path); tsc emits the JavaScript anyway, so the run continues and the reported Error running command: ... npx tsc ... line in the output is expected.
  • instrumentType3.js : Different code from the /instrumentation/instrumentType3.js, but does the same thing. Usage: node tsinstrumentation/instrumentType3.js <path>
  • testMutation.js : Different code from the /instrumentation/testMutation.js, but does the same thing. Its CLI differs from the JS one — usage: node tsinstrumentation/testMutation.js <path> <excludeTs>
  • processmismatches.js : helper used for post-processing the recorded mismatches.

• log-mismatches (Directory) : local helper npm package for TS package testing, building this package (npm run build) after making changes to /src/index.js is neccessary.

• packages and tspackages (Directories) : the analysis scripts that are run after the test suite of a package finishes. packages/ is used by run.py (typematching.js, getLocations.js, metrics.js, primitiveMismatches.js) and tspackages/ by run_ts.py (getLocations.js, logmetrics.js, mismatchmetrics.js). packages/ has its own dependencies and needs npm i; tspackages/ has none.

• Testpackages and TestTspackages (Directories) : the per-iteration analysis scripts (metrics.js, findDiff.js, matchTypes.py, ...). At startup, run.py symlinks Testpackages/*.* into benchmarkIterDir and run_ts.py symlinks TestTspackages/*.*, so the iteration scripts can invoke them as ../metrics.js from inside a cloned package. Their dependencies must be installed (npm i) as well.

Other scripts in this directory

• iterTesting.py / tsiterTesting.py : run the mutation iterations for one package. Invoked by run.py / run_ts.py; usage: python3 iterTesting.py <basePkgPath> <mutatedPkgPath> <testCommand> <packageName> [MUTATION_PERCENTAGE]
• getType3.py : compares the base and mutated logs of common-method/equality checks; called at the end of the iterations.
• mutation_coverage.py : computes the mutation-coverage metrics; must be run by hand after the experiments (see above).
• commonMethoderr.py, uniqueEntries.py, test_1package.py : helper/analysis scripts.
• Graphs.ipynb : notebook used to produce the plots from the *_percent_results directories.

Setup

• Install the necessary packages for instrumentation, logging and analysis — run npm i in instrumentation/, instrumentation/instrumentationpackage/, tsinstrumentation/, log-mismatches/, packages/, Testpackages/ and TestTspackages/ (refer to the package.json in each directory; tspackages/ has no dependencies of its own).
• Register the local packages globally by running npm link in the instrumentation/instrumentationpackage/ and log-mismatches/ directories.
• Build log-mismatches (npm run build) — required before the first TypeScript run and after any change to log-mismatches/src/index.js.
• The packages to be analysed do not have to be downloaded by hand: run.py / run_ts.py clone them into benchmarkDir and benchmarkIterDir, and create the runtime/ results directory inside each of them. The scripts also link the local package into every cloned package, i.e. they run for you:

npm link instrumentationpackage // for JavaScript packages
npm link log-mismatches         // for TypeScript packages

• To run a single package, set skip: true for all the other packages in packageConfig.yaml / packageConfigTs.yaml.
• Re-running the scripts reuses the already cloned directories and skips instrumentation for them (a package is instrumented only when it has just been cloned). Delete the package directory from benchmarkDir / benchmarkIterDir to force a clean run.