Script-Free Testing

This project integrates the script-free GUI-testing technique introduced by Kirinuki et al. (ICSME 2021 & SANER 2022). Instead of brittle, locator-based Selenium code, each test step is expressed in plain English (e.g., click "Login"), and the framework uses NLP and heuristic search to identify the correct element at runtime.

Objectives

• Lower maintenance costs by eliminating explicit locators and reducing test flakiness after UI changes.
• Assess robustness of the script-free approach on evolving web applications and real-world test suites.
• Provide a reproducible benchmark (build scripts, Docker setup, and example branches) so others can compare against traditional locator-repair techniques.
• Document best practices and pitfalls discovered while adopting script-free testing in a standard Node/Playwright workflow.

---

Getting Started

1 · Prerequisites

Tool	Recommended Version	Notes
Node.js	10.x	nvm use will auto-select the right version.
Yarn	≥ 1.22	Install via npm i -g yarn or follow the Yarn docs.

2 · Script-Free Preparation

Follow the step-by-step setup in test/e2e/script-free-implementation/readme.md before continuing.

---

Building Local Test Bundles

git checkout <branch-to-test>   # e.g., before_app_change_b7eae4b
yarn                            # install dependencies
yarn build:test                 # outputs uncompressed files to /dist and compressed builds to /builds

---

Running Script-Free End-to-End Tests

yarn test:e2e:scriptfree

---

Example Branch Pairs for Locator-Change Experiments

Before app change	After app change, before locator change	After app change and locator change	impacted files
before_app_change_b7eae4b	after_app_change_before_locator_change_b7eae4b	after_app_change_and_locator_change_b7eae4b	metamask-ui.spec.js, from-import-ui.spec.js
before_app_change_a6d4725	after_app_change_before_locator_change_a6d4725	after_app_change_and_locator_change_a6d4725	incrimental-security.spec.js

Descriptions

• Before app change
  Snapshot of the application before the changes that cause locator failures in the E2E tests.
  Significance: Run script-free tests here to assess baseline robustness.

• After app change and before locator change
  Snapshot of the application with the app changes that trigger locator failures, but without locator fixes in the E2E tests.
  Significance: Confirms that failures are due to locator changes.

• After app change and locator change
  Snapshot of the application with the app changes and updated locators in the E2E tests.
  Significance: Both the normal E2E tests and the script-free tests should pass.

---

Running the Experiment

1. Verify failures on the “after app change, before locator change” branch

   git checkout after_app_change_before_locator_change_b7eae4b
   yarn install
   yarn install-chrome
   yarn build:test
   yarn test:e2e:chrome

   Expected result: Normal E2E tests fail due to elements not being located.

2. Run script-free on the “before app change” branch

   git checkout before_app_change_b7eae4b
   yarn install
   yarn install-chrome
   yarn build:test
   ./test/e2e/get_extension_id.sh

   Now you will get the right extension id in test/e2e/extension_id.txt.

   Then adjust the urls in the script-free tests in test/e2e/script-free-implementation/test_cases_metamsk_experiment to your extension_id. only these steps:

   - open "chrome-extension://{extension_id}/home.html#"

   Then run:

   yarn test:e2e:scriptfree

   After script generation, adjust the MetaMask runner at
   test/e2e/script-free-implementation/test_script/metamask_exp/metamaskui_tests_runner.py and test/e2e/script-free-implementation/test_script/metamask_exp/from_import_ui_tests_runner.py to execute the newly generated Selenium script.

   Then run:

   ./test/e2e/metamask-ui-run-generated-script.sh
   ./test/e2e/from-import-ui-run-generated-script.sh

   Expected result: The script runs successfully. (Tip: check screenshots.)

3. Confirm success on the “after app change and locator change” branch

   git checkout after_app_change_and_locator_change_b7eae4b
   yarn install
   yarn build:test
   # Repeat the same execution steps as above

   Expected result: Both normal E2E tests and script-free tests succeed.

---

MetaMask Browser Extension

The original MetaMask extension source is available at
https://github.com/MetaMask/metamask-extension.

---

MetaMask Locator Changes — Commits

The table below is imported from metamask_locator_changes_commits.csv and lists the commits related to locator changes used in the experiment.

Locator only Changes
Repo	SHA Commit	Description/affected files	Results Description	Number of test cases	Nr. Tc fail on before locator change and run after	Number of converted cases	Number of locator changes	Number Test case passes in Script-free	Number of locator passes in Script-free	Test case pass rate in %	Locator pass rate in %
MetaMask/metamask-snaps-beta	f501f62b2988afb5b33a6543e44361c628eefed3	test/e2e/*	Original Tests pass in before and after change (prophylactic locator change)	25	0	0	25	0	0	0	0
	b7eae4ba80e06f1d2069575d9b96dae66ea4496f	test/e2e/metamask-ui.spec.js, test/e2e/from-import-ui.spec.js	5 of 6 test case pass in script-free and 9 of 9 Locator Changes passed in Script-free	6	6	6	9	5	9	83.3333333333333	100
	9d5be5d29fcdab1273e30810f87de4624b8622a1	test/e2e/incremental-security.spec.js	Tests not convertible, requires opening two tabs	1	1	0	2	0	0	0	0
	a6d4725e5c946e66f1f79afdc80563b807500104	test/e2e/incremental-security.spec.js	1 of 1 test case pass in script-free and 1 of 1 locator change passed in script-free	1	1	1	1	1	1	100	100
	32a3f5ad7b8d30c1d4843152e575770b189146ea	test/e2e/metamask-ui.spec.js	Test case not running on both versions	1	0	0	2	0	0	0	0

---

Further Reading

• Kirinuki et al. “NLP-Assisted Web Element Identification Toward Script-Free Testing” — ICSME 2021.
• Kirinuki et al. “Web Element Identification by Combining NLP and Heuristic Search for Web Testing” — SANER 2022.
• Jasem, “Assessing the Robustness of Script-Free GUI Testing in Evolving Web Applications” — Bachelor Thesis Proposal, Ruhr-Uni Bochum (2025).