OLAP-BENCH Submission Bundle

This folder is the self-contained submission bundle for reproducing the main benchmark summaries, plots, and tables from the anonymized experiment outputs.

Reviewer Starting Point

Reviewers who want to inspect the artifacts by Research Question should start with:

• Artifacts_Organized_by_Research_Questions/

That folder groups the paper-facing evidence into RQ1/, RQ2/, and RQ3/, with a short README.md in each subfolder explaining which claims the files support and which figures, tables, CSVs, or JSON files to inspect first.

The organized RQ folders are curated for review. The same results can be reproduced from the included experiment data by running the notebook described below. The notebook rebuilds the metrics, tables, and figures from results_cluster/ and writes regenerated outputs to experiment_sum_ups/.

For additional methodological details, supplementary figures, and extended examples, reviewers can also inspect:

• APPENDIX.pdf

The primary entry point is:

• rebuild_experiment_sum_ups_from_results_cluster.ipynb

That notebook rebuilds the derived metrics and paper-facing artifacts from the included run results in:

• results_cluster/

The notebook is intended to be the main reproduction path for reviewers or other readers. Most users should start there rather than running the scripts manually.

What This Bundle Contains

The bundle includes:

• benchmark run artifacts copied from the cluster: results_cluster/
• benchmark question / SQL summary inputs: query_screening/, data/
• experiment plans and fitted hyperparameters: experiment_run_records/
• the scripts used to rebuild metrics, tables, and figures: utils/
• a notebook that executes the end-to-end rebuild flow: rebuild_experiment_sum_ups_from_results_cluster.ipynb

The notebook writes regenerated outputs to:

• experiment_sum_ups/

Recommended Reproduction Path

Use the notebook, in order:

1. Open rebuild_experiment_sum_ups_from_results_cluster.ipynb.
2. Run the setup cells.
3. Run the rebuild sections in order.
4. Inspect regenerated outputs under experiment_sum_ups/.

The notebook is organized as a staged pipeline:

1. rebuild absolute metrics from results_cluster
2. filter to the benchmark antipattern families used in the paper
3. generate relative-to-ground-truth JSONs
4. generate relative-to-raw-database JSONs
5. verify the expected outputs
6. regenerate figures and tables used in the paper

This notebook is the intended guide for understanding which scripts are run, in what order, and which artifacts they produce.

Main Inputs

These are the main inputs used by the notebook:

• results_cluster/ Anonymized cluster run folders containing plans, result CSVs, timing summaries, logs, and per-run scripts.
• query_screening/ Supporting benchmark-selection and SQL-tiering inputs used by the rebuild scripts.
• experiment_run_records/cluster_scripts/combined_linear_regression_hyperparameters.json The fitted hyperparameters used for EIEC reconstruction.

Main Outputs

The notebook regenerates artifacts under:

• experiment_sum_ups/figures/
• experiment_sum_ups/tables/
• experiment_sum_ups/*.json

Important outputs include:

• rebuilt merged-run metrics JSONs
• relative-to-ground-truth JSONs
• relative-to-raw-database JSONs
• model-level summary plots
• antipattern robustness tables
• benchmark overview tables

Folder Guide

Use this as a navigation map, not as a required execution order.

• utils/ The submitted scripts used by the notebook to rebuild metrics, tables, and figures.
• results_cluster/ The anonymized experiment outputs copied from the controlled cluster runs. This is the main raw input to the rebuild notebook.
• experiment_sum_ups/ The destination for regenerated JSONs, CSVs, and PDF figures.
• experiment_run_records/ Controlled experiment plans, successful-run summaries, mappings, and fitted hyperparameter files used by the rebuild process.
• query_screening/ Runtime-tier and benchmark-selection supporting artifacts.
• data/ Submission-local benchmark metadata and supporting JSON inputs.
• antipattern_database_generation/ Scripts and JSON specifications for generating antipattern databases.
• upgraded_refinement_framework/ Scripts and configs for generating the upgraded-refined SQL predictions.
• controlled_environment/ The controlled cluster execution stack used to produce the original experiment runs.
• local_docker_experiment/ A local Docker adaptation of the controlled environment, included for the TPCH ground-truth local reproduction demo.
• local_experiment_runs/ Output location for the local Docker reproduction notebook.
• system_information/ Environment specification files describing the controlled runtime and system assumptions.
• final_metrics_and_insights/ Included paper-facing artifacts from the original workflow.
• FILE_LIST.txt A full file manifest for the submission bundle.

How The Notebook Maps to the Scripts

The notebook is a thin orchestration layer over the scripts in utils/. In practice, each notebook section calls one or more submitted Python scripts and writes the outputs into experiment_sum_ups/.

This means:

• the notebook is the easiest way to reproduce the results
• the scripts remain available if you want to rerun only one stage
• the outputs can be inspected after each stage instead of only at the end

Cluster Run Artifacts

The copied results_cluster/ tree preserves the original run structure:

• one top-level folder per experiment
• benchmark_runs/ folders for individual (model, query, database, repeat) executions
• plans/, results/, samples/, and logs/ subfolders within each run

These artifacts are sufficient for the submitted rebuild scripts to recompute:

• correctness matching
• row-flow and execution-overhead derived metrics
• EIEC-based summaries
• merged-run benchmark artifacts

Anonymization

This bundle has been anonymized for submission. Local usernames, institution- specific paths, shared account paths, and identifiable hostnames were replaced with generic placeholders.

Local TPCH Ground-Truth Demo

This submission also includes a local Docker reproduction path for the raw TPCH SF1 ground-truth experiment. That path is separate from the main notebook above.

Use:

• reproduce_tpch_ground_truth_local_docker.ipynb

This local demo writes its outputs to:

• local_experiment_runs/

Most readers interested in the benchmark summary artifacts do not need this local Docker path. It is mainly included to document how the controlled execution stack can also be reproduced on a local machine for a small case.

If You Only Want the Main Paper Artifacts

Use this minimal path:

1. open rebuild_experiment_sum_ups_from_results_cluster.ipynb
2. run all cells in order
3. inspect experiment_sum_ups/figures/ and experiment_sum_ups/tables/

That is the shortest path from the submitted run artifacts to the regenerated paper-facing outputs.