feat: CSV-first model API for cluster scripts

[Jammy2211]

Overview

Make CSV the first-class API for cluster lens modeling end to end. Today only the scaling-tier members live in a CSV (scaling_galaxies.csv); main galaxies, the host halo, and source profiles are still composed inline in Python. We'll extend the CSV layer to cover every cluster component, add a new scripts/cluster/csv_api.py guide that round-trips a full cluster model through CSVs, and refactor the existing 3 cluster scripts so the canonical workflow is "edit a CSV, re-run". The CSV layer becomes the definition of a cluster lens model.

Plan

• Library helpers (PyAutoGalaxy): extend autogalaxy/galaxy/galaxy_table.py (or sibling module) with readers/writers for named-galaxy CSVs keyed by a galaxy column and a profile_class column. One CSV per profile family — mass, light, point. The reader returns a typed result the workspace can feed into af.Collection(...); the writer takes a list of (galaxy_name, profile) and emits the family CSV. Profile-class lookup goes through al.mp.* / al.lp.* / al.ps.* via getattr. Sparse columns supported (different profile classes use different parameter columns; empty cells are tolerated).
• scripts/cluster/csv_api.py — new canonical workspace guide. Builds a small cluster model end-to-end in plain Python, writes it out to the family of CSVs, loads it back from those CSVs, prints the round-trip diff so every column has a visible Python counterpart. Also documents point_datasets.csv (existing) in the same place — convention is "load dataset before modeling", so the CSV story is presented in that order.
• simulator.py — load the truth model from the CSVs csv_api.py writes (rather than hardcoded inline). The simulator then writes data.fits + point_datasets.csv + copies of the model CSVs into the dataset folder so downstream scripts pick them up. This becomes the auto-simulate chain.
• modeling.py — refactor model composition to happen entirely from the model CSVs (main mass, main light, host halo, source light, source point, scaling tier). The inline af.Model(...) composition is shown as the alternative for users who prefer it, kept short.
• start_here.py — mirror modeling.py from CSVs, keeping the intro-script tone (Beta-feature warning, JAX speed pitch, Google Colab setup, simpler prose).
• Workspace audit — scan autolens_workspace for other CSV usage to confirm we're not breaking anything else; per the prompt the only existing case is scaling_galaxies.csv, verify.
• Schema decisions to lock during implementation:
  • Column convention for tuple parameters (ell_comps, etc.) — name_y / name_x vs name_0 / name_1.
  • Whether the host halo (NFWMCRLudlowSph with redshift_object / redshift_source) goes in main_lens_mass.csv or its own host_halo.csv — likely the latter since its parameter set is fundamentally different from cluster-member dPIE mass.
  • Whether scaling-tier members keep the existing 3-column y, x, luminosity schema or migrate to the new galaxy, profile_class, y, x, luminosity, ... schema. Probably keep existing — the scaling tier is implicitly one profile class per member and naming each scaling galaxy is more overhead than signal.

Detailed implementation plan

Affected Repositories

• PyAutoGalaxy (library — helpers + tests)
• autolens_workspace (workspace — new guide + refactor of 3 cluster scripts)

Work Classification

Both — library ships first (PyAutoGalaxy PR), workspace follows after the library lands.

Branch Survey

Repository	Current Branch	Dirty?
PyAutoGalaxy	main	clean
autolens_workspace	main	clean (dataset/* artifacts ignored)

Suggested branch: feature/cluster-csv-api
Worktree root: ~/Code/PyAutoLabs-wt/cluster-csv-api/

Implementation Steps

1. PyAutoGalaxy: library helpers. In autogalaxy/galaxy/galaxy_table.py (or split into a new galaxy_csv.py sibling), add:
   • galaxy_models_to_csv(galaxy_profile_pairs, file_path, profile_family) — writes one CSV per family. Each row carries galaxy, profile_class, <param columns…>, redshift?.
   • galaxy_models_from_csv(file_path) — returns a typed GalaxyModelTable carrying the rows grouped by galaxy name with profile_class dispatched to the right al.mp / al.lp / al.ps class.
   • Helper for stitching multiple family CSVs into an af.Collection (or list of af.Model(Galaxy)) keyed by the shared galaxy column.
   • Sparse-column tolerance: rows in the same CSV can use different profile classes with non-overlapping parameter columns; the reader only consumes the columns the row's class needs.
2. PyAutoGalaxy: tests. Round-trip unit tests in test_autogalaxy/galaxy/test_galaxy_table.py (or test_galaxy_csv.py) covering: single-profile-class round-trip, multi-class sparse-column round-trip, missing-column rejection, name-join across families.
3. autolens_workspace: scripts/cluster/csv_api.py — new guide demonstrating the round-trip with prose for every section.
4. autolens_workspace: simulator/modeling/start_here refactor — load model CSVs via the new library helpers; simulator emits both the model CSVs and the derived point_datasets.csv into the dataset folder.
5. Auto-sim guard — modeling/start_here check both data.fits and the model CSVs; trigger simulator if missing.
6. Workspace audit — grep -rn '\.csv' autolens_workspace/scripts/ to confirm we haven't missed another CSV consumer.

Key Files

• PyAutoGalaxy/autogalaxy/galaxy/galaxy_table.py (or new sibling) — library helpers.
• PyAutoGalaxy/test_autogalaxy/galaxy/test_galaxy_table.py — round-trip tests.
• PyAutoLens/autolens/__init__.py — re-export new helpers under al.*.
• autolens_workspace/scripts/cluster/csv_api.py — new guide.
• autolens_workspace/scripts/cluster/simulator.py — load truth from CSVs.
• autolens_workspace/scripts/cluster/modeling.py — compose model from CSVs.
• autolens_workspace/scripts/cluster/start_here.py — compose model from CSVs.

Open Design Questions (to resolve during implementation)

• Tuple parameter column convention (name_y / name_x vs name_0 / name_1).
• Whether NFWMCRLudlowSph host halo gets its own CSV.
• Whether scaling-tier CSV format migrates or stays.
• How to encode priors vs truth values in CSV cells when the simulator and modeling scripts share the same family CSV (the simulator wants concrete floats; modeling wants UniformPrior / fixed values — likely the CSV carries only concrete values, and modeling promotes selected columns to priors at compose time).

Original Prompt

Click to expand starting prompt

For autolens_workspace/scripts/cluster/simulator.py, can we make it so that all parameters are in .csv form
and loaded from there, to establish that the base way to interact with the autolens API for clusters
is via csv.

I thin kthe way to make this work is to write a guide, autolens_workspace/scripts/cluster/csv_api.py,
which illustrates how to set up lens models using the normal autolens API and then output them to csv,
and showing how all those featres linked together.

This csv file can then act as an "Auto Simulate" type siutation for simulator.py, which loads the csv outputsof this file.
The simulator will put the csv files int he lens it simulates at the end, meaning other scripts only need the
auto simulate performed here.

Things I am still unclear on that this guide could help with are:

Shouild main galaxies, extra galaxies and scaling galaies use their own csv files or can they all be combined into one?
I think it would be good if they could all be combined into one, but hiswould mean the .csv needs to know a lot more
than just parameters, but feasible mass profile class, lens name (e.g. when its used to name light and mass profiles in the Galaxy), redshift and
others. I think I like the idea of a single .csv API being used for all cluster interfaces.

The flip side is this could get complicated because if the same galaxy has light and mass profiles then the notion of column
heads breaks down, so maybe the rule is "one csv file per light or mass profile", and the reuse of light profile names, mass profile
names and galaxy names is exploited when building the model? Most cluster models will apply the same thing over loads
of galaxies so I think that works, so we can just build it in an extensible way.

This would mean it also needs the galaxy names, even though in simulator.py galaxies are not named when used in a Tracer
these names would be used for performing model composition, again the csv_api.py script could explain and cover this.

I would then go so far as to make it so that this guide also explains point_datasets.csv, which functionally looks a lot
more complete to me and just needs an explanation. I would explain this before doing galaxy API, as convention is normally load
dataset before modeling. Note that point_datasets.csv itself is made by simulator.py, thus I think csv_api.py
can just make an example one which is not paired to the model in the guide making it clear its for illustrative purposes
but that simultor.py makes the actual one.

There are also no .csv's used at all for defining the point source model, which obviously need to be paired
with the point dataset.

Source-side composition (the SersicCore light + Point model loops in both simulator.py and the JAX registration mirror) should also be migrated to two CSV files (source_point_models.csv and source_light_models.csv), making the whole cluster experience a first-class csv experience.

I guess at this point users should easily not just load .csv's into models but also be able to print the csv
contents in python / Notebook cells and have clear print statements of the loaded objects showing how their
csv load parameters link to autolens objects, I guess the csv_api does that.

Finally, do a quick scan through autolens_workspace of other csv uses but I think at the moment its just scaling galaxies
which are already implemented well.

Do deep research when coming to this csv API and once you're happy with it make csv interface the version used on all 3 cluster
scripts that exist. This is a huge issue -- the csv interface defines cluster modeling throughout, so dont be afraid
to ask hard questions about balancing the need for modeling large amoiunts of gaalxies to making a user friendly API.
Feel free to ask if some of this work would benefit going into the source code more so than it alredy has.

[Jammy2211]

Library PR Created — 2026-05-19

PRs

• feat(galaxy): named-galaxy CSV reader/writer for full model round-trips PyAutoGalaxy#428 — new autogalaxy/galaxy/galaxy_model_csv.py module + tests + re-exports
• feat: re-export galaxy_model_csv helpers under al.* PyAutoLens#526 — al.* re-exports of the new helpers

Status

Library helpers landed. Next: /start_workspace — scripts/cluster/csv_api.py guide + simulator/modeling/start_here refactor.

API Summary

Six new public symbols on PyAutoGalaxy (re-exported under al.*):

• al.GalaxyModelRow / al.GalaxyModelTable — typed CSV-row dataclasses.
• al.galaxy_models_to_csv(profiles_by_galaxy, file_path, family, redshifts=None) — write one CSV per profile family.
• al.galaxy_models_from_csv(file_path, family) — read; dispatches profile_class against mp/lp/ps namespaces.
• al.galaxies_from_csv_tables(*tables) — join family tables → Dict[str, Galaxy].
• al.galaxy_af_models_from_csv_tables(*tables) — same but yields Dict[str, af.Model[Galaxy]] ready to mutate selected params into priors.

Sparse columns supported, so dPIE + NFW rows can share one mass.csv. centre tuples use existing y, x columns; other tuples (e.g. ell_comps) use <name>_0 / <name>_1. Existing galaxy_table.py (3-column scaling-tier schema) untouched.

[Jammy2211]

Smoke Test Results — 2026-05-19

Workspace	Passed	Failed	Total
autolens_workspace	7	0	7

All 7 smoke tests passed across autolens_workspace.

[Jammy2211]

Shipped — 2026-05-19

PRs

• PyAutoGalaxy feat(galaxy): named-galaxy CSV reader/writer for full model round-trips PyAutoGalaxy#428 — new autogalaxy/galaxy/galaxy_model_csv.py module + 9 tests + re-exports under ag.*
• PyAutoLens feat: re-export galaxy_model_csv helpers under al.* PyAutoLens#526 — re-exports under al.*
• autolens_workspace feat(cluster): adopt named-galaxy CSV API across all cluster scripts #189 — scripts/cluster/csv_api.py + simulator/modeling/start_here refactor

Summary

Made CSV the first-class API for cluster lens modelling. The cluster simulator now writes the truth model into three family-level CSVs (mass.csv, light.csv, point.csv) keyed by a galaxy column with profile_class dispatch. modeling.py and start_here.py compose their lens model entirely from those CSVs via the new PyAutoGalaxy helpers (al.galaxy_models_from_csv, al.galaxy_af_models_from_csv_tables), and the new pedagogical scripts/cluster/csv_api.py walks through every cluster CSV end to end. The per-tier JSON centre files (main_lens_centres.json, host_halo_centre.json, source_centres.json) are dropped — centres now live in the CSVs.

Session Notes

• Schema decisions locked early via AskUserQuestion. One CSV per profile family (mass/light/point), profile-class dispatch via getattr against autogalaxy.profiles.{mass,light.standard,point_sources}, sparse columns tolerated. centre tuple keeps existing y, x precedent from galaxy_table.py; other tuples use <name>_0/<name>_1. Concrete values only in CSVs — priors applied at compose time by mutating af.Model[Galaxy] returned by galaxy_af_models_from_csv_tables.
• attr_name column was the key insight. Each row carries the Galaxy attribute name to bind the profile under (e.g. mass, bulge, point_0). Without it, name-pairing for the point-source convention (point_0 PointDataset ↔ point_0 model attr on source_0 galaxy) would have been ambiguous.
• Scaling-tier CSV deliberately kept as-is. scaling_galaxies.csv retains its 3-column y, x, luminosity schema. The new helpers are a parallel code path, not a replacement — scaling members are implicitly one profile class per row, naming each one would add overhead with no signal.
• Workspace stays self-contained inside PyAutoGalaxy via namespace constants. No PyAutoLens dependency in the library helper — _FAMILY_NAMESPACES = {"mass": autogalaxy.profiles.mass, ...}. PyAutoLens just re-exports the top-level symbols.
• Writer validation added after workspace bug. Initially missed that the helper accepted any-family profiles silently; passing a light profile under family="mass" would write a malformed CSV that only failed on read. Added a writer-side check that each profile's class is in the declared family namespace, plus a test for the new guard. Caught while writing csv_api.py (the bug surfaced as profile_class 'dPIEMassSph' not found in light namespace on read).
• Constructor-arg introspection via inspect.signature(cls). Both writer and reader use inspect.signature to discover constructor parameters per profile class. Tuple parameters detected via isinstance(default, tuple) on the signature default. Sparse columns handled automatically by autoconf.csvable.output_to_csv with headers=None (collects union of dict keys).
• PyAutoLens re-exports almost got deferred. Started development with PyAutoLens held by weak-fit; planned to ship PyAutoGalaxy alone and add re-exports later. weak-fit shipped mid-session so re-exports got folded back into the same PR pair.
• start_here.py survived its full rewrite from cluster-scaling-members. Was a stale group-scale extended-imaging copy when that task started. The CSV refactor here was a straightforward port of the modeling.py changes (load CSVs, mutate priors, compose).
• Source centre priors deliberately ignore the CSV truth. modeling.py and start_here.py initialize source point centres from the mean of observed multiple-image positions (np.mean(positions[:, 0/1])), not the truth value in point.csv — in a real analysis you don't know the source's true position, you only have its multiple-image positions.

Future Work (queued in z_features/cluster_lensing.md)

• cluster/3_test_workspace.md — autolens_workspace_test/scripts/cluster/ likelihood sanity checks + JAX likelihood functions
• cluster/4_likelihood_function.md — step-by-step likelihood_function.py walkthrough
• cluster/5_profiling.md — profiling scripts for cluster likelihood paths
• Deferred: aplt plotter promotion from the visualization prototype

Per feedback_no_bulk_issue_queues: these will be issued one at a time when their predecessor is close to shipping.