perf(cpa): use AST for templates #14648

denolfe · 2025-11-17T22:05:51Z

Overview

Refactor create-payload-app from string manipulation to AST-based transformations using ts-morph,
enabling safer and more maintainable template configuration.

Key Changes

AST transformation engine (packages/create-payload-app/src/lib/ast/)
- Type-safe payload config manipulation (database/storage adapters, imports, plugins)
- Package.json dependency management
- Prettier integration for formatted output
Template normalization
- Removed comment markers from all templates
- Standardized config structures for consistent AST processing
Improved reliability
- Replaced fragile regex patterns with structural code analysis
- Proper handling of import aliases and property positions

Design Decisions

AST over string manipulation

String-based replacements were brittle and error-prone with variations in formatting, import order, and code
structure. Using ts-morph provides:

Type-aware transformations that understand TypeScript syntax
Preservation of code formatting via Prettier
Maintainable operations that won't break on whitespace changes

Adapter configuration centralization

Database and storage adapter configs are now centralized in adapter-config.ts rather than scattered across
template files and replacement logic. This reduces duplication and makes adding new adapters
straightforward.

Template cleanup

Removed comment markers (e.g., // database-adapter-import-start) from templates since AST can locate
insertion points structurally. Templates now serve as clean, production-ready code without scaffolding
artifacts.

Overall Flow

sequenceDiagram
    participant CLI as create-payload-app
    participant AST as AST Engine
    participant TS as ts-morph
    participant Template as Template Files

    CLI->>Template: Copy template
    CLI->>AST: configurePayloadConfig(options)
    AST->>TS: Parse payload.config.ts
    AST->>TS: Transform config structure
    Note over AST,TS: Replace DB/storage adapters, add/remove plugins, update imports
    TS->>AST: Modified AST
    AST->>AST: Format with Prettier
    AST->>Template: Write updated file
    AST->>Template: Update package.json

…cture

Implement AST-based detection of buildConfig call structure in Payload config files using TDD approach. Detects: - buildConfig CallExpression - Import declarations - db property assignment - plugins array All tests pass (3/3).

Implements removeSharp function using TDD to remove sharp import and property from payload config files. This completes Phase 3 - Transformation Implementation.

Refactored configure-payload-config.ts to use AST-based transformations instead of string manipulation: - Replaced string manipulation logic with calls to configurePayloadConfig() and updatePackageJson() from AST modules - Added helper functions: mapDbType(), mapStorageAdapter(), and getEnvVarName() - Enhanced AST payload-config module with removeCommentMarkers() to clean up template markers - Fixed quote style conversion (double to single quotes) - Added indentation normalization to handle ts-morph's formatting quirks - Updated database and storage adapter templates with proper indentation - All existing integration tests pass

Removed AST comment markers from all template payload.config.ts files: - storage-adapter-import-placeholder - database-adapter-import - database-adapter-config-start - database-adapter-config-end - storage-adapter-placeholder - sharp-import Normalized template structures with clean imports and empty plugins arrays where applicable. Templates cleaned: - templates/_template/src/payload.config.ts - templates/blank/src/payload.config.ts - templates/website/src/payload.config.ts - templates/ecommerce/src/payload.config.ts - templates/with-cloudflare-d1/src/payload.config.ts - templates/with-vercel-website/src/payload.config.ts

Removed string-based replacement utilities that were superseded by AST-based config manipulation. Updated test to verify adapter configuration directly instead of using replacement templates.

Fixed .js extension in AST utils import and updated pnpm-lock.yaml with prettier version.

Add integration tests for sampled template × database × storage combinations. Tests verify AST transformations are correctly applied to payload.config.ts and package.json for representative combinations: - blank + mongodb + localDisk - blank + postgres + vercelBlobStorage - website + mongodb + s3Storage - website + postgres + localDisk - ecommerce + mongodb + localDisk - ecommerce + postgres + r2Storage Tests copy template to temp directory, apply transformations via configurePayloadConfig(), and verify imports/config properties are correctly modified. Skip TypeScript compilation for speed.

- Add .js extension to types import in package-json.ts - Add type assertion for DB_PACKAGE_NAMES lookup - Import QuoteKind enum from ts-morph - Add null checks for regex match results in indentation normalization - Remove invalid indentationText setting

…, azureStorage, gcsStorage, uploadthingStorage)

…uce duplication

Convert all interfaces to types in AST implementation: - types.ts: 7 interfaces (DetectionError, PayloadConfigStructures, etc.) - adapter-config.ts: 2 interfaces (DatabaseAdapterConfig, StorageAdapterConfig) - utils.ts: 2 internal interfaces (FormatErrorOptions, AddImportOptions) - package-json.ts: 2 internal interfaces (PackageJsonTransformOptions, PackageJsonStructure) ConfigureOptions now uses intersection type (WriteOptions & {...}) instead of extends.

…ations Add debug logging throughout AST transformation pipeline when --debug flag is passed: **payload-config.ts:** - Detection phase: buildConfig found/missing, import counts, structure details - Transformations: adapter types, imports added/removed, special imports (vercel-postgres, d1-sqlite) - Validation: structure validation results - Write phase: quote/indent normalization, prettier status, file operations **utils.ts:** - Import operations: additions to existing vs new imports, removals **package-json.ts:** - Dependency updates: adapter changes, sharp removal, package name updates **Integration:** - configure-payload-config.ts: accepts debugMode parameter - create-project.ts: passes cliArgs['--debug'] to AST functions All debug messages use [AST] prefix with status indicators (✓ success, ✗ failure, ⚠ warning).

… code - Set process.env.DEBUG in main.ts when --debug flag is passed - Update debug() function to check process.env.DEBUG internally - Remove debugMode parameters from all AST functions (56 if-blocks eliminated) - Update all call sites to remove debugMode arguments - Remove debugMode from WriteOptions and ConfigureOptions types This eliminates ~150 LOC and improves code readability by removing conditional debug blocks throughout the codebase.

… with 2+ params - addDatabaseAdapter: (sourceFile, adapter, envVarName) → (sourceFile, { adapter, envVarName }) - addStorageAdapter: (sourceFile, adapter) → (sourceFile, { adapter }) - findImportDeclaration: (sourceFile, moduleSpecifier) → (sourceFile, { moduleSpecifier }) - removeImportDeclaration: (sourceFile, moduleSpecifier) → (sourceFile, { moduleSpecifier }) Benefits: - Improved backwards-compatibility (can add new options without breaking changes) - More explicit at call sites - Consistent with existing functions (addImportDeclaration, configurePayloadConfig) All 64 tests passing.

… AST code Replace dynamic prettier import with CLI execution to resolve Jest/ESM issues. - Add detectPackageManager() for pnpm/npm/yarn/bun detection - Refactor writeTransformedFile() to call prettier CLI after save - Remove 50+ lines of manual normalization code - Enable formatWithPrettier by default - Update tests to use prettier (remove formatWithPrettier: false)

… replacing DB adapters Enhance AST transformations to maintain original positions of imports and properties during database adapter replacement. Changes (done alongside prettier refactoring): - Update removeImportDeclaration() to return removed import index - Add insertIndex parameter to addImportDeclaration() - Modify addDatabaseAdapter() to track and preserve import positions - Preserve db property position in buildConfig object - Add tests for import and property position preservation Behavior: - When replacing adapter: import and db property stay in original positions - When adding new adapter: inserted at end to avoid disrupting existing structure Tests: 36/36 unit tests passing, 6/6 integration tests passing

github-actions · 2025-11-17T22:13:11Z

📦 esbuild Bundle Analysis for payload

This analysis was generated by esbuild-bundle-analyzer. 🤖
This PR introduced no changes to the esbuild bundle! 🙌

denolfe added 30 commits November 14, 2025 20:47

chore(create-payload-app): add ts-morph and create AST directory stru…

fab0708

…cture

feat(create-payload-app): add AST transformation types

68ba02e

feat(create-payload-app): add error formatting utility

d791fea

feat(create-payload-app): add findImportDeclaration helper

a1934a6

feat(create-payload-app): add addImportDeclaration helper

57d21cf

feat(create-payload-app): add removeImportDeclaration helper

3101b0d

feat(create-payload-app): add buildConfig detection

02741b7

Implement AST-based detection of buildConfig call structure in Payload config files using TDD approach. Detects: - buildConfig CallExpression - Import declarations - db property assignment - plugins array All tests pass (3/3).

feat(create-payload-app): add database adapter transformation

10f80c9

feat(create-payload-app): add storage adapter transformation

849e2cf

feat(create-payload-app): add sharp removal transformation

92e1fc7

Implements removeSharp function using TDD to remove sharp import and property from payload config files. This completes Phase 3 - Transformation Implementation.

feat(create-payload-app): add structural validation

9bfb7ae

feat(create-payload-app): add file writing with prettier formatting

157d93a

feat(create-payload-app): add package.json transformation

0aad07e

feat(create-payload-app): add high-level configurePayloadConfig API

ef4671b

chore(create-payload-app): remove obsolete replacements.ts

cca5fee

Removed string-based replacement utilities that were superseded by AST-based config manipulation. Updated test to verify adapter configuration directly instead of using replacement templates.

chore(create-payload-app): update imports and dependencies

d3d58a0

Fixed .js extension in AST utils import and updated pnpm-lock.yaml with prettier version.

docs(create-payload-app): document AST architecture

2db29e6

chore(create-payload-app): remove empty plugin-project.ts file

fefa73c

fix(create-payload-app): add missing storage adapter types (s3Storage…

ab5bce8

…, azureStorage, gcsStorage, uploadthingStorage)

chore(create-payload-app): remove obsolete replacements.ts

2018bae

refactor(create-payload-app): centralize adapter configuration to red…

323537d

…uce duplication

denolfe added 7 commits November 15, 2025 00:49

chore: cleanup, db types as const

0ca30a6

chore: add upload test artifact to .gitignore

39a84bd

fix: use proper adapter replacements

56a8ba4

fix: always use npx for executing prettier

7c37f33

fix: remove @vercel/postgres import

a64d413

chore: template cleanup

9bda19e

github-actions bot added the created-by: Payload team label Nov 17, 2025

denolfe added 2 commits November 17, 2025 17:19

fix: properly add to plugins array

69317ec

refactor: co-locate unit tests

20d136e

denolfe changed the title ~~chore(create-payload-app): add ts-morph and create AST directory structure~~ chore(cpa): add ts-morph and create AST directory structure Nov 26, 2025

denolfe changed the title ~~chore(cpa): add ts-morph and create AST directory structure~~ chore(cpa): use AST for templates Nov 26, 2025

denolfe added 7 commits November 25, 2025 21:36

chore: adjust ast parsing approach

964c77c

test: add more int tests, handle alias

3099bbb

test: remove unnecessary/redundant tests

dc72293

test: remove more

be0c578

chore: improve package management flow

eca5750

chore: update CLAUDE.md

5812c22

fix: handle storage adapters in gen-templates, add AST workaround

1544a86

denolfe requested a review from paulpopus November 26, 2025 18:18

denolfe changed the title ~~chore(cpa): use AST for templates~~ perf(cpa): use AST for templates Nov 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(cpa): use AST for templates #14648

perf(cpa): use AST for templates #14648

Uh oh!

denolfe commented Nov 17, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 17, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

perf(cpa): use AST for templates #14648

Are you sure you want to change the base?

perf(cpa): use AST for templates #14648

Uh oh!

Conversation

denolfe commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Key Changes

Design Decisions

AST over string manipulation

Adapter configuration centralization

Template cleanup

Overall Flow

Uh oh!

github-actions bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 esbuild Bundle Analysis for payload

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

denolfe commented Nov 17, 2025 •

edited

Loading

github-actions bot commented Nov 17, 2025 •

edited

Loading