Add `llm-transpile` command with Switch runner and integration tests #2078

hiroyukinakazato-db · 2025-10-06T06:29:13Z

Changes

This PR adds the llm-transpile command for LLM-powered SQL conversion using the Switch transpiler.

What does this PR do?

Adds llm-transpile CLI command that runs Switch transpiler jobs with parameter passing support.

Relevant implementation details

CLI Integration:

Add llm-transpile command to Lakebridge CLI
Input source validation (workspace paths and local files)
Parameter passing to Switch job runs

Switch Runner Implementation:

SwitchConfig: manages Switch resources and job ID retrieval from InstallState
SwitchRunner: orchestrates Switch job execution with parameters

Testing:

Unit tests for Switch runner with parameter verification
Integration tests for Switch installation lifecycle

Development Environment:

Add .env to .gitignore for local development credentials

Caveats/things to watch out for when reviewing:

Parameter design: Follows transpile and recon command patterns
Catalog/schema usage: Uses values configured during Switch installation (following recon pattern)
Output parameter naming: Uses --output-ws-folder (not --output-folder) to explicitly indicate workspace folder
Dependencies: Requires PR Add Switch transpiler with --include-llm-transpiler flag #2066 (Switch installation) to be merged first

Linked issues

Resolves #2047

Functionality

added relevant user documentation
added new CLI command: databricks labs lakebridge llm-transpile
modified existing command

Tests

manually tested
added unit tests
added integration tests

github-actions · 2025-10-06T06:33:58Z

✅ 46/46 passed, 7 flaky, 3m5s total

Flaky tests:

🤪 test_validate_non_empty_tables (10ms)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (14.375s)
🤪 test_transpile_teradata_sql_non_interactive[True] (15.492s)
🤪 test_transpile_teradata_sql (19.182s)
🤪 test_transpile_teradata_sql_non_interactive[False] (4.998s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (8.237s)
🤪 test_transpiles_informatica_to_sparksql (9.502s)

_{Running from acceptance #2826}

Implement llm-transpile command for LLM-based code transpilation: - Add SwitchInstaller for Switch transpiler package management - Install Switch package and deploy to workspace - Create and manage Databricks jobs with job-level parameters - Configure Switch resources (catalog, schema, volume) - Add SwitchRunner for executing Switch transpilation jobs - Upload source files to workspace volume - Execute transpilation via Databricks job - Download results and handle job lifecycle - Add llm-transpile CLI command with Switch transpiler support - Add comprehensive unit and integration tests

Move _get_switch_package_path() from WorkspaceInstallation to SwitchDeployment as a protected method, following Single Responsibility Principle. SwitchDeployment now resolves its own package path internally. Changes: - Add _get_switch_package_path() protected method to SwitchDeployment - Update SwitchDeployment.install() signature to remove path parameter - Remove duplicate _get_switch_package_path() from WorkspaceInstallation - Remove unused sys and TranspilerRepository imports from installation.py - Update tests to use new interface with mocked path resolution

Update test_installation.py to match the refactored SwitchDeployment.install() interface that now takes only resources parameter (path resolution is internal). Changes: - Remove switch_repository fixture parameter from test methods - Delete unused _StubTranspilerRepository stub class - Remove unused imports (Path, TranspilerRepository) - Update assertions to check only resources argument The tests verify that: 1. Switch installation uses configured resources correctly 2. Missing resources logs appropriate error message

…le command

Sync with main branch to incorporate latest documentation updates # Conflicts: # labs.yml

The wait_for_completion option is intended for local CLI execution only and should not be included in Databricks job parameters. This change filters it out when building job parameter definitions. Changes: - Add excluded_options set to filter local-only options - Skip wait_for_completion when converting config.yml options - Add test using FriendOfSwitchDeployment pattern to verify exclusion

…ntrol Add test_switch_install_with_transpile for full workflow testing including job execution and output verification. Test automatically skips without LAKEBRIDGE_SWITCH_E2E=true environment variable. Refactor existing test to test_switch_install and extract helper functions for DRY implementation. All changes in single file with no CI impact.

…que suffix generation

Merged latest changes from main branch including: - Profiler skeleton and Synapse profiler scripts - Transpiler product_name to transpiler_id rename - Additional transpile command arguments support - Test improvements for MSSQL and transpiler repository

…transpile

sundarshankar89

LGTM

sundarshankar89 · 2025-10-08T06:19:37Z

src/databricks/labs/lakebridge/cli.py

+) -> None:
+    """Transpile source code to Databricks using LLM Transpiler (Switch)"""
+    ctx = ApplicationContext(w)
+    ctx.add_user_agent_extra("cmd", "llm-transpile")


Suggested change

ctx.add_user_agent_extra("cmd", "llm-transpile")

ctx.add_user_agent_extra("cmd", "switch")

sundarshankar89 · 2025-10-08T06:21:18Z

tests/unit/transpiler/test_switch_runner.py

+from databricks.labs.lakebridge.transpiler.switch_runner import SwitchConfig, SwitchRunner
+
+
+class FriendOfSwitchRunner(SwitchRunner):


Suggested change

class FriendOfSwitchRunner(SwitchRunner):

class MockSwitchRunner(SwitchRunner):

labs.yml

…itch

…transpile

gueniai

LGTM

gueniai · 2025-10-30T18:13:00Z

src/databricks/labs/lakebridge/cli.py

+    if ctx is None:
+        ctx = ApplicationContext(w)
+    del w
+    ctx.add_user_agent_extra("cmd", "transpile-switch")


the command is llm-transpile

I will modify this.

hiroyukinakazato-db added enhancement New feature or request feat/cli actions that are visible to the user labels Oct 6, 2025

hiroyukinakazato-db temporarily deployed to tool October 6, 2025 06:29 — with GitHub Actions Inactive

hiroyukinakazato-db temporarily deployed to tool October 6, 2025 06:56 — with GitHub Actions Inactive

hiroyukinakazato-db temporarily deployed to tool October 7, 2025 00:21 — with GitHub Actions Inactive

hiroyukinakazato-db force-pushed the feature/llm-transpile branch from 804507e to d884a20 Compare October 8, 2025 00:21

hiroyukinakazato-db had a problem deploying to tool October 8, 2025 00:21 — with GitHub Actions Failure

hiroyukinakazato-db temporarily deployed to tool October 8, 2025 01:33 — with GitHub Actions Inactive

hiroyukinakazato-db temporarily deployed to tool October 8, 2025 01:57 — with GitHub Actions Inactive

hiroyukinakazato-db marked this pull request as ready for review October 8, 2025 02:15

hiroyukinakazato-db requested a review from a team as a code owner October 8, 2025 02:15

hiroyukinakazato-db added 4 commits October 9, 2025 16:49

fix: update error messages to include 'true' flag for install-transpi…

bacd5f6

…le command

hiroyukinakazato-db force-pushed the feature/llm-transpile branch from 4f551ce to bacd5f6 Compare October 9, 2025 08:59

hiroyukinakazato-db temporarily deployed to tool October 9, 2025 09:00 — with GitHub Actions Inactive

Merge branch 'main' into feature/llm-transpile

21b6629

Sync with main branch to incorporate latest documentation updates # Conflicts: # labs.yml

hiroyukinakazato-db temporarily deployed to tool October 9, 2025 12:26 — with GitHub Actions Inactive

hiroyukinakazato-db added 2 commits October 10, 2025 17:02

chore: update Switch wheel with wait_for_completion fix

13bcc15

hiroyukinakazato-db temporarily deployed to tool October 10, 2025 08:06 — with GitHub Actions Inactive

hiroyukinakazato-db added 3 commits October 14, 2025 15:59

feat: enhance E2E testing for Switch with resource management and uni…

83678b8

…que suffix generation

hiroyukinakazato-db temporarily deployed to tool October 14, 2025 08:45 — with GitHub Actions Inactive

sundarshankar89 changed the base branch from main to switch-integration October 27, 2025 06:31

sundarshankar89 added the stacked PR Should be reviewed, but not merged label Oct 27, 2025

sundarshankar89 changed the base branch from switch-integration to main October 27, 2025 06:32

sundarshankar89 temporarily deployed to tool October 27, 2025 13:27 — with GitHub Actions Inactive

Intermediate check in

bb7c3d6

sundarshankar89 had a problem deploying to tool October 27, 2025 13:50 — with GitHub Actions Error

Intermediate check in

bd70638

sundarshankar89 temporarily deployed to tool October 27, 2025 14:00 — with GitHub Actions Inactive

Intermediate check in

0eb1570

sundarshankar89 temporarily deployed to tool October 27, 2025 14:26 — with GitHub Actions Inactive

Merge branch 'feature/switch-installer-integration' into feature/llm-…

9823201

…transpile

sundarshankar89 temporarily deployed to tool October 28, 2025 03:26 — with GitHub Actions Inactive

initial tests

eb46f24

sundarshankar89 temporarily deployed to tool October 28, 2025 09:44 — with GitHub Actions Inactive

Merge branch 'feature/switch-installer-integration' into feature/llm-…

c54e68f

…transpile

sundarshankar89 temporarily deployed to tool October 28, 2025 11:32 — with GitHub Actions Inactive

Merge branch 'feature/switch-installer-integration' into feature/llm-…

fab0e87

…transpile

sundarshankar89 temporarily deployed to tool October 29, 2025 06:33 — with GitHub Actions Inactive

sundarshankar89 approved these changes Oct 29, 2025

View reviewed changes

added flag to fail if users use regular transpile after installing sw…

1e10b60

…itch

sundarshankar89 had a problem deploying to tool October 30, 2025 05:43 — with GitHub Actions Error

sundarshankar89 requested review from asnare and gueniai October 30, 2025 05:44

Merge branch 'feature/switch-installer-integration' into feature/llm-…

394ad9d

…transpile

sundarshankar89 had a problem deploying to tool October 30, 2025 05:45 — with GitHub Actions Error

Merge branch 'feature/switch-installer-integration' into feature/llm-…

aeff475

…transpile

sundarshankar89 temporarily deployed to tool October 30, 2025 05:45 — with GitHub Actions Inactive

gueniai approved these changes Oct 30, 2025

View reviewed changes

gueniai reviewed Oct 30, 2025

View reviewed changes

added additional tests improved cuj

7a336f9

sundarshankar89 temporarily deployed to tool October 31, 2025 11:14 — with GitHub Actions Inactive

added user agent extra

2602577

sundarshankar89 temporarily deployed to tool October 31, 2025 13:53 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `llm-transpile` command with Switch runner and integration tests #2078

Add `llm-transpile` command with Switch runner and integration tests #2078

Uh oh!

hiroyukinakazato-db commented Oct 6, 2025

Uh oh!

github-actions bot commented Oct 6, 2025 •

edited

Loading

Uh oh!

sundarshankar89 left a comment

Uh oh!

sundarshankar89 Oct 8, 2025

Uh oh!

sundarshankar89 Oct 8, 2025

Uh oh!

Uh oh!

gueniai left a comment

Uh oh!

gueniai Oct 30, 2025

Uh oh!

sundarshankar89 Oct 31, 2025

Uh oh!

sundarshankar89 Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	ctx.add_user_agent_extra("cmd", "llm-transpile")
	ctx.add_user_agent_extra("cmd", "switch")

		from databricks.labs.lakebridge.transpiler.switch_runner import SwitchConfig, SwitchRunner


		class FriendOfSwitchRunner(SwitchRunner):

	class FriendOfSwitchRunner(SwitchRunner):
	class MockSwitchRunner(SwitchRunner):

Add llm-transpile command with Switch runner and integration tests #2078

Are you sure you want to change the base?

Add llm-transpile command with Switch runner and integration tests #2078

Uh oh!

Conversation

hiroyukinakazato-db commented Oct 6, 2025

Changes

What does this PR do?

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

Functionality

Tests

Uh oh!

github-actions bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

gueniai Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add `llm-transpile` command with Switch runner and integration tests #2078

Add `llm-transpile` command with Switch runner and integration tests #2078

github-actions bot commented Oct 6, 2025 •

edited

Loading