diff --git a/_data/migration-scenarios.yml b/_data/migration-scenarios.yml index 6e808345a7a..a87767451fc 100644 --- a/_data/migration-scenarios.yml +++ b/_data/migration-scenarios.yml @@ -107,3 +107,13 @@ scenarios: url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/reroute-traffic-from-capture-proxy-to-target/" - title: "Teardown" url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/remove-migration-infrastructure/" + + - title: "Scenario 4: Kubernetes experimental workflow-based backfill" + steps: + - title: "Workflow CLI overview" + url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-overview/" + substeps: + - title: "Getting started" + url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-getting-started/" + - title: "Backfill workflow" + url: "{{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill-workflow/" diff --git a/_migration-assistant/migration-console/workflow-cli-getting-started.md b/_migration-assistant/migration-console/workflow-cli-getting-started.md new file mode 100644 index 00000000000..3ac3cd5289a --- /dev/null +++ b/_migration-assistant/migration-console/workflow-cli-getting-started.md @@ -0,0 +1,463 @@ +--- +layout: default +title: Workflow CLI getting started +parent: Migration console +grand_parent: Migration Assistant for OpenSearch +nav_order: 11 +permalink: /migration-assistant/migration-console/workflow-cli-getting-started/ +--- + +# Getting started with Workflow CLI + +{: .warning } +> **Experimental Feature**: The Workflow CLI is an experimental feature for Kubernetes-based deployments (3.x release). For the stable ECS-based approach (2.x), see [Migration console commands]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/migration-console-commands-references/). + +This guide walks you through your first OpenSearch migration using the Workflow CLI. You'll learn how to configure, submit, and monitor a migration workflow. + +## Prerequisites + +Before starting, ensure: + +- You have access to the migration console on your Kubernetes cluster +- Argo Workflows is installed and running +- Source and target clusters are accessible +- You've reviewed the [Workflow CLI overview]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-overview/) + +## Step 1: Access the migration console + +Connect to the migration console pod: + +```bash +kubectl exec -it -n migration-assistant -- bash +``` +{% include copy.html %} + +Verify the `workflow` command is available: + +```bash +workflow --help +``` +{% include copy.html %} + +## Step 2: View a sample configuration + +Before creating your configuration, review a sample: + +```bash +workflow configure sample +``` +{% include copy.html %} + +This displays a template showing all available options: + +```yaml +sourceClusters: + : + endpoint: https://your-source-cluster:9200 + version: "7.10.2" + authConfig: + basic: + username: admin + password: password + +targetClusters: + : + endpoint: https://your-target-cluster:9200 + version: "2.11.0" + authConfig: + basic: + username: admin + password: password + +migrations: + - sourceCluster: + targetCluster: + snapshotMigrations: + - indices: ["index-pattern-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true +``` + +## Step 3: Load the sample as a starting point + +Load the sample configuration: + +```bash +workflow configure sample --load +``` +{% include copy.html %} + +You should see: + +``` +Sample configuration loaded successfully + +Use 'workflow configure view' to see it +Use 'workflow configure edit' to modify it +``` + +## Step 4: Edit the configuration + +Customize the configuration for your clusters: + +```bash +workflow configure edit +``` +{% include copy.html %} + +This opens your default editor. Update these sections: + +**Source cluster:** + +```yaml +sourceClusters: + my-es-source: + endpoint: https://source.example.com:9200 + version: "7.10.2" + authConfig: + basic: + username: admin + password: your-password +``` + +**Target cluster:** + +```yaml +targetClusters: + my-os-target: + endpoint: https://target.example.com:9200 + version: "2.11.0" + authConfig: + basic: + username: admin + password: your-password +``` + +**Migration configuration:** + +```yaml +migrations: + - sourceCluster: my-es-source + targetCluster: my-os-target + snapshotMigrations: + - indices: ["logs-*", "metrics-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true +``` + +Save and exit (in vi: `Esc`, then `:wq`, then `Enter`). + +## Step 5: Verify your configuration + +Check that your configuration was saved: + +```bash +workflow configure view +``` +{% include copy.html %} + +Review the output to ensure: +- Cluster endpoints are correct +- Credentials are accurate +- Index patterns match what you want to migrate +- Cluster names are referenced correctly + +## Step 6: Submit the workflow + +Submit your migration workflow: + +```bash +workflow submit +``` +{% include copy.html %} + +You'll see output like: + +``` +Initializing workflow from session: default +Workflow initialized with prefix: a1b2c3d4-e5f6-7890-abcd-ef1234567890 + +Submitting workflow to namespace: migration-assistant + +Workflow submitted successfully + Name: migration-xwz9r + Prefix: a1b2c3d4-e5f6-7890-abcd-ef1234567890 + Namespace: migration-assistant +``` + +**Important:** Note the workflow name (`migration-xwz9r` in this example) for monitoring. + +## Step 7: Monitor workflow progress + +Check the status of your workflow: + +```bash +workflow status +``` +{% include copy.html %} + +For detailed status of your specific workflow: + +```bash +workflow status migration-xwz9r +``` +{% include copy.html %} + +You'll see a tree view showing progress: + +``` +[*] Workflow: migration-xwz9r + Phase: Running + Started: 2024-01-15T10:30:00Z + +πŸ“‹ Workflow Steps +β”œβ”€β”€ βœ“ Initialize (Succeeded) +β”œβ”€β”€ β–Ά Create Snapshot (Running) +β”‚ β”œβ”€β”€ βœ“ Validate Indices (Succeeded) +β”‚ └── β–Ά Snapshot Creation (Running) +β”œβ”€β”€ β—‹ Register Snapshot (Pending) +β”œβ”€β”€ β—‹ Migrate Metadata (Pending) +└── β—‹ Restore Documents (Pending) +``` + +**Status symbols:** +- `βœ“` - Step completed successfully +- `β–Ά` - Step currently running +- `β—‹` - Step pending (waiting to run) +- `βœ—` - Step failed +- `⟳` - Step waiting for approval + +## Step 8: Wait for completion + +The workflow runs asynchronously. You can monitor it in several ways: + +**Option A:** Check manually + +```bash +workflow status migration-xwz9r +``` +{% include copy.html %} + +**Option B:** Auto-refresh with watch + +```bash +watch -n 10 workflow status migration-xwz9r +``` +{% include copy.html %} + +(Refreshes every 10 seconds. Press Ctrl+C to exit.) + +**Option C:** Submit with wait flag + +```bash +workflow submit --wait --timeout 3600 +``` +{% include copy.html %} + +## Step 9: Handle approvals (if required) + +If the workflow requires manual approval, you'll see: + +``` +β”œβ”€β”€ ⟳ Manual Approval Gate - WAITING FOR APPROVAL +``` + +Approve to continue: + +```bash +workflow approve migration-xwz9r +``` +{% include copy.html %} + +Then check status again: + +```bash +workflow status migration-xwz9r +``` +{% include copy.html %} + +## Step 10: Verify success + +When complete, you'll see: + +``` +[+] Workflow: migration-xwz9r + Phase: Succeeded + Started: 2024-01-15T10:30:00Z + Finished: 2024-01-15T11:45:00Z + +πŸ“‹ Workflow Steps +β”œβ”€β”€ βœ“ Initialize (Succeeded) +β”œβ”€β”€ βœ“ Create Snapshot (Succeeded) +β”œβ”€β”€ βœ“ Register Snapshot (Succeeded) +β”œβ”€β”€ βœ“ Migrate Metadata (Succeeded) +└── βœ“ Restore Documents (Succeeded) +``` + +Verify data in your target cluster to confirm the migration succeeded. + +## Common scenarios + +### Migrate specific indices only + +```yaml +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + - indices: ["user-data-*", "application-logs-2024-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true +``` + +### Metadata only (no documents) + +```yaml +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + - indices: ["*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: false +``` + +### Multiple index groups + +```yaml +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + # Critical data with metadata + - indices: ["orders-*", "customers-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true + + # Logs without metadata + - indices: ["logs-*"] + metadataMigration: + enabled: false + documentBackfill: + enabled: true +``` + +## Troubleshooting + +### Configuration won't save + +**Problem:** Editor closes but configuration isn't saved. + +**Solution:** +- Save the file before exiting (vi: `Esc` β†’ `:wq` β†’ `Enter`) +- Check for YAML syntax errors + +### Workflow fails to submit + +**Problem:** Error: "No workflow configuration found" + +**Solution:** + +```bash +# Check if configuration exists +workflow configure view + +# If empty, create one +workflow configure edit +``` +{% include copy.html %} + +### Workflow stuck in Pending + +**Problem:** All steps show "Pending" + +**Solution:** + +1. Check workflow templates are deployed: + + ```bash + kubectl get workflowtemplates -n migration-assistant + ``` + {% include copy.html %} + +2. Check pod status: + + ```bash + kubectl get pods -n migration-assistant + ``` + {% include copy.html %} + +3. View workflow logs: + + ```bash + kubectl logs -n migration-assistant -l workflows.argoproj.io/workflow=migration-xwz9r + ``` + {% include copy.html %} + +## Quick reference + +```bash +# Configuration commands +workflow configure sample --load # Start with sample +workflow configure edit # Edit configuration +workflow configure view # Show configuration +workflow configure clear # Clear configuration + +# Execution commands +workflow submit # Submit workflow +workflow submit --wait # Submit and wait for completion +workflow status # Show all workflows +workflow status # Show specific workflow + +# Management commands +workflow approve # Approve manual steps +workflow stop # Stop a running workflow +``` + +## Next steps + +Now that you've completed your first migration: + +1. Verify data in your target cluster +2. Learn about [Backfill workflows]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill-workflow/) for more details +3. Review the [Workflow CLI overview]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-overview/) for architectural details +4. Explore more complex scenarios and configurations + +## Getting help + +If you encounter issues: + +1. Check command help: + + ```bash + workflow --help + workflow configure --help + workflow submit --help + ``` + {% include copy.html %} + +2. Enable verbose logging: + + ```bash + workflow -vv status + ``` + {% include copy.html %} + +3. Check workflow logs: + + ```bash + kubectl logs -n migration-assistant -l workflows.argoproj.io/workflow= + ``` + {% include copy.html %} + +4. Provide feedback through the [opensearch-migrations repository](https://github.com/opensearch-project/opensearch-migrations) diff --git a/_migration-assistant/migration-console/workflow-cli-overview.md b/_migration-assistant/migration-console/workflow-cli-overview.md new file mode 100644 index 00000000000..2a92697aaec --- /dev/null +++ b/_migration-assistant/migration-console/workflow-cli-overview.md @@ -0,0 +1,253 @@ +--- +layout: default +title: Workflow CLI overview +parent: Migration console +grand_parent: Migration Assistant for OpenSearch +nav_order: 10 +permalink: /migration-assistant/migration-console/workflow-cli-overview/ +--- + +# Workflow CLI overview + +{: .warning } +> **Experimental Feature**: The Workflow CLI is an experimental feature for Kubernetes-based deployments (3.x release). For the stable ECS-based approach (2.x), see [Migration console commands]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/migration-console-commands-references/). + +The Workflow CLI provides a declarative, workflow-driven approach to managing OpenSearch migrations on Kubernetes. Instead of running individual commands, you define your migration configuration in YAML and submit it as a workflow that Argo Workflows orchestrates. + +## What is the Workflow CLI? + +The Workflow CLI is a command-line tool available in the migration console that lets you: + +- **Configure** migrations using declarative YAML +- **Submit** migrations as Argo Workflows +- **Monitor** workflow execution and progress +- **Manage** workflow lifecycle (approve, stop, etc.) + +The CLI abstracts the complexity of Kubernetes and Argo Workflows, providing a simple interface for migration tasks. + +## Key concepts + +### Declarative configuration + +Instead of running commands step-by-step, you declare what you want to migrate: + +```yaml +sourceClusters: + my-source: + endpoint: https://source-cluster:9200 + version: "7.10.2" + +targetClusters: + my-target: + endpoint: https://target-cluster:9200 + version: "2.11.0" + +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + - indices: ["logs-*", "metrics-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true +``` + +The system determines how to execute the migration based on your configuration. + +### Workflow templates + +Pre-built workflow templates handle common migration scenarios: + +- **Full migration**: Snapshots, metadata, and document backfill +- **Snapshot migration**: Create and restore snapshots +- **Metadata migration**: Migrate index templates and settings +- **Document bulk load**: Efficient document loading using RFS + +These templates are pre-deployed on your Kubernetes cluster and don't need to be managed individually. + +### Argo Workflows orchestration + +Argo Workflows provides: + +- **Parallel execution**: Multiple migration tasks run simultaneously +- **Retry logic**: Automatic retry on transient failures +- **Progress tracking**: Visual workflow status +- **Resource management**: Controlled pod scheduling + +You don't interact with Argo Workflows directlyβ€”the Workflow CLI handles this for you. + +### State management with etcd + +Workflow state and coordination data are stored in etcd: + +- **Configuration**: Runtime settings accessible to all workflow steps +- **Coordination**: Synchronization between parallel tasks +- **Status**: Shared state for monitoring progress + +This state is managed automatically; you don't need to interact with etcd directly. + +## How it works + +### 1. Configuration phase + +You create a migration configuration file using the `workflow configure` commands: + +```bash +# Load a sample configuration +workflow configure sample --load + +# Edit the configuration +workflow configure edit + +# View the configuration +workflow configure view +``` + +### 2. Submission phase + +Submit your configuration as a workflow: + +```bash +workflow submit +``` + +This command: +- Validates your configuration +- Transforms it to Argo Workflow format +- Initializes etcd state +- Submits to Argo Workflows +- Returns a workflow name for tracking + +### 3. Execution phase + +Argo Workflows executes the migration in stages: + +1. **Initialize**: Set up workflow environment +2. **Create snapshots**: Snapshot source cluster indices +3. **Register snapshots**: Make snapshots available to target +4. **Migrate metadata**: Transfer index templates and settings +5. **Load documents**: Bulk load documents from snapshots +6. **Cleanup**: Remove temporary state + +### 4. Monitoring phase + +Track progress with status commands: + +```bash +# View all workflows +workflow status + +# Check specific workflow +workflow status migration-abc123 + +# Approve manual steps if needed +workflow approve migration-abc123 +``` + +## Workflow CLI vs. console commands + +The Workflow CLI differs from traditional migration console commands: + +| Aspect | Workflow CLI | Console Commands | +|--------|--------------|------------------| +| Approach | Declarative | Imperative | +| Configuration | YAML file | Command parameters | +| Orchestration | Argo Workflows | Manual | +| Parallelism | Automatic | Manual coordination | +| Retry | Built-in | Manual | +| State | etcd | In-memory | +| Platform | Kubernetes | ECS | + +## When to use Workflow CLI + +Consider using the Workflow CLI if you: + +- Want declarative migration configuration +- Need automatic retry and error handling +- Prefer workflow orchestration over manual coordination +- Are deploying on Kubernetes +- Have complex migrations with multiple index groups + +For simpler migrations or ECS deployments, traditional console commands may be more appropriate. + +## Architecture + +The Workflow CLI system architecture: + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ User Configuration (YAML) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Workflow CLI β”‚ +β”‚ - Validation β”‚ +β”‚ - Transformation β”‚ +β”‚ - Submission β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Argo Workflows β”‚ +β”‚ - Orchestration β”‚ +β”‚ - Parallelism β”‚ +β”‚ - Retry logic β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Migration Execution β”‚ +β”‚ - Snapshot operations β”‚ +β”‚ - Metadata migration β”‚ +β”‚ - Document loading β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## Benefits + +### Declarative configuration + +- Define what to migrate, not how +- Configuration is version-controlled +- Easy to review and audit +- Repeatable migrations + +### Workflow orchestration + +- Automatic parallelism +- Built-in retry logic +- State management +- Progress tracking + +### Kubernetes-native + +- Leverages Kubernetes scheduling +- Resource limits and requests +- Integration with Kubernetes ecosystem +- Scalable execution + +## Limitations + +{: .warning } +The Workflow CLI has some limitations in this experimental release: + +- Limited to backfill migrations (no live capture/replay) +- Requires Kubernetes and Argo Workflows expertise +- Less mature than console commands approach +- Fewer examples and documentation +- API may change in future releases + +## Next steps + +1. Follow the [Getting started guide]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-getting-started/) for your first workflow +2. Learn about [Backfill workflows]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill-workflow/) + +## Related documentation + +- [Traditional console commands]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/migration-console-commands-references/) - ECS-based approach + +## Feedback + +The Workflow CLI is experimental. If you encounter issues or have suggestions, provide feedback through the [opensearch-migrations repository](https://github.com/opensearch-project/opensearch-migrations). diff --git a/_migration-assistant/migration-phases/backfill-workflow.md b/_migration-assistant/migration-phases/backfill-workflow.md new file mode 100644 index 00000000000..ed5d5661800 --- /dev/null +++ b/_migration-assistant/migration-phases/backfill-workflow.md @@ -0,0 +1,426 @@ +--- +layout: default +title: Backfill workflow +parent: Migration phases +grand_parent: Migration Assistant for OpenSearch +nav_order: 50 +permalink: /migration-assistant/migration-phases/backfill-workflow/ +--- + +# Backfill workflow + +{: .warning } +> **Experimental Feature**: This workflow-based backfill approach is experimental and intended for the 3.x Kubernetes-based release. For the stable 2.x approach, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/). + +This page describes how to perform document backfill using the workflow CLI on Kubernetes. The workflow-based approach provides declarative configuration, automatic retry handling, and progress monitoring through Argo Workflows. + +## Overview + +The backfill workflow migrates documents from your source cluster to your target cluster using snapshot-based reindexing (RFS). Unlike the traditional approach where you run commands manually, the workflow approach uses Argo Workflows to orchestrate the entire process. + +### What the workflow does + +The backfill workflow performs these steps automatically: + +1. **Create snapshot**: Creates a snapshot of specified indices on the source cluster +2. **Register snapshot**: Makes the snapshot available to the target cluster +3. **Migrate metadata** (optional): Transfers index templates and settings +4. **Load documents**: Reindexes documents from the snapshot to the target cluster +5. **Cleanup**: Removes temporary state and coordination data + +## Prerequisites + +Before running a backfill workflow: + +- Have access to the migration console on your Kubernetes cluster +- Have Argo Workflows installed and running +- Ensure source and target clusters are accessible +- Have an S3 bucket configured for snapshots (if not using existing snapshots) + +## Configuring a backfill workflow + +### Basic configuration + +A minimal backfill configuration includes: + +```yaml +sourceClusters: + my-source: + endpoint: https://source-cluster:9200 + version: "7.10.2" + authConfig: + basic: + username: admin + password: password + +targetClusters: + my-target: + endpoint: https://target-cluster:9200 + version: "2.11.0" + authConfig: + basic: + username: admin + password: password + +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + - indices: ["*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true +``` + +### Configuration options + +#### Index selection + +Specify which indices to migrate using patterns: + +```yaml +snapshotMigrations: + # All indices + - indices: ["*"] + + # Specific patterns + - indices: ["logs-*", "metrics-*"] + + # Multiple specific indices + - indices: ["users", "orders", "products"] +``` + +#### Metadata migration + +Control whether to migrate index templates and settings: + +```yaml +metadataMigration: + enabled: true # Migrate templates, mappings, settings + # or + enabled: false # Skip metadata, documents only +``` + +#### Document backfill + +Enable or disable document migration: + +```yaml +documentBackfill: + enabled: true # Migrate documents + # or + enabled: false # Skip documents (metadata only) +``` + +#### S3 snapshot configuration + +If using S3 for snapshots: + +```yaml +sourceClusters: + my-source: + endpoint: https://source-cluster:9200 + version: "7.10.2" + snapshotRepo: + s3RepoPathUri: s3://my-bucket/snapshots + aws_region: us-east-1 +``` + +## Running a backfill workflow + +### Step 1: Create configuration + +Access the migration console and create your configuration: + +```bash +# Load sample and customize +workflow configure sample --load +workflow configure edit +``` +{% include copy.html %} + +### Step 2: Submit workflow + +Submit the workflow to Argo: + +```bash +workflow submit +``` +{% include copy.html %} + +Note the workflow name from the output for monitoring. + +### Step 3: Monitor progress + +Check workflow status: + +```bash +workflow status +``` +{% include copy.html %} + +Example output: + +``` +[*] Workflow: migration-abc123 + Phase: Running + Started: 2024-01-15T10:30:00Z + +πŸ“‹ Workflow Steps +β”œβ”€β”€ βœ“ Initialize (Succeeded) +β”œβ”€β”€ βœ“ Create Snapshot (Succeeded) +β”œβ”€β”€ βœ“ Register Snapshot (Succeeded) +β”œβ”€β”€ β–Ά Migrate Metadata (Running) +β”œβ”€β”€ β—‹ Restore Documents (Pending) +└── β—‹ Cleanup (Pending) +``` + +### Step 4: Wait for completion + +The workflow runs asynchronously. Monitor until all steps succeed: + +```bash +watch -n 10 workflow status +``` +{% include copy.html %} + +Or submit with the `--wait` flag to block until completion: + +```bash +workflow submit --wait --timeout 3600 +``` +{% include copy.html %} + +## Advanced scenarios + +### Multiple index groups + +Migrate different sets of indices with different configurations: + +```yaml +migrations: + - sourceCluster: my-source + targetCluster: my-target + snapshotMigrations: + # Critical data: full migration + - indices: ["orders-*", "customers-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: true + + # Historical logs: documents only + - indices: ["logs-2023-*"] + metadataMigration: + enabled: false + documentBackfill: + enabled: true + + # Templates only: no documents + - indices: ["templates-*"] + metadataMigration: + enabled: true + documentBackfill: + enabled: false +``` + +### Parallel migrations + +Migrate from multiple sources simultaneously: + +```yaml +sourceClusters: + source-1: + endpoint: https://source1:9200 + source-2: + endpoint: https://source2:9200 + +targetClusters: + target: + endpoint: https://target:9200 + +migrations: + - sourceCluster: source-1 + targetCluster: target + snapshotMigrations: + - indices: ["source1-*"] + + - sourceCluster: source-2 + targetCluster: target + snapshotMigrations: + - indices: ["source2-*"] +``` + +The workflow engine executes these migrations in parallel where possible. + +### Using existing snapshots + +If you have existing snapshots, configure the snapshot repository: + +```yaml +sourceClusters: + my-source: + endpoint: https://source:9200 + snapshotRepo: + s3RepoPathUri: s3://existing-bucket/existing-snapshots + aws_region: us-west-2 +``` + +## Monitoring and troubleshooting + +### Check workflow status + +View all running workflows: + +```bash +workflow status +``` +{% include copy.html %} + +View specific workflow details: + +```bash +workflow status +``` +{% include copy.html %} + +### View workflow logs + +Check logs for a specific workflow: + +```bash +kubectl logs -n migration-assistant -l workflows.argoproj.io/workflow= +``` +{% include copy.html %} + +### Common issues + +#### Workflow stuck in Pending + +**Problem:** Workflow shows all steps as Pending. + +**Solutions:** +- Check if workflow templates are deployed: `kubectl get workflowtemplates -n migration-assistant` +- Verify Argo Workflows is running: `kubectl get pods -n argo` +- Check resource availability: `kubectl describe pod -n migration-assistant` + +#### Snapshot creation failed + +**Problem:** Snapshot creation step fails. + +**Solutions:** +- Verify S3 bucket permissions +- Check source cluster has snapshot repository configured +- Ensure indices exist and are accessible +- Review snapshot step logs for specific errors + +#### Document loading slow or stalled + +**Problem:** Document restore is taking too long or appears stuck. + +**Solutions:** +- Check target cluster health and capacity +- Monitor shard allocation: `curl http://target:9200/_cat/shards?v` +- Verify network connectivity between clusters +- Check if target cluster has sufficient disk space + +#### Metadata migration errors + +**Problem:** Metadata migration fails with compatibility errors. + +**Solutions:** +- Review breaking changes between source and target versions +- Check for unsupported field types +- See [Migrate metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/) for handling breaking changes + +### Stopping a workflow + +If you need to stop a running workflow: + +```bash +workflow stop +``` +{% include copy.html %} + +This gracefully terminates the workflow and cleans up resources. + +## Performance considerations + +### Parallelism + +The workflow engine supports parallel execution of independent tasks. Configure parallelism based on your cluster capacity: + +- Default: Up to 100 concurrent pods +- Limited by Kubernetes cluster resources +- Balanced against source and target cluster capacity + +### Resource limits + +Workflow pods use default resource limits. For large migrations, you may need to adjust: + +- CPU and memory requests/limits +- Storage for temporary data +- Network bandwidth + +### Migration duration + +Factors affecting migration time: + +- Total data volume +- Number of indices and shards +- Source cluster load +- Target cluster capacity +- Network bandwidth between clusters +- S3 snapshot transfer speeds + +## Verification + +After the workflow completes: + +1. **Check workflow status**: + + ```bash + workflow status + ``` + {% include copy.html %} + +2. **Verify document counts**: + + ```bash + # Source cluster + curl http://source:9200/_cat/indices?v + + # Target cluster + curl http://target:9200/_cat/indices?v + ``` + {% include copy.html %} + +3. **Compare index settings**: + + ```bash + # Check target cluster indices + curl http://target:9200//_settings + ``` + {% include copy.html %} + +4. **Test queries** on the target cluster to ensure data is accessible + +## Next steps + +After completing the backfill workflow: + +1. Verify all data migrated successfully +2. Test application connectivity to the target cluster + +## Related documentation + +- [Workflow CLI overview]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-overview/) - Learn about the workflow approach +- [Getting started with Workflow CLI]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/workflow-cli-getting-started/) - Step-by-step tutorial +- [Traditional backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/) - ECS-based approach + +## Feedback + +This is an experimental feature. Provide feedback through the [opensearch-migrations repository](https://github.com/opensearch-project/opensearch-migrations). + +{% include migration-phase-navigation.html %}