vipdatasync

CLI utility for WordPress VIP data sync management.

Usage

Validate

To validate the environment-specific YAML config file against production URLs:

1. Download the environment-specific YAML config file from the master branches of wpcomvip repos
2. Export the production URLs via:

   vip @my-app.produciton -- wp site list --fields=url --format=json

3. Run the validate command:

   vipdatasync validate --config=config.yml --urls=sites.json

Example: Bash Script

gh api \
  -H "Accept: application/vnd.github.raw+yaml" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  /repos/wpcomvip/my-app/contents/config/.vip.my-app.staging.yml\?ref=preprod \
  > ~/Desktop/.vip.my-app.staging.yml

vip @my-app.production --yes -- wp site list --fields=url --format=json > ~/Desktop/my-app-production-sites.json

vipdatasync validate --config=~/Desktop/.vip.my-app.staging.yml --urls=~/Desktop/my-app-production-sites.json

Duplicated Destinations

Multiple domain map items to the same destination.

Important

Special case: It is considered valid if both www and non-www sources are mapped to the same destination where the www item comes first.

Duplication

data_sync:
  domain_map:
    vip-example.com: example-com-staging.go-vip.net
    example.go-vip.net: example-com-staging.go-vip.net

$ vipdatasync validate

DUPLICATED DESTINATIONS

     1. example-com-staging.go-vip.net
         - vip-example.com
         - example.go-vip.net

See validate_duplicated_destinations.txtar.

Special case: www and non-www with good ordering

data_sync:
  domain_map:
    www.vip-example.com: example-com-staging.go-vip.net
    vip-example.com: example-com-staging.go-vip.net

$ vipdatasync validate

DUPLICATED DESTINATIONS

        No problems found

See validate_duplicated_destinations_www_ordering_good.txtar.

Exact duplication

data_sync:
  domain_map:
    vip-example.com: example-com-staging.go-vip.net
    vip-example.com: example-com-staging.go-vip.net

$ vipdatasync validate

DUPLICATED DESTINATIONS

     1. example-com-staging.go-vip.net
         - vip-example.com
         - vip-example.com

See validate_duplicated_destinations_exactly.txtar.

www and non-www with extra items

data_sync:
  domain_map:
    www.vip-example.com: example-com-staging.go-vip.net
    vip-example.com: example-com-staging.go-vip.net
    foo.com: example-com-staging.go-vip.net

$ vipdatasync validate

DUPLICATED DESTINATIONS

     1. example-com-staging.go-vip.net
         - www.vip-example.com
         - vip-example.com
         - foo.com

See validate_duplicated_destinations_www_extra.txtar.

www and non-www with bad ordering

data_sync:
  domain_map:
    vip-example.com: example-com-staging.go-vip.net
    www.vip-example.com: example-com-staging.go-vip.net

$ vipdatasync validate

DUPLICATED DESTINATIONS

     1. example-com-staging.go-vip.net
         - vip-example.com
         - www.vip-example.com

See validate_duplicated_destinations_www_ordering_bad.txtar.

Duplicated Catch-All Destinations

Note

It is safe to ignore catch-all items which map both convenience and production domains to the same destination.

The command reports duplicated destination problems for catch-all items. There is no plan to fix this false-alarm because of the complexity to reliably identify the main production domain.

Example: Catch-all

This is safe to ignore.

data_sync:
  domain_map:
    vip-example.com: example-com-staging.go-vip.net
    vip-example.go-vip.net: example-com-staging.go-vip.net

# This is safe to ignore

$ vipdatasync validate

DUPLICATED DESTINATIONS

     1. example-com-staging.go-vip.net
         - vip-example.com
         - vip-example.go-vip.net

Duplicated Tos

Multiple production URLs map to the same environment-specific URL.

Duplication

data_sync:
   domain_map:
      vip-example.com: example-com-staging.go-vip.net
      example.go-vip.net/zh: example-com-staging.go-vip.net/en

[
  {
     "url": "https://vip-example.com/en"
  },
  {
     "url": "https://example.go-vip.net/zh"
  }
]

$ vipdatasync validate

DUPLICATED TOS

     1. https://example-com-staging.go-vip.net/en
         - https://vip-example.com/en
           vip-example.com
             -> example-com-staging.go-vip.net
         - https://example.go-vip.net/zh
           example.go-vip.net/zh
             -> example-com-staging.go-vip.net/en

See validate_duplicated_tos.txtar.

Unreplaced

data_sync:
   domain_map:
      vip-example.com: example-com-staging.go-vip.net

[
   {
      "url": "https://vip-example.com/en"
   },
   {
      "url": "https://example-com-staging.go-vip.net/en"
   }
]

$ vipdatasync validate

DUPLICATED TOS

     1. https://example-com-staging.go-vip.net/en
         - https://vip-example.com/en
           vip-example.com
             -> example-com-staging.go-vip.net
         - https://example-com-staging.go-vip.net/en

See validate_duplicated_tos_unreplaced.txtar.

Unreplaced URLs

Production URLs that are not replaced by any domain map item.

Unreplaced

data_sync:
   domain_map:
      vip-example.com: example-com-staging.go-vip.net
      example.go-vip.net/zh: example-com-staging.go-vip.net/zh

[
   {
      "url": "https://example.go-vip.net"
   },
   {
      "url": "https://vip-example.com/en"
   },
   {
      "url": "https://example.go-vip.net/en"
   },
   {
      "url": "https://example.go-vip.net/zh"
   }
]

$ vipdatasync validate

UNREPLACED URLS

     1. https://example.go-vip.net
     2. https://example.go-vip.net/en

See validate_duplicated_tos_unreplaced.txtar.

Unused Domain Map Items

Domain map items that are not used by any production URLs.

Unused

data_sync:
   domain_map:
      foo.com/zh: foo-com-staging.go-vip.net/zh
      foo.com: foo-com-staging.go-vip.net

      bar.com/zh: bar-com-staging.go-vip.net/zh
      bar.com/fr: bar-com-staging.go-vip.net/fr
      bar.com: bar-com-staging.go-vip.net

      vip-example.com: example-com-staging.go-vip.net

[
   {
      "url": "https://vip-example.com"
   },
   {
      "url": "https://vip-example.com/en"
   },
   {
      "url": "https://foo.com/zh"
   },
   {
      "url": "https://bar.com/en"
   },
   {
      "url": "https://bar.com/zh"
   },
   {
      "url": "https://bar.com"
   }
]

$ vipdatasync validate

UNUSED DOMAIN MAP ITEMS

     1. foo.com
          -> foo-com-staging.go-vip.net
     2. bar.com/fr
          -> bar-com-staging.go-vip.net/fr

See validate_unused_domain_map_items.txtar.

Bad ordering

data_sync:
   domain_map:
      foo.com: foo-com-staging.go-vip.net
      foo.com/zh: foo-com-staging.go-vip.net/zh

      bar.com: bar-com-staging.go-vip.net
      bar.com/fr: bar-com-staging.go-vip.net/fr
      bar.com/zh: bar-com-staging.go-vip.net/zh

      vip-example.com: example-com-staging.go-vip.net

[
   {
      "url": "https://vip-example.com"
   },
   {
      "url": "https://vip-example.com/en"
   },
   {
      "url": "https://foo.com/zh"
   },
   {
      "url": "https://bar.com/en"
   },
   {
      "url": "https://bar.com/zh"
   },
   {
      "url": "https://bar.com"
   }
]

$ vipdatasync validate

UNUSED DOMAIN MAP ITEMS

     1. foo.com/zh
          -> foo-com-staging.go-vip.net/zh
     2. bar.com/fr
          -> bar-com-staging.go-vip.net/fr
     3. bar.com/zh
          -> bar-com-staging.go-vip.net/zh

See validate_unused_domain_map_items_bad_ordering.txtar.

Install

Prebuilt Binaries (Preferred)

Tip

Find your operating system and hardware architecture via:

$ uname -om
Darwin arm64

Prebuilt binaries are available for a variety of operating systems and architectures. Visit the latest release page, and scroll down to the Assets section.

1. Download the archive for the desired edition, operating system, and architecture
2. Extract the archive
3. (Optional) Verify the integrity and provenance of the executable

   $ gh attestation verify /path/to/vipdatasync_999.888.777_xxx_yyy/vipdatasync --repo typisttech/vipdatasync --signer-repo typisttech/vipdatasync
   
   Loaded digest sha256:xxxxxxxxxxxxxxxxxx for file:///path/to/vipdatasync_999.888.777_xxx_yyy/vipdatasync
   Loaded 1 attestation from GitHub API
   ✓ Verification succeeded!
   
   sha256:xxxxxxxxxxxxxxxxxx was attested by:
   REPO                    PREDICATE_TYPE                  WORKFLOW
   typisttech/vipdatasync  https://slsa.dev/provenance/v1  .github/workflows/go-release.yml@refs/tags/v999.888.777

4. Move the executable to the desired directory
5. Add this directory to the PATH environment variable
6. Verify that you have execute permission on the file

Please consult your operating system documentation if you need help setting file permissions or modifying your PATH environment variable.

If you do not see a prebuilt binary for the desired edition, operating system, and architecture, install using one of the methods described below.

Install from Source

Tip

See $ go help install or https://golang.org/ref/mod#go-install for details.

go install github.com/typisttech/vipdatasync/cmd/vipdatasync@latest

Develop

# Run the tests
go test -v -count=1 -race -shuffle=on ./...

# Update testscript golden files
# Note: This doesn't update the README.md examples
UPDATE_SCRIPTS=true go test ./cmd/...

# Lint
golangci-lint run
goreleaser check