Skip to content

[Fleet] Update "Remote Elasticsearch output” #2048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

vishaangelova
Copy link
Contributor

@vishaangelova vishaangelova commented Jul 5, 2025

This PR is a review/update of the documentation created for [Fleet] Multi Cluster Fleet support - Phase 1: Global Visibility and Control, Local Data Plane, and updates the Remote Elasticsearch output doc with the following:

  • Updated the doc frontmatter to include applies_to
  • Removed references to specific subscriptions, added a reworded note in the “Automatic integrations synchronization” section
  • Clarified the remote address required for setting up a remote cluster
  • Added links to deploy-manage guides for setting up remote clusters and cross-cluster replication in the relevant sections
  • Restructured and updated the steps in the document for clarity and a more logical workflow
  • Added info about the available remote cluster data views in a new section
  • Added a new troubleshooting section

Closes elastic/ingest-docs#1817

Preview

reference/fleet/remote-elasticsearch-output.md

Copy link

github-actions bot commented Jul 5, 2025

🔍 Preview links for changed docs

applies_to:
stack: ga
deployment:
eck: ga
Copy link
Contributor Author

@vishaangelova vishaangelova Jul 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What deployments is this document valid for? Should the focus be on ECH, ECE and self-managed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think ECH, ECE and self-managed sounds good.


To configure a remote {{es}} cluster for your {{agent}} data:

1. In {{fleet}}, open the **Settings** tab.
1. In your main {{es}} cluster (Cluster A), open {{kib}}, and search for **Fleet settings** in the search bar. Select **Fleet/Settings** in the results.
Copy link
Contributor Author

@vishaangelova vishaangelova Jul 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we refer to the main cluster as the “management cluster" instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think we can use "management cluster" here.


::::{dropdown} Find the remote host address of the remote cluster
:open:
1. In the remote cluster (Cluster B), open {{kib}}, and search for **Fleet settings** in the search bar. Select **Fleet/Settings** in the results.
Copy link
Contributor Author

@vishaangelova vishaangelova Jul 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using Cluster A and Cluster B to designate the main and remote clusters, respectively, could help users identify more easily the cluster where the step in question takes place. I’d appreciate any feedback on whether this is useful, or if it makes the doc more cumbersome to read.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually like this, but I can see the argument for cumbersome. I'd like to phone a friend. @karenzone what do you think of this approach?

@vishaangelova vishaangelova force-pushed the 1817-cross-cluster-integration-updates branch from a88458e to a21b40e Compare July 5, 2025 21:47
@vishaangelova vishaangelova marked this pull request as ready for review July 5, 2025 21:54
6. Choose whether integrations should be automatically synchronized on the remote {{es}} cluster (Cluster B). To configure this feature, refer to the [Automatic integrations synchronization](#automatic-integrations-synchronization) section.

::::{note}
This feature is only available with certain subscriptions. For more information, check [Subscriptions](https://www.elastic.co/subscriptions).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention that minimum Enterprise license is required on both clusters? The subscriptions doc is not yet updated with this feature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For legal reasons we can no longer call out specific subscription details in the docs. Do you know when this will be added to the subscriptions doc?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, do you know where to update the subscriptions doc? Can we do that directly?

Copy link
Member

@bmorelli25 bmorelli25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvements. A few comments.

GET fleet-synced-integrations-ccr-<output_name>/_search
```

The response should match the the contents of the leader index on the main cluster.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The response should match the the contents of the leader index on the main cluster.
The response should match the contents of the leader index on the main cluster.

- `<remote_cluster>:logs-*`
- `<remote_cluster>:metrics-*`

To display these data views, open {{kib}} in your main {{es}} cluster, then go to **Management** -> **Stack management** → **Data Views**.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To display these data views, open {{kib}} in your main {{es}} cluster, then go to **Management** -> **Stack management****Data Views**.
To display these data views, open {{kib}} in your main {{es}} cluster, then go to **Management** **Stack management****Data Views**.

4. Choose whether uninstalled integrations should also be uninstalled on the remote cluster.
5. In the remote output configuration on the main cluster (Cluster A), add the {{kib}} URL of the remote cluster (Cluster B) in the **Remote Kibana URL** field.
6. In the **Remote Kibana API Key** field, add an API key to access Kibana on the remote cluster (Cluster B).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key should be lowercase in this sentence and later ones

When you use a remote {{es}} output, {{fleet-server}} performs a test to ensure connectivity to the remote cluster. The result of that connectivity test is used to report the ES Remote output as healthy or unhealthy on the **Fleet** > **Settings** > **Outputs** page, under the **Status** column. In some cases, the remote {{es}} output used for data from {{agent}} may be reachable only by those agents and not by {{fleet-server}}, so the unhealthy state and an associated `Unable to connect` error that appears on the UI can be ignored.
When you use a remote {{es}} output, {{fleet-server}} performs a test to ensure connectivity to the remote cluster. The result of that connectivity test is used to report whether the remote output is healthy or unhealthy, and is displayed on the **{{fleet}}** → **Settings** → **Outputs** page, in the **Status** column.

In some cases, the remote {{es}} output used for {{agent}} data can be reached by the {{agent}}s but not by {{fleet-server}}. In those cases, you can ignore the resulting unhealthy state of the output and the associated `Unable to connect` error on the UI.
::::

## Automatic integrations synchronization
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is an optional feature with its own set of prereq's (like CCR). I wonder if this should be moved to its own page. It's 2/3 of the content on this page as is.


## Configuration
## Configuration [remote-output-config]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could consider using a stepper here to make the flow stand out more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[REQUEST]: Document Multi Cluster Fleet support - Phase 1
3 participants