Skip to content

[aws] Ingest managed insights from Security Hub #14354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

chrisberkhout
Copy link
Contributor

@chrisberkhout chrisberkhout commented Jun 27, 2025

Proposed commit message

[aws] Ingest managed insights from Security Hub

The AWS Security Hub API's GetInsights endpoint[1] returns only custom
insights. Managed insights[2] are built-in, and can only be retrieved
by ARN.

Fetch a hard-coded list of managed insights first, then fetch pages of
custom insights as before.

This has been tested against the live API.

[1]: https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetInsights.html
[2]: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-managed-insights.html

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

@chrisberkhout chrisberkhout self-assigned this Jun 27, 2025
@chrisberkhout chrisberkhout requested a review from a team as a code owner June 27, 2025 16:50
@chrisberkhout chrisberkhout added enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jun 27, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@chrisberkhout chrisberkhout requested review from a team as code owners June 27, 2025 16:52
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jun 27, 2025

🚀 Benchmarks report

Package aws 👍(13) 💚(7) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
guardduty 1061.57 887.31 -174.26 (-16.42%) 💔
inspector 2012.07 1492.54 -519.53 (-25.82%) 💔

To see the full report comment with /test benchmark fullreport

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit message, s/retieved/retrieved/.

Comment on lines 81 to 84
# This transform's job is to fail and end pagination when finished.
# It will set a value if the last request was the first in the sequence
# (which collects the managed insights), or if there is another page of
# custom insights to fetch.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's enough HTTPJSON magic here that I think the comment needs to explain more about how it is actually working.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a sentence. Here's the full new comment:

      # This transform's job is to fail and end pagination when finished.
      # It will set a value if the last request was the first in the sequence
      # (which collects the managed insights), or if there is another page of
      # custom insights to fetch. Otherwise, there will be a value template
      # error (.last_response.body.NextToken doesn't exist) and the pagination
      # sequence will end (because fail_on_template_error is set).

@chrisberkhout chrisberkhout force-pushed the aws-securityhub_insights-managed-insights branch from ea0efa3 to 0444f3e Compare June 30, 2025 08:43
@chrisberkhout chrisberkhout requested a review from efd6 June 30, 2025 08:44
Copy link

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @chrisberkhout

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@chrisberkhout chrisberkhout merged commit 08dd8a2 into elastic:main Jul 1, 2025
7 checks passed
@elastic-vault-github-plugin-prod

Package aws - 3.9.0 containing this change is available at https://epr.elastic.co/package/aws/3.9.0/

robester0403 pushed a commit to robester0403/integrations that referenced this pull request Jul 8, 2025
The AWS Security Hub API's GetInsights endpoint[1] returns only custom
insights. Managed insights[2] are built-in, and can only be retrieved
by ARN.

Fetch a hard-coded list of managed insights first, then fetch pages of
custom insights as before.

This has been tested against the live API.

[1]: https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetInsights.html
[2]: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-managed-insights.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[aws] Security Hub integration to ingest managed insights
3 participants