[aws_billing] Add new aws_billing integration

[gpop63]

Summary

This PR introduces a new integration for ingesting AWS Cost and Usage Report (CUR) 2.0 data from S3 buckets. The integration uses S3 polling with a minimum 24-hour interval (default) to avoid duplicate ingestion, as AWS CUR files are cumulative and updated multiple times daily.

The integration includes an hourly ES transform that groups billing data by key dimensions (service, account, usage type, region, tags) and aggregates unblended costs. This transform deduplicates the raw CUR data by creating uniquely identifiable data points.

For future upgrades, the transform configuration will require bumping both the fleet_transform_version and the dest.index. The move_on_creation: true setting is required as it ensures the destination index becomes the only member of the alias when created, automatically removing previous indices from the alias. This allows dashboards to continue displaying the data correctly.

I tested an actual upgrade by adding a new field to the transform's group by, then bumping the fleet_transform_version and dest.index. On upgrade, a new index aws_billing.billing_latest-v2 was created and got the aws_billing.billing_latest alias. The old index aws_billing.billing_latest-v1 lost the alias. The dashboards kept working fine because they use the alias, not the specific index names.

The integration uses defaults specific for the AWS CUR usecase:

• bucket_list_interval is set to 24h to avoid duplicate data
• content_type defaults to text/csv, the standard CUR format
• file_selectors defaults to \.csv\.gz$ (this doesn't work as a default yet)

Removed Configuration Options for aws-s3

I removed some config options that I thought are not relevant or applicable to this integration.

Option
expand_event_list_from_field
include_s3_metadata
max_number_of_messages
queue_url
sqs_max_receive_count
sqs_wait_time
visibility_timeout

Minor inconveniences

• Transform name starts with logs-* for some reason, this doesn't affect the functionality
• Unable to set file_selectors default value to \.csv\.gz$

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates https://github.com/elastic/obs-integration-team/issues/202
• Closes https://github.com/elastic/obs-integration-team/issues/369

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[mykola-elastic]

is it not meant to be supported on ^9.0.0?

[gpop63]

the functionality we need is already available in version 8.18.0 (probably even in older versions), so unless there are significant bug fixes in the filebeat awss3 input or other stack features in general, there's no strong justification for bumping the version 🤔

we could change it to ^8.16.5 || ^9.0.0"

[gpop63]

Dashboards updated @agithomas @daniela-elastic

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2f9109c

History

• 💔 Build #28000 failed 85dafba
• 💚 Build #27744 succeeded 00bf033
• 💚 Build #27720 succeeded 2f71d1b
• 💔 Build #27719 failed 2ebe919
• 💔 Build #27718 failed 6651b5e
• 💚 Build #27650 succeeded a8077d3

cc @gpop63

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube