Skip to content

Commit 7189d7b

Browse files
maxcoldkcreddy
andauthored
Add new AWS Security Hub Findings Full Posture data stream and update misconfig transform to use it (#13372)
* implement full aws security hub posture data stream * build readme * update transform * add full posture data stream to security hub bundle * adjust transform event.dataset mapping * add new securityhub_findings_full_posture data stream to codeowners * fix pipeline test * fix typo Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]> * add pr link to changelog Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]> * update maxResults to 500 to ingest data faster * build docs * revert MaxResults to 100 * revert changes outside of security hub integration * do not filter out resolved findings for AWS Security Hub Full Posture data stream --------- Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>
1 parent 3e63dee commit 7189d7b

File tree

27 files changed

+8071
-15
lines changed

27 files changed

+8071
-15
lines changed

.github/CODEOWNERS

+1
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@
5858
/packages/aws/data_stream/s3_storage_lens @elastic/obs-infraobs-integrations
5959
/packages/aws/data_stream/s3access @elastic/obs-ds-hosted-services
6060
/packages/aws/data_stream/securityhub_findings @elastic/security-service-integrations
61+
/packages/aws/data_stream/securityhub_findings_full_posture @elastic/security-service-integrations
6162
/packages/aws/data_stream/securityhub_insights @elastic/security-service-integrations
6263
/packages/aws/data_stream/sns @elastic/obs-infraobs-integrations
6364
/packages/aws/data_stream/sqs @elastic/obs-infraobs-integrations

packages/aws/_dev/build/docs/securityhub.md

+13
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ The [AWS Security Hub](https://docs.aws.amazon.com/securityhub/) integration col
2222

2323
1. For the current integration package, it is recommended to have interval in hours.
2424
2. For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
25+
3. Findings Full Posture data stream request all the historical findings every 24 hours.
2526

2627
## Logs
2728

@@ -37,6 +38,18 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
3738

3839
{{fields "securityhub_findings"}}
3940

41+
### Findings Full Posture
42+
43+
This is the [`securityhub_findings_full_posture`](https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetFindings.html#API_GetFindings_ResponseElements) data stream.
44+
45+
{{event "securityhub_findings_full_posture"}}
46+
47+
**ECS Field Reference**
48+
49+
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
50+
51+
{{fields "securityhub_findings_full_posture"}}
52+
4053
### Insights
4154

4255
This is the [`securityhub_insights`](https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetInsights.html#API_GetInsights_ResponseElements) data stream.

packages/aws/changelog.yml

+12-7
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "3.0.0"
3+
changes:
4+
- description: Add new Security Hub Findings Full Posture data stream. If you rely on Findings > Misconfigurations view, enable this new data stream.
5+
type: breaking-change
6+
link: https://github.com/elastic/integrations/pull/13372
27
- version: "2.45.2"
38
changes:
49
- description: Update grok pattern for AWS S3 access ingest pipeline
@@ -11,14 +16,14 @@
1116
link: https://github.com/elastic/integrations/pull/13350
1217
- version: "2.45.0"
1318
changes:
14-
- description: Update default data_stream.dataset to aws.cloudwatch_logs for cloudwatch_logs data stream.
15-
type: breaking-change
16-
link: https://github.com/elastic/integrations/pull/13370
19+
- description: Update default data_stream.dataset to aws.cloudwatch_logs for cloudwatch_logs data stream.
20+
type: breaking-change
21+
link: https://github.com/elastic/integrations/pull/13370
1722
- version: "2.44.0"
1823
changes:
19-
- description: Add `actor.entity.id` and `target.entity.id`
20-
type: enhancement
21-
link: https://github.com/elastic/integrations/pull/12685
24+
- description: Add `actor.entity.id` and `target.entity.id`
25+
type: enhancement
26+
link: https://github.com/elastic/integrations/pull/12685
2227
- version: "2.43.0"
2328
changes:
2429
- description: Set `event.type` and `event.action` fields in vpcflow logs.
@@ -41,7 +46,7 @@
4146
link: https://github.com/elastic/integrations/pull/12755
4247
- version: "2.40.0"
4348
changes:
44-
- description: Add support for Kibana `9.0.0`
49+
- description: Add support for Kibana `9.0.0`
4550
type: enhancement
4651
link: https://github.com/elastic/integrations/pull/12637
4752
- version: "2.39.0"
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
version: '2.3'
2+
services:
3+
securityhub_full_posture:
4+
image: docker.elastic.co/observability/stream:v0.15.0
5+
hostname: securityhub.xxxx.amazonaws.cn
6+
ports:
7+
- 443
8+
volumes:
9+
- ./files:/files:ro
10+
environment:
11+
PORT: "443"
12+
command:
13+
- http-server
14+
- --addr=:443
15+
- --config=/files/config.yml
16+
- --tls-cert=/files/certificate.crt
17+
- --tls-key=/files/private.key
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
-----BEGIN CERTIFICATE-----
2+
MIIDUjCCAjoCCQDQ1VVKJuqgWjANBgkqhkiG9w0BAQsFADBrMQswCQYDVQQGEwJY
3+
WDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBh
4+
bnkgTHRkMScwJQYDVQQDDB5zZWN1cml0eWh1Yi54eHh4LmFtYXpvbmF3cy5jb20w
5+
HhcNMjIwNzA2MDg1MTUwWhcNMjMwNzA2MDg1MTUwWjBrMQswCQYDVQQGEwJYWDEV
6+
MBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBhbnkg
7+
THRkMScwJQYDVQQDDB5zZWN1cml0eWh1Yi54eHh4LmFtYXpvbmF3cy5jb20wggEi
8+
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDhyLkZGxIdXMUb8UuD16U67hGi
9+
/W7SvhtHLkQGbHTWAD7+AAg5ybbBFa2LTf3G5lprgJ/nUAl5N2i7CnSOlRxm6yKU
10+
VeyXPzQ8327sb7Y1pm07hU2Y+unKXcCjQi4lgF9GUXgRFYGxzIiwbG52XgZNJ4Cq
11+
TWXAlRi8J4nJbSPty3R6wt2+bxIGf9/v6VoBpj0Ltal7aM9/YTGYkc+PprcoK6+x
12+
o5IzXha4iedNLjVRl7MLkP57BmDTTJpdO8OraddWjm1/I3kG5Lyu19A9URMg47vW
13+
L7IOtOZzfDNyCYbFwqNMHk62AVpTOYqL/icNlX+EpUxX4kyVhd4W0Y9xBs5HAgMB
14+
AAEwDQYJKoZIhvcNAQELBQADggEBAFA+VI+UgD2ldDLkfoCG+BNtasm9dyJvuer+
15+
9+R8IyMDL0O8ppLSpKny7MbTLFKymIkTFJzCKf3+q5cL/y4W5YRPsm3tYD8wzBfN
16+
o+sG2e1UlmMtv0vU4dsmoeHqYFyuxuDlgtH0FynCYgh+Xo6s6zPpNi48QsLebIf9
17+
Bp0lgklIyHpVhMTwUua5P0t00ecKvkCNf51x/apqyRYBdoAvrwQ9IRVPmvu/iQCR
18+
3AMQH0dhaDjS3aVzKyRrhu+jjEAFRV5yVr64LTkQAWzMb6yz1KaQa0OjXNV1wX4F
19+
/k5zhqX0C0HAvDkSKXqwtUXl8jKyvP3Ogwddzg17932lVJe/3jc=
20+
-----END CERTIFICATE-----
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
rules:
2+
- path: /findings
3+
methods: ["POST"]
4+
responses:
5+
- status_code: 200
6+
body: |
7+
{"Findings":[{"Action":{"ActionType":"PORT_PROBE","PortProbeAction":{"PortProbeDetails":[{"LocalPortDetails":{"Port":80,"PortName":"HTTP"},"LocalIpDetails":{"IpAddressV4":"1.128.0.0"},"RemoteIpDetails":{"Country":{"CountryName":"Example Country"},"City":{"CityName":"Example City"},"GeoLocation":{"Lon":0,"Lat":0},"Organization":{"AsnOrg":"ExampleASO","Org":"ExampleOrg","Isp":"ExampleISP","Asn":64496}}}],"Blocked":false}},"AwsAccountId":"111111111111","CompanyName":"AWS","Compliance":{"RelatedRequirements":["Req1","Req2"],"Status":"PASSED","StatusReasons":[{"ReasonCode":"CLOUDWATCH_ALARMS_NOT_PRESENT","Description":"CloudWatch alarms do not exist in the account"}]},"Confidence":42,"CreatedAt":"2017-03-22T13:22:13.933Z","Criticality":99,"Description":"The version of openssl found on instance i-abcd1234 is known to contain a vulnerability.","FindingProviderFields":{"Confidence":42,"Criticality":99,"RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"}],"Severity":{"Label":"MEDIUM","Original":"MEDIUM"},"Types":["Software and Configuration Checks/Vulnerabilities/CVE"]},"FirstObservedAt":"2017-03-22T13:22:13.933Z","GeneratorId":"acme-vuln-9ab348","Id":"us-west-2/111111111111/98aebb2207407c87f51e89943f12b1ef","LastObservedAt":"2017-03-23T13:22:13.933Z","Malware":[{"Name":"Stringler","Type":"COIN_MINER","Path":"/usr/sbin/stringler","State":"OBSERVED"}],"Network":{"Direction":"IN","OpenPortRange":{"Begin":443,"End":443},"Protocol":"TCP","SourceIpV4":"1.128.0.0","SourceIpV6":"2a02:cf40::","SourcePort":"42","SourceDomain":"example1.com","SourceMac":"00:0d:83:b1:c0:8e","DestinationIpV4":"1.128.0.0","DestinationIpV6":"2a02:cf40::","DestinationPort":"80","DestinationDomain":"example2.com"},"NetworkPath":[{"ComponentId":"abc-01a234bc56d8901ee","ComponentType":"AWS::EC2::InternetGateway","Egress":{"Destination":{"Address":["1.128.0.0/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}},"Ingress":{"Destination":{"Address":["175.16.199.1/24"],"PortRanges":[{"Begin":443,"End":443}]},"Protocol":"TCP","Source":{"Address":["175.16.199.1/24"]}}}],"Note":{"Text":"Don't forget to check under the mat.","UpdatedBy":"jsmith","UpdatedAt":"2018-08-31T00:15:09Z"},"PatchSummary":{"Id":"pb-123456789098","InstalledCount":"100","MissingCount":"100","FailedCount":"0","InstalledOtherCount":"1023","InstalledRejectedCount":"0","InstalledPendingReboot":"0","OperationStartTime":"2018-09-27T23:37:31Z","OperationEndTime":"2018-09-27T23:39:31Z","RebootOption":"RebootIfNeeded","Operation":"Install"},"Process":{"Name":"syslogd","Path":"/usr/sbin/syslogd","Pid":12345,"ParentPid":56789,"LaunchedAt":"2018-09-27T22:37:31Z","TerminatedAt":"2018-09-27T23:37:31Z"},"ProductArn":"arn:aws:securityhub:us-east-1:111111111111:product/111111111111/default","ProductFields":{"generico/secure-pro/Count":"6","Service_Name":"cloudtrail.amazonaws.com","aws/inspector/AssessmentTemplateName":"My daily CVE assessment","aws/inspector/AssessmentTargetName":"My prod env","aws/inspector/RulesPackageName":"Common Vulnerabilities and Exposures"},"ProductName":"Security Hub","RecordState":"ACTIVE","Region":"us-east-1","RelatedFindings":[{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"123e4567-e89b-12d3-a456-426655440000"},{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/guardduty","Id":"AcmeNerfHerder-111111111111-x189dx7824"}],"Remediation":{"Recommendation":{"Text":"Run sudo yum update and cross your fingers and toes.","Url":"http://myfp.com/recommendations/dangerous_things_and_how_to_fix_them.html"}},"Resources":[{"Type":"AwsEc2Instance","Id":"i-cafebabe","Partition":"aws","Region":"us-west-2","Tags":{"billingCode":"Lotus-1-2-3","needsPatching":"true"},"Details":{"IamInstanceProfileArn":"arn:aws:iam::123456789012:role/IamInstanceProfileArn","ImageId":"ami-79fd7eee","IpV4Addresses":["175.16.199.1"],"IpV6Addresses":["2a02:cf40::"],"KeyName":"testkey","LaunchedAt":"2018-09-29T01:25:54Z","MetadataOptions":{"HttpEndpoint":"enabled","HttpProtocolIpv6":"enabled","HttpPutResponseHopLimit":1,"HttpTokens":"optional","InstanceMetadataTags":"disabled"},"NetworkInterfaces":[{"NetworkInterfaceId":"eni-e5aa89a3"}],"SubnetId":"PublicSubnet","Type":"i3.xlarge","VirtualizationType":"hvm","VpcId":"TestVPCIpv6"}}],"Sample":true,"SchemaVersion":"2018-10-08","Severity":{"Label":"CRITICAL","Original":"8.3"},"SourceUrl":"http://threatintelweekly.org/backdoors/8888","ThreatIntelIndicators":[{"Type":"IPV4_ADDRESS","Value":"175.16.199.1","Category":"BACKDOOR","LastObservedAt":"2018-09-27T23:37:31Z","Source":"Threat Intel Weekly","SourceUrl":"http://threatintelweekly.org/backdoors/8888"}],"Threats":[{"FilePaths":[{"FileName":"b.txt","FilePath":"/tmp/b.txt","Hash":"sha256","ResourceId":"arn:aws:ec2:us-west-2:123456789012:volume/vol-032f3bdd89aee112f"}],"ItemCount":3,"Name":"Iot.linux.mirai.vwisi","Severity":"HIGH"}],"Title":"EC2.20 Both VPN tunnels for an AWS Site-to-Site VPN connection should be up","Types":["Software and Configuration Checks/Vulnerabilities/CVE"],"UpdatedAt":"2018-08-31T00:15:09Z","UserDefinedFields":{"reviewedByCio":"true","comeBackToLater":"Check this again on Monday"},"VerificationState":"UNKNOWN","Vulnerabilities":[{"Cvss":[{"BaseScore":4.7,"BaseVector":"AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N","Version":"V3"},{"BaseScore":4.7,"BaseVector":"AV:L/AC:M/Au:N/C:C/I:N/A:N","Version":"V2"}],"Id":"CVE-2020-12345","ReferenceUrls":["http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12418","http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17563"],"RelatedVulnerabilities":["CVE-2020-12345"],"Vendor":{"Name":"Alas","Url":"https://alas.aws.amazon.com/ALAS-2020-1337.html","VendorCreatedAt":"2020-01-16T00:01:43Z","VendorSeverity":"Medium","VendorUpdatedAt":"2020-01-16T00:01:43Z"},"VulnerablePackages":[{"Architecture":"x86_64","Epoch":"1","Name":"openssl","Release":"16.amzn2.0.3","Version":"1.0.2k"}]}],"Workflow":{"Status":"NEW"},"WorkflowState":"NEW"}]}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
-----BEGIN PRIVATE KEY-----
2+
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDhyLkZGxIdXMUb
3+
8UuD16U67hGi/W7SvhtHLkQGbHTWAD7+AAg5ybbBFa2LTf3G5lprgJ/nUAl5N2i7
4+
CnSOlRxm6yKUVeyXPzQ8327sb7Y1pm07hU2Y+unKXcCjQi4lgF9GUXgRFYGxzIiw
5+
bG52XgZNJ4CqTWXAlRi8J4nJbSPty3R6wt2+bxIGf9/v6VoBpj0Ltal7aM9/YTGY
6+
kc+PprcoK6+xo5IzXha4iedNLjVRl7MLkP57BmDTTJpdO8OraddWjm1/I3kG5Lyu
7+
19A9URMg47vWL7IOtOZzfDNyCYbFwqNMHk62AVpTOYqL/icNlX+EpUxX4kyVhd4W
8+
0Y9xBs5HAgMBAAECggEAF21MR16XspQ9n3iZ7UQi0MqC6faB2TwAeJJEXKZEOTAt
9+
WQ2HzPcxDzfAmgOtoUWlfCIMdWPIl9s38rBTB7hRChy7qciAk/Dq6qYETGQK8+Yg
10+
z1w1gPoH6AdyRX5Ia3u2ZwVs/9jLbDdct3GIxJ9c6ASBRSpitGjD+EHh+hRo9fNE
11+
bnTNCS9roukGIyXbRDJMplAoCLNI+HVjTkjWPq4mff6EeYuTCjPKoJVzsrp8Ecai
12+
Rf9a444KeUFlE4rcNmFtHJiVohJiPpIF85DUb8RBfVr8xSdoG6QHxaTcjbk3nPd2
13+
/x+NSY5O5PkEXbQpsBZmEo1Aba1qjeRg2pCNsP9tgQKBgQDxNBtroNv7uWMeQMKf
14+
fj4FtyvFfgfBt4fdUZblW60sWbRu2PnrwDyFxGGX+KKVFrKauS2R8SfSvX230kGl
15+
vbKXSxo10XjmmY0Kaulet7z9awjK+yTcj3HKqVpjCdZK0KO1FXwZ45hwM7ewB6KI
16+
xukbZPORJwbwIjBYAGt0mfSaTQKBgQDvondtX11L0qjDoqcW5a6o2cdkj1MjBfP+
17+
AKZqOKDNNeHG3hT/YWfcFUis/UXMV7TBG4NQuIRGu5xZn3WbxgynHx3/QiVKG90/
18+
m56hsAStcVHTVcPcAh48jgYF60u60jgUhBcyrAZpsskul+oY/v16Eutx5QqjGjnc
19+
3bmFZe/s4wKBgB2SeOYqM65aHVfhMrthO/NxcLFm8UaD3Ol6jliSc9njKacJfSK1
20+
T/ZKjHiYaD6FKOKlX3vsKCjDSL2XzqqmZlX8RDti8kK7grpLP094kXg0fkB8qBlO
21+
kPH673UDCL3ldJzIBI4cBF2FSbkQRpIkaQINz3r1YPliB7FSY9pI4d9lAoGAWGyz
22+
8vjonUz7l00SqQFR5N6PlAzLGbZdpVGqFrIUrASA7ngOeXoA8BYufh7rPY7zlPpJ
23+
B2U+8jbSZ8POiw+Wpah20jUfO2xyxMDw1Sr1Xubc0cXpAusJK0Eg+dgsVqCxruhb
24+
Awi1SRV+5SGLcXPOJtiKZrmkpjDMPzLV/WJzGQ8CgYAbcMtnLshdYVNXfutWgSm2
25+
TqYfGm/L+njAFXfSnIxotIw0jQVt/uB0okcNAHKTn1elCxC0v0BZDsSUhxToUGk+
26+
x1wfip3SVhR5sYg8HBYbDCkTKZerleeW5PzcFFf+BY4DxR+8yWNEA1PrAejKyXk5
27+
Id0GFdKT0A2niGndkyL7/A==
28+
-----END PRIVATE KEY-----
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
fields:
2+
tags:
3+
- preserve_original_event
4+
- preserve_duplicate_custom_fields
5+
dynamic_fields:
6+
"@timestamp": ".*"

0 commit comments

Comments
 (0)