[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

moxarth-rathod · 2025-07-18T11:59:11Z

Proposed commit message

office_365: populate ECS message field with alert titles from ComplianceDLPExchange 

This PR updates the mapping of message ECS field for the ComplianceDLPExchange events
from mixture of o365 fields.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/o365 directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes [Office365] Populate ECS message Field with Alert Titles for DLP Exchange Alerts #12598

…events

elasticmachine · 2025-07-18T11:59:41Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG · 2025-07-21T09:58:33Z

packages/o365/changelog.yml

- version: "2.18.6"
+- version: "2.19.0"
  changes:
-    - description: Stricter enforcement of maximum age limits.
-      type: bugfix
-      link: https://github.com/elastic/integrations/pull/14567
+    - description: Populate `message` field from the O365 Audit Log fields instead of `Subject` field in ComplianceDLPExchange events to better reflect Alert Titles.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14587


why are we replacing an existing changelog ?

Oh, this is happened because #14567 was merged after i raised this PR. Need to take pull from main.

kcreddy · 2025-07-21T09:45:06Z

packages/o365/changelog.yml

- version: "2.18.6"
+- version: "2.19.0"
  changes:
-    - description: Stricter enforcement of maximum age limits.
-      type: bugfix
-      link: https://github.com/elastic/integrations/pull/14567
+    - description: Populate `message` field from the O365 Audit Log fields instead of `Subject` field in ComplianceDLPExchange events to better reflect Alert Titles.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14587


Please add new changelog entry instead of updating existing one.

I need to take pull from main as #14567 was merged few days back.

kcreddy · 2025-07-21T09:49:27Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+  - script:
+      lang: painless
+      description: Construct a message from operation, user and subject fields for 'ComplianceDLPExchange' events
+      if: 'ctx.event?.code != null && ctx.event?.code == "ComplianceDLPExchange"'


Suggested change

if: 'ctx.event?.code != null && ctx.event?.code == "ComplianceDLPExchange"'

if: ctx.event?.code == "ComplianceDLPExchange"

kcreddy · 2025-07-21T09:51:31Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      description: Construct a message from operation, user and subject fields for 'ComplianceDLPExchange' events
+      if: 'ctx.event?.code != null && ctx.event?.code == "ComplianceDLPExchange"'
+      source: >
+        def operation = ctx.event?.action != null ? ctx.event.action : ctx.o365audit?.Operation;


Much earlier in the pipeline we have:

integrations/packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

Lines 163 to 166 in 648e999

- rename:

field: o365audit.Operation

target_field: event.action

ignore_missing: true

I think L#738 this is same as: def operation = ctx.event?.action

kcreddy · 2025-07-21T09:51:57Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      if: 'ctx.event?.code != null && ctx.event?.code == "ComplianceDLPExchange"'
+      source: >
+        def operation = ctx.event?.action != null ? ctx.event.action : ctx.o365audit?.Operation;
+        def user = ctx.user?.id != null ? ctx.user.id : ctx.o365audit?.UserId;


Same here, based on

integrations/packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

Lines 154 to 158 in 648e999

- convert:

field: o365audit.UserId

target_field: user.id

type: string

ignore_missing: true

kcreddy · 2025-07-21T09:57:00Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        def operation = ctx.event?.action != null ? ctx.event.action : ctx.o365audit?.Operation;
+        def user = ctx.user?.id != null ? ctx.user.id : ctx.o365audit?.UserId;
+        def subject = ctx.o365audit?.ExchangeMetaData?.Subject != null ? ctx.o365audit.ExchangeMetaData.Subject : (ctx.email?.subject != null ? ctx.email.subject : '');
+        ctx.message = "Office365 Alert: " + operation + " detected in email sent by " + user + " with subject '" + subject + "'";


It seems to be this only handles part of the suggested fix: #12598 (comment) ?
I don't see any message of the format DLP Alert: {Operation} detected in email or Office 365 alert: {Operation} detected

@raqueltabuyo can you please update the issue as we discussed?
@kcreddy i had a discussion with her and confirmed the format.

I don't see this clarified in the issue. Can this be done?

kcreddy · 2025-07-21T09:57:19Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

@@ -730,6 +730,22 @@ processors:
      ignore_missing: true
      if: 'ctx.event?.code == "SecurityComplianceAlerts" && ctx.rule?.ruleset == "MalwareFamily"'
  # DLP Schema
+  - script:
+      lang: painless


kcreddy · 2025-07-21T09:59:59Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        - set:
+            field: message
+            value: 'DLP Alert: Unable to construct message due to script failure.'


I don't think the processor reaches a failure as all conditions are handled.

Oh, yes then there's no meaning of the on_failure block, @kcreddy can we keep it for just unexpected script failures?

kcreddy

Please update README as per CI error:

 | o365.audit.ExchangeMetaData.Sent |  | date |
+| o365.audit.ExchangeMetaData.Subject |  | ketword |
 | o365.audit.ExchangeMetaData.To |  | keyword |

Error: checking package failed: checking readme files are up-to-date failed: files do not match
[o365] run_tests_package failed

ShourieG · 2025-07-21T10:04:57Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        def operation = ctx.event?.action != null ? ctx.event.action : ctx.o365audit?.Operation;
+        def user = ctx.user?.id != null ? ctx.user.id : ctx.o365audit?.UserId;
+        def subject = ctx.o365audit?.ExchangeMetaData?.Subject != null ? ctx.o365audit.ExchangeMetaData.Subject : (ctx.email?.subject != null ? ctx.email.subject : '');


One query here,

In scenarios when ctx.event.action == null, ctx.user.id == null and ctx.o365audit.ExchangeMetaData?.Subject == null are we guaranteed to have the alternatives populated ? If not, we might end up with an incomplete message with the current logic.

@kcreddy, what do you think, should we have a fallback message just in-case ?

I think it might be worth to have <unknown>. But I will leave it to @raqueltabuyo to confirm.
Also waiting for #14587 (comment)

elastic-vault-github-plugin-prod · 2025-07-21T11:32:50Z

🚀 Benchmarks report

Package `o365` 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`audit`	1712.33	1190.48	-521.85 (-30.48%)	💔

To see the full report comment with /test benchmark fullreport

efd6 · 2025-07-25T03:27:14Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      source: >
+        def operation = ctx.event?.action;
+        def user = ctx.user?.id;
+        def subject = ctx.o365audit?.ExchangeMetaData?.Subject != null ? ctx.o365audit.ExchangeMetaData.Subject : (ctx.email?.subject != null ? ctx.email.subject : '');


This kind of density is why ternaries are problematic.

If this must be done, suggest instead the slightly less unpleasant:

def subject = ctx.o365audit?.ExchangeMetaData?.Subject ?: (ctx.email?.subject != null ? ctx.email.subject : '');

elastic-sonarqube · 2025-07-25T05:51:47Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-07-25T05:51:55Z

💚 Build Succeeded

Buildkite Build
Commit: 3e41680

History

💚 Build #28970 succeeded 3d9ef9c
💔 Build #28969 failed f8ae6e9
💚 Build #28813 succeeded bd49309
💚 Build #28810 succeeded b51abad
💔 Build #28807 failed 42fcc75
💔 Build #28756 failed 9c7b681

cc @moxarth-rathod

[O365] Update the mapping of message field for ComplianceDLPExchange …

122219b

…events

moxarth-rathod requested a review from a team as a code owner July 18, 2025 11:59

moxarth-rathod self-assigned this Jul 18, 2025

moxarth-rathod added 2 commits July 18, 2025 18:17

Add PR link to changelog

9c7b681

fix data type for ExchangeMetaData.Subject field

42fcc75

ShourieG reviewed Jul 21, 2025

View reviewed changes

kcreddy reviewed Jul 21, 2025

View reviewed changes

ShourieG reviewed Jul 21, 2025

View reviewed changes

Pull from main and address PR comments

b51abad

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 21, 2025

moxarth-rathod added 3 commits July 21, 2025 17:37

Remove unnecessary on_failure block from the script processor

bd49309

Update changelog for older versions

f8ae6e9

resolve merge conflicts

3d9ef9c

efd6 reviewed Jul 25, 2025

View reviewed changes

Update the turnary operation for subject value

3e41680

	if: 'ctx.event?.code != null && ctx.event?.code == "ComplianceDLPExchange"'
	if: ctx.event?.code == "ComplianceDLPExchange"

	- rename:
	field: o365audit.Operation
	target_field: event.action
	ignore_missing: true

	- convert:
	field: o365audit.UserId
	target_field: user.id
	type: string
	ignore_missing: true

[O365] Update the mapping of ECS message field for ComplianceDLPExchange events #14587

Are you sure you want to change the base?

[O365] Update the mapping of ECS message field for ComplianceDLPExchange events #14587

Conversation

moxarth-rathod commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moxarth-rathod Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

ShourieG Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package o365 👍(0) 💚(0) 💔(1)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Jul 25, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Jul 25, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

moxarth-rathod commented Jul 18, 2025 •

edited

Loading

kcreddy Jul 21, 2025 •

edited

Loading

moxarth-rathod Jul 21, 2025 •

edited

Loading

ShourieG Jul 21, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Jul 21, 2025 •

edited

Loading

Package `o365` 👍(0) 💚(0) 💔(1)