[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

moxarth-rathod · 2025-07-18T11:59:11Z

Proposed commit message

office_365: populate ECS message field with alert titles from ComplianceDLPExchange 

This PR updates the mapping of message ECS field for the ComplianceDLPExchange events
from mixture of o365 fields.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/o365 directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes [Office365] Populate ECS message Field with Alert Titles for DLP Exchange Alerts #12598

…events

elasticmachine · 2025-07-18T11:59:41Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG · 2025-07-21T09:58:33Z

packages/o365/changelog.yml

- version: "2.18.6"
+- version: "2.19.0"
  changes:
-    - description: Stricter enforcement of maximum age limits.
-      type: bugfix
-      link: https://github.com/elastic/integrations/pull/14567
+    - description: Populate `message` field from the O365 Audit Log fields instead of `Subject` field in ComplianceDLPExchange events to better reflect Alert Titles.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14587


why are we replacing an existing changelog ?

Oh, this is happened because #14567 was merged after i raised this PR. Need to take pull from main.

packages/o365/changelog.yml

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

kcreddy

Please update README as per CI error:

 | o365.audit.ExchangeMetaData.Sent |  | date |
+| o365.audit.ExchangeMetaData.Subject |  | ketword |
 | o365.audit.ExchangeMetaData.To |  | keyword |

Error: checking package failed: checking readme files are up-to-date failed: files do not match
[o365] run_tests_package failed

ShourieG · 2025-07-21T10:04:57Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+        def operation = ctx.event?.action != null ? ctx.event.action : ctx.o365audit?.Operation;
+        def user = ctx.user?.id != null ? ctx.user.id : ctx.o365audit?.UserId;
+        def subject = ctx.o365audit?.ExchangeMetaData?.Subject != null ? ctx.o365audit.ExchangeMetaData.Subject : (ctx.email?.subject != null ? ctx.email.subject : '');


One query here,

In scenarios when ctx.event.action == null, ctx.user.id == null and ctx.o365audit.ExchangeMetaData?.Subject == null are we guaranteed to have the alternatives populated ? If not, we might end up with an incomplete message with the current logic.

@kcreddy, what do you think, should we have a fallback message just in-case ?

I think it might be worth to have <unknown>. But I will leave it to @raqueltabuyo to confirm.
Also waiting for #14587 (comment)

If everything is null, I would change the message to "Office365 Alert", without giving additional details as the rest is unknown, but we still alert the user of the existence of a suspicious behavior in their O365 environment.

elastic-vault-github-plugin-prod · 2025-07-21T11:32:50Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-07-25T03:27:14Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      source: >
+        def operation = ctx.event?.action;
+        def user = ctx.user?.id;
+        def subject = ctx.o365audit?.ExchangeMetaData?.Subject != null ? ctx.o365audit.ExchangeMetaData.Subject : (ctx.email?.subject != null ? ctx.email.subject : '');


This kind of density is why ternaries are problematic.

If this must be done, suggest instead the slightly less unpleasant:

def subject = ctx.o365audit?.ExchangeMetaData?.Subject ?: (ctx.email?.subject != null ? ctx.email.subject : '');

elasticmachine · 2025-07-31T10:51:19Z

💚 Build Succeeded

Buildkite Build
Commit: caeaad5

History

💚 Build #29056 succeeded 3e41680
💚 Build #28970 succeeded 3d9ef9c
💔 Build #28969 failed f8ae6e9
💚 Build #28813 succeeded bd49309
💚 Build #28810 succeeded b51abad
💔 Build #28807 failed 42fcc75

cc @moxarth-rathod

elastic-sonarqube · 2025-07-31T10:51:25Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

kcreddy

LGTM for my comments.

raqueltabuyo

LGTM

elastic-vault-github-plugin-prod · 2025-08-01T06:08:27Z

Package o365 - 2.19.0 containing this change is available at https://epr.elastic.co/package/o365/2.19.0/

[O365] Update the mapping of message field for ComplianceDLPExchange …

122219b

…events

moxarth-rathod requested a review from a team as a code owner July 18, 2025 11:59

moxarth-rathod self-assigned this Jul 18, 2025

moxarth-rathod added 2 commits July 18, 2025 18:17

Add PR link to changelog

9c7b681

fix data type for ExchangeMetaData.Subject field

42fcc75

ShourieG reviewed Jul 21, 2025

View reviewed changes

kcreddy reviewed Jul 21, 2025

View reviewed changes

ShourieG reviewed Jul 21, 2025

View reviewed changes

Pull from main and address PR comments

b51abad

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 21, 2025

moxarth-rathod added 3 commits July 21, 2025 17:37

Remove unnecessary on_failure block from the script processor

bd49309

Update changelog for older versions

f8ae6e9

resolve merge conflicts

3d9ef9c

efd6 reviewed Jul 25, 2025

View reviewed changes

moxarth-rathod added 2 commits July 25, 2025 10:58

Update the turnary operation for subject value

3e41680

Update message if each field is null

caeaad5

moxarth-rathod requested review from kcreddy, raqueltabuyo, efd6 and ShourieG July 31, 2025 10:24

kcreddy approved these changes Jul 31, 2025

View reviewed changes

raqueltabuyo approved these changes Jul 31, 2025

View reviewed changes

efd6 approved these changes Jul 31, 2025

View reviewed changes

moxarth-rathod merged commit dd1b1f7 into elastic:main Aug 1, 2025
9 checks passed

[O365] Update the mapping of ECS message field for ComplianceDLPExchange events #14587

[O365] Update the mapping of ECS message field for ComplianceDLPExchange events #14587

Uh oh!

Conversation

moxarth-rathod commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jul 18, 2025

Uh oh!

ShourieG Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

moxarth-rathod Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

ShourieG Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShourieG Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

raqueltabuyo Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

efd6 Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 31, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jul 31, 2025

Quality Gate passed

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

raqueltabuyo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 1, 2025

Uh oh!

Uh oh!

[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

[O365] Update the mapping of ECS `message` field for `ComplianceDLPExchange` events #14587

moxarth-rathod commented Jul 18, 2025 •

edited

Loading

ShourieG Jul 21, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Jul 21, 2025 •

edited

Loading