Skip to content

[cisco_ios] Improve hostname parsing #13816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

[cisco_ios] Improve hostname parsing #13816

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f4f8893
Use script to determine hostname vs seq_num
mjwolf May 5, 2025
aa046f6
In case sequence is incorrectly parsed, remove it
mjwolf May 7, 2025
a1ee0e5
Merge remote-tracking branch 'upstream/main' into cisco_ios_hostname_2
mjwolf May 7, 2025
cde4c94
Update changelog
mjwolf May 7, 2025
9cde5f1
Merge branch 'main' into cisco_ios_hostname_2
mjwolf May 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/cisco_ios/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.30.2"
changes:
- description: Correct parsing of FQDN hostnames
type: bugfix
link: https://github.com/elastic/integrations/pull/13816
- version: "1.30.1"
changes:
- description: Correct parsing of FQDN hostnames
Expand Down
40 changes: 36 additions & 4 deletions packages/cisco_ios/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,26 +36,58 @@ processors:
tag: grok_header
patterns:
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{TIMESTAMP_ISO8601:_temp_.cisco_timestamp} %{CISCO_HOSTNAME:log.syslog.hostname} %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{SYSLOGTIMESTAMP} %{IP} %{CISCO_HOSTNAME:log.syslog.hostname}: (?:%{NUMBER:cisco.ios.sequence}: )?(?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP}): %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{SYSLOGTIMESTAMP} %{IP} (?:%{CISCO_HOSTNAME:log.syslog.hostname}: )?(?:%{NUMBER:cisco.ios.sequence}: )?(?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP}): %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{SYSLOGTIMESTAMP} (?:%{IP}|%{CISCO_HOSTNAME:log.syslog.hostname}) %{NUMBER:cisco.ios.sequence}: (?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP}): %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?(?:(?:%{IP}|%{CISCO_HOSTNAME:log.syslog.hostname})(?:: \*%{DATA}:|[:]?)? )?(?:%{NUMBER:cisco.ios.sequence}: )?(?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP}): %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?(?:(?:%{IP}|%{CISCO_HOSTNAME:log.syslog.hostname})(?:: \*%{DATA}:|:?)? )?(?:%{NUMBER:cisco.ios.sequence}: )?(?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP:_temp_.timestamp}): %{GREEDYDATA:_temp_.message}$'
- '^%{SYSLOGTIMESTAMP} (?:%{IP}|%{HOSTNAME:log.syslog.hostname}) %{CISCO_PRIORITY_MSGCOUNT}?(?:%{NUMBER:cisco.ios.sequence}: )(?:(?:%{CISCO_UPTIME:cisco.ios.uptime}|%{CISCO_TIMESTAMP}): )?%{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{SYSLOGTIMESTAMP} (?:%{IP:log.syslog.hostname}|%{CISCO_HOSTNAME:log.syslog.hostname}) %{GREEDYDATA:_temp_.message}$'
- '^%{CISCO_PRIORITY_MSGCOUNT}?%{SYSLOGTIMESTAMP} (?:%{IP:log.syslog.hostname}|%{CISCO_HOSTNAME:log.syslog.hostname}|%{WORD:log.syslog.hostname}) %{GREEDYDATA:_temp_.message}$'
pattern_definitions:
ISO8601_TIMEZONE: "(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))"
TIMESTAMP_ISO8601: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE:_temp_.tz}?"
CISCO_PRIORITY_MSGCOUNT: '<%{NONNEGINT:log.syslog.priority:long}>(?:%{NONNEGINT:cisco.ios.message_count})?(?:: )?'
CISCO_TIMESTAMP: '[*]?%{CISCOTIMESTAMP_EX:_temp_.cisco_timestamp}(?: %{CISCO_TZ:_temp_.tz})?'
CISCOTIMESTAMP_EX: '(%{CISCOTIMESTAMP})|(%{YEAR} %{MONTH} %{MONTHDAY} %{TIME})'
CISCO_UPTIME: '(?:\d{1,4}:\d{2}:\d{2}|(?:(\d+)y)?(?:(\d+)w)?(?:(\d+)d)?(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?)'
CISCO_HOSTNAME: '[a-zA-Z][.0-9a-zA-Z_-]{0,253}[0-9a-zA-Z]?'
CISCO_HOSTNAME: '[0-9a-zA-Z][.0-9a-zA-Z_-]{0,253}[0-9a-zA-Z]?'
CISCO_TZ: '[a-zA-Z]{1,4}([+-]\d{1,2}|[+-]\d{2}:\d{2})?'
- grok:
field: _temp_.message
tag: grok_message
patterns:
- '^%%{GREEDYDATA:message}$'
- '^%{GREEDYDATA:_temp_.generic_message}$'
# Do not allow all-digit hostnames, these are sequence numbers. Due limitations of grok, and how
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the Painless script pattern matching for number really necessary? It looks like we can just grok log.syslog.hostname as NUMBER into cisco.ios.sequence – if that works, we have a sequence.

We might need to match it against the timestamp pattern, but can't that also be done with grok?

# many various log formats are supported, it's not possible to do this entirely with patterns.
# See https://github.com/elastic/integrations/pull/13816 for details.
- script:
lang: painless
tag: convert_host_to_seq
source: |
if (ctx?.log?.syslog != null && ctx.log.syslog.containsKey('hostname')) {
def val = ctx.log.syslog.hostname;
Pattern p = /\d+/;
if (p.matcher(val).matches()) {
if (ctx.cisco == null) {
ctx.cisco = new HashMap();
}
if (ctx.cisco.ios == null) {
ctx.cisco.ios = new HashMap();
}
ctx.cisco.ios.sequence = val;
ctx.log.syslog.remove('hostname');
// sequence may actually be part of a timestamp in some cases. Remove it, if it appears to be parsed from a timestamp
if (ctx?._temp_?.timestamp != null) {
val = ctx._temp_.timestamp;
Pattern p2 = /^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d{1,2} \d{2}:\d{2}:\d{2}.*/;
if (p2.matcher(val).matches()) {
ctx.cisco.ios.remove('sequence');
}
}
if (ctx.log.syslog.isEmpty()) {
ctx.log.remove("syslog");
}
}
}
- set:
field: event.sequence
copy_from: cisco.ios.sequence
Expand Down
2 changes: 1 addition & 1 deletion packages/cisco_ios/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: cisco_ios
title: Cisco IOS
version: "1.30.1"
version: "1.30.2"
description: Collect logs from Cisco IOS with Elastic Agent.
type: integration
categories:
Expand Down