Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Bambenek parser to better handle description #1451

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from
Draft

Improved Bambenek parser to better handle description #1451

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
debd147
Improved parser to better handle description
Sep 13, 2019
e5b4372
Removed whitespace and broke down regex expression
Sep 13, 2019
56021be
TST: add bambenek tests for commas in descriptions
Sep 18, 2019
f8dfa9b
Corrected line-break in regex expression
Sep 19, 2019
87a11ed
Merge pull request #1 from wagner-certat/pr-1451
amojamo Sep 19, 2019
4065494
Merge branch 'develop' of https://github.com/amojamo/intelmq into dev…
Sep 19, 2019
ef4ec5c
Merge branch 'develop' into pr-1451
May 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions intelmq/bots/parsers/bambenek/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
""" IntelMQ parser for Bambenek DGA, Domain, and IP feeds """

import re
from intelmq.lib.bot import ParserBot


Expand Down Expand Up @@ -32,32 +33,33 @@ def parse_line(self, line, report):
self.tempdata.append(line)

else:
value = line.split(',')
m = re.match(r"(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?P<description>.*), \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not seem to work with IPv6

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we change .* (greedy) to .*? (non-greedy) here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'm sorry, I didn't consider IPv6. Regex for an IPv6 pattern is out of my scope.
The greedy to non-greedy works for the description group, but not for the URL group.

(?P<timestamp>\d{4}-\d{2}-\d{2}[ ]\d{2}[:]\d{2}),(?P<url>.*)", line)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we change .* (greedy) to .*? (non-greedy) here?

Can we $ at the end to indicate the regular expression should match the full line?

values = m.groups()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That line raises an exception in the tests

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be my line split. I'm not sure how to break the regex line into two lines in order not to trigger the "Line too long" warning when it comes to code style checking.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just split the line like this:

            m = re.match(r"(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?P<description>.*), "
                         r"(?P<timestamp>\d{4}-\d{2}-\d{2}[ ]\d{2}[:]\d{2}),(?P<url>.*)", line)

event = self.new_event(report)

event.add('event_description.text', value[1])
event.add('event_description.url', value[3])
event.add('event_description.text', values[1])
event.add('event_description.url', values[3])
event.add('raw', line)

# last column is a url with malware named txt file link
malware_name = value[-1].split('/')[-1].split('.')[0]
malware_name = values[-1].split('/')[-1].split('.')[0]
event.add('malware.name', self.MALWARE_NAME_MAP.get(malware_name, malware_name))

if report['feed.url'] in BambenekParserBot.IPMASTERLIST:
event.add('source.ip', value[0])
event.add('time.source', value[2] + ' UTC')
event.add('source.ip', values[0])
event.add('time.source', values[2] + ' UTC')
event.add('classification.type', 'c2server')
event.add('status', 'online')

elif report['feed.url'] in BambenekParserBot.DOMMASTERLIST:
event.add('source.fqdn', value[0])
event.add('time.source', value[2] + ' UTC')
event.add('source.fqdn', values[0])
event.add('time.source', values[2] + ' UTC')
event.add('classification.type', 'c2server')
event.add('status', 'online')

elif report['feed.url'] in BambenekParserBot.DGA_FEED:
event.add('source.fqdn', value[0])
event.add('time.source', value[2] + ' 00:00 UTC')
event.add('source.fqdn', values[0])
event.add('time.source', values[2] + ' 00:00 UTC')
event.add('classification.type', 'dga domain')

else:
Expand Down