Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

STIX Parser Crashing After Reading 65535 Bytes #58

Open
TheDr1ver opened this issue Dec 21, 2018 · 2 comments
Open

STIX Parser Crashing After Reading 65535 Bytes #58

TheDr1ver opened this issue Dec 21, 2018 · 2 comments

Comments

@TheDr1ver
Copy link

TheDr1ver commented Dec 21, 2018 •
edited
Loading

I'm not sure if this is some weird issue with my server or Python or if it's something dealing with one of the underlying libraries used by the MISP Taxii Service, but I'm kind of stumped.

I'm attempting to import STIX files obtained from the Anomali TAXII feed into MISP, but every STIX file I tried was crashing. After banging my head against the wall for a bit, I realized that while lxml is parsing the STIX file it crashes while reading the 65535th character of the STIX file.

Poll Anomali to get the latest Phish Tank STIX file
taxii-poll --host limo.anomali.com --https --collection Phish_Tank_F107 --discovery /api/v1/taxii/taxii-discovery-service/ --username guest --password guest --begin 2018-11-01T00:00:00Z --end 2018-12-22T00:00:00Z > phishtank.xml

Attempt to push the STIX file to opentaxii-run-dev
taxii-push --path http://localhost:9000/services/inbox -f phishtank.xml --dest collection --username MYUSERNAME --password MYPASSWORD

Error Log - line 847, column 228 is the 65535th char in the STIX file

    127.0.0.1 - - [21/Dec/2018 12:59:04] "POST /services/inbox HTTP/1.1" 200 -
'cm9vdDpyb290' 12
Posting STIX...
2018-12-21T18:00:36.436787Z [opentaxii.middleware] warning: Status exception {exception=Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 112, in load_stix
    stix_package = STIXPackage.from_json(data)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 495, in from_json
    d = json.loads(json_doc)
  File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in proce
ss
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers
.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers
.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_c
ontent
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 62, in post_stix
    package = pymisp.tools.stix.load_stix(StringIO(block))
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 16, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 119, in load_stix
    stixXml = etree.fromstring(stix.read())
  File "src/lxml/etree.pyx", line 3213, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1757, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1067, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 710, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 639, in lxml.etree._raiseParseError
  File "<string>", line 847
lxml.etree.XMLSyntaxError: Premature end of data in tag Description line 847, line 847, column 228

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 62, in post_stix
    package = pymisp.tools.stix.load_stix(StringIO(block))
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 16, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 119, in load_stix
    stixXml = etree.fromstring(stix.read())
  File "src/lxml/etree.pyx", line 3213, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1757, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1067, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 710, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 639, in lxml.etree._raiseParseError
opentaxii.taxii.exceptions.FailureStatus: Premature end of data in tag Description line 847, line 847, column 228 (<string>, line 847), logger=opentaxii.middleware, timestamp=2018-12-21T18:00:36.436787Z, event=Status exception, level=warning}

I'm wondering if it has something to do w/ the TEXT data object for MySQL maxing out at 65535 characters, but I'm not sure where I could find the type of data object being used to send the STIX docs to the database and possibly change it to MEDIUMTEXT or LONGTEXT.

Thanks in advance for any help you might be able to provide!
@TheDr1ver
Copy link
Author

Actually, now that I'm looking at it I'm wondering if this pull request addressed this very issue:

eclecticiq/OpenTAXII@882e2a5#diff-ce3f7b939e5c540480ac655aef32c513

Any idea if MISP-Taxii-Server can be successfully linked to the latest version of OpenTAXII without breaking? It looks like the one it's linked to is from June of last year, while the pull request addressing this issue was in October of last year.

@FloatingGhost
Copy link
Member

Managed to get around to updating to the latest version of OpenTAXII. Installation instructions and dependencies are updated.

Should probably fix the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@FloatingGhost @TheDr1ver