TypeError: '<' not supported between instances of 'NoneType' and 'str'

[himynameissebbo]

Infos

We're using DD to monitor our systems. From time to time we see the following errors popping up in our Sentry environment.

Maybe related to #3019?

Which version of dd-trace-py are you using?

0.56.0

Which version of pip are you using?

latest

What is the result that you get?

TypeError: '<' not supported between instances of 'NoneType' and 'str'
  File "ddtrace/profiling/scheduler.py", line 53, in flush
    exp.export(events, start, self._last_export)
  File "ddtrace/profiling/exporter/http.py", line 150, in export
    profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)
  File "ddtrace/profiling/exporter/pprof.pyx", line 630, in ddtrace.profiling.exporter.pprof.PprofExporter.export
  File "ddtrace/profiling/exporter/pprof.pyx", line 314, in ddtrace.profiling.exporter.pprof._PprofConverter._build_profile

[Apakottur]

We just had a similar issue on version 0.56.1, I've also updaed in #3019 .

[jd]

Backport for 0.56 here: #3062

[Apakottur]

We are still having the error in 0.57:

File "ddtrace/profiling/exporter/pprof.pyx", line 594, in ddtrace.profiling.exporter.pprof.PprofExporter.export
File "ddtrace/profiling/exporter/pprof.pyx", line 493, in ddtrace.profiling.exporter.pprof.PprofExporter._group_stack_events
TypeError: '<' not supported between instances of 'int' and 'NoneType'

[jd]

Ok, this is weird. Line 493 is:

sorted(events, key=self._stack_event_group_key),

The error makes since comparison is being used here.
What's weird is that _stack_event_group_key returns a StackEventGroupKey whose typing has no None:

StackEventGroupKey = typing.NamedTuple(
    "StackEventGroupKey",
    [
        ("thread_id", str),
        ("thread_native_id", str),
        ("thread_name", str),
        ("task_id", str),
        ("task_name", str),
        ("local_root_span_id", str),
        ("span_id", str),
        ("trace_resource", str),
        ("trace_type", str),
        ("frames", HashableStackTraceType),
        ("nframes", int),
    ],
)

Only nframes is an int, and worst case it's set to 0 — never None as far as I can tell glancing at the code.
_stack_event_group_key does convert all data to the right type.

@Apakottur do you see 100% of the time or randomly from time to time?

[Apakottur]

@jd I see it randomly from time to time, I think.
I don't see any impact on APM as well, just this error popping on Sentry.
We've had 4 occurences of the error in the last few days, and I was using APM to analyze thousands of requests successfully.

I can't see the contents of self._stack_event_group_key on Sentry, but I do have some details from

ddtrace/profiling/scheduler.py:56
exp.export(events, start, self._last_export)

Which calls

ddtrace.profiling.exporter.pprof.PprofExporter.export

{
<class 'ddtrace.profiling.collector.memalloc.MemoryAllocSampleEvent'>: [
MemoryAllocSampleEvent(timestamp=1640358086011282246, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011320577, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 168, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011348129, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 168, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011374247, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre...
]
}

Maybe this helps?

[jd]

That actually gave me an idea. You might have a stack trace where lineno is None. That should not happen, but who knows, it's possible that in some cases (maybe eval?) the lineno is set to None and then comparison would fail:

>>> stack1 = [("somefile", 1, "funcname")]
>>> stack2 = [("somefile", None, "funcname")]
>>> stack1 < stack2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'NoneType'

I'll write up a PR to fix that and make sure all line numbers that are None are actually set to 0.