Streaming Accumulate() fails with 'invalid character' JSON error on malformed API responses

[avedmala]

Description

The Message.Accumulate() method fails with JSON marshaling errors when processing streaming responses that contain malformed or truncated JSON chunks. This appears to be related to issue #164 but with a different error pattern.

Error Messages

We're seeing errors like:

error accumulating message: error converting content block to JSON: json: error calling MarshalJSON for type json.RawMessage: invalid character 's' after object key

This is different from issue #164 which reports unexpected end of JSON input. Our errors suggest the accumulated JSON is syntactically malformed mid-stream rather than just truncated.

Environment

• SDK Version: v1.13.0 (also verified issue exists in v1.19.0)
• Go Version: 1.22+
• API: Messages API with streaming enabled

Reproduction

The error occurs intermittently during streaming tool use responses. We haven't been able to create a minimal reproduction case since it appears to depend on specific API response patterns.

The error occurs in message.go around lines 2730-2736:

case ContentBlockStopEvent:
    if len(acc.Content) == 0 {
        return fmt.Errorf("received event of type %s but there was no content block", event.Type)
    }
    contentBlock := &acc.Content[len(acc.Content)-1]
    cbJson, err := json.Marshal(contentBlock)  // <-- Fails here
    if err != nil {
        return fmt.Errorf("error converting content block to JSON: %w", err)
    }

Root Cause Analysis

The issue stems from how InputJSONDelta events are accumulated (lines 2710-2716):

case InputJSONDelta:
    if len(delta.PartialJSON) != 0 {
        if string(cb.Input) == "{}" {
            cb.Input = []byte(delta.PartialJSON)
        } else {
            cb.Input = append(cb.Input, []byte(delta.PartialJSON)...)
        }
    }

When the API sends malformed partial JSON chunks (possibly due to network issues or API glitches), appending them results in invalid JSON like {"key s instead of proper JSON. The json.RawMessage type doesn't validate its contents, so the error only surfaces when json.Marshal is called on the content block.

Impact

• These errors are transient and typically succeed on retry
• They cause streaming requests to fail even though the API response was partially received
• Multiple occurrences observed in production with various "invalid character" patterns

Related Issues

• Accumulation failure when parsing tool_use with 0 param tool #164 - Accumulation failure when parsing tool_use with 0 param tool (same error location, different cause)
• Guidance on handling streaming errors #200 - Guidance on handling streaming errors