QA Parsing Becomes Unstable When Using Local vLLM with Markdown-Formatted Inputs

[mdjhacker]

Problem Description

When using the local vLLM backend to generate QA pairs (e.g. atomic / aggregated QA), inputs that contain Markdown-formatted content (such as #, ``` , *, `>`, `---`) may cause QA parsing failures.

---

Reproduction Conditions

• Use local vLLM backend

• Input text is Markdown-formatted and contains one or more of:

  • Headings (#, ##)
  • Separators (---, ***)
  • Code blocks (```)
  • Comment-style or annotation-heavy text

• Generate and parse QA pairs (e.g. atomic / aggregated QA)

---

Bad Case Example

Below is two real input example that reliably triggers the issue.
Although the overall request is valid JSON, the content field contains a large amount of Markdown separators and annotation-style text:

Observed Behavior

When processed by local vLLM, this input may result in:

• Markdown separators such as --- being interpreted as semantic or structural boundaries
• Question and Answer delimiters being duplicated, shifted, or merged
• QA extraction logic failing to reliably identify the true Q/A boundaries
• Final QA outputs becoming malformed or unparseable

This behavior is especially prominent in prompts that contain annotation-heavy Markdown content.

---

Root Cause Analysis

1. Markdown Special Characters Are Not Handled During Input Construction

In VLLMWrapper._build_inputs, conversation history and prompts are constructed via plain string concatenation:

@staticmethod
def _build_inputs(prompt: str, history: Optional[List[str]] = None) -> str:
    msgs = history or []
    lines = []
    for m in msgs:
        if isinstance(m, dict):
            role = m.get("role", "")
            content = m.get("content", "")
            lines.append(f"{role}: {content}")
        else:
            lines.append(str(m))
    lines.append(prompt)
    return "\n".join(lines)

This implementation:

• Does not escape or normalize Markdown structural symbols
• Directly injects Markdown syntax into the model context

As a result, the model may misinterpret Markdown markers as semantic or QA boundaries, leading to unstable QA generation and parsing failures.

[ChenZiHong-Gavin]

Can you please tell me which model you are using? @mdjhacker

[mdjhacker]

Can you please tell me which model you are using? @mdjhacker

Hi Gavin,

I'm using the Qwen3-4B-Instruct-2507 model for this testing. Interestingly, the QA parsing instability issue only occurs when feeding markdown-formatted inputs to the local vLLM directly. When I run the local vLLM independently and generate questions via the HTTP API, this problem does not arise at all.

The command I use to launch the vLLM service is:

vllm serve /cluster/llm/Qwen3-4B-Instruct-2507 --tensor-parallel-size 4 --gpu-memory-utilization 0.88 --max-model-len 262144

[ChenZiHong-Gavin]

I have successfully reproduced the issue, but I believe it's unrelated to markdown formatting—it's about the synthesis model being used. Some models lack sufficient instruction-following capability, causing their outputs to include extraneous information in both answers and questions.
When using gpt-4o-mini or open-source models above 32B, regardless of the document format, the aforementioned issue occurs with very low probability.
To address this, I modified the prompt to an XML version and modified the corresponding parse function, as can be seen in PR #137. This way, even small models like 4B can basically generate Q&A in the correct format, and there is no redundant output in the final results.

However, a small portion of data may be discarded due to parsing failures.
This is a trade-off, as we generally believe that data with format errors could be harmful to the model even if used for training.
If this is unacceptable, you can use models with stronger instruction-following capability as the synthesis model.

[mdjhacker]

I have successfully reproduced the issue, but I believe it's unrelated to markdown formatting—it's about the synthesis model being used. Some models lack sufficient instruction-following capability, causing their outputs to include extraneous information in both answers and questions. When using gpt-4o-mini or open-source models above 32B, regardless of the document format, the aforementioned issue occurs with very low probability. To address this, I modified the prompt to an XML version and modified the corresponding parse function, as can be seen in PR #137. This way, even small models like 4B can basically generate Q&A in the correct format, and there is no redundant output in the final results.

However, a small portion of data may be discarded due to parsing failures. This is a trade-off, as we generally believe that data with format errors could be harmful to the model even if used for training. If this is unacceptable, you can use models with stronger instruction-following capability as the synthesis model.

Thanks a lot for the thorough investigation and clear explanation — the XML-based prompt works well:-). Apologies for my earlier incorrect assumption about markdown being the cause