Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

@Vdaleke I am in principle OK with this, but I would like to understand which problem it solves. In what examples do we not want to escape underscores?

In what examples do we not want to escape underscores?

I gave an explanation in #134. This solves the problem when using docling to prepare documents for RAG, where text search of chunks is used among other things. When the document contains underscores, but the user query does not, the corresponding chunks are not found.

The output without escaping underscores would be an illegal markdown, e.g. having non-closed _ which have a semantic meaning.

@Vdaleke are you sure you actually need Markdown for your downstream application? Alternatively, you can use the option strict_text=True which is producing plain text.

note: in case strict_text=True is also escaping underscores, I would consider that a bug which should be fixed.

@Vdaleke are you sure you actually need Markdown for your downstream application? Alternatively, you can use the option strict_text=True which is producing plain text.

Hello @dolfim-ibm !

In our case we want to use docling as data preparation for RAG. The data should be used both during search and during LLM context processing. LLM interacts better with Markdown, since it was learned largely from it (depends on the LLM I think, but in our case we are sure of it).

So we want to convert arbitrary files to Markdown for search and processing by the model. For model processing, underscores don't seem to be a problem. But for search, particularly for text search (as part of hybrid search) text is just text and if the word A_B_C occurs in the text, it is not the same text as in a user query containing A_B_C.

So we want to use markdown for model processing, but also markdown without escaping for search. Using strict text for lookup may be worse, as we may lose the importance of headers and table structure, which can be important when processing for vector search (again as part of hybrid search).

We realize that such a flag will generate incorrect syntax, but since docling is positioned as a tool for preparing data for LLM too, we hope that adding such a flag will not be a problem.