fix: fix single newline handling in MD backend

[vagenas]

Resolves #822.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[vagenas]

For reference, Marko (the Markdown parsing lib employed under the hood) handles new lines as follows:

• captures a BlankLine in case of multiple (more than one) successive new lines,
• otherwise, in case of single new line, captures a LineBreak

[ceberam]

While this PR fixes the problem described in this issue, it may at the same time disregard hard line breaks, which could be important for the document author.

For instance, consider the following text (rendered here as text):

First line with no space after.
The first line continues.

First line with two spaces after.  
And the next line.

According to the markdown syntax and, for instance, GitHub Flavored Markdown the first block represents a single line while the second block should have 2 lines:

First line with no space after. The first line continues.

First line with two spaces after.
And the next line.

However this docling fix puts the second block as a single line and thus remove the hard line break. Running docling --from md --to md on the document above would result in:

First line with no space after. The first line continues.

First line with two spaces after. And the next line.

[vagenas]

@ceberam I would anyway regard the hard line break only as a visual wrap / formatting, in which case it should not correspond to a new DoclingDocument text item / paragraph.

To make it clear, in the discussed "rendering" example:

First line with no space after. The first line continues.

First line with two spaces after.
And the next line.

these are conceptually still only two paragraphs.

In this sense, the produced MD export is as expected (given that DoclingDocument will capture the paragraphs):

First line with no space after. The first line continues.

First line with two spaces after. And the next line.