JSON BOM serialization "trims" whitespace from DPKG license text (XML does not)

[gdgib]

Background

On debian & ubuntu systems the dpkg copyright files are (in modern times, thank goodness) intended to be machine readable according to this spec. The CycloneDX linux generator on Ubuntu faithfully replicates the text of the copyright file into components/[]/licenses/[]/license/text/content as one might expect.

According to the JSON AbstractBomGenerator.java line 68 it would appear ALL STRINGS, when serialized to JSON, are serialized with TrimStringSerialize which not only trims whitespace but removes it similar to how an HTML processor might.

The XML AbstractBomXmlGenerator.java does not remove whitespace, which would seem to be the correct behavior.

Bug

1. I would argue that not all strings in BOMs should have their whitespace remove & coalesced when converted to JSON. Copyright and license file text in particular is a good example where replicating the original is probably best.
2. I think the JSON & XML formats of the same BOM should contain identical data, this includes text/strings and their whitespace.

History

• @stevespringett implemented TrimStringSerializer in 0fab7fb#diff-9f5ef24a21ed10eaae782875e4efc0cd90cec8b7f598bee89d096f50431db5cc.
• That was in turn a cleanup of his earlier work in 2e5ace4#diff-1ba2a4fedacc21e8c4b7d22713a15269ac6f187cd10332015d28de1c341e0465.

Without test cases to accompany either of those changes, it's hard for me to understand why they were made. The history of July 9th 2020 doesn't show PRs or groups of commits that seem to help me understand either. The problem is that this behavior was obviously desired, but I'm not clear why or how it would be helpful.

Potential Solutions

1. I can use the XML formatted output (at least for now), which does not appear to mangle the structure of the dpkg copyright files when converting them to license text.
2. I'd be happy to submit a PR with appropriate fixes, but I'm really hoping @stevespringett might somehow remember the reason behind this before I go writing code that could break something important as per my note above about at least some part of this being desired behavior.

Personal Note

This is my first comment to this project, and I look forward to working with you if possible. I have both personal and professional interest in this area, and I hope to both integrate with and contribute to CycloneDX.

[stevespringett]

TrimStringSerializer is used in JSON because JSON does not support the concept of CDATA. There were too many instances where people (or various Java implementations) would use the library without properly performing input validation before sending data to create the BOM. TrimStringSerializer exists to reduce the amount of errors people were encountering while serializing JSON.

For license text, the recommended way to include full license text that would result in identical JSON and XML is to base64 encode it which is supported by both formats. Properties are encoding and content-type which are both optional and supported by XML and JSON.

[ben-c8y]

Not sure that transforming whitespace (but ignoring characters like ? < > ) really counts of input validation?

Unless the API documents that it is making a transformation, I think it should pass through the text unchanged.

Also passing it through unchanged is much more useful behaviour, since it gives the user of the API the choice of behaviour, rather than blocking them in.

Use of base64 is pretty unusual inside a json file, and certainly complicates understanding the contents when looked at by a human e.g. for debugging, so it should be a choice not something enforced.

[nscuro]

What's the desired way forward here? Based on @stevespringett's response this seems to be working as designed.

[benken-parasoft]

I provided a bit of detail and analysis in #394 but I have some additional feedback.

Concerning base64, this is a universally accepted way to encode binary data in both XML and JSON documents, like if you needed to include the contents of a pdf document. In contrast, license and copyright text are indeed text. They should be correctly encoded as a JSON String without any characters arbitrarily stripped out or modified or replaced. A standards compliant JSON generator/parser will correctly encode/decode the textual data, preserving the original text.

Concerning CDATA, whether license and copyright text is encoded as CDATA or with characters escaped (with entity references) is irrelevant, a bit of a stylistic choice. A standards compliant XML generator/parser will correctly encode/decode the textual data, preserving the original text.