CVE Record Format Rules

[ccoffin]

Are there any rules or principles that we should define for the CVE Record Format? If appropriate, we can open a separate discussion for the individual ideas.

Some ideas/examples might include:

• The CVE Record Format JSON should NOT be considered a "presentation" format.
• The CVE Record Format should avoid including redundant information wherever possible, e.g., CVSS data currently
• The CVE Record Format input schema should avoid containerized data formats, e.g., Purl

[alilleybrinker]

Just as a point of order, what do you mean by "containerized data formats"?

[ccoffin]

Maybe there is a better word for this, please include and update if you are aware. I think this term was used by someone to describe Purl at one point. The basic idea being that a Purl tries to combine a lot of different pieces of data into a single value. Also applies to CVSS vector strings and CPEs. Should the CVE Record format accept these things on input or should it only accept the discrete parts, and then generate the combined values on output where needed?

[darakian]

I think this term was used by someone to describe Purl at one point. The basic idea being that a Purl tries to combine a lot of different pieces of data into a single value.

That's probably me. Not a formal definition, but purl is one spec with many "types" which I think of as sub-specs as they all have their own rules. By adopting the "containerized data format" purl we implicitly adopt all the sub-specs which are themselves "contained" (unless we explicitly disallow some, see the discussion about disallowing some purl types etc... etc... )

This leads to the state where we could represent the package available at
https://files.pythonhosted.org/packages/c6/af/77b403926025dc6f7fd7b31256394d643469418965eb528eab45d0505358/django-5.2.3.tar.gz
as
pkg:pypi/[email protected]
or as a collection url pypi.org with a lookup key of django and a version of 5.2.3 or maybe the collection url is pythonhosted.org or maybe both work, but I think the idea is clear. One could also imagine a less ambiguous take on the collection url idea 👀

[andrewpollock]

https://github.com/ossf/osv-schema/blob/main/GUIDING_PRINCIPLES.md may be useful for comparison purposes?

[alilleybrinker]

Related, I highly commend anything we can do to be clear about non-goals of the CVE Record Format. Non-goals are fantastic, they help to exclude future possibilities in a way that keeps the project focused and constrains scope.

[ccoffin]

@andrewpollock Thank you for pointing this out and i think it will be super useful as a starting point! My thinking now is that a guiding principles document should be our next step.

@alilleybrinker I agree completely with the inclusion of the non-goals as well!

[ccoffin]

Guiding Principles draft can be found here:
https://docs.google.com/document/d/1FFA4V4TIOg1_Jo3A1ILEKvTE0iefDWAduGrA0BChUXQ/edit?tab=t.0

[alilleybrinker]

One goal I think the project should have:

Prefer structured information.

This means both that CNAs should prefer to use structured fields, rather than placing things in freeform text fields, and that the QWG should work to codify structures for common patterns and then work with CNAs to adopt those structured patterns as they arive.

For example, we have the ongoing discussions on introducing new versionType values which would codify common patterns for expressing versions. As new versionTypes are added to the specification, CVE should work to get CNAs to adopt them.

[ElectricNroff]

I'd like to explore moving away from a "record format" in favor of a system in which producers, as well as their customers and other interested parties, can submit any information that they feel is helpful, and let the CVE Program use the best available AI technologies to allow consumers to ask questions about the data, or extract all of the data in a format that the consumer specifies. This, of course, does not mean that the record format would be abruptly abandoned, but may mean that all efforts to improve the record format could be deprioritized.

Often, there's a large series of questions I'd like to ask about a specific vulnerability, and I suspect that AI-generated answers will usually be good enough for my purposes, will often be better answers than are realistically achievable by having a producer's human staff type the answers into individual "record format" fields, and will be available to me much more quickly than if I have to wait for "record format" fields to be standardized.

Just a few examples that I've thought about today:

Is it sufficient to update to the latest version in the way that is normal for this product (e.g., it's not a situation where the latest version is 1.3, but the system is in an insecure state if 1.2 was ever installed, and the Supplier is not providing a programmatic way to fix this)?

I don't know anything about this product or its use cases other than my company has it installed on a nonzero number of machines. What's the probability that we can be successfully attacked?

It's expensive for us to update to the latest version. How do I explore whether we can be successfully attacked?

What is the typical Total Cost of Ownership of a remediation?

Is it plausible that my company has the vulnerability even though the named products are not present? For example, was the official name of the product changed? Are there forks? Is it possible that a file or function from this product was re-used elsewhere, either because a human copied it or because it is in LLM training data?

Can I get the complete set of file checksums that represent all plausible instances of the vulnerable code? How would we tell whether this is complete?

Is the vulnerable part of the code different depending on how I installed it (e.g., NPM versus GitHub or .msi versus .zip)?

I see a CWE mentioned, but is this weakness the root cause?

Is there any workaround at all, even drastic ones in which I delete a file and accept that not all of the product's functionality will still be present?

Has anyone observed that something broke after an update? (the product's own functionality or any interoperability)?

I don't want to update the product that was affected. Is there a replacement product?

Is it plausible that I have the fixed code but a vulnerable version number?

Is this CVSS score considered useful for prioritization? Why?

Was any information keyed in manually by the data producer? What is the likelihood that the information is correct?

Is it likely that hundreds of similar reports about this product, with the same importance, could have been submitted but weren't?

Is it a mandatory update in the context of a supplier/consumer relationship?

Will this automatically be updated in some or all cases? How soon?

Can automated analysis of the patch tell me whether it's likely to be an incomplete fix?

[alilleybrinker]

I'm going to break my response into two parts.

First, while many of the questions you raised are interesting or important questions for CVE consumers to care about, many are 1) not answerable by LLMs, 2) not answerable by a CNA, or 3) not answerable at all. Even if we pursued your proposed path of deprecating the CVE Record Format in favor of a freeform text protocol that could be mined with LLMs by consumers, this would not result in an ecosystem which can answer the questions you're posing.

I've provided a detailed breakdown of these limitations, question-by-question, at the bottom of my reply.

Second, given this, pursuing deprecation of the CVE Record Format in favor of freeform text to be interpreted by LLMs would be an enormous mistake.

The CVE Record Format is the ground truth for an ecosystem of consumers trying to manage vulnerabilities for production systems. It is a critical piece of security infrastructure, and quality failures in this ecosystem have real effects downstream: mitigations not taken or taken too slowly, vulnerabilities unnoticed, enormous cost spent by consumers to determine what vulnerabilities apply to them, and more. Avoiding and reducing those harms is why the Quality Working Group exists.

Since CVE's inception, improvements in the quality of CVE Records have been driven by two forces: improved ability of CVE Records or downstream enrichment forms to express the necessary information for consumers, and higher quality input into the CVE system by CNAs. The CVE Record Format, as it stands today, is a central lever for quality; it sets the minimum floor for what an acceptable CVE Record is, ensuring downstream consumers can rely on having the data they need to make decisions and take action.

Abandoning that lever, and thus any minimum quality floor, by deprecating the Record Format, would be a quality own-goal with substantial downstream effects. While many CNAs do their best to produce and share quality data, the backstop of CVE's own minimum requirements, enforced by the Record Format, ensure that budget changes, shifting priorities, or staff churn at CNAs do not lead to substantial quality drops. If anything, the priority of the QWG of late has been (and ought to continue to be) to drive the Record Format to raise the quality floor for CVE Records, to serve the mission of empowering vulnerability managers in their critical operations.

---

Breakdown analysis of the questions posed above.

For the questions, I've grouped them into categories, and placed my analyses of them as sub-bullets.

• Counterfactual Questions
  • Is it likely that hundreds of similar reports about this product, with the same importance, could have been submitted but weren't?
    • No LLM can answer counterfactual questions like this. The information necessary to credibly estimate likelihood like this doesn't exist. No human could produce a credible answer here either. An LLM will produce an answer, but it will not be based in reality and cannot be trusted.
• Data Quality Questions
  • Was any information keyed in manually by the data producer? What is the likelihood that the information is correct?
    • No LLM can tell you with certainty what information was keyed in manually, nor can they estimate likelihood of correctness. An LLM will produce an answer, but that answer will not be based in reality and cannot be trusted.
• "Am I Affected?" Questions
  • Is it plausible that my company has the vulnerability even though the named products are not present? For example, was the official name of the product changed? Are there forks? Is it possible that a file or function from this product was re-used elsewhere, either because a human copied it or because it is in LLM training data?
    • While an LLM, equipped with MCP hookups to relevant data sources, may be able to gather some of the factual information to attempt an answer here, it's not clear that LLMs would currently perform better than existing Software Composition Analysis tools. It's also worth noting that in order to do this, the LLM would need information like where to find the official source of the package (via software identity information, which we've recently discussed in the QWG). If the Record Format is eliminated, and there is thus no guarantee that information would be present in a CVE Record, then there is no guarantee consumers would be able to answer this question at all, even if an LLM could do it.
  • Can I get the complete set of file checksums that represent all plausible instances of the vulnerable code? How would we tell whether this is complete?
    • An LLM, equipped with MCP hookups to relevant data sources, probably could gather this information. It would need to be checked, and should not be relied upon without checking. It's also not clear how an LLM + checking workflow for this question would be better or faster than simply gathering the information yourself, since the steps for checking vs. getting the information would be the same.
  • Is the vulnerable part of the code different depending on how I installed it (e.g., NPM versus GitHub or .msi versus .zip)?
    • An LLM may be able to attempt an answer here, if the necessary level of detail is present in the CVE Record, but if the CVE Record is now a freeform text field, there's no guarantee that prerequisite information would be present.
  • Is it plausible that I have the fixed code but a vulnerable version number?
    • Same as the last question.
• Remediation Questions
  • Is it sufficient to update to the latest version in the way that is normal for this product (e.g., it's not a situation where the latest version is 1.3, but the system is in an insecure state if 1.2 was ever installed, and the Supplier is not providing a programmatic way to fix this)?
    • Same as the last question.
  • What is the typical Total Cost of Ownership of a remediation?
    • An LLM could probably answer this in a general case, or at least attempt an answer (which may be of uncertain credibility).
  • Is there any workaround at all, even drastic ones in which I delete a file and accept that not all of the product's functionality will still be present?
    • If there's no guidance about this in the CVE Record, there's no way an LLM could produce a credible answer.
  • Has anyone observed that something broke after an update? (the product's own functionality or any interoperability)?
    • An LLM with MCP connections to gather information from social media sites, forums, and other community data sources may be able to answer this, but it's uncertain.
  • I don't want to update the product that was affected. Is there a replacement product?
    • An LLM may be able to answer this.
  • Is this CVSS score considered useful for prioritization? Why?
    • I think what you're getting at here is that some CVSS scores are produced by CNAs whose positioning in their value chain is so early that the CNA is unable to produce a sensible-enough CVSS score for most of their downstream, as they have to make worst-case assumptions which may not apply to all users. This may be answerable by an LLM in some cases.
  • Will this automatically be updated in some or all cases? How soon?
    • An LLM may be able to answer this.
  • Can automated analysis of the patch tell me whether it's likely to be an incomplete fix?
    • A specialized LLM for code analysis may be able to attempt an answer to this, though it would need to be validated.
• Exploitation Questions
  • I don't know anything about this product or its use cases other than my company has it installed on a nonzero number of machines. What's the probability that we can be successfully attacked?
    • This kind of architectural analysis and likelihood estimation, without the ability to validate the answer in any meaningful way, is exactly the kind of quicksand question you should not ask an LLM.
  • It's expensive for us to update to the latest version. How do I explore whether we can be successfully attacked?
    • An LLM can probably answer this question.
• Weakness / Cause Questions
  • I see a CWE mentioned, but is this weakness the root cause?
    • An LLM may be able to attempt an answer based on whether some CWEs tend to be "root" causes (often vulnerabilities may be multi-causal, so "root cause" is itself an often false and limiting frame).
• Legal Questions
  • Is it a mandatory update in the context of a supplier/consumer relationship?
    • Do not replace a lawyer with an LLM.

I'll make one additional note here: when asking LLMs to produce information, given the propensity LLMs have for hallucination (based in the fact that they will generally produce a response even if they lack any of the actual information necessary to do so), you should only ask for information for which you have some ability to validate correctness. For example, asking an LLM to generate source code is a reasonable use case, because you can validate the function of the code through code review, testing, static analysis, and more. Similarly, asking an LLM to generate factual information about a field you understand can be reasonable, because you as an expert can check the LLM's information for accuracy.

Asking an LLM to answer factual questions, where it lacks the necessary information to do so, and where there is no capacity for you to validate the answers, is asking to be misled.

[andrewpollock]

I'd like to explore moving away from a "record format" in favor of a system in which producers, as well as their customers and other interested parties, can submit any information that they feel is helpful, and let the CVE Program use the best available AI technologies to allow consumers to ask questions about the data, or extract all of the data in a format that the consumer specifies. This, of course, does not mean that the record format would be abruptly abandoned, but may mean that all efforts to improve the record format could be deprioritized.

This is an interesting vision, and somewhat orthogonal to data structures.

I would argue that it's a prerequisite for the very interesting use cases you propose to have broadly available, machine readable, accurate and complete vulnerability metadata before interested parties could attempt to answer the sorts of questions you posit.

[ElectricNroff]

I'm going to break my response into two parts.

I don't believe it's true that the CVE Record Format is a central lever for quality. There may be fixed resource allocations on the producer side in many cases. For example, if we specify that the minimum is, in some way, "more" then producers may react in several ways:

• a lower level of attention to detail in other parts of the CVE Record
• assigning CVE Record production to the most junior staff
• assigning CVE IDs to vulnerabilities at a lower rate, e.g., setting a higher bar for what enters their CVE process
• etc.

Also, I have a different perspective on the "Breakdown analysis of the questions" - for example, this one seemed to be the most adamant that it was impossible:

• Item: Is it likely that hundreds of similar reports about this product, with the same importance, could have been submitted but weren't?

• Response: No LLM can answer counterfactual questions like this. The information necessary to credibly estimate likelihood like this doesn't exist.

For that question, I feel that I receive good answers from LLMs based on frontier models. For example, in the case of CVE-2024-11706, the answer was ultimately "Yes" and seemed to be based on the LLM using training data and making web searches, covering:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1923767#c4 is incorrect because that library code is part of the Firefox executable program
• there are tens of thousands of unique crash signatures every day for Firefox
• only a fraction are evaluated during each Firefox release cycle (where each set of CVE Records from the Mozilla CNA is established)
• the decision on whether to assign a CVE ID is based on inspection of the crash signature and stack trace, but without publicly documented decision criteria
• a mention of two other bug reports for the same source file that were fixed without a CVE ID assignment ever occurring
• producing a CVE Record involves additional effort beyond fixing the bug
• etc.

It seems to me that applicable information does exist today, and more applicable information might exist if there were some type of process for CNAs to submit that information. In this instance, the question is largely about "is my level of risk in using this Supplier's products well covered by the CVE List" versus "would I benefit from both the CVE List data and a different data set that attempted to explain each CNA's triage process." Whether structured data or unstructured data would better serve LLM applications is a separate question.

The same principle might apply to other types of data that is often omitted from CVE Records today, and might or might not have a natural structure that could be re-used across CNAs of different types, e.g., the remediation information that was often mentioned in the CVE consumer survey responses.

[alilleybrinker]

Matt, when I say "central lever for quality," I mean it is the central mechanism with the CVE project has for influencing the quality floor of data submitted by CNAs. Yes, there are limits on the CNA side which may influence their own quality ceiling, but that's a different issue.

On the second part of your response, the argument that an LLM can credibly answer counterfactuals like the one you posed as an example, I am now unclear on your position. It seems you're arguing that there should be a structured means to elicit additional information from CNAs, like what their decision criteria are, to feed to LLMs. Is that correct? If so, that's a different and wholly inconsistent position from your original one.

I also do not find your remembered example convincing for the idea that an LLM can be trusted to provide reliable answers to counterfactuals.

[ElectricNroff]

I don't agree that the CVE Record Format is the central mechanism for influencing quality.

My responses have been consistently about 'explore moving away from a "record format"' and 'submit any information that they feel is helpful,' without constraining implementation. You then stated 'your proposed path of deprecating the CVE Record Format in favor of a freeform text protocol' but I don't feel that that's equivalent.

Although it's possible that a producer would rewrite information into freeform text and that an audience exists for which this is helpful, I feel it's more likely that producers would directly pass along what they already have, e.g., how they exchange information within engineering teams or how they compose information for their customers. Or, possibly they already use a different specification:

https://docs.oasis-open.org/csaf/csaf/v2.0/os/csaf-v2.0-os.md has

  * `/vulnerabilities[]/notes`
    > Provides details about the vulnerability.

(for "Security")

  * `/document/notes` with at least one item which has a `category` of
    `description`, `details`, `general` or `summary`
    > Reasoning: Without at least one note item which contains information
      about the "issue" which is the topic of the advisory it is useless.

(for "Informational")

https://ossf.github.io/osv-schema/ has

The details field gives additional English textual details about the
vulnerability.

https://gcve.eu/bcp/gcve-bcp-03/ has

GCVE-BCP-03 does not enforce a specific JSON format for vulnerability
publication.

Finally, I believe that seeking answers to counterfactuals is a fundamental part of the human condition, that neither other humans nor any types of AI systems always have reliable answers to counterfactuals, but that answers still are important on the pathway to choosing a course of action.

[alilleybrinker]

Arguing that we should consider "moving away from" the Record Format and that CNAs could "submit any information that they feel is helpful" is absolutely an argument to abandon the record format in favor of freeform text. In fact in the very examples you're giving, information is provided in freeform text. Arguing that your argument is not somehow an argument against structured data and in favor of freeform text is disingenuous.

Your point that producers might stick in information in their own ad hoc structure, perhaps reusing structure from another format, is separate from the point that without a Record Format, CVE would not enforce substantial structure.

CVE used to be less structured, and has become more structured over time because that's what works well for raising the quality floor, that's what enables greater automation, and that's what enables greater integration with other systems. It is the central quality lever that CVE has over CVE Records, and abandoning it would be a colossal mistake.

---

Regarding counterfactuals: asking an LLM to answer counterfactuals for you is asking to be lied to. LLMs will almost always produce an answer, even if that answer is entirely divorced from reality. In the case of counterfactuals, an LLM cannot produce a valid answer, because you are asking for an alternate history of what would have happened had some prior event gone differently. Humans can't answer that, and neither can LLM's. The fact that an LLM will produce an answer in response to such a question does not make the answer credible, and you should neither ask such questions of an LLM nor trust such an answer when given.