Schema update: Make 'Extra' object or a similar flexible object an "optional" property of 'Test.Steps.step' object

[apatelia]

The existing schema of Step object within steps array of Test object supports only two properties, name and status. While this may serve well as a minimal schema, it leaves out support for quite a few properties that modern test frameworks provide for a step.

For example, some nice to have and sometimes need to have properties like the ones listed below are left out.

• execution duration of a step
• type/category of a step (i.e. hook/test step/assertion etc.)
• location of the step within a test file
• any child steps (current flat "array of objects" structure cannot support any step hierarchy in reporting).

Proposal:
If Extra object which is already available to Results, Tool, Summary and Test objects, is also made an "optional" property of Step object, it will allow implementing additional properties for a step.

If this proposal is accepted and implemented, it will allow for a report like below.

"tests": [
  {
    "name": "Test that the correct product is added to the cart",
    "status": "passed",
    "duration": 1618,
    "start": 1725036983,
    "stop": 1725036984,
    "rawStatus": "passed",
    "tags": [],
    "type": "e2e",
    "filePath": "/projectDir/tests/my-test.spec.js",
    "retries": 0,
    "flaky": false,
    "steps": [
      {
        "name": "Before Hooks",
        "status": "passed",
        "extra": {
          "duration": 125,
          "category": "hook",
          "childSteps":[
            {
              "name": "BrowserContext"
              "status": "passed",
              "extra":{
                "category": "Framework API Call"
              }
            }
          ]
        }
      },
      {
        "name": "Click on a button",
        "status": "passed",
        "extra":{
          "duration": 265,
          "category": "Test Step",
          "childSteps":[
            {
              "name": "waitForElementToBeVisible",
              "status": "passed",
              "extra":{
                "duration": 65,
                "category": "Framework API Call"
              }
            },
            {
              "name": "waitForElementToBeClickable",
              "status": "passed",
              "extra":{
                "duration": 80,
                "category": "Framework API Call"
              }
            },
            {
              "name": "clickElement",
              "status": "passed",
              "extra":{
                "duration": 120,
                "category": "Framework API Call"
              }
            }
          ]
        }
      },
      {
        "name": "Another step but without any child steps",
        "status": "passed",
        "extra": {
          "duration": 150,
          "category": "Test Step",
          "childSteps":[]
        }
      },
      {
        "name": "After Hooks",
        "status": "passed",
        "extra": {
          "duration": 40,
          "category": "Test Step",
          "childSteps":[]
        }
      }
    ],
  }
]

As you can see, this enriches test step information with meaningful details.

Also, such enriched report can be transformed in a static HTML report which can provide test result details on par with a given framework's native report. Below is an example from Playwright's native report.

My proposed change can make an enormous difference in what details we can report as test result.

I would like to hear what the community thinks.

[apatelia]

If my proposal above is too long to read or understand, let me try to summarize it here.

Proposal: Make optional Extra object a property of Test Step object. This will allow for more detailed reports with specific details of test steps.

Currently, Extra is an optional property of every other object in schema, which does not have any downside. Similarly, adding Extra to Test Step will not have any downside, as far as I can think of.

Hope, this time around, the community will either engage in discussion.

[Ma11hewThomas]

Hello @apatelia,

Thank you for your suggestion. We agree that adding extra to steps is a practical enhancement, and we've updated the schema accordingly: https://ctrf.io/docs/schema/test#step-object.

We'd like to keep this discussion open as we explore further ways to improve support for steps and BDD-style frameworks with built-in CTRF properties.