Offline Development Collection Test Suite

[pieterocp]

I have checked the following:

• I use the newest version of bruno.
• I've searched existing issues and found nothing related to my issue.

Describe the bug

When working on my fork of the whole bruno project to replicate a defect that my colleague is having, I checked out and started running the test suite as detailed in packages/bruno-tests/readme.md, but when I tried to run the Local tests, some of them work, but some of them are failing.

This is probably something where if we fixed this, and part of that would probably entail making the Local suite also run inside the github action, it might help accelerate any/all developers working on the project since they would be able to have the suite run much faster and can continue working when their internet dies or is slow.

I've done a bit of an investigation, but the answer will probably require some changes to both the collections and the testbench, as well as a bit of an extension, e.g. have a very barebones graphql implementation instead of using spacex's in Local/Prod tests. I've detailed a bit of work required to be done below

Failing Steps

Looking through each test case(s) folder by folder in the tree structure:

• auth/: ~ 75ms locally, ~ 4000 ms using Prod.
• echo/: This fails out the box since it's using echo-host locally, when the prod has this. With a slight modification (which can come in a pull request), 2586 ms using Prod, ~150ms if I remove some of the xml scenarios that seem to hang locally.
• graphql/: This calls spacex - 837 ms, will require a bit inside the Local to emulate graphql, but should be a lot faster.
• preview/: Both making network requests, ~ 500ms
• redirections/: ~ 41ms ~ 2016 ms using Prod
• scripting/: 12 failed, ~ 1400ms, ~11,494 ms using Prod but nearly all passed. One with a couple of changes needed. There's a setTimeout with 1000ms which could be shaved down and the axios-pre-req-script.bru uses a hardcoded testbench too.
• string interpolation/: 3 failed, 39ms vs ~1300ms using Prod, all passing.

In total with failures, getting the local experience of ~2567 ms vs ~19101 ms using Prod without major changes. Which once we swap out the timeout and network requests, it should be able to run around a second.

Below is the local run visualised using a helpful website to show which tests dominate the time taken in any test suite.

.bru file to reproduce the bug

No response

Screenshots/Live demo link

There is nothing to screenshot, this is a test suite speedup.

[sreelakshmi-bruno]

Hi, thanks for the detailed feedback. We'll look into it.

[pieterocp]

Did have a play with it myself, got it down to ~130 ms, feel free to pick and choose the bits folk like and dislike, some tests fail and will need adjustment but keeping all of the requests local (with the fallback of having the option of a call using "Prod") should hopefully speedup feedback cycles when developing new functionality.

Note this is a WIP but parking it here for now, can eventually make a tidier branch if someone else doesn't steal bits/ideas from it.

main...pieterocp:bruno:make-collection-tests-work-locally

[pieterocp]

Updated the above branch, does go down to 1 second, with the 3 inside the packages/bruno-tests/collection/auth/digest folder dominating the timing of the test running. These are ones where replicating this behaviour is a bit of a pain so one I'd prefer to defer for a while.

Requests:    96 passed, 7 failed, 103 total
Tests:       62 passed, 5 failed, 67 total
Assertions:  139 passed, 2 failed, 141 total
Ran all requests - 1095 ms

I think this is one for discussion on the discord.

Addition: Got it down to ~280ms again, will require adding a digest mock to replicate the results of https://httpbin.org.

[ramki-bruno]

@pieterocp
I can relate to the preference over having the tests offline and I ended up fixing some of those tests which were failing(when Local env is selected) in #4709.

Also had discussions internally and will be getting rid of the external dependencies(like Auth, GraphQL etc.).
If you think you can raise a PR, we would be glad to pick some of the changes you've already done(with proper author credits).