Running a single test case of a parameterized test in IDE

[Youssef1313]

I didn't try to repro with VSTest, but at least with MTP, this appears to be problematic.

We get TestNodeUidListFilter from Test Explorer, but that doesn't have any info regarding which test case is actually being executed.

With VSTest, I think we send TestCase to VS and we serialize the test method arguments, which is already very problematic.

[Youssef1313]

@drognanar For MTP, is it guaranteed that discovery and execution happen in the same process?

[thomhurst]

@drognanar For MTP, is it guaranteed that discovery and execution happen in the same process?

Isn't that up to the framework? If I get a discovery request, I do discovery. If I get a run request, I do discovery and execution. (because without discovery I won't have any test objects populated into memory)

[Youssef1313]

@thomhurst My question was more like for the discovery request and execution request. But I'm now thinking that even they are done in the same process, that's not a behavior that a test framework should depend on and it's up to the frameworks how they can recognize the test cases. I'm moving the area to be on MSTest side actually.

[Evangelink]

It is definitely up to the test framework!

It could be interesting to brainstorm across test frameworks as we all face a similar problem: with .NET there is no good way to consistently uniquely identify a "data" without asking the user.

Let's assume the user is creating a "data source" returning a Person object. Now, we need to produce a unique string identifier for a given instance of this object. How do we do? Serialization? It's possible that the user data is not serializable or uses a different serialization mechanism that the one we support. Another solution is to have an index based lookup, problem is that runtime said that we cannot rely on reflection returning the data in the same order AND if user provides randomic data, the index based is screwed. Based off ALL my experimentations, the only way I found was to ask user to provide a unique ID linked to each data returned that would help to uniquely identify this data across multiple runs.

[nohwnd]

I agree and I think the problem needs to left to the data provider to "solve".

DataRow data provider can reasonably assume that the attribute data can be serialized to some string. And matched later, even if the test case moves in order.

Database source can ask the user to provider unique key for data, which they hopefully have.

The biggest problem are just normal objects that are created in process via DynamicData source, there we don't have a good way to relate instance created in one process to instance that was created in subsequent run, in a totally new process. We have only serialization which assumes that the object can be fully and uniquely serialized. Or that the object can provide some stable unique identifier, which such objects often don't have.

---

Then there is one more aspect which is preferring a replay of the test, or prefer re-run of the test.

For example if I make an edit in code like this:

[DataRow("a")]
public void TestMethod1

[DataRow("b")]
public void TestMethod1

And then tell TE to run TestMethod("a") should it:

• deserialize the data and replay it, even though no such test case exist anymore in code
• fail because we cannot find test in codebase

Imho the first behavior is what XUnit does because it serializes whole test case to data, and uses that as source for the test run.
The second behavior would be MSTest because the serialized data are used as ID for the test to match it, and it is not there anymore.

For me the second behavior is more accurate, because I get more up-to-date data.

---

Another aspect is how aware the IDE should be about the fact that the data under the tests can change. When we say RunAll tests, TE is currently sending all test cases to be run. Instead of sending a command that says "run all tests in this assembly", similar approach can be applied to running tests in class or on method.

To me this is not great, because I as user want to know if the system is working against the current set of data, and current set of tests, and not against set of tests I discovered 20 minutes ago.