I’ve noticed that one of our unit test seems to “fail but pass” strangely.
After investigating: It looks as if the “Total” count of UnitTestResult - is not taking nested results into account, screenshot:
There are total of 40 sections in the TRX file that was loaded.
11 of the 40 are nested inside other UnitTestResult sections
As these aren’t counted for the “Total” - the Math doesn’t work:
29 total is not equal to 36 passed +4 failed
I’m using the following expression in order to evaluate whether or not I should fail the unit test job:
The conditional ‘$Utils.GetNumber($Build.Metrics.UnitTests.Passed$)$’ GreaterThanOrEqual ‘$Utils.GetNumber($Build.Metrics.UnitTests.CaseCount$)$’ has been evaluated to true.
This expands to:
[ If [‘36’ GreaterThanOrEqual ‘29’]]
Also (a different but possibly related issue): strangely enough:
Does not equal zero -
Both stages finish staged as “success”
The 2nd stage is staged as “success” because of the Metrics.UnitTests expression that was shown above was evaluated to “True” but the stage gate condition after this stage - fails the build (as I would expect)
The thing is, although the build is shown in the UI as failed:
The final status reported back to our GitHub server is “Successful”:
The “Update Github Status” type event handler is configured this way:
I’d appreciate your input about both the unit test count issue and this failed but passed issue.
We get the total count from the
<ResultSummary> element in the trx file, rather that counting the unit tests. So it seems that there maybe something different with how MSTest it writing the totals in your case. Which version of MSTest are you running? If you send us a copy of the trx file (either by direct message to me her to to our support email) we’ll investigate what is happening with the counts tomorrow.
Regarding the second issue, this outcome would be expected as the stage gate expression fails the build (as there are failures) but not the stage which is already in a successful state. The stage gate runs after the stage result is complete.
Thank you Dave!
I’ve sent you the .trx file.
We’re using dotnet 6.0.202 to run the test in this case, with a command line that looks like this:
Execute Program [ cmd.exe /c dotnet test C:\…\Service.Test.csproj /p:IsTestProject=true --logger “trx;LogFileName=C:\CCIAgentShare\Ws\151043\Source\DevOps\Deployment\UnitTesting\Service.Test.26052022_145303.trx” ]
About the 2nd issue: only now I see that the event handler is operating in stage scope rather than in a build scope; So I understand why the passed/failed discrepancy occurred.
Could you tell me why the event handler works with a stage scope and not with the scope of the full build?
(Just asking out of interest)
No reason, that’s just how we it’s set up so that a stage is selected rather than a build. I guess we could add an option post at the end of the build - I’ll look into it.
What type of unit tests cause the nested UnitTestResult sections to be created? Are they parameterised tests using the
Looks like parameterized tests indeed, but using a regular old
We’ve been looking at this today and there some oddities about the content of your trx file.
The counters in the
ResultSummary appear to be counting the outer
UnitTestResult as well as the
UnitTestResult nodes under
InnerResults. This means that it counts 5 tests for the test you posted in your last comment. We think that it should be 4 and that the total for the whole file should be 37 not 40.
Also when we run
dotnet test on a .Net 6 project with parameterised tests defined in a similar way to yours, all test results are listed at the same level - there are no
We’re currently planning to ignore the
ResultSummary counters and count the results as we iterate through the file. We hope to have a patch ready by the beginning of next week.
See Extra test cases for DataRow methods · Issue #488 · microsoft/testfx · GitHub. Which version of the
MSTest.TestAdapter NuGet package is your project using?
MSTest.TestAdapter version for this test project is 2.1.1
The MSTest results parser in v220.127.116.112 now counts UnitTestResult sections.