# Inconclusive Tests
**URL:** https://www.finalbuilder.com/forums/t/inconclusive-tests/5464
**Category:** Discussion
**Created:** [March 11, 2016, 11:57am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464 "2016-03-11T11:57:17Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 11, 2016, 11:57am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/1 "2016-03-11T11:57:17Z")
It seems that inconclusive tests cause the test action to fail even if they are shelved?
Test output:
Test Run Inconclusive.
Passed 261
Inconclusive 2
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 11, 2016, 1:34am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/2 "2016-03-11T01:34:20Z")
seems to happen only if the option “fail action if any tests error” is checked. so you mean an inconclusive test is an error?
what if a “real” error happens or what is meant with “error”?
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 11, 2016, 1:40am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/3 "2016-03-11T01:40:50Z")
btw: “new failures” also shows our 8 inconclusives … slightly confusing for me
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 14, 2016, 6:24am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/4 "2016-03-14T06:24:25Z")
Hi Christian,
Is it MSTest that you are using? We’re having trouble reproducing an inconclusive test with MSTest. The command Assert.Inconclusive() produces an AssertInconclusiveException with a Failed outcome in our tests. Can you provide us with some example code which produces an inconclusive outcome?
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 14, 2016, 6:25am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/5 "2016-03-14T06:25:16Z")
Can you also let us know which version of MSTest you are using?
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 23, 2016, 10:28am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/6 "2016-03-23T10:28:33Z")
we are still using mstest boundled with vs2010. meanwhile we unshelved all tests.
if i run a build now, all tests are green and the 10 inconsluive are also covered. the test stage is green, too. so “shelving” has nothing to do with this.
if i activate “Fail action if any tests error” the test stage fails. so question for me is: what means “error”?
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 23, 2016, 12:04pm UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/7 "2016-03-23T12:04:29Z")
Hi Christian,
Error tests should only include tests which raise an exception. Inconclusive tests should not be classed as an error. They are interpreted as a failure in the code. It’s possible that MSTest is returning an error code along with an inconclusive state, which would cause this result.
We are trying to reproduce an inconclusive result so that we work out what is happening here, but cannot find an working example. With Nunit we would use Assert.Inconclusive(), but this throws an exception with MSTest. If you could share an example inconclusive test, then we will then attempt to reproduce and see what we can do to fix the code to interpret an inconclusive test correctly.
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 23, 2016, 1:18am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/8 "2016-03-23T01:18:27Z")
we simply use Assert.Inconclusive(“message”), nothing special. however, these tests always throw a “AssertInconclusiveException” (as described in the documentation). the .TRX files contain a separate entry “inconclusive”.
So i am not sure how you measure the test results internally.
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 24, 2016, 2:57am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/9 "2016-03-24T02:57:50Z")
Hi Christian,
When we run an MSTest.VisualStudio.2010 action on the following test:
`[TestMethod]`
`public` `void` `InconclusiveTest()`
`{`
` ``Assert.Inconclusive(``“this is inconclusive”``);`
`}`
we get the following result in the trx file:
`<``UnitTestResult` `executionId``=``“e2c06fa8-41ca-4ba6-ba65-27e757658369”` `testId``=``“6817d349-ecf5-36c4-231a-020d18dc26b4”` `testName``=``“InconclusiveTest”` `computerName``=``“SPARKY”` `duration``=``“00:00:00.0142090”` `startTime``=``“2016-03-24T11:38:17.1627687+11:00”` `endTime``=``“2016-03-24T11:38:17.1783835+11:00”` `testType``=``“13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b”` `outcome``=``“Failed”` `testListId``=``“8c84fa94-04c1-424b-9868-57a2d4851a1d”` `relativeResultsDirectory``=``“e2c06fa8-41ca-4ba6-ba65-27e757658369”``>`
` ``<``Output``>`
` ``<``ErrorInfo``>`
` ``<``Message``>Test method BuildTest.Tests.InconclusiveTest threw exception: `
`Microsoft.VisualStudio.TestTools.UnitTesting.AssertInconclusiveException: Assert.Inconclusive failed. this is inconclusive``Message``>`
` ``<``StackTrace``> at BuildTest.Tests.InconclusiveTest() in d:\Continua\MasterAgent\Ws\1522\Source\BuildTest\Calc\Tests.cs:line 74`
```StackTrace``>`
` ````ErrorInfo``>`
` ````Output``>`
` ````UnitTestResult``>`
This is interpreted by ContinuaCI as a failure and shown under the Failed indicator of the Unit Tests view. No tests are shown under the Inconclusive indicator
The MSTest Action indicates Success in the build log however that Stage fails due the the default Stage Gate condition of "$Stage.Metrics.UnitTests.Failed$ Equals 0"
Are you sure that it is not your stage gate which is failing rather than the action? Are you seeing the tests marked as Inconclusive in ContinuaCI at all or just as Failed? Can you send a copy of your trx file to [support@finalbuilder.com](mailto:support@finalbuilder.com)?
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 24, 2016, 9:33am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/10 "2016-03-24T09:33:15Z")
its not the stage gate. in my build log the action “MSTest” is marked “Failed” and not “Success”.
the stage gate is:
$Stage.Metrics.UnitTests.Failed$ \<= 0
$Stage.Metrics.UnitTests.Error$ \<= 0
i agree that the single unit test result is the same as you posted. however, at the top, we can see that there is none failed and 2 inconclusive:
[code] \ \ \[/code]
continua is showing everything correct (2 inconclusive, none failed). just the action fails if i mark “any tests error”. seems a little bit inconsistent by mstest.
for sure, i could switch off “if any tests error” and everything would be fine. just want to make sure that the testing stage isn’t skipping any “errors” by accident.
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 24, 2016, 9:47am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/11 "2016-03-24T09:47:08Z")
see here: [http://picpaste.de/pics/Inconclusive-oY5Srp0C.1458805625.png](http://picpaste.de/pics/Inconclusive-oY5Srp0C.1458805625.png)
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 25, 2016, 1:32am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/12 "2016-03-25T01:32:06Z")
Hi Christian,
Thanks for this additional information. We’ve now worked out what is happening. MSTest is returning an exit code of 1 when there are inconclusive tests.
The ContinuaCI MSTest action returns an error outcome in the following cases:
1. There is a error count in the result summary
2. There is a non-zero exit code and the failed count is zero
MSTest returns an exit code of 1 for both errors and failure - we now see that this is also for an inconclusive tests. So we need to change case 2 to exclude the inconclusive tests (and possibly others). We’ll fix that for the next version.
The reason we were not seeing this in our environment, is because MSTest was returning a failure count in the ResultSummary for an inconclusive test e.g.
`<``ResultSummary` `outcome``=``“Failed”``>`
` ``<``Counters` `total``=``“1”` `executed``=``“1”` `passed``=``“1”` `error``=``“0”` `failed``=``“1”` `timeout``=``“0”` `aborted``=``“0”` `inconclusive``=``“0”` `passedButRunAborted``=``“0”` `notRunnable``=``“0”` `notExecuted``=``“0”` `disconnected``=``“0”` `warning``=``“0”` `completed``=``“0”` `inProgress``=``“0”` `pending``=``“0”` `/>`
```ResultSummary``>`
We are still interested in understanding why you are getting different results to us in your trx file. Can you let us know the file version of “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe” and also any non-default settings in your MSTest action?
---
### Author:  [@christian.leh](https://www.finalbuilder.com/forums/u/christian.leh)
#### Post date: [March 29, 2016, 8:36am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/13 "2016-03-29T08:36:16Z")
The version ist 10.0.30319.1. We don’t use any special. We have a test setttings file, however, running the tests without this settings file, the result is the same (and correct from my point of view).
This is how Continua is running it (copied from build log):
[code]Working Directory: C:\BuildAgent\Ws\157863 Executable: c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exeArguments: /testcontainer:C:\BuildAgent\Ws\157863\Source\INFOsys\Bin\Tests\INFOsys.Framework.Tests.dll /testcontainer:C:\BuildAgent\Ws\157863\Source\INFOsys\Bin\Tests\INFOsys.Objects.Test.dll /resultsfile:C:\BuildAgent\Ws\157863\Output\Tests\INFOsys.trx[/code]
---
### Author:  [@Sparky](https://www.finalbuilder.com/forums/u/Sparky)
#### Post date: [March 30, 2016, 10:03am UTC](https://www.finalbuilder.com/forums/t/inconclusive-tests/5464/14 "2016-03-30T10:03:10Z")
Hi Christian,
We are now able to reproduce inconclusive test outcomes after making changes to the references in our test project. We have made some changes so they are no longer by treated as errors. We hope to get a new version out in the next couple of days with this fix and others.