Introduction
Whether from the need to test in production environments, or run tests that integrate with tools that cannot run within the Microsoft Build and Development environments, there is still a need to report test case results back to the Team Foundation Server (TFS). This is especially true for automated tests, requiring an automated posting of Test Case Results. Here is one way to approach this from an externally run utility that integrates with the Microsoft.TestManagment
.NET assemblies.
Caveats
- Please use at your own risk and [in]convenience.
- Dependencies on Microsoft's Team Explorer. Microsoft's Team Explorer (TE) is freely available, with multiple versions to match your versions of Visual Studio and TFS. Because of this variety of environments, as well as respect for redistribution licensing (or lack thereof), the dependent TE assemblies are not included and must be acquired and configured to run independently.
- Dense, Partial Code Samples. Apologies in advance for the inappropriately dense coding style. Please beautify as needed. For brevity, error checking and variable acquisition has been removed from the code snippets in this article.
Posting Test Case Results
Step One: Find and Add TestManager References
To start, you must download and install Microsoft's Team Explorer. Next, map references from your project to Microsoft.TeamFoundation.TestManagement.Client
, and Microsoft.TeamFoundation.TestManagement.Common
.
You will also need to reference Microsoft.TeamFoundation.Client
, Microsoft.TeamFoundation.Common
, Microsoft.TeamFoundation.WorkItemTracking.Client
, and Microsoft.TeamFoundation.WorkItemTracking.Common
assemblies.
Though this will place about 30 assemblies into your output directory, two important files from the Team Explorer assemblies are left out and cannot be referenced directly. The workaround to this is to find the DLLs and place them into your project as 'Build Action = None' and 'Copy if newer' files.
If you add Microsoft.WITDataStore32.dll and Microsoft.WITDataStore64.dll from the Add|Existing Item,... project menu, a static copy of the DLLs will be copied into your project source files folder. My OCD makes me close the project, delete the copied Microsoft DLLs, and edit the .csproj file to point back to the original directory, e.g.:
Before:
<ItemGroup>
<None Include="Microsoft.WITDataStore32.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="Microsoft.WITDataStore64.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
After:
<ItemGroup>
<None Include="..\..\..\..\..\Program Files (x86)\Microsoft Visual Studio\2017\TestPro\
Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Microsoft.WITDataStore32.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="..\..\..\..\..\Program Files (x86)\Microsoft Visual Studio\2017\TestPro\
Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Microsoft.WITDataStore64.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
When you deploy your 'TCResults
' executable, you will have to include the entourage of thirty or so DLLs that get stuffed in the project build output directory along with your executable.
Step Two: Secure the TfsTeamProjectCollection Object
An initial step when integrating with the TFS .NET objects is to get the ProjectCollection
from a TFS server URI and the name of the project. This static routine returns the ProjectCollection
, if it can be found.
static TfsTeamProjectCollection GetTeamProjectCollection(Uri TFSServer, string ProjectName)
{
TfsConfigurationServer configServer = TfsConfigurationServerFactory.GetConfigurationServer(TFSServer);
ReadOnlyCollection<CatalogNode> projects = configServer.CatalogNode.QueryChildren(new[]
{ CatalogResourceTypes.ProjectCollection }, false, CatalogQueryOptions.None);
foreach (CatalogNode pc in projects)
{
Guid collection_id = new Guid(pc.Resource.Properties["InstanceId"]);
ReadOnlyCollection<CatalogNode> teamProjects = pc.QueryChildren(new[]
{ CatalogResourceTypes.TeamProject }, false, CatalogQueryOptions.None);
foreach (CatalogNode tp in teamProjects)
{
if (string.Compare(ProjectName, tp.Resource.DisplayName,
CLU.Option.Collection.CaseSensitivity) == 0)
return configServer.GetTeamProjectCollection(collection_id);
}
}
return null;
}
Step Three: Secure the TestPlan to Post to.
(Bring your own bolded variable values)
TfsTeamProjectCollection teamCollection =
GetTeamProjectCollection(_tfs_server_uri, _project_name);
ITestManagementService tms = teamCollection.GetService<ITestManagementService>();
ITestManagementTeamProject testProject = tms.GetTeamProject(_project_name);
string wis_command =
(string.IsNullOrWhiteSpace(_test_plan_name))
? " SELECT * FROM TestPlan WHERE [PlanState] = 'Active'"
: " SELECT * FROM TestPlan WHERE [PlanState] = 'Active'
AND [Title] = '" + _test_plan_name + "' ";
ITestPlanCollection testplanCollection = testProject.TestPlans.Query(wis_command);
</code>
Explanatory Interlude
Now it gets messy and confusing, muddled with my own conjectures based upon working with a TFS 2013 on-premise server.
Here at fluentbytes.com is an excellent implementation guide, posted by Marcel de Vries, on the consumption of TFS TestCase
s and TestCaseResult
s, through the TestManagement interface
s. From this, I got a basic idea of how the .NET objects of the TestManagement
assembly were organized, crudely shown below.
But writing new objects into the store is much more constrained than implied by this read-only diagram. Below is an interface usage diagram of the related TestManagement
objects.
What is not discernable from Marcel de Vries' article are the relationship and sequencing requirements of TestCaseResult
s, TestRun
s, TestPoint
s, TestCase
s, TestSuite
s, and TestPlan
s.
The Basic Constraints
TestCaseResult
s can only be assigned to the TestPoint
s of a TestRun
. - (In my exerience)
TestPoint
s can only be added to a TestRun
after it is created, and before it is saved. TestPoint
s are a matrix intersection of TestCases
with TestSuite
configurations. - The
TestRun
must be created under a valid, preferably "Active" TestPlan
. - The
TestCaseResult
objects are instantiated when the TestRun
is initially saved. - The
TestCase
associated with the TestCaseResult
, must also be associated with a targeted TestSuite
under the targeted TestPlan
. This connotes a sibling relationship between the new TestRun
and the TestSuite
, both having the same parent TestPlan
. And a cousin relationship between the TestCase
and the TestPoint
.
Step Four: Indirectly Create the TestCaseResult Placeholders
To have the placeholder TestCaseResult
objects instantiated for the TeamProject
's TestCase
(s) for which there are results, it must be verified that the TestCase
(s) are associated with the designated TestSuite
. Then a new TestRun
must be created with the appropriate TestPoint
(s) added that are associated with the TestCase
(s). When the TestRun
is saved, the placeholder TestCaseResult
objects are instantiated. Below the TestCase
s are queried by Id
or by the AutomatedTestName
field.
(bring your own bolded variable values)
<code>
foreach (ITestPlan testPlan in testplanCollection)
{
int suite_index = 0;
ITestSuiteBase testSuite =
(testPlan.RootSuite.SubSuites.Count > suite_index)
? testPlan.RootSuite.SubSuites[suite_index] : testPlan.RootSuite;
while (testSuite != null)
{
if (((ITestSuiteBase2)testSuite).Status.Equals("In Progress")
&&
(string.IsNullOrWhiteSpace(_test_suite_name)
|| _test_suite_name.Equals(testSuite.Title))
&&
(!string.IsNullOrWhiteSpace(_test_case_id)
|| !string.IsNullOrWhiteSpace(_automated_test_name))
)
{
string testcase_query = null;
if (!string.IsNullOrWhiteSpace(_test_case_id))
testcase_query = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Test Case'
AND [Id] = '" + _test_case_id + "'";
else if (!string.IsNullOrWhiteSpace(_automated_test_name))
testcase_query = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Test Case'
AND [Microsoft.VSTS.TCM.AutomatedTestName] = '" + _automated_test_name + "'";
foreach (ITestCase test_case in testProject.TestCases.Query(testcase_query))
{
ITestRun testRun = testPlan.CreateTestRun(true);
ITestPointCollection testpointCollection = testPlan.QueryTestPoints
("SELECT * FROM [TestPoint] WHERE [TestCaseId] = '" + test_case.Id.ToString() + "'");
if (testpointCollection.Count == 0)
{
testSuite.TestCases.Add(test_case);
testPlan.Save();
testPlan.Refresh();
testSuite.Refresh(true);
testpointCollection = testPlan.QueryTestPoints
("SELECT * FROM [TestPoint] WHERE [TestCaseId] = '" + test_case.Id.ToString() + "'");
}
foreach (ITestPoint test_point in testpointCollection)
{
if (test_point.ConfigurationName.Equals(_test_configuration_name))
testRun.AddTestPoint(test_point, null);
}
testRun.Title = test_case.Title;
testRun.Save();</code>
Step Five: Post and Save the TestCaseResult
Now that the placeholder TestCaseResut
objects have been instantiated, their internal values can be set.
(bring your own bolded variable values)
foreach (ITestCaseResult tcr in testRun.QueryResults(false))
{
switch (_tcr_outcome)
{
case "NONE":
tcr.Outcome = TestOutcome.None;
tcr.State = TestResultState.Completed;
break;
case "PASS":
case "PASSED":
tcr.Outcome = TestOutcome.Passed;
tcr.State = TestResultState.Completed;
break;
case "FAIL":
case "FAILED":
tcr.Outcome = TestOutcome.Failed;
tcr.State = TestResultState.Completed;
break;
case "INCONCLUSIVE":
tcr.Outcome = TestOutcome.Inconclusive;
tcr.State = TestResultState.Completed;
break;
case "TIMEOUT":
tcr.Outcome = TestOutcome.Timeout;
tcr.State = TestResultState.Completed;
break;
case "ABORTED":
tcr.Outcome = TestOutcome.Aborted;
tcr.State = TestResultState.Completed;
break;
case "BLOCK":
case "BLOCKED":
tcr.Outcome = TestOutcome.Blocked;
tcr.State = TestResultState.Pending;
break;
case "NE":
case "NOTEXECUTED":
tcr.Outcome = TestOutcome.NotExecuted;
tcr.State = TestResultState.Pending;
break;
case "WARNING":
tcr.Outcome = TestOutcome.Warning;
tcr.State = TestResultState.Unspecified;
break;
case "ERROR":
tcr.Outcome = TestOutcome.Error;
tcr.State = TestResultState.Unspecified;
break;
case "NA":
case "NOT APPLICABLE":
tcr.Outcome = TestOutcome.NotApplicable;
tcr.State = TestResultState.Completed;
break;
case "PAUSED":
tcr.Outcome = TestOutcome.Paused;
tcr.State = TestResultState.Paused;
break;
case "INPROGRESS":
tcr.Outcome = TestOutcome.InProgress;
tcr.State = TestResultState.InProgress;
break;
case "UNSPECIFIED":
default:
tcr.Outcome = TestOutcome.Unspecified;
tcr.State = TestResultState.Unspecified;
break;
}
tcr.Comment = _tcr_comment;
tcr.Save();
}
testRun.QueryResults(false).Save(false);
}
}
testSuite = (testPlan.RootSuite.SubSuites.Count > ++suite_index) ?
testPlan.RootSuite.SubSuites[suite_index] : null;
}
}
Conclusion
With the ability to post more exhaustive Test Case results from external sources, your projects and teams will have better visibility into your product's quality metrics.
History
- 2017.07.27 - Initial release