Visual Studio Team System 2008 : Command Line (part 1)

MSTest utility

This is the command line utility supported by Visual Studio. Using this MSTest tool, we can run the tests created in Visual Studio.

To access the MSTest tool, add the Visual Studio install directory to the path or access the Visual Studio command prompt from the Tools section of the Visual Studio group in the start menu.

After opening the command prompt, type MSTest.

The MSTest command expects the parameter to be specified, which is the name of the test to be run. To know the options of MSTest, just type MSTest/help or MSTest/? in the command prompt.

The help option lists the different parameter options that can be used with the MSTest and the description of each parameter and its usage.

Option	Description
/help	Displays this usage message <Short form: /? or /h>
/nologo	Does not display the startup banner and the copyright message
/testcontainer:[file name]	Loads a file that contains tests; you can specify this option more than once to load multiple test files Examples: /tescontainer:mytestproject.dll /testcontainer:loadtest1.loadtest
/testmetadata:[file name]	Loads a metadata file Example: /testmetadata:testproject1.vsmdi
/runconfig:[file name]	Uses the specified run configuration file Example: /runconfig:mysettings.testrunconfig
/resultsfile:[file name]	Saves the test run results to the specified file Example: /resultsfile:c:\temp\myresults.trx
/testlist:[test list path]	The test list to run as specified in the metadata file; you can specify this option multiple times to run more than one test list Example: /testlist:checkintests/clientteam
/test:[file name]	The name of a test to be run; you can specify this option multiple times to run more than one test
/unique	Runs a test only if one unique match is found for any given /test
/noisolation	Runs a test within the MSTest.exe process. This choice improves test run speed but increases risk to the MSTest process
/noresults	Does not save the test results in a TRX file; the choice improves test run speed but does not save the test run results
/detail:[property id]	The name of a property that you want to show values for, in addition to the test outcome; please examine a test results file to see the available properties. Example: /detail:errormessage

In addition to the above options, there are many other options which can be used with MSTest if team explorer is used.&;&;

Option	Description
/publish:[server name]	Publishes results to the TFS
/publishbuild:[build name]	The build identifier to be used to publish test results
/publishresultsfile: [file name]	The name of the test results file to be published; if none is specified, use the file produced by the current test run
/teamproject: [team project name]	The name of the team project to which the build belongs; specify this when publishing test results
/platform:[platform]	The platform of the build against which to publish the test results
/flavor:[flavor]	The flavor of the build against which to publish test results

We will run and see some test for the above options using the MSTest command line tool.

Running a test from the command line

Let us see some of the options for MSTest and see the results produced by these options. The MSTest utility does not work for manual tests. It is applicable only for the automated tests. Even if we apply the command for the manual test, the tool will remove the non-automated test from the test run. For example, if we run the test container command for the manual test, the reply would be similar to the one shown below:

/testcontainer

T&;&;he testcontainer option requires the file that contains information about the tests that must be run. The test container file differs across tests. For example, the unit test information is contained in the unit test project assembly file. For the ordered test, it would be the<ordertestproject>.orderedtest file, which contains information about the ordered test.

If we have multiple test projects created under the solution, each of these projects has its own container for the tests within the project.

Let us consider an ordered test which contains four different tests such as generictest1.generictest, GetTotalPrice Unit test, and couple of manual tests. The name of the ordered test file is OrderedTest1.orderedtest. Now let us try using the MSTest for running the tests in the ordered test file. You can see the output of the test as shown in the following image. The automated tests removed the manual non-automated tests from the list and ran only the automated tests.

For Unit tests and Web tests, the test project file will be an assembly. For example, the following command will load the TestProjforTestingApp.dll assembly and run the test within that assembly.

First, the MSTest will load all the tests within the ordered test and then start them executing one by one. The result of each test run is also shown. The results are not shown but the detailed test run is also stored in the test trace file. We can load the trace file and see the details of the test run.

/t&;estmetadata

The&; testmetadata option is used for running the tests in multiple test projects under a solution. This is based on the metadata file created under the solution. The metadata file contains a list of all the tests added to the Test List Editor. Using this Test Editor, we can create a new list of tests from all the available tests under the solution irrespective of the projects.

The testcontainer option is specific to a test project whereas testmetadata is for multiple test containers with the flexibility of choosing tests from each container.

For example, the following image shows two different test lists which contain different tests added to the list from all the available tests in the solution:

Any change or update made to the test list editor, will be saved in the metadata file. Now, to run only the tests added to the test lists in the editor, we use the option metadata. To run a specific test list, we should use the /testlist option along with the testmetadata option as shown:

mstest /testmetadata:[file name] /testlist:[test list name]

The following command runs all the tests added to the list, TestListOne:

If there are no tests added to the test list, and if it is an empty list, the Mstest command will clearly display a message saying there are no tests to be run in the list.

/tes&;t

There&; are situations where we might have to run a particular test from the available list of tests. In that case, we can use the /test option with the testmetatadata option or the testcontainer option. For example, the following command runs only the CalculateTotalPriceTest test from the list:

The /test option can be used along with testmetadata or testcontainer, but not both. There are different usages for the /test option:

1. We can specify any number of tests using the /test multiple times against testmetadata or testcontainer option.

2. The name used against the /test option is the search keyword of the fully qualified test names. For example, if there are test names with fully qualified names such as:

   TestProjforTestingApp.Class1Test.CalculateTotalPriceTest 
   TestProjforTestingApp.Service1Test.RegisterNewUserTest 
   TestProjforTestingApp.Service1Test.GetMyOrdersTest
   

   And if the command contains the option, /test:TestingApp then all the above three tests will run as the name contains the Testingapp string in it. Even though we specify only the name to the /test option, the result displays the fully qualified name of the tests run in the results window.

/uniq&;ue

This o&;ption will make sure that only one test, which matches the given name, is run. In the above examples, we have three different tests with the string TestingApp in its fully qualified name. If we run the following command, all the above three tests will be executed:

mstest /testmetadata:c:\workspace\testingapp\TestingApp.vsmdi /test:
TestingApp

But if we specify the /unique option along with the above command, the MSTest will return the message saying more than one test was found in the same name. It means that the test will be successful only if a unique test name is found with the name given.

The following command will execute successfully as there is only one test with the name Webtest3:

/noisolation

This opt&;ion runs tests with the MStest.exe process. This choice improves test run speed, but increases risk to the MSTest.exe process.

Usually, the tests are run in a separate process which takes its own memory. If we launch the MSTest.exe with the /noisolation option, the tests are run in a process which avoids more memory usage.

/runconfig

This is to specify to the test run to use a specific configuration file. If the configuration file is not specified, the MSTest uses the default configuration file. The following example forces the test to use the TestRunConfig2 configuration file:

/resultsfile

In all th&;e command executions, the MSTest stores the test results to a trace file. By default, the trace file name is assigned by the MSTest with the login user ID, the machine name and the current date and time. We can force the test tool to use a specific file to store the test results. For example, the above test result is stored in the file User1_MyPerPC_2008-06-09 20_53_32_.trx at the default Visual Studio folder. We can use the resultsfile option to change the location and the filename of the results file.

The previous image shows the test results stored at the c:\temp location in the results file, StoreTestResult.trx.

/noresult&;s

This optio&;n is to inform the MSTest application to not store the test results to the TRX file. This choice does not store the results but increases the performance of the test execution.

/nologo

This optio&;n is to inform the MSTest tool to not display the copyright information which is shown at the beginning of the test run.

/detail

This is th&;e option used for getting the values of the properties of each test run result. Each test result provides information about the test such as error messages, start time, end time, test name, description, test type and many more properties. Using this option we can get the property value after the test run. For example, the following command shows the start and end time of the test run and also the type of the test run.

The detailed option can be specified multiple times to get multiple properties value after the test run.