Living Documentation and test reports in VSTS/TFS pipelines

Mario Majcica May 7, 2018 0 Comments

Introduction

In order to truly get advantage from all of the hard work that we put into our tests, we need to present our test run results and share our specifications in more convenient and accessible way. On Windows platform in order to make this tasks happen, we can leverage tools that you are probably already using, SpecFlow itself and a sidekick project of it called Pickles. If none of what I just said does make sense, you are reading the wrong post, so please check the SpecFlow documentation and read about BDD which is partly in PDF so using software as Soda PDF could be useful for this. However, if you are already familiar with it, you are using SpecFlow and are looking for a decent way to automate the above-mentioned tasks, please continue reading as I may have a valid solution to it.

All of the implementations that I came across till now, involved scripts, MSBuild tasks and a lot of other cumbersome solutions. I saw potential in VSTS/TFS build/release pipeline that through some specific build/release tasks are a neat solution automating these requirements.

Let’s start.

Generating SpecFlow reports in VSTS

Generating a nice and easy to consult report over our your test runs is relatively easy. SpecFlow NuGet package already includes all of the necessary to do so, that is the SpecFlow executable itself. In my demo case, I am using NUnit 3, however other frameworks are also supported.

Let’s check first what are the necessary manual steps to get the desired report.
After executing my tests with NUnit Console Runner with the following parameters

nunit3-console.exe --labels=All "--result=TestResult.xml;format=nunit2" SpecFlowDemo.dll

I am ready to generate my report. Now I just need to invoke the SpecFlow executable with the following parameters

specflow.exe nunitexecutionreport SpecFlowDemo.csproj /xmlTestResult:TestResult.xml /out:MyReport.html

And voila, the report is generated and it looks like following

Details over the various parameters accepted by SpecFlow executable can be found here, Test Execution Report.

Now, how do we integrate this into our VSTS build pipeline?

First, we need to run our tests in order to get the necessary test results. These may vary based on the testing framework that we are using. Two supported ones are NUnit or MSTest.

Be aware that if you run your MsTest’s with vstest runner, the output trx file will not be compatible with the format generated by mstest runner and the report will not render correctly. For that to work, you’ll need a specific logger for your vstest runner.

Once the tests are completed we are going to use the SpecFlow Report Generator task that is part of the homonymous extension that you can find here.
After adding the SpecFlow Report Generator task in your build definition, it will look similar to this.

In case you are interested in how each parameter influences the end result, check the above-mentioned link pointing to the SpecFlow documentation as the parameters are the same as on per tool invocation via the console.

Now that your report is ready you can process it further like making it part of the artifact or send it via email to someone.

Generating Pickles living documentation

Support documentation based on your specifications can be generated by Pickles in many ways, such us MSBuild task that could be part of your project, PowerShell library or simply by invoking the Pickles executable on the command line.

In case of trying to automate this task, you will probably use the console application or PS cmdlets. Let’s suppose the first case, then the command that we are looking for is like following

Pickles.exe --feature-directory=c:\dev\my-project\my-features --output-directory=c:\Documentation --documentation-format=dhtml

All of the available arguments are described here.

As the result you will get all of the necessary files to render the following page:

Back to VSTS. In order to replicate the same in your build definition, you can use the Pickles documentation generator task. Add the task to your definition and it should look like somewhere like this

All of the parameters do match the ones offered by the console application. You are now left to choose how and where further to ship this additional material.

Conclusion

In this post, I illustrated a way on how to get more out of the tooling that you are probably already using. Once you automate these steps chance is that they are going to stay up to date and thus probably get to be actually used. All of the tasks I used can also be used in the VSTS/TFS release pipeline.

I hope it helps.

Using global application host file in Visual Studio 2015+

Mario Majcica March 15, 2018 0 Comments

In moving one of the projects from VS2013 to VS2015 I realized that IISExpress settings specified in the users applicationhost.config file got overridden. After a quick search, it came to me that another applicationhost.config file was created by the VS2015 and placed in the .vs/config folder under the root of my project.
This is, however, an undesired behaviour as I do not want to check in the source code the .vs folder, being that a bad practice also described by Microsoft itself. Files under the .vs folder are all files that you would never check in, since they are generated from a build or contain machine-specific information.

Then how to proceed with what concerns my applicationhost.config file?

There is a solution for this and it lies in a setting that is part of our project file. If you edit your web project .csproj file with a text editor, you should find a MSBuild property called UseGlobalApplicationHostFile. By default, the UseGlobalApplicationHostFile is specified and the value of it is not set, so you should see the following:

<UseGlobalApplicationHostFile />

As you can imagine from the property name, if set to true, it will indicate that for the project in question the global applicationhost.config file should be used, and that is precisely what we are looking for. Set this property to true:

<UseGlobalApplicationHostFile>True</UseGlobalApplicationHostFile>

Once it is set, we can see that our global applicationhost.config file (which is located in %USERPROFILE%\Documents\IIS Express\config\applicationHost.config) is again used and that my application can be launched for debug from Visual Studio correctly.

Job done!

CodedUI Tips and Tricks – Unavailable search attributes

Mario Majcica June 14, 2015 0 Comments

Searching and filtering for the UITestControl objects is a straight forward process you can read about in plenty of places. All of them will tell you on how to leverage the SearchProperties and FilterProperties of the object in order to get the access to the desired control.

In both of these properties once you add the new PropertyExpression to the collection, you need to provide the property name. Each class that is based on HtmlControl base class, by contract, implements and exposes a property called PropertyNames. In the property implementation all of the controls do inherit from HtmlControl.PropertyNames which on it’s own brings the content of the class UITestControl.PropertyNames, and extend them with some control specific properties. This class is just a collection of strings, indicating the name of the element we are in search off. As this can sound confusional I’ll make a simple example of my typical control search:

HtmlButton okButton = new HtmlButton(browserWindow);
okButton.SearchProperties.Add(HtmlControl.PropertyNames.Id, "btnOk");
okButton.FilterProperties.Add(HtmlControl.PropertyNames.InnerText, "OK");

Now, no matter that the first parameter of the Add method is a string, and that PropertyNames collection is just that, a collection of strings, if the element you are searching for is not listed inside the PropertyNames collection and you try to set it by your custom constant like “style”, the control search will not be correctly performed.

popUpDiv.SearchProperties.Add("style", "display: block");

What is left for us to do in cases like this, is to explore the property called ControlDefinition. All of the attributes of the element we are exploring will be listed inside this property which is part of HtmlControl object. Luckily we can bring this property also inside the search and filtering criteria and we do not need to check the content of the controls in nasty loops and conditional operators.
In the following sample I am searching for my style setting and leveraging the PropertyExpressionOperator to imply a property value contains the given string.

popUpDiv.SearchProperties.Add(
    HtmlControl.PropertyNames.ControlDefinition, 
    "display: block", 
    PropertyExpressionOperator.Contains);

You also need to be sure on how is your element and it’s value written inside this property as it often has some custom way of picking these values up. The quickest way that works for me is to debug my automation code and check the content of this property.
What I usually do is, narrow the search as much as possible and then resolve all of the controls that match that criteria.

UITestControlCollection controls = popUpDiv.FindMatchingControls();

Once I found my control I do annotate the property value and use this information to setup my search or filtering.

I hope this tip saved you some time as this kind of things are not documented on MSDN nor are popular topic in the blogging community.

Happy coding!