Living Documentation and test reports in VSTS/TFS pipelines

Mario Majcica May 7, 2018 0 Comments

Introduction

In order to truly get advantage from all of the hard work that we put into our tests, we need to present our test run results and share our specifications in more convenient and accessible way. On Windows platform in order to make this tasks happen, we can leverage tools that you are probably already using, SpecFlow itself and a sidekick project of it called Pickles. If none of what I just said does make sense, you are reading the wrong post, so please check the SpecFlow documentation and read about BDD which is partly in PDF so using software as Soda PDF could be useful for this. However, if you are already familiar with it, you are using SpecFlow and are looking for a decent way to automate the above-mentioned tasks, please continue reading as I may have a valid solution to it.

All of the implementations that I came across till now, involved scripts, MSBuild tasks and a lot of other cumbersome solutions. I saw potential in VSTS/TFS build/release pipeline that through some specific build/release tasks are a neat solution automating these requirements.

Let’s start.

Generating SpecFlow reports in VSTS

Generating a nice and easy to consult report over our your test runs is relatively easy. SpecFlow NuGet package already includes all of the necessary to do so, that is the SpecFlow executable itself. In my demo case, I am using NUnit 3, however other frameworks are also supported.

Let’s check first what are the necessary manual steps to get the desired report.
After executing my tests with NUnit Console Runner with the following parameters

nunit3-console.exe --labels=All "--result=TestResult.xml;format=nunit2" SpecFlowDemo.dll

I am ready to generate my report. Now I just need to invoke the SpecFlow executable with the following parameters

specflow.exe nunitexecutionreport SpecFlowDemo.csproj /xmlTestResult:TestResult.xml /out:MyReport.html

And voila, the report is generated and it looks like following

Details over the various parameters accepted by SpecFlow executable can be found here, Test Execution Report.

Now, how do we integrate this into our VSTS build pipeline?

First, we need to run our tests in order to get the necessary test results. These may vary based on the testing framework that we are using. Two supported ones are NUnit or MSTest.

Be aware that if you run your MsTest’s with vstest runner, the output trx file will not be compatible with the format generated by mstest runner and the report will not render correctly. For that to work, you’ll need a specific logger for your vstest runner.

Once the tests are completed we are going to use the SpecFlow Report Generator task that is part of the homonymous extension that you can find here.
After adding the SpecFlow Report Generator task in your build definition, it will look similar to this.

In case you are interested in how each parameter influences the end result, check the above-mentioned link pointing to the SpecFlow documentation as the parameters are the same as on per tool invocation via the console.

Now that your report is ready you can process it further like making it part of the artifact or send it via email to someone.

Generating Pickles living documentation

Support documentation based on your specifications can be generated by Pickles in many ways, such us MSBuild task that could be part of your project, PowerShell library or simply by invoking the Pickles executable on the command line.

In case of trying to automate this task, you will probably use the console application or PS cmdlets. Let’s suppose the first case, then the command that we are looking for is like following

Pickles.exe --feature-directory=c:\dev\my-project\my-features --output-directory=c:\Documentation --documentation-format=dhtml

All of the available arguments are described here.

As the result you will get all of the necessary files to render the following page:

Back to VSTS. In order to replicate the same in your build definition, you can use the Pickles documentation generator task. Add the task to your definition and it should look like somewhere like this

All of the parameters do match the ones offered by the console application. You are now left to choose how and where further to ship this additional material.

Conclusion

In this post, I illustrated a way on how to get more out of the tooling that you are probably already using. Once you automate these steps chance is that they are going to stay up to date and thus probably get to be actually used. All of the tasks I used can also be used in the VSTS/TFS release pipeline.

I hope it helps.

Handling TPL exceptions in MSTest unit tests

Mario Majcica October 2, 2014 0 Comments

Introduction

In the past couple of years TPL (Task Parallel Library) and language features it brought became more and more popular in the .NET community. One of the differences that stood up is the exception handling. Unhandled exceptions that are thrown by user code that is running inside a task are propagated back to the joining thread. To propagate all the exceptions back to the calling thread, the Task infrastructure wraps them in an AggregateException instance.

Now this is all clear and quite straight forward to manage. I will not go in the detail about this topic as you can find all of the necessary information on MSDN at http://msdn.microsoft.com/en-us/library/dd997415(v=vs.110).aspx.

Once I started writing unit tests for my async methods I found out that I can’t anymore use my usual technique.

In order to assert that a certain unit is throwing a certain exception in well determined circumstances, I always went for the ExpectedException attribute. To make it even clearer, I will make an example of usage.

Now imagine the following class:

public bool MySimpleMethod(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    return true;
}

We are going to write a test that will assert that in case of a null being passed as a parameter to this method call, the method should raise an ArgumentNullException.
In order to do so, consider the following code.

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void MySimpleMethod_Throws_ArgumentNullException()
{
    sut.MySimpleMethod(null);
}

Once executed, this test will pass.
Now let’s write an async analog of the previous method.

public async Task MySimpleMethodAsync(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    await Task.Delay(100);
 
    return true;
}

If we now try to execute the following test, it will fail.

public async Task MySimpleMethodAsync(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    await Task.Delay(100);
 
    return true;
}

As we do know that our exception is wrapped in an AggregateException, it is expected our test to fail. How should we than test this kind of situations?

Building a new attribute

Wait, I really liked the ExpectedException attribute, I would like to have an attribute that will check if any of the aggregated exceptions are of the type I do expect.
After a quick search I hadn’t found anything that suites me. That was the moment the ExpectedAggregateExceptionAttribute was born.
I followed the pattern on which ExpectedException was made and this is what I came up with.

/// 

/// Indicates that an exception is expected during test method execution.
/// It also considers the AggregateException and check if the given exception is contained inside the InnerExceptions.
/// This class cannot be inherited.
/// 
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public sealed class ExpectedAggregateExceptionAttribute : ExpectedExceptionBaseAttribute
{
    protected override string NoExceptionMessage
    {
        get
        {
            return string.Format("{0} - {1}, {2}, {3}",
                this.TestContext.FullyQualifiedTestClassName,
                this.TestContext.TestName,
                this.ExceptionType.FullName,
                base.NoExceptionMessage);
        }
    }
 
    /// 

    /// Gets the expected exception type.
    /// 
    /// 
    /// 
    /// A  object.
    /// 
    public Type ExceptionType { get; private set; }
 
    public bool AllowDerivedTypes { get; set; }
 
    /// 

    /// Initializes a new instance of the  class with and expected exception type and a message that describes the exception.
    /// 
    /// An expected type of exception to be thrown by a method.
    public ExpectedAggregateExceptionAttribute(Type exceptionType)
        : this(exceptionType, string.Empty)
    {
    }
 
    /// 

    /// Initializes a new instance of the  class with and expected exception type and a message that describes the exception.
    /// 
    /// An expected type of exception to be thrown by a method.
    /// If the test fails because an exception was not thrown, this message is included in the test result.
    public ExpectedAggregateExceptionAttribute(Type exceptionType, string noExceptionMessage)
        : base(noExceptionMessage)
    {
        if (exceptionType == null)
        {
            throw new ArgumentNullException("exceptionType");
        }
 
        if (!typeof(Exception).IsAssignableFrom(exceptionType))
        {
            throw new ArgumentException("Given type is not an exception.", "exceptionType");
        }
 
        this.ExceptionType = exceptionType;
    }
 
    /// The exception that is thrown by the unit test.
    protected override void Verify(Exception exception)
    {
        Type type = exception.GetType();
 
        if (this.AllowDerivedTypes)
        {
            if (!this.ExceptionType.IsAssignableFrom(type))
            {
                base.RethrowIfAssertException(exception);
 
                throw new Exception(string.Format("Test method {0}.{1} threw exception {2}, but exception {3} was expected. Exception message: {4}",
                    base.TestContext.FullyQualifiedTestClassName,
                    base.TestContext.TestName,
                    type.FullName,
                    this.ExceptionType.FullName,
                    GetExceptionMsg(exception)));
            }
        }
        else
        {
            if (type == typeof(AggregateException))
            {
                foreach (var e in ((AggregateException)exception).InnerExceptions)
                {
                    if (e.GetType() == this.ExceptionType)
                    {
                        return;
                    }
                }
            }
 
            if (type != this.ExceptionType)
            {
                base.RethrowIfAssertException(exception);
 
                throw new Exception(string.Format("Test method {0}.{1} threw exception {2}, but exception {3} was expected. Exception message: {4}",
                    base.TestContext.FullyQualifiedTestClassName,
                    base.TestContext.TestName,
                    type.FullName,
                    this.ExceptionType.FullName,
                    GetExceptionMsg(exception)));
            }
        }
    }
 
    private string GetExceptionMsg(Exception ex)
    {
        StringBuilder stringBuilder = new StringBuilder();
        bool flag = true;
 
        for (Exception exception = ex; exception != null; exception = exception.InnerException)
        {
            string str = exception.Message;
 
            FileNotFoundException notFoundException = exception as FileNotFoundException;
            if (notFoundException != null)
            {
                str = str + notFoundException.FusionLog;
            }
 
            stringBuilder.Append(string.Format((IFormatProvider)CultureInfo.CurrentCulture, "{0}{1}: {2}", flag ? (object)string.Empty : (object)" ---> ", (object)exception.GetType(), (object)str));
            flag = false;
        }
 
        return ((object)stringBuilder).ToString();
    }
}

To cut it short, I do check if the thrown exception is of AggregateException type, and if so I do again check if any of the inner exceptions is of the requested type. In case this condition is not matched a new exception is thrown which will make your unit test fail.
If we now use our new attribute instead of the ExpectedExceptionAttribute this test will be successful.

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void MySimpleMethodAsync_Throws_ArgumentNullException_2()
{
    sut.MySimpleMethodAsync(null).Wait();
}

In order to be sure that our new attribute is working properly I will make some other tests. First I do expect the unit test to fail in case no exception was thrown:

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void ArgumentNullException_Fail_If_No_Exception()
{
    sut.MySimpleMethodAsync(new List()).Wait();
}

Mind that this test is not meant to pass (do not include it in your solution).
Also, if in the inner exceptions there is no expected exception type, the test should fail:

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void ArgumentNullException_Fail_If_Wrong_Inner_Exception()
{
    Task.Factory.StartNew(() =>
    {
        throw new ArgumentOutOfRangeException();
    });
}

As both test failed, I can conclude this new attribute works as supposed. You can package it as you wish, in your own common library, in a new or existing custom testing library or in any of your projects.

Downloads and the source code

You can find the source code and the demo project on Github at this address here.

Happy unit testing!