Resetting the Visual Studio Experimental instance

Recently, I got into developing a couple of extensions for Visual Studio. As many of you know, once you create Visual Studio Package project (or any VSIX project), by default your application will be set to start debugging in a Visual Studio Experimental instance.

To safeguard your Visual Studio development environment from untested extensions that might change it, the Visual Studio provides an alternative Visual Studio instance that you can use to experiment. You are going to develop new extensions by using Visual Studio as usual, but you will run them by using this experimental instance.

This behavior is set as default by the Visual Studio Package project template and you can see it the following screenshot:

ExperimentalInstanceSettings

As you can see, we are pointing to the Visual Studio executable (we can vary that parameter for testing our extension with different versions of Visual Studio) and we are passing the following command line arguments to it:

/RootSuffix Exp

This is sufficient to start Visual Studio in Experimental instance mode and once you hit F5 and start debugging you will notice that the started instace of visual studio in the application title bar contains the “Experimental Instance” suffix.

ExperimentalInstanceDebugging

The default location of the experimental instance depends on the Visual Studio version number. For example, for Visual Studio 2013, the location is

%localappdata%\Microsoft\VisualStudio\12.0Exp\

All the files in the directory location are considered part of that instance. Any additional experimental instances will not be loaded by Visual Studio unless the directory name is changed to the default location.

Till here all fine, but what happens once you have “ruined” your experimental instance environment?
Well, you will need to reset it. Although you can find all of this information on MSDN, it isn’t linked or clear enough on how to proceed.

Inside the

C:\Program Files (x86)\Microsoft Visual Studio 12.0\VSSDK\VisualStudioIntegration\Tools\Bin

folder you will find a utility called CreateExpInstance. Open your command prompt and position yourself in just mentioned directory (this is the default path for Visual Studio 2013 SDK, the same applies also for Visual Studio 2012, just change your path accordingly).

All you need to do now is to execute the following command

CreateExpInstance /Reset /VSInstance=12.0 /RootSuffix=Exp

Once the cleanup is performed, the next experimental instance you lunch will found itself in a clean environment. In case you are working with Visual Studio 2012 change the VSInstance parameter to 11.0.

You can find more details about the CreateExpInstance Utility here.

moq.Callback(), The Unknown

Introduction

More and more often I do see people having trouble testing certain type of code. As a result, code coverage is dropping down, unverified logics are shown up, lowering the quality and rising frustrations. This is what pushed me to write this post and describe this kind of situations and a decent solution to it.
This kind of situations are commonly found in all the cases in which the tested unit manipulates the arguments that are passed to our mock. In this case we need to verify that this transformation did what we expected it to do. However, it ain’t that simple and straightforward as it seems. As code speaks more than thousand words, let’s illustrate this with an example.
Consider the following class:

public class ProductService
{
    private readonly IOrderRepository m_orderRepository;

    public ProductService(IOrderRepository orderRepository)
    {
        m_orderRepository = orderRepository;
    }

    public List GetProducts(int customerId, int orderId)
    {
        OrderSearchCriteria orderSearchCriteria = new OrderSearchCriteria
        {
            OrderId = customerId // THIS IS THE PROBLEM WE ARE GOING TO SEARCH FOR

            // Set some other search criteria...
        };

        Order retrievedOrder = m_orderRepository.GetOrder(orderSearchCriteria);

        // Do something else
        return retrievedOrder.Products;
    }
}

What you can see here is a hypothetical product service that we are going to test. More precisely we are going to write unit tests for the GetProducts method. What this method does in particular is composing another object that will be passed to our dependency, the order repository. Now you can argue that this is a bad practice, that the object composition should be handled in a different manner, as usually in this cases the single responsibility principle is not met. And you are right, but we do not live in a perfect world and often we can’t easily change what is already there. However, we need to keep extending and improving our software.

Still, I do need to write a test for that method. What should I do, how do I spot this bug that we just introduced?

There are two ways of writing a unit test that will test, verify and spot our bug. Let’s start with the first one.

Shaping an expected instance

We can tackle this problem by setting up manually, in our test, an instance of OrderSearchCriteria class, as we expect it to be, base on the parameters that we are passing in, and make sure that our mock accepts only an instance that equals ours, on purpose created class.

Let’s check our unit test.

[TestMethod]
public void GetProducts_Creates_OrderSearchCriteria_Correctly()
{
    const int customerId = 56789;
    const int orderId = 12345;

    OrderSearchCriteria orderSearchCriteria = new OrderSearchCriteria
    {
        OrderId = orderId
    };

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(orderSearchCriteria))
        .Returns(new Order());

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(customerId, orderId);
}

Now, first thing first. In order this example to even work, your parameter class needs to implement the equality members. This is necessary as Moq, in order to determine equality of parameters, rightly, relays on Equals() method.

Another disadvantage of this technique is the fact that construction of our own object can sometimes be hard or even not possible. Not to even mention the maintenance problem we are going to introduce.

As Moq in the case of wrong parameter will return a null from the method call, often null value is managed and interpreted as a possible state. In that case it will be very hard or impossible to discover our bug.
Luckily there is a cleaner way to approach this kind of situations.

Extracting the parameter via Callback method

As it is not often used, many developers tend to ignore the Callback method that is provided by Moq framework. In this kind of situations it can be very handy.

Check out the following test.

[TestMethod]
public void GetProducts_Creates_OrderSearchCriteria_Correctly_2()
{
    const int customerId = 56789;
    const int orderId = 12345;

    OrderSearchCriteria recievedOrderSearchCriteria = null;

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(It.IsAny<OrderSearchCriteria>()))
        .Returns(new Order())
        .Callback(o => recievedOrderSearchCriteria = o);

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(customerId, orderId);

    Assert.IsNotNull(recievedOrderSearchCriteria);
    Assert.AreEqual(orderId, recievedOrderSearchCriteria.OrderId);
}

You can see that I’m setting up my mock and I’m specifying what the callback should do. What I’m telling him in this case is that for the parameter of type OrderSearchCriteria, once the method is invoked, copy it to the locally defined object called recievedOrderSearchCriteria. This will give me the possibility to check what came in the call of the GetOrder method, and verify that is what I expect to receive.
Once I start my assertions I do a check on the recievedOrderSearchCriteria and I do make sure that what came in, is what I do expect.

This test will fail and we will succeed in our intent. Also the message that we are receiving is far way clearer than one in the previous example. It states at this moment

Assert.AreEqual failed. Expected:<12345>. Actual:<56789>.

Beside this we are actually asserting the expected result thus specifying behavior in an explicit way.
Now, this on my opinion is much better!

Other considerations about the Callback method

For a less experienced developers, I’ll also make an example of how to get a callback working if you have more then one parameter accepted by your mocked object.
I’m going to extend our IOrderRepository interface by adding a overload of the GetOrder method that accepts two parameters. Also I will implement another method on ProductService class that uses this newly created method.

public interface IOrderRepository
{
    Order GetOrder(OrderSearchCriteria searchCriteria);

    Order GetOrder(int orderId, bool archieved);
}

public List GetProducts(int orderId)
{
    Order retrievedOrder = m_orderRepository.GetOrder(orderId, true);

    return retrievedOrder.Products;
}

In order to mock my order repository and have on callback the necessary values, the following test is used.

[TestMethod]
public void GetProducts_With_Archieved_Orders()
{
    const int orderId = 12345;

    int receivedOrderId = 0;
    bool receivedArchieved = false;

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(It.IsAny(), It.IsAny()))
        .Returns(new Order())
        .Callback((o, a) =>
        {
            receivedOrderId = o; 
            receivedArchieved = a;
        });

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(orderId);

    Assert.AreEqual(orderId, receivedOrderId);
    Assert.AreEqual(true, receivedArchieved);
}

As you can see, I just added an extra type in my generic definition and then adjusted my lambda expression accordingly. If more than two parameters are required, you can just follow the same pattern and define them as many as need. At example .Callback((o, a, i) etc.

You always need to respect the exact parameter signature of the method you are setting up. The number of parameters accepted by the mocked method need to mach in type, order and number the parameters accepted by your mocked method.

There is another way to express the same statement, just by using the lambda expression instead of generics. I can rewrite the above examples in the following way:

.Callback((OrderSearchCriteria o) => recievedOrderSearchCriteria = o);

// ...

.Callback((int o, bool a) =>
{
    receivedOrderId = o;
    receivedArchieved = a;
});

The effect is the same, it is only about your preference which of the two ways to use.

It is also possible to define it before and after the method invocation and as I can’t think of a nice example I will just provide the one from the Moq documentation:

// callbacks can be specified before and after invocation
mock.Setup(foo => foo.Execute("ping"))
    .Callback(() => Console.WriteLine("Before returns"))
    .Returns(true)
    .Callback(() => Console.WriteLine("After returns"));


Conclusion

Each time you need to check the arguments that are passed to the method you are setting up, Callback() will help you get to them. Also, if you need to execute any code before or after the method is invoked, Callback() will let you do so.
Hopefully you are not going to use it on a daily basis, but it is handy to know about it as sooner or later you are going to face a situation where a Callback() will help you achieve your goal.

Handling TPL exceptions in MSTest unit tests

Introduction

In the past couple of years TPL (Task Parallel Library) and language features it brought became more and more popular in the .NET community. One of the differences that stood up is the exception handling. Unhandled exceptions that are thrown by user code that is running inside a task are propagated back to the joining thread. To propagate all the exceptions back to the calling thread, the Task infrastructure wraps them in an AggregateException instance.

Now this is all clear and quite straight forward to manage. I will not go in the detail about this topic as you can find all of the necessary information on MSDN at http://msdn.microsoft.com/en-us/library/dd997415(v=vs.110).aspx.

Once I started writing unit tests for my async methods I found out that I can’t anymore use my usual technique.

In order to assert that a certain unit is throwing a certain exception in well determined circumstances, I always went for the ExpectedException attribute. To make it even clearer, I will make an example of usage.

Now imagine the following class:

public bool MySimpleMethod(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    return true;
}

We are going to write a test that will assert that in case of a null being passed as a parameter to this method call, the method should raise an ArgumentNullException.
In order to do so, consider the following code.

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void MySimpleMethod_Throws_ArgumentNullException()
{
    sut.MySimpleMethod(null);
}

Once executed, this test will pass.
Now let’s write an async analog of the previous method.

public async Task MySimpleMethodAsync(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    await Task.Delay(100);
 
    return true;
}

If we now try to execute the following test, it will fail.

public async Task MySimpleMethodAsync(List param)
{
    if (param == null)
    {
        throw new ArgumentNullException("param");
    }
 
    await Task.Delay(100);
 
    return true;
}

As we do know that our exception is wrapped in an AggregateException, it is expected our test to fail. How should we than test this kind of situations?

Building a new attribute

Wait, I really liked the ExpectedException attribute, I would like to have an attribute that will check if any of the aggregated exceptions are of the type I do expect.
After a quick search I hadn’t found anything that suites me. That was the moment the ExpectedAggregateExceptionAttribute was born.
I followed the pattern on which ExpectedException was made and this is what I came up with.

/// 
/// Indicates that an exception is expected during test method execution.
/// It also considers the AggregateException and check if the given exception is contained inside the InnerExceptions.
/// This class cannot be inherited.
/// 
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public sealed class ExpectedAggregateExceptionAttribute : ExpectedExceptionBaseAttribute
{
    protected override string NoExceptionMessage
    {
        get
        {
            return string.Format("{0} - {1}, {2}, {3}",
                this.TestContext.FullyQualifiedTestClassName,
                this.TestContext.TestName,
                this.ExceptionType.FullName,
                base.NoExceptionMessage);
        }
    }
 
    /// 
    /// Gets the expected exception type.
    /// 
    /// 
    /// 
    /// A  object.
    /// 
    public Type ExceptionType { get; private set; }
 
    public bool AllowDerivedTypes { get; set; }
 
    /// 
    /// Initializes a new instance of the  class with and expected exception type and a message that describes the exception.
    /// 
    /// An expected type of exception to be thrown by a method.
    public ExpectedAggregateExceptionAttribute(Type exceptionType)
        : this(exceptionType, string.Empty)
    {
    }
 
    /// 
    /// Initializes a new instance of the  class with and expected exception type and a message that describes the exception.
    /// 
    /// An expected type of exception to be thrown by a method.
    /// If the test fails because an exception was not thrown, this message is included in the test result.
    public ExpectedAggregateExceptionAttribute(Type exceptionType, string noExceptionMessage)
        : base(noExceptionMessage)
    {
        if (exceptionType == null)
        {
            throw new ArgumentNullException("exceptionType");
        }
 
        if (!typeof(Exception).IsAssignableFrom(exceptionType))
        {
            throw new ArgumentException("Given type is not an exception.", "exceptionType");
        }
 
        this.ExceptionType = exceptionType;
    }
 
    /// The exception that is thrown by the unit test.
    protected override void Verify(Exception exception)
    {
        Type type = exception.GetType();
 
        if (this.AllowDerivedTypes)
        {
            if (!this.ExceptionType.IsAssignableFrom(type))
            {
                base.RethrowIfAssertException(exception);
 
                throw new Exception(string.Format("Test method {0}.{1} threw exception {2}, but exception {3} was expected. Exception message: {4}",
                    base.TestContext.FullyQualifiedTestClassName,
                    base.TestContext.TestName,
                    type.FullName,
                    this.ExceptionType.FullName,
                    GetExceptionMsg(exception)));
            }
        }
        else
        {
            if (type == typeof(AggregateException))
            {
                foreach (var e in ((AggregateException)exception).InnerExceptions)
                {
                    if (e.GetType() == this.ExceptionType)
                    {
                        return;
                    }
                }
            }
 
            if (type != this.ExceptionType)
            {
                base.RethrowIfAssertException(exception);
 
                throw new Exception(string.Format("Test method {0}.{1} threw exception {2}, but exception {3} was expected. Exception message: {4}",
                    base.TestContext.FullyQualifiedTestClassName,
                    base.TestContext.TestName,
                    type.FullName,
                    this.ExceptionType.FullName,
                    GetExceptionMsg(exception)));
            }
        }
    }
 
    private string GetExceptionMsg(Exception ex)
    {
        StringBuilder stringBuilder = new StringBuilder();
        bool flag = true;
 
        for (Exception exception = ex; exception != null; exception = exception.InnerException)
        {
            string str = exception.Message;
 
            FileNotFoundException notFoundException = exception as FileNotFoundException;
            if (notFoundException != null)
            {
                str = str + notFoundException.FusionLog;
            }
 
            stringBuilder.Append(string.Format((IFormatProvider)CultureInfo.CurrentCulture, "{0}{1}: {2}", flag ? (object)string.Empty : (object)" ---> ", (object)exception.GetType(), (object)str));
            flag = false;
        }
 
        return ((object)stringBuilder).ToString();
    }
}

To cut it short, I do check if the thrown exception is of AggregateException type, and if so I do again check if any of the inner exceptions is of the requested type. In case this condition is not matched a new exception is thrown which will make your unit test fail.
If we now use our new attribute instead of the ExpectedExceptionAttribute this test will be successful.

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void MySimpleMethodAsync_Throws_ArgumentNullException_2()
{
    sut.MySimpleMethodAsync(null).Wait();
}

In order to be sure that our new attribute is working properly I will make some other tests. First I do expect the unit test to fail in case no exception was thrown:

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void ArgumentNullException_Fail_If_No_Exception()
{
    sut.MySimpleMethodAsync(new List()).Wait();
}

Mind that this test is not meant to pass (do not include it in your solution).
Also, if in the inner exceptions there is no expected exception type, the test should fail:

[TestMethod]
[ExpectedAggregateException(typeof(ArgumentNullException))]
public void ArgumentNullException_Fail_If_Wrong_Inner_Exception()
{
    Task.Factory.StartNew(() =>
    {
        throw new ArgumentOutOfRangeException();
    });
}

As both test failed, I can conclude this new attribute works as supposed. You can package it as you wish, in your own common library, in a new or existing custom testing library or in any of your projects.

Downloads and the source code

You can find the source code and the demo project on Github at this address here.

Happy unit testing!