Living Documentation and test reports in VSTS/TFS pipelines

Mario Majcica May 7, 2018 0 Comments

Introduction

In order to truly get advantage from all of the hard work that we put into our tests, we need to present our test run results and share our specifications in more convenient and accessible way. On Windows platform in order to make this tasks happen, we can leverage tools that you are probably already using, SpecFlow itself and a sidekick project of it called Pickles. If none of what I just said does make sense, you are reading the wrong post, so please check the SpecFlow documentation and read about BDD which is partly in PDF so using software as Soda PDF could be useful for this. However, if you are already familiar with it, you are using SpecFlow and are looking for a decent way to automate the above-mentioned tasks, please continue reading as I may have a valid solution to it.

All of the implementations that I came across till now, involved scripts, MSBuild tasks and a lot of other cumbersome solutions. I saw potential in VSTS/TFS build/release pipeline that through some specific build/release tasks are a neat solution automating these requirements.

Let’s start.

Generating SpecFlow reports in VSTS

Generating a nice and easy to consult report over our your test runs is relatively easy. SpecFlow NuGet package already includes all of the necessary to do so, that is the SpecFlow executable itself. In my demo case, I am using NUnit 3, however other frameworks are also supported.

Let’s check first what are the necessary manual steps to get the desired report.
After executing my tests with NUnit Console Runner with the following parameters

nunit3-console.exe --labels=All "--result=TestResult.xml;format=nunit2" SpecFlowDemo.dll

I am ready to generate my report. Now I just need to invoke the SpecFlow executable with the following parameters

specflow.exe nunitexecutionreport SpecFlowDemo.csproj /xmlTestResult:TestResult.xml /out:MyReport.html

And voila, the report is generated and it looks like following

Details over the various parameters accepted by SpecFlow executable can be found here, Test Execution Report.

Now, how do we integrate this into our VSTS build pipeline?

First, we need to run our tests in order to get the necessary test results. These may vary based on the testing framework that we are using. Two supported ones are NUnit or MSTest.

Be aware that if you run your MsTest’s with vstest runner, the output trx file will not be compatible with the format generated by mstest runner and the report will not render correctly. For that to work, you’ll need a specific logger for your vstest runner.

Once the tests are completed we are going to use the SpecFlow Report Generator task that is part of the homonymous extension that you can find here.
After adding the SpecFlow Report Generator task in your build definition, it will look similar to this.

In case you are interested in how each parameter influences the end result, check the above-mentioned link pointing to the SpecFlow documentation as the parameters are the same as on per tool invocation via the console.

Now that your report is ready you can process it further like making it part of the artifact or send it via email to someone.

Generating Pickles living documentation

Support documentation based on your specifications can be generated by Pickles in many ways, such us MSBuild task that could be part of your project, PowerShell library or simply by invoking the Pickles executable on the command line.

In case of trying to automate this task, you will probably use the console application or PS cmdlets. Let’s suppose the first case, then the command that we are looking for is like following

Pickles.exe --feature-directory=c:\dev\my-project\my-features --output-directory=c:\Documentation --documentation-format=dhtml

All of the available arguments are described here.

As the result you will get all of the necessary files to render the following page:

Back to VSTS. In order to replicate the same in your build definition, you can use the Pickles documentation generator task. Add the task to your definition and it should look like somewhere like this

All of the parameters do match the ones offered by the console application. You are now left to choose how and where further to ship this additional material.

Conclusion

In this post, I illustrated a way on how to get more out of the tooling that you are probably already using. Once you automate these steps chance is that they are going to stay up to date and thus probably get to be actually used. All of the tasks I used can also be used in the VSTS/TFS release pipeline.

I hope it helps.

Manage with CodedUI sites that are using Windows authentication

Mario Majcica April 2, 2015 6 Comments

Introduction

It can happen that you have a need to automate tests via CodedUI on Web Sites that do use Windows authentication. This means that no login forms will be presented and your browser, by default, will pass the currently logged in user credentials to the IIS. This is not handy when it comes to automated tests, as for sure, you are planning to use a specific user to run your tests and not the one that is running the test agent services.
When it comes to Internet Explorer we do have an option to set, so that the browser will always request the credentials for the authentications process. You can find this setting inside the Internet Options setting panel Security Tab:

In most of the cases, as you probably will be running your tests inside your intranet, you are going to select the “Local Intranet” zone and edit the settings for it by clicking the “Custom Level” button. The following screen will be shown:

Once you scroll to the bottom you will find the settings in question under User authentication -> Logon. By default, it is set to “Automatic logon only in intranet zone”. If we change it to “Prompt for user name and password” you will see that once we now reopen our site, a request to insert our credentials will be presented. In this way we can login wit a different user than the one we are logged in on this system.

Note that in some cases this settings can be disabled by a Group Policy. This is often the case in the larger organizations.

This is very handy and let’s see how to achieve this via our CodedUI test.

Setting the User Logon options from code

Instead of using the interface, we are going to try to set this option through the registry. This is quite handy as it can also bypass the group Policy restriction in case we do have sufficient rights. After some Googling I came across the following page Internet Explorer security zones registry entries for advanced users. As you can see from the title it enlists all of the settable options for Internet Explorer security zones via the registry entries. In between others there is the one we are interested to, precisely, “1A00 User Authentication: Logon”.
Our task is now clear, we need to open the key SoftwareMicrosoftWindowsCurrentVersionInternet SettingsZones1 under the Current User hive and change the value of 1A00 setting.
Let’s write some code that will help us with that.

First of all I am going to declare a couple of constants that will be used in our methods and a property that will expose the interested key:

private const string LocalIntranetZoneKeyPath =
	@"SoftwareMicrosoftWindowsCurrentVersionInternet SettingsZones1";
private const string LogonSettingValueName = "1A00";

private static RegistryKey _localIntranetZone;

/// 

/// Gets a key-level node in the registry regarding the ID Local Intranet Zone settings.
/// 
protected static RegistryKey LocalIntranetZone
{
	get
	{
		if (_localIntranetZone == null)
		{
			_localIntranetZone = Registry.CurrentUser.OpenSubKey(LocalIntranetZoneKeyPath, true);
		}

		return _localIntranetZone;
	}
}

To follow are the methods that will allow me to retrieve and set the correct logon settings. In order to have a cleaner and more precise overview over the possible options, I am going to create and use an enumeration in order to manipulate them. I will call my enumeration LogonSetting and it will list all of the accepted values.

public enum LogonSetting
{
    NotSet = -1,
    AutomaticallyLogonWithCurrentUsernameAndPassword = 0x00000,
    PromptForUserNameAndPassword = 0x10000,
    AutomaticLogonOnlyInTheIntranetZone = 0x20000,
    AnonymousLogon = 0x30000
}

All of the necessary information came from the MSDN document I previously mentioned.
Plus I added a NotSet value for the cases in which the value 1A00 doesn’t exists, which is plausible in certain cases.
At the end, the two methods that will set and retrieve the values in question:

/// 

/// Sets the IE Logon setting to the desired value.
/// 
/// The desired value to assign to the Logon Setting.
public static void SetLogonSettings(LogonSetting logonSetting)
{
	if (logonSetting == LogonSetting.NotSet)
	{
		LocalIntranetZone.DeleteValue(LogonSettingValueName);
	}
	else
	{
		LocalIntranetZone.SetValue(LogonSettingValueName, (int)logonSetting);
	}
}

/// 

/// Retrieves the current IE Logon setting.
/// 
public static LogonSetting GetLogonSettings()
{
	object logonSettingValue = LocalIntranetZone.GetValue(LogonSettingValueName);

	if (logonSettingValue == null)
	{
		return LogonSetting.NotSet;
	}

	return (LogonSetting)logonSettingValue;
}

This is all of the necessary code that we need to comfortably interact with this settings.
You can note that we are using an extra state in our enumerator that indicates the value not being set at all. If that is the case, once we are setting the value, we need to eventually remove it from the registry.

Using the code in CodedUI

Once we start laying down our CodedUI test code we need to choose what strategy we are going to adopt for preforming our task of changing the IE setting. There are couple of places in which we can do that, more precisely three events we can consider for this task. CoudedUI put’s on our disposition three attributes that we can use to trigger the execution of our code at certain, predetermined moment. This attributes are respectively TestInitialize, ClassInitialize and AssemblyInitialize. You can read more about this attributes at the following page Anatomy of a Unit Test. As most of you probably already came across this attributes I will not get in the details about them and I will pick the TestInitialize which runs given code before the run of each test. Based on your situation you may prefer to perform this only once per assembly or at the class level. The choice is yours and the implementation may vary based on your needs.

What we need to do before our test is executed is:

1. Retrieve the current value of this setting, so that we can restore it once the test is done.
2. Change the setting to the desire state.

This is how it translates to the code:

private const LogonSetting DesiredLogonSetting = LogonSetting.PromptForUserNameAndPassword;
private LogonSetting originalLogonSetting;

[TestInitialize()]
public void MyTestInitialize()
{
    originalLogonSetting = Page.GetLogonSettings();

    if (originalLogonSetting != DesiredLogonSetting)
    {
        Page.SetLogonSettings(DesiredLogonSetting);
    }
}

As you can see we are persisting the original value inside a variable on the class level by using our previously created method GetLogonSettings() then checking if perhaps it is already set to our desired value (so that we may be do not need to change it) and if not we are using our SetLogonSettings() method to set it to the desired value.

Now our browser will be set to always prompt for user name and password. The next thing is to restore the original condition. We are going to use the antagonistic attribute to TestInitialize which is called TestCleanup.

[TestCleanup()]
public void MyTestCleanup()
{
    if (originalLogonSetting != DesiredLogonSetting)
    {
        Page.SetLogonSettings(originalLogonSetting);
    }
}

Again, we check if the desired setting is not our original setting (and in that case we do not need to do nothing), otherwise we set our setting to the original value.

All done!

In the following paragraph we will see on how this works and how to authenticate via the Windows Security window.

Last missing piece

In order to intercept the Windows Security window, with whom we are going to interact and automatically provide the credentials, we need to declare it in the way that CodedUI can recognize it and map the elements we are going to interact with. To be clear which window we are speaking about, here is a picture of it.

This is the window that will be presented once the IE is asked to provide the credentials.
With the following code we will declare this window so that CodedUI is able to recognize it.

public class WindowsSecurityWindow : WinWindow
{
    private WinText uiUseanotheraccountText;
    private WinEdit uiUsernameEdit;
    private WinEdit uiPasswordEdit;
    private WinButton uiokButton;

    public WindowsSecurityWindow()
    {
        SearchProperties[PropertyNames.Name] = "Windows Security";
        SearchProperties[PropertyNames.ClassName] = "#32770";
        TechnologyName = "MSAA";
        WindowTitles.Add("Windows Security");
    }

    public WinText UseAnotherAccountText
    {
        get
        {
            if ((uiUseanotheraccountText == null))
            {
                uiUseanotheraccountText = new WinText(this);
                uiUseanotheraccountText.SearchProperties[WinText.PropertyNames.Name] = "Use another account";
            }

            return uiUseanotheraccountText;
        }
    }

    public WinEdit UsernameEdit
    {
        get
        {
            if ((uiUsernameEdit == null))
            {
                uiUsernameEdit = new WinEdit(this);
                uiUsernameEdit.SearchProperties[WinEdit.PropertyNames.Name] = "User name";
            }

            return uiUsernameEdit;
        }
    }

    public WinEdit PasswordEdit
    {
        get
        {
            if ((uiPasswordEdit == null))
            {
                uiPasswordEdit = new WinEdit(this);
                uiPasswordEdit.SearchProperties[WinEdit.PropertyNames.Name] = "Password";
            }

            return uiPasswordEdit;
        }
    }

    public WinButton OkButton
    {
        get
        {
            if ((uiokButton == null))
            {
                uiokButton = new WinButton(this);
                uiokButton.SearchProperties[WinButton.PropertyNames.Name] = "OK";
            }

            return uiokButton;
        }
    }

    public void Authenticate(string userName, string password)
    {
        if (UseAnotherAccountText.Exists)
        {
            Mouse.Click(UseAnotherAccountText);
        }

        UsernameEdit.Text = userName;
        PasswordEdit.Text = password;

        Mouse.Click(OkButton);
    }
}

Aside the standard code, you can see that we are searching for an element called “Use another account”. It can happen in certain cases that a variation of the window we saw in the previous image gets presented. The variation looks like following:

If this is the case we are still able to handle it correctly. We are going first to select the right tab:

And insert the credentials as in the ordinary case (luckily element names are always called the same).
What remains is to recall it in the following way and pass in the desired credentials.

WindowsSecurityWindow windowsSecurityWindow = new WindowsSecurityWindow();
windowsSecurityWindow.Authenticate("userName", "password");

Putting all together

We saw all the pieces of the puzzle. Now let’s see how to prepare the IE, start it and authenticate. In the bottom of this post you will find a link where you can download my example project. There is an ASP.NET web site which uses Windows Authentication and relative CodedUI project which executes the test. In this way you can have the complete picture.

Our CodedUI test will state the following

[TestMethod]
public void CodedUITestMethod1()
{
    BrowserWindow.Launch(new Uri("http://localhost:59542/"));

    WindowsSecurityWindow windowsSecurityWindow = new WindowsSecurityWindow();
    windowsSecurityWindow.Authenticate(@"Home8test", "test");
}

As you can see, we are launching the browser window and pointing it to our application (make sure IIS Express is running before you do execute your test) and just specifying our WindowsSecurityWindow object. After that authenticate will kick in and all the actions will be performed as expected. Thanks to the code we wrote earlier, the browser will request the credentials to be provided and it will not try to login with our current user.

Note: In order this example to work, with IIS Express, you will need to enable windows authentication for IIS Express.

In order to enable windows authentication in IIS express open the file called applicationhost.config which is located in My DocumentsIISExpressconfig which again translated in my case is C:UsersMaiODocumentsIISExpressconfig. Once you open this file for edit and search the “windowsAuthentication enabled” string. You will land on the right spot. By default, this element is set to false. Just set it to true and you are ready to go.

This is an abstract of the final state of applicationhost.config.

...
  
...
    
      
    
...
  
...

One more thing, make sure that you create an user that you are going to use to test this scenario. Open the computer management and create a dummy user (in my case called test with a strong password equaling to test).

Last thing to do is to change the application web config with the right user and update the user name and credentials into your test.

You can now open your Test Explorer and hit run! You should see IE starting and pointing to your application page, authentication window popping out, credentials being passed and finally you are logged in!

That’s all folks

I hope you enjoyed reading this article and that you fancy the neat technique of managing the windows authentication. In this way you do not need to pre-prepare your clients on which you are going to execute your tests and more important you do not need to mess up wit the Credential Manager.
Stay tuned for more articles about CodedUI and testing automation.

Happy coding!

Download the complete example

moq.Callback(), The Unknown

Mario Majcica October 14, 2014 0 Comments

Introduction

More and more often I do see people having trouble testing certain type of code. As a result, code coverage is dropping down, unverified logics are shown up, lowering the quality and rising frustrations. This is what pushed me to write this post and describe this kind of situations and a decent solution to it.
This kind of situations are commonly found in all the cases in which the tested unit manipulates the arguments that are passed to our mock. In this case we need to verify that this transformation did what we expected it to do. However, it ain’t that simple and straightforward as it seems. As code speaks more than thousand words, let’s illustrate this with an example.
Consider the following class:

public class ProductService
{
    private readonly IOrderRepository m_orderRepository;

    public ProductService(IOrderRepository orderRepository)
    {
        m_orderRepository = orderRepository;
    }

    public List GetProducts(int customerId, int orderId)
    {
        OrderSearchCriteria orderSearchCriteria = new OrderSearchCriteria
        {
            OrderId = customerId // THIS IS THE PROBLEM WE ARE GOING TO SEARCH FOR

            // Set some other search criteria...
        };

        Order retrievedOrder = m_orderRepository.GetOrder(orderSearchCriteria);

        // Do something else
        return retrievedOrder.Products;
    }
}

What you can see here is a hypothetical product service that we are going to test. More precisely we are going to write unit tests for the GetProducts method. What this method does in particular is composing another object that will be passed to our dependency, the order repository. Now you can argue that this is a bad practice, that the object composition should be handled in a different manner, as usually in this cases the single responsibility principle is not met. And you are right, but we do not live in a perfect world and often we can’t easily change what is already there. However, we need to keep extending and improving our software.

Still, I do need to write a test for that method. What should I do, how do I spot this bug that we just introduced?

There are two ways of writing a unit test that will test, verify and spot our bug. Let’s start with the first one.

Shaping an expected instance

We can tackle this problem by setting up manually, in our test, an instance of OrderSearchCriteria class, as we expect it to be, base on the parameters that we are passing in, and make sure that our mock accepts only an instance that equals ours, on purpose created class.

Let’s check our unit test.

[TestMethod]
public void GetProducts_Creates_OrderSearchCriteria_Correctly()
{
    const int customerId = 56789;
    const int orderId = 12345;

    OrderSearchCriteria orderSearchCriteria = new OrderSearchCriteria
    {
        OrderId = orderId
    };

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(orderSearchCriteria))
        .Returns(new Order());

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(customerId, orderId);
}

Now, first thing first. In order this example to even work, your parameter class needs to implement the equality members. This is necessary as Moq, in order to determine equality of parameters, rightly, relays on Equals() method.

Another disadvantage of this technique is the fact that construction of our own object can sometimes be hard or even not possible. Not to even mention the maintenance problem we are going to introduce.

As Moq in the case of wrong parameter will return a null from the method call, often null value is managed and interpreted as a possible state. In that case it will be very hard or impossible to discover our bug.
Luckily there is a cleaner way to approach this kind of situations.

Extracting the parameter via Callback method

As it is not often used, many developers tend to ignore the Callback method that is provided by Moq framework. In this kind of situations it can be very handy.

Check out the following test.

[TestMethod]
public void GetProducts_Creates_OrderSearchCriteria_Correctly_2()
{
    const int customerId = 56789;
    const int orderId = 12345;

    OrderSearchCriteria recievedOrderSearchCriteria = null;

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(It.IsAny<OrderSearchCriteria>()))
        .Returns(new Order())
        .Callback(o => recievedOrderSearchCriteria = o);

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(customerId, orderId);

    Assert.IsNotNull(recievedOrderSearchCriteria);
    Assert.AreEqual(orderId, recievedOrderSearchCriteria.OrderId);
}

You can see that I’m setting up my mock and I’m specifying what the callback should do. What I’m telling him in this case is that for the parameter of type OrderSearchCriteria, once the method is invoked, copy it to the locally defined object called recievedOrderSearchCriteria. This will give me the possibility to check what came in the call of the GetOrder method, and verify that is what I expect to receive.
Once I start my assertions I do a check on the recievedOrderSearchCriteria and I do make sure that what came in, is what I do expect.

This test will fail and we will succeed in our intent. Also the message that we are receiving is far way clearer than one in the previous example. It states at this moment

Assert.AreEqual failed. Expected:<12345>. Actual:<56789>.

Beside this we are actually asserting the expected result thus specifying behavior in an explicit way.
Now, this on my opinion is much better!

Other considerations about the Callback method

For a less experienced developers, I’ll also make an example of how to get a callback working if you have more then one parameter accepted by your mocked object.
I’m going to extend our IOrderRepository interface by adding a overload of the GetOrder method that accepts two parameters. Also I will implement another method on ProductService class that uses this newly created method.

public interface IOrderRepository
{
    Order GetOrder(OrderSearchCriteria searchCriteria);

    Order GetOrder(int orderId, bool archieved);
}

public List GetProducts(int orderId)
{
    Order retrievedOrder = m_orderRepository.GetOrder(orderId, true);

    return retrievedOrder.Products;
}

In order to mock my order repository and have on callback the necessary values, the following test is used.

[TestMethod]
public void GetProducts_With_Archieved_Orders()
{
    const int orderId = 12345;

    int receivedOrderId = 0;
    bool receivedArchieved = false;

    Mock orderRepositoryMock = new Mock();
    orderRepositoryMock
        .Setup(m => m.GetOrder(It.IsAny(), It.IsAny()))
        .Returns(new Order())
        .Callback((o, a) =>
        {
            receivedOrderId = o; 
            receivedArchieved = a;
        });

    ProductService sut = new ProductService(orderRepositoryMock.Object);

    List result = sut.GetProducts(orderId);

    Assert.AreEqual(orderId, receivedOrderId);
    Assert.AreEqual(true, receivedArchieved);
}

As you can see, I just added an extra type in my generic definition and then adjusted my lambda expression accordingly. If more than two parameters are required, you can just follow the same pattern and define them as many as need. At example .Callback((o, a, i) etc.

You always need to respect the exact parameter signature of the method you are setting up. The number of parameters accepted by the mocked method need to mach in type, order and number the parameters accepted by your mocked method.

There is another way to express the same statement, just by using the lambda expression instead of generics. I can rewrite the above examples in the following way:

.Callback((OrderSearchCriteria o) => recievedOrderSearchCriteria = o);

// ...

.Callback((int o, bool a) =>
{
    receivedOrderId = o;
    receivedArchieved = a;
});

The effect is the same, it is only about your preference which of the two ways to use.

It is also possible to define it before and after the method invocation and as I can’t think of a nice example I will just provide the one from the Moq documentation:

// callbacks can be specified before and after invocation
mock.Setup(foo => foo.Execute("ping"))
    .Callback(() => Console.WriteLine("Before returns"))
    .Returns(true)
    .Callback(() => Console.WriteLine("After returns"));

Conclusion

Each time you need to check the arguments that are passed to the method you are setting up, Callback() will help you get to them. Also, if you need to execute any code before or after the method is invoked, Callback() will let you do so.
Hopefully you are not going to use it on a daily basis, but it is handy to know about it as sooner or later you are going to face a situation where a Callback() will help you achieve your goal.