Monitoring TFS 2015 availability from F5 LTM

Introduction

If redundancy and performance are the thing you are looking for your TFS application tier setup, for sure you stumbled upon the term Network Load Balancing (NLB). Microsoft describes the benefits of such a setup and prerequisites in the document named How to: Create a Team Foundation server farm (high availability), thus I will not go in the details about these topics if you continue reading. However, in the documentation, Microsoft encourages you to setup the NLB feature that is integrated in the Windows Server operating system. In many situations that is not an option due to the network restrictions or company policies and the only choice is to use preexisting networking appliances. Reasons for using a hardware based NLB can also be a performance as it offloads the AppTier machines from this task that, for how minor it can be on today’s machines, it adds some load.

Monitoring

In case of using the Windows NLB feature, nodes participating in the pool of the machines used for the load distribution are monitored directly by the system itself, meanwhile for the hardware based solutions we need to setup a health monitor. This is essential as the load balancer needs to know if the node is available and in healthy state, otherwise it is excluded from the pool and the traffic is not sent towards that node.

Now, what is the best practice when it comes to the health status of TFS? Googling around you can’t find much, there are some pointers towards a SOAP method called GetServerStatus exposed, however it doesn’t bring the necessary information.
Luckily there is a non documented rest resource that is exposed on TFS 2015 and beyond and you can reach it at the URL

http(s)://your.tfs.address:port/tfs/_apis/health

It will return just a simple current time stamp by default using the JSON notation. Accessing this resource still requires the user to be authenticated.

When it comes to the F5 in particular, you need to create HTTPS Health Monitor (Local Traffic > Monitors > Create…)

f5-health-monitor

The most important fields to set are Send and Receive string. Here we will send a request towards TFS at the above mentioned address and expect a status code 200 in the response. We can ignore the time stamp in the response body.
The send string will be:

GET /tfs/_apis/health HTTP1.1\r\nHost: your.tfs.address:port\r\n

meanwhile the receive string should be set to:

HTTP/1.1 200 OK

A simple check that the request succeeded (we are not interested in the timestamp in this case).

Do not also forget to provide a username and password of the account that has sufficient rights to access this resource on your TFS server. Username needs to be provided in the form of DOMAIN\UserName. A bare minimum of access rights are necessary for accessing this resource and a View instance-level information permission on the server level is more than sufficient. You can set server-level permissions from the Team Foundation Administration Console or using the TFSSecurity command line tool. Now assign the newly created health monitor to your NLB pool and you are ready to go.

In case you are trying to do so from a script for some of your custom dashboards, I wrote a CmdLet that will return true or false based on the response received from the call to the above mentioned REST resource.

function Get-ServerStatus()
{
	[CmdletBinding()]
    [OutputType([bool])]
	param
	(
		[parameter(Mandatory = $true)]
		[Uri]$url,
        [System.Management.Automation.PSCredential]$credential
	)
	BEGIN
    {
        if ($url.AbsoluteUri)
        {
            $url = $url.AbsoluteUri.TrimEnd('/')
        }

        Add-Type -AssemblyName System.Net.Http
	}
	PROCESS
	{
		$httpClientHandler = New-Object System.Net.Http.HttpClientHandler
 
        if ($Credential)
        {
		    $networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password)
		    $httpClientHandler.Credentials = $networkCredential
        }
        else
        {
            $httpClientHandler.UseDefaultCredentials = $true
        }

        $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler

        try
        {
			$response = $httpClient.GetAsync("$url/_apis/health").Result
 
			if ($response.IsSuccessStatusCode)
			{
				return $true
			}
 
			return $false
        }
        catch [Exception]
        {
			$PSCmdlet.ThrowTerminatingError($_)
        }
        finally
        {
            if($null -ne $httpClient)
            {
                $httpClient.Dispose()
            }
 
            if($null -ne $response)
            {
                $response.Dispose()
            }
        }

		return $false
	}
	END { }
}

It is sufficient to invoke this cmdlet by passing in the URL of your TFS instance and eventually the credentials. If no credentials are provided, current process credentials will be used.

$uri = "http(s)://your.tfs.address:port/tfs"
$credential = Get-Credential

$state = Get-ServerStatus $uri $credential

A simple solution is now in place that will keep other tools informed about the availability of our TFS instance.

Good luck!

Custom nuget.exe for TFS 2015 build

Introduction

I had a couple of users complaining about not being able to restore a specific version of AutoMapper package during their build. A quick search showed me that they are not the only one facing this issues and that this is quite a common problem. I verified that I’m able to reproduce this issue and I saw that it is presented based on the version of nuget client. As by default the build agent does use the nuget.exe that ships with the agent itself, I verified the version of it and saw that in my case (TFS 2015.3) it is 3.2.1.10581. With the 3.2.1.10581 version of nuget client I was unable to restore the package in question (AutoMapper.5.1.1) meanwhile from Visual Studio with version 3.4.3.855 all went well. The error I could see in the log is the following:

##[error]Unable to find version '5.1.1' of package 'AutoMapper'.

Without digging into details of why this is happening, I’ll show you how to push your build to use a different version of a nuget client.

Preparing the build server

As a first thing, let’s “install” the latest nuget version on our build server. Just download the latest version of nuget client and place it in a folder of your choice. Make sure that account on which your build agent is running has sufficient rights to access that path. For me it will be ‘D:\Program Files(x86)\Nuget‘.
Once placed your nuget.exe in the above mentioned folder, let’s add a system environment variable that will point to this executable. Open Control Panel > System and Security > System and choose Advanced system settings. In System properties dialog click Environment variables button and add a new System variable by click the new button (and be sure it is a System property and not a user variable). As a variable name choose NugetPath and as value set the path towards your nuget.exe file, which in my case is D:\Program Files(x86)\Nuget\nuget.exe

nuget-path-system

Now you should restart your agent services so that the new system variable is picked up by the agents. If everything went well you should see the following capability in the agent capability list:

agent-capability-nuget

If you can see it listed correctly, it all went well till now.

Setting up the build

Now it’s the time for the build. I suppose you are using the NuGet Installer build step in order to restore your packages before the build. If not, you should, as resorting the packages from Visual Studio Build step is obsolete and should not be used.
In order to force NuGet Installer build step to use our new nuget client, we need to expand the Advance group settings and set the Path to NuGet.exe option value to $(NuGetPath):

nuget-installer-build-step

Once this is done, just to be sure that only the build agents having the custom nuget version installed will be used, we are going also to specify a demand for our build. In the general tab of you build definition add new demand of type exists and set it to NuGetPath:

nugetpath-demand

Now, queue a new build and check in the log file that our new nuget client is used instead of the default one that ships with the build agent. You should find a similar line in your log:

D:\Program Files (x86)\NuGet\nuget.exe restore "E:\a1\_work\29\s\SimpleWebProject.sln" -NoCache -NonInteractive

That’s all folks, an easy way to push you build to use a specific version of the NuGet client instead of the default one.

TFS Job “Reporting Service Path Rename” failing

Introduction

After migrating our TFS 2015 server to a different AD domain we started seeing in our Job Monitoring page a large number of “Reporting Service Path Rename” job runs failing.

jobmonitoring

In the job details result message we can see the following, The permissions granted to user 'RABOSVC\prd.TFSService' are insufficient for performing this operation., where 'RABOSVC\prd.TFSService' is the application tier service account.

Solution

After a quick check, all the permissions set in the reporting services seemed to be all right. In the ‘Site Settings’ our service account was added in ‘System Administrator’ role, as on the ‘TfsReports’ folder it was set as ‘Team Foundation Content Manager’, and also inside ‘TfsReports’ folder it had also ‘Team Foundation Content Manager’ association on the Team Project Collection folder.

tfs-reports-security

As it turned out this was not sufficient. On the project level folders we also needed to associate our new TFS Service account to ‘Team Foundation Content Manager’ role. Once done, the “Reporting Service Path Rename” job managed to run successfully.

reporting-add-rights

Obviously this needs to be set only once, for the projects created previous to the domain change. Newly created projects will inherit the correct right from the Collection folder.

Although this is not a very common operation/situation, still I hope this can help someone in a similar situation as not many references to “Reporting Service Path Rename” job can be found on Google.

Also I would like to say a big thanks to Microsoft support which helped us diagnosing this issue.