TFS 2015 Build System overview

Mario Majcica December 1, 2015 1 Comments

Introduction

With TFS 2015 a new build system was introduced. If you were using Visual Studio Online you may already have seen it. With TFS 2015 the build system has been completely revamped.

Team Foundation Server has included a build system since its first release. Historically, we started running our builds with XML based MSBuild scripts using a windows service as host. With TFS 2010 a new build system was introduced. It included a more robust infrastructure based on build controllers and agents and the build process was driven by a Workflow Foundation XAML script. Although it had its own limitations and very steep learning curve it was maintained through two versions of TFS, 2012 and 2013. Luckily, with TFS 2015, Microsoft decided to solve major issues of the build system and came up with a new solution. The new build system is finally cross platform and provides more flexibility for infrastructure.

What’s new

A great news for all of you who do not fancy the WWF and XAML (and I’m part of that group), the process is no longer based on it! Infrastructure got simplified and as already mentioned, there is now support for Mac OSX and Linux. On Microsoft website you can find an overview of the new features at https://msdn.microsoft.com/Library/vs/alm/Build/feature-overview

Infrastructure

Some new concepts are introduced into the build infrastructure landscape. There is no longer the necessity for having build controllers and all of the limitations and complexity they brought. Instead, two new concepts are now there to help us with grouping and organizing our build resources. Pool and queues are only “organizational” units and do not require any specific software to be installed or configured.

Pools

A pool is essentially an association of a collection of one or more build agents to one or more queues. Pools are defined at the TFS application tier level. You can create as many pools as you like and you can assign pool administrators to each of them. Different people/groups can be assigned to different pools allowing different groups of people to manage your various build assets.

In synthesis, agent pools are used to organize and define permission boundaries around your build agents.

Queues

Queues are defined at the project collection level and are tied directly to a pool. A queue can be tied to at most one pool; however, a pool can be tied to more than one queue assuming the queues are configured on multiple project collections. As build definitions are created they are associated with a queue. The queue you select for a build definition ultimately limits the set of build agents that can be utilized for the builds generated by that definition.

When you create a new queue you must immediately select the pool that is associated with it. You can also choose to have the queue created and associated with the new pool automatically at the time the pool is created (it’s an option on the new pool dialog).

Agents

The build agent is a small application residing on a build server, a machine that is intended to be used for executing automated builds. The build agents are responsible for building your application (based on the build definitions). Although a pool can be associated with many build agents, a build agent is associated with at most a single pool. With the new build system a Microsoft Cross Platform Build Agent is available and Mac OSX and Linux platforms are supported. We will see later on how to proceed with the installation of a build agent.

The following diagram is a schematic overview of these concepts:

Build process

Build process has drastically changed. We do not have any more the build process template, which is substituted by a collection of build tasks (steps). In order to define your build process, you will add one or more build tasks in your build definition. By default several build tasks are available as Visual Studio Build, Visual Studio Test and many more. As you can guess, each of them perform an operation in order to accomplish the process you had in mind for your build.

More tasks are available online as also all the source code of all the tasks that are available on TFS. You can find a list of the ones supplied by Microsoft at the following address https://msdn.microsoft.com/Library/vs/alm/Build/steps/index and the relevant source code at https://github.com/Microsoft/vso-agent-tasks.

Aside of this important change, there are several things we can benefit of, such as, versioning and real-time output.

Versioning

Build definition versioning and auditing is introduced with the new system so now you can easily see who made the change and exactly what change was made on your build definition. Any change to build definition is logged and you can add a note corresponding to your changes. You can see what changed and revert to the desired version directly from the web interface.

Real-time output

Some of the most noticeable things about the way that build runs under this new system are the ability to get a real-time visibility while the build runs, which means that we will spend less time trying to dig through logs to see what really happened with your build. You can expect that you will get the same output as your build run locally.

Interface

Unlike the previous version, where you can only make changes to your build definition from Visual Studio, new build definition management is all web based. You do not need any other tool than your web browser in order to create, execute and manage your builds.

Build task anatomy

Every build task is composed from at least a file describing the build task. Describing a task and defining the elements in the interface is done via a predefined structure expressed via a JSON format. At example:

{
   "id": "61ed2e1d-efb7-406e-a31b-81f5d22e6d54",
   "name": "TestTask",
   "friendlyName": "Name that is displayed in the list",
   "description": "Testing new TFS 2015 build system.",
   "category": "Package",
   …

At the bottom of the task file you need to indicate the name of the associated PowerShell script which actually implements the logic of the task itself, or its node .js equivalent in case the task is meant to run cross-platform. If the task should run on both windows and xplat agents, you can indicate both:

"execution": {
   "Node": {
      "target": "ant2.js",
      "argumentFormat": ""
   },
   "PowerShell": {
      "target": "$(currentDirectory)\ant.ps1",
      "argumentFormat": "",
      "workingDirectory": "$(currentDirectory)"
   }
}

Other custom files such as executable can also be shipped with the task. In case you need to localize the task, it can be done by adding the necessary resources. Also you can specify the icon that will represent the task.

Once all of the necessary items are in the folder, you need to zip it and upload to the server. Uploading is done by calling a specific resource on TFS REST API or by utilities provided by Microsoft. Currently only ‘Agent Pool Administrators’ are able to add/update or remove tasks. Tasks are server-wide, this means that you will upload to the server, not to a specific collection or project.

Once you start creating your custom task, the best thing is to check how certain functionalities are implemented by peeking at the source code of the tasks available on GitHub.

TFS 2015 Update 1

As of today, Microsoft started shipping Update 1 for TFS 2015. It brings a couple of things in regard to a build system. With this update, build administrators can now add permissions to agent queue, which will restrict who can use that queue in a build definition. Several new build tasks are added and several minor improvements are present.
For a full list of the improvements you can consult the release notes.

I hope this post gave you an idea about what the new build system brings and made some of the new concepts clearer. If you are not enthusiastic about it maybe the following article could help you getting the right reasons to onboard on TFS 2015 build system.
In the following months I plan to extensively write about TFS 2015 and it’s new build system, bringing several examples from a real life implementation and answers to the questions I got during a training course I held on this argument.

I bet you will enjoy it.

Happy building!

PowerShell tips and tricks – Decoding SecureString

Mario Majcica November 17, 2015 6 Comments

Introduction

It is quite common to get across SecureString type in PowerShell. If you ever needed to supply a credential object, at example to the Invoke-RestMethod cmdlet, then you for sure came across a SecureString. I will describe a couple of “tricks” that can be useful meanwhile working with SecureString objects.

Creating a SecureString

This is one of the most common cases. If you are constructing your System.Management.Automation.PSCredential object instance you need to provide a SecureString for the password argument in the constructor of just mentioned class. This implies that you need to transform your string to a SecureString. Luckily this is a trivial task since we have a cmdlet that can help us with that.

$securePassword = ConvertTo-SecureString –String $password -AsPlainText -Force

ConvertTo-SecureString will do just that. Be aware that you need to set two parameters in order to process your plain string correctly, and those are AsPlainText and Force.
AsPlainText indicates that you are providing a plain text as the input and that variable is not protected. With Force parameter you just confirm that you understand the implications of using the AsPlainText parameter and still want to use it.

Another way of creating a SecureString is to interactively prompt a user for information. You can achieve that in the following way.

$securePassword = Read-Host "Please enter the password" -AsSecureString

This is a indicated way of retrieving sensitive data if user interaction is required. Also you could request it through well known authentication dialog and retrieve the above mentioned PSCredential object. This object does expose a property called Password which is of type SecureString.

$credential = Get-Credential

As you can see, I used a Get-Credential cmdlet to trigger the authentication dialog.

If you need to persist your sensitive data in a text file or in a database, you can leverage ConvertFrom-SecureString cmdlet to transform your SecureString value into an encrypted string. Consider the following code:

ConvertFrom-SecureString $securePassword | Out-File C:\tmppassword.txt

Once stored, you can retrieve it again by

ConvertTo-SecureString -String (Get-Content C:\tmppassword.txt)

For using a custom key for the AES algorithm that is used for encryption by the ConvertFrom-SecureString cmdlet, you can specify it via a Key or SecureKey parameter. More info about this you can get directly on Technet site ConvertFrom-SecureString.
If no key is specified, the Windows Data Protection API (DPAPI) is used to encrypt the standard string representation.

Back to the string

We saw till now that we are able to create a SecureString from a plain text, express it as encoded string, and load it back to the SecureString object. What about getting a plain text out of our SecureString?
This is not an easy task to achieve. Luckily I came across the following article on MSDN which shows the right way to achieve this and translated it to PowerShell cmdlet. Next step was translating the shown code into PowerShell and encapsulating it in a cmdlet. This is the outcome.

function Get-PlainText()
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.Security.SecureString]$SecureString
	)
	BEGIN { }
	PROCESS
	{
		$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($SecureString);

		try
		{
			return [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr);
		}
		finally
		{
			[Runtime.InteropServices.Marshal]::FreeBSTR($bstr);
		}
	}
	END { }
}

Now it is sufficient to pass your secure string as a parameter and you will retrieve the plain text behind it.

You may ask yourself, why? When can this be handy? It happened to me to be in need to construct an Authentication Header for Basic access authentication.
What is expected by the server is a header in your http request that equals to Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ== where the last string is a Base64 encoded string called token and it is composed of the username and password in the following disposition “username:password”. All of my authentication methods are using System.Management.Automation.PSCredential object to express and move credentials. This is where decoding a secure string was important.
For completeness I’ll show you other cmdlets I wrote to accomplish this task.

function Get-AuthenticationToken()
{
	[CmdletBinding()]
	param
	(
		[parameter(Mandatory = $true)]
		[System.Management.Automation.PSCredential]$Credential
	)
	BEGIN
	{
	}
	PROCESS
	{
		$password = Get-PlainText $Credential.Password;
		$key = [string]::Format("{0}:{1}", $Credential.UserName, $password)

		return [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($key))
	}
	END { }
}

As you can see, I do get a PSCredential object as a parameter, I decode the password, compose the token string and at the end create a Base64 representation of it.
That’s about it.

How do you intend to use this tip?

UPDATE

Lately I came to know about another way of decoding a SecureString object and that is via the PSCredential object itself. In the above case we could do the following to retrieve the decrypted password:

$Credential.GetNetworkCredential().password

Meanwhile you can create the PSCredential object like this:

$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "dummyUsername", $securePassword

You can read more about this method at Decrypt PowerShell Secure String Password.

PowerShell tips and tricks – Retrieving TFS collections and projects

Mario Majcica November 15, 2015 4 Comments

Introduction

The following post is not really about a tip or a trick regarding a PowerShell itself. It’s more about on how to leverage some TFS libraries in order to automate processes regarding TFS via a PowerShell script. It will not show any fancy PowerShell technique but a way to query a TFS server of your choice, extract the necessary information and eventually make changes.

In this first tip we will see the essential, how to connect to the TFS, retrieve all of the available collections and list all of the projects for each TPC. Imagine that you are working on a TFS instance containing multiple collections, each one again having many projects (for many I do intend hundreds of projects in total). Verifying and changing settings on each one manually will not be easy nor convenient. As there is no UI for executing bulk operations on TFS, the easiest and most logical way to interact with it is through PowerShell. Let’s see how we can achieve that.

Prerequisites

Before we start, in order to be able to use these scripts you need to have at least PowerShell v3 installed and the necessary TFS libraries registered in GAC. The easiest way to make sure that you do have just mentioned libraries installed is to make sure that you do have Visual Studio installed on the machine you are executing this script from. Also it is good that the Visual Studio you added does match in version your TFS install. It means that if you do have TFS2013 installed, the best option will be to have Visual Studio 2013. For what concerns PowerShell, you do probably already have version 3 installed on your machine. The easiest way to check is to execute the following command:

$PSVersionTable.PSVersion

If is not the case that you do have PowerShell version 3 installed, you can do so by following this link: Windows Management Framework 3.0

In order to perform the call to the TFS server you need to have sufficient rights to do so. Managing collection objects require Edit instance-level information permission level, which is granted by default only to TFS admin. In case you do not have sufficient rights, you may encounter an exception reporting
Exception calling "GetCollections" with "0" argument(s): "Access Denied: Mario Majcica needs the following permission(s) to perform this action: Edit instance-level information"

Just for a sake of completeness I will add that this is not the only way of establishing a dialog with TFS, you could extract and set some of the information also through the REST API. Also some of the objects I do treat can be retrieved in a slightly more efficient way at the expense of simplicity.
In case you are interested about just mentioned techniques you can read the following post Building a TFS 2015 PowerShell Module using Nuget. Bare in mind that it is not targeting beginners as this blog post and it requires a deeper understanding on TFS Object Model and PowerShell.

Again, some utilities as Microsoft Visual Studio Team Foundation Server 2013 Power Tools do deliver PowerShell Cmdlets that can be used to work with different features of TFS such as changesets, shelvesets, workspaces and more. If interested in details about this approach I can advise you to Google out the argument or get your hands of a book called Windows PowerShell 4.0 for .NET Developers.

Let’s script

First things first. In order to reference the necessary libraries we need to issue the following command.

Add-Type -AssemblyName "Microsoft.TeamFoundation.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

This may seem a quite complicated way to load an assembly, unfortunately only a certain pre-defined set of assemblies can be loaded by their partial name. For all the rest a fully qualified name is necessary. In case you decide to use some other TFS functionalities, it may be necessary to reference other libraries.

Note, in case of Visual Studio 2015, the object model client libraries are removed from GAC. However the necesseray libraries are still available. In order to load them you need to approach the load in a different manner.

Add-Type -Path "C:Program Files (x86)Microsoft Visual Studio 14.0Common7IDECommonExtensionsMicrosoftTeamFoundationTeam ExplorerMicrosoft.TeamFoundation.Client.dll"

You need to point the Add-Type cmdlet to a path. It may vary based on the folder in which your Visual Studio is installed.

There is another option that may be a nice way to comply to this dependency. If you are an early adopter of PowerShell 5, you may retrieve the necessary package via OneGet.

Walking through TFS objects

All TFS libraries do have a same entry point. There are multiple factory classes, exposing static methods, that will give us the instance of classes implementing the requested interface thought whom we will perform desired operations. For a less experienced DevOps (at least less Dev’s more Ops) this may not make sense, thus let me try to explain it a bit better.
In order to use an object (a non-static class in this particular case) we need to construct an instance of it (and probably pass some parameters for its initialization). Via the factory pattern, adopted by Microsoft for this particular set of libraries, we are able to get the right instance of the class implementing the interface we are in need for. Still too complicated? Then I’m sorry, I’m already going way out of scope here.

In order to get the object through whom we will get the services we are in need for, we need to call a static method GetConfigurationServer on TfsConfigurationServerFactory class.
This call will return an instance of ConfigurationServer class which will be our main entry point for all the services.

$uri = "http://my.tfs.local:8080/tfs"
$tfsConfigurationServer = [Microsoft.TeamFoundation.Client.TfsConfigurationServerFactory]::GetConfigurationServer($uri)

As you can see I’m passing to the GetConfigurationServer method a parameter, which is the address of our TFS. It means that all of the operations performed through its service calls will be pointing to TFS indicated in this address.

Once our entry point is obtained, we can ask him to get us an instance of the class that implements the necessary logic to perform actions we are in need for.
In our case this is a class that implements ITeamProjectCollectionService interface.
We can request it with the following command

$tpcService = $tfsConfigurationServer.GetService("Microsoft.TeamFoundation.Framework.Client.ITeamProjectCollectionService")

Now that I do have a correct instance of the class that implements the interface which declares the methods I’m interested in, I will just call a method that will return a collection of TeamProjectCollection objects.
This method is called GetCollections and it is invoked as follows.

$tpcService.GetCollections() 

I’m now able to iterate through this collection and retrieve, beside others, the name of each project collection. Each object in the collection represents a project collection on our TFS. There should be always at least one element in it.

Let’s recap our script before continuing.

Add-Type -AssemblyName "Microsoft.TeamFoundation.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

$uri = "http://tfs:8080/tfs"
$tfsConfigurationServer = [Microsoft.TeamFoundation.Client.TfsConfigurationServerFactory]::GetConfigurationServer($uri)
$tpcService = $tfsConfigurationServer.GetService("Microsoft.TeamFoundation.Framework.Client.ITeamProjectCollectionService")

$sortedCollections = $tpcService.GetCollections() | Sort-Object -Property Name

foreach($collection in $sortedCollections) {
    Write-Host $collection.Name
}

The only small detail I haven’t mentioned until now, is the fact that once I do retrieve the collections I do sort them base on the value of Name property.

Now that we have a list of all TPC’s we can construct a URL that will be used to get other services which as a starting point do require a reference to the TPC.
As just mentioned in order to construct the URL I will declare another variable and concatenate the TFS URL and the project name after which I can request an instance of TfsTeamProjectCollectionFactory class that I will use as the entry point for all the operations on the given TPC.

As already seen for the ITeamProjectCollectionService we need to obtain a service that will provide us with the necessary data. In our case this is ICommonStructureService3. It is all achieved by the following code.

Now we are able to invoke a method called ListProjects in order to get all of the projects part of that TPC.

$collectionUri = $uri + "/" + $collection.Name
$tfsTeamProject = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection($collectionUri)
$cssService = $tfsTeamProject.GetService("Microsoft.TeamFoundation.Server.ICommonStructureService3")   
$sortedProjects = $cssService.ListProjects() | Sort-Object -Property Name

Each value in $sortedProjects will be of type ProjectInfo and within we will find all of the necessary information about that Team Project.
In between other properties and methods, as expected, we do have a property called Name. We will output the name of all the projects in that TCP.

The end result

I will add some of the formatting for our messages so that the output of the script is easier to read. Also I will collect the total number of projects. This code I do hope is not needed to be explained in detail. Following the complete script.

Add-Type -AssemblyName "Microsoft.TeamFoundation.Client, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

$uri = "http://tfs:8080/tfs"
$tfsConfigurationServer = [Microsoft.TeamFoundation.Client.TfsConfigurationServerFactory]::GetConfigurationServer($uri)
$tpcService = $tfsConfigurationServer.GetService("Microsoft.TeamFoundation.Framework.Client.ITeamProjectCollectionService")

$sortedCollections = $tpcService.GetCollections() | Sort-Object -Property Name
$numberOfProjects = 0

foreach($collection in $sortedCollections) {
    $collectionUri = $uri + "/" + $collection.Name
    $tfsTeamProject = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection($collectionUri)
    $cssService = $tfsTeamProject.GetService("Microsoft.TeamFoundation.Server.ICommonStructureService3")   
    $sortedProjects = $cssService.ListProjects() | Sort-Object -Property Name

    Write-Host $collection.Name "- contains" $sortedProjects.Count "project(s)"

    foreach($project in $sortedProjects)
    {
        $numberOfProjects++
        Write-Host (" - " + $project.Name)
    }
}

Write-Host
Write-Host "Total number of project collections" $sortedCollections.Count
Write-Host "Total number of projects           " $numberOfProjects

As there are plenty of properties available, your are able to change or check a setting on all of the projects in all of the TCP’s you have. This can be very handy for the maintenance task as we are going to see later in a blog post that I’m currently working on.

Happy coding!