Passing values between TFS 2015 build steps

Introduction

It may happen that you need to pass or make available a value from one build step in your build definition, to all other build steps that are executed after that one. It is not well documented, but there are two ways of passing values in between build steps.
Both of the approaches I am going to illustrate here are based on setting a variable on the task context. The first task can set a variable, and following tasks are able to use the variable. The variable is exposed to the following tasks as an environment variable. When ‘issecret’ option is set to true, the value of the variable will be saved as secret and masked out from log.

In order to test my approaches, I made 4 build tasks, two of them for the first approach and two of them for the second one. They look something like this:

task-1-set

You can download all of the examples here.

Task Logging Command

Build task context supports several types of logging commands. Logging command are a facility available in the run context of the build engine and do allow us to perform some particular actions. A full list of available logging commands is available here. Be aware that some of them are only available since a certain version of the build agent, and before using them check the minimum agent version for clarity, eventually limit your build task to the indicated version.
In our example, I will use a variable provided as a parameter to my task with a value set from the UI, print the value out and store it in the task context. This will be my build task called task1. Once I set the task manifest correctly, I will upload it to my on-premise TFS (it will also work on VSTS/VSO). Let’s check the code.

Write-Output "Message is: $msg"
Write-Output ("##vso[task.setvariable variable=task1Msg;]$msg")

Now, I will create another task and call it task2. This task will try to retrieve the value from the task context, then print the value out. Following is the code I’m using.

$task1Msg = $env:task1Msg

if ($task1Msg)
{
	Write-Output "Value of the message is set and equals to $task1Msg"
}
else
{
	Write-Output "Value of the message is not set."
}

You can find all of the examples I’m using available for download here.

If you now set your build definition with tasks task1 and task2, you will see that based on what you set as a Message in task1, will be visualized in the console by the task2.

task-1-2-run

Set-TaskVariable and Get-TaskVariable cmdlets

The same result can be achieved by using some cmdlets that are made available by the build execution context. We are going to create a task called task3 which will have the same functionality as task1, but it will use a different approach to achieve the same. Let’s check the code:

Import-Module "Microsoft.TeamFoundation.DistributedTask.Task.Common"
    
Set-TaskVariable "task1Msg" $msg 

As you can see we are importing modules from a specific library called Microsoft.TeamFoundation.DistributedTask.Task.Common. You can read more about the available modules and exposed cmdlets in my post called Available modules for TFS 2015 build tasks.
After that we are able to invoke the Set-TaskVariable cmdlet and pass the variable name and value as parameters. This will do the job.

In order to retrieve task we need to import another module which is Microsoft.TeamFoundation.DistributedTask.Task.Internal. Although those are quite connected actions, for some reason Microsoft decided to package them in two different libraries. We are going to retrieve our variable value in the following way:

Import-Module "Microsoft.TeamFoundation.DistributedTask.Task.Internal"

$task1Msg = Get-TaskVariable $distributedTaskContext "task1Msg"

if ($task1Msg)
{
	Write-Output "Value of the message is set and equals to $task1Msg"
}
else
{
	Write-Output "Value of the message is not set."
}

As you can see we are invoking the Get-TaskVariable cmdlet with two parameters. First one is the task context. As for other things, build execution context will be populated with a variable called distributedTaskContext which is of type Microsoft.TeamFoundation.DistributedTask.Agent.Worker.Common.TaskContext and beside other exposes information like ScopeId, RootFolder, ScopeType, etc. Second parameter is the name of the variable we would like to retrieve.

The result still stays the same as you can see from the following screenshot:

task-3-4-run

Conclusion

Both approaches will get the job done. It is on you to choose the preferred way of storing and retrieving values. A mix of both also works. You need to experiment a bit with the build task context and see what works for you the best. Be aware that this is valid at the moment I’m writing this, and it is working with TFS 2015, TFS 2015.1 and with VSTS as of 2016-02-19. Seems however that in the near future all of this will be heavily refactored by Microsoft, then my guide will not be relevant anymore.

Stay tuned!

TFS 2015 Build System overview

Introduction

With TFS 2015 a new build system was introduced. If you were using Visual Studio Online you may already have seen it. With TFS 2015 the build system has been completely revamped.

Team Foundation Server has included a build system since its first release. Historically, we started running our builds with XML based MSBuild scripts using a windows service as host. With TFS 2010 a new build system was introduced. It included a more robust infrastructure based on build controllers and agents and the build process was driven by a Workflow Foundation XAML script. Although it had its own limitations and very steep learning curve it was maintained through two versions of TFS, 2012 and 2013. Luckily, with TFS 2015, Microsoft decided to solve major issues of the build system and came up with a new solution. The new build system is finally cross platform and provides more flexibility for infrastructure.

What’s new

A great news for all of you who do not fancy the WWF and XAML (and I’m part of that group), the process is no longer based on it! Infrastructure got simplified and as already mentioned, there is now support for Mac OSX and Linux. On Microsoft website you can find an overview of the new features at https://msdn.microsoft.com/Library/vs/alm/Build/feature-overview

Infrastructure

Some new concepts are introduced into the build infrastructure landscape. There is no longer the necessity for having build controllers and all of the limitations and complexity they brought. Instead, two new concepts are now there to help us with grouping and organizing our build resources. Pool and queues are only “organizational” units and do not require any specific software to be installed or configured.

Pools

A pool is essentially an association of a collection of one or more build agents to one or more queues. Pools are defined at the TFS application tier level. You can create as many pools as you like and you can assign pool administrators to each of them. Different people/groups can be assigned to different pools allowing different groups of people to manage your various build assets.

In synthesis, agent pools are used to organize and define permission boundaries around your build agents.

Queues

Queues are defined at the project collection level and are tied directly to a pool. A queue can be tied to at most one pool; however, a pool can be tied to more than one queue assuming the queues are configured on multiple project collections. As build definitions are created they are associated with a queue. The queue you select for a build definition ultimately limits the set of build agents that can be utilized for the builds generated by that definition.

When you create a new queue you must immediately select the pool that is associated with it. You can also choose to have the queue created and associated with the new pool automatically at the time the pool is created (it’s an option on the new pool dialog).

Agents

The build agent is a small application residing on a build server, a machine that is intended to be used for executing automated builds. The build agents are responsible for building your application (based on the build definitions). Although a pool can be associated with many build agents, a build agent is associated with at most a single pool. With the new build system a Microsoft Cross Platform Build Agent is available and Mac OSX and Linux platforms are supported. We will see later on how to proceed with the installation of a build agent.

The following diagram is a schematic overview of these concepts:

PoolsAndQueues

Build process

Build process has drastically changed. We do not have any more the build process template, which is substituted by a collection of build tasks (steps). In order to define your build process, you will add one or more build tasks in your build definition. By default several build tasks are available as Visual Studio Build, Visual Studio Test and many more. As you can guess, each of them perform an operation in order to accomplish the process you had in mind for your build.

TFS 2015 Build Tasks

More tasks are available online as also all the source code of all the tasks that are available on TFS. You can find a list of the ones supplied by Microsoft at the following address https://msdn.microsoft.com/Library/vs/alm/Build/steps/index and the relevant source code at https://github.com/Microsoft/vso-agent-tasks.

Aside of this important change, there are several things we can benefit of, such as, versioning and real-time output.

Versioning

Build definition versioning and auditing is introduced with the new system so now you can easily see who made the change and exactly what change was made on your build definition. Any change to build definition is logged and you can add a note corresponding to your changes. You can see what changed and revert to the desired version directly from the web interface.

TFS 2015 Build definition history

Real-time output

Some of the most noticeable things about the way that build runs under this new system are the ability to get a real-time visibility while the build runs, which means that we will spend less time trying to dig through logs to see what really happened with your build. You can expect that you will get the same output as your build run locally.

Interface

Unlike the previous version, where you can only make changes to your build definition from Visual Studio, new build definition management is all web based. You do not need any other tool than your web browser in order to create, execute and manage your builds.

Build task anatomy

Every build task is composed from at least a file describing the build task. Describing a task and defining the elements in the interface is done via a predefined structure expressed via a JSON format. At example:

{
   "id": "61ed2e1d-efb7-406e-a31b-81f5d22e6d54",
   "name": "TestTask",
   "friendlyName": "Name that is displayed in the list",
   "description": "Testing new TFS 2015 build system.",
   "category": "Package",
   …

At the bottom of the task file you need to indicate the name of the associated PowerShell script which actually implements the logic of the task itself, or its node .js equivalent in case the task is meant to run cross-platform. If the task should run on both windows and xplat agents, you can indicate both:

"execution": {
   "Node": {
      "target": "ant2.js",
      "argumentFormat": ""
   },
   "PowerShell": {
      "target": "$(currentDirectory)\ant.ps1",
      "argumentFormat": "",
      "workingDirectory": "$(currentDirectory)"
   }
}

Other custom files such as executable can also be shipped with the task. In case you need to localize the task, it can be done by adding the necessary resources. Also you can specify the icon that will represent the task.

Once all of the necessary items are in the folder, you need to zip it and upload to the server. Uploading is done by calling a specific resource on TFS REST API or by utilities provided by Microsoft. Currently only ‘Agent Pool Administrators’ are able to add/update or remove tasks. Tasks are server-wide, this means that you will upload to the server, not to a specific collection or project.

Once you start creating your custom task, the best thing is to check how certain functionalities are implemented by peeking at the source code of the tasks available on GitHub.

TFS 2015 Update 1

As of today, Microsoft started shipping Update 1 for TFS 2015. It brings a couple of things in regard to a build system. With this update, build administrators can now add permissions to agent queue, which will restrict who can use that queue in a build definition. Several new build tasks are added and several minor improvements are present.
For a full list of the improvements you can consult the release notes.

I hope this post gave you an idea about what the new build system brings and made some of the new concepts clearer. If you are not enthusiastic about it maybe the following article could help you getting the right reasons to onboard on TFS 2015 build system.
In the following months I plan to extensively write about TFS 2015 and it’s new build system, bringing several examples from a real life implementation and answers to the questions I got during a training course I held on this argument.

I bet you will enjoy it.

Happy building!

Available modules for TFS 2015 build tasks

It is not yet well documented but if you are writing a custom build task for your TFS 2015 build system, you get at your disposition some of the modules that are made available by your VsoWorker.
As I couldn’t find a list of available modules and cmdlets they expose, I decided to dig into them and see what is there. I also wanted to check if with the Update 1 (and the new version of the build agent) there will be more of them.
I wrote a handy task that will list all of the available cmdlets for all of the modules shipped by the agent worker. Following are my results.

With TFS 2015 the agent version delivered is 1.83.2. It offers the following modules:

  • Microsoft.TeamFoundation.DistributedTask.Task.Common.dll
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Azure.psm1
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Chef.psm1
  • Microsoft.TeamFoundation.DistributedTask.Task.DevTestLabs.dll
  • Microsoft.TeamFoundation.DistributedTask.Task.DTA.dll
  • Microsoft.TeamFoundation.DistributedTask.Task.Internal.dll
  • Microsoft.TeamFoundation.DistributedTask.Task.TestResults.dll

I will now list of all importable functions and cmdlets in those modules.

  • Microsoft.TeamFoundation.DistributedTask.Task.Common.dll
    • Add-TaskIssue
    • Complete-Task
    • Find-Files
    • Get-LocalizedString
    • Set-TaskProgress
    • Set-TaskVariable
    • Write-TaskDetail
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Azure.psm1
    • Get-AzureCmdletsVersion
    • Get-AzureModuleLocation
    • Get-AzureVersionComparison
    • Get-RequiresEnvironmentParameter
    • Get-SelectNotRequiringDefault
    • Import-AzurePowerShellModule
    • Initialize-AzurePowerShellSupport
    • Initialize-AzureSubscription
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Chef.psm1
    • Get-DetailedRunHistory
    • Get-PathToNewtonsoftBinary
    • Get-ShouldWaitForNodeRuns
    • Get-TemporaryDirectoryForChef
    • Initialize-ChefRepo
    • Invoke-GenericMethod
    • Invoke-Knife
    • Invoke-WithRetry
    • Wait-ForChefNodeRunsToComplete
  • Microsoft.TeamFoundation.DistributedTask.Task.DevTestLabs.dll
    • Complete-EnvironmentOperation
    • Complete-EnvironmentResourceOperation
    • Complete-ResourceOperation
    • Copy-FilesToAzureBlob
    • Copy-FilesToRemote
    • Copy-FilesToTargetMachine
    • Copy-ToAzureMachines
    • Get-Environment
    • Get-EnvironmentProperty
    • Get-EnvironmentResources
    • Get-ProviderData
    • Invoke-BlockEnvironment
    • Invoke-EnvironmentOperation
    • Invoke-PsOnRemote
    • Invoke-ResourceOperation
    • Invoke-UnblockEnvironment
    • New-OperationLog
    • Register-Environment
    • Register-EnvironmentDefinition
    • Register-Provider
    • Register-ProviderData
    • Remove-Environment
    • Remove-EnvironmentResources
  • Microsoft.TeamFoundation.DistributedTask.Task.DTA.dll
    • Invoke-DeployTestAgent
    • Invoke-RunDistributedTests
  • Microsoft.TeamFoundation.DistributedTask.Task.Internal.dll
    • Add-BuildArtifactLink
    • Add-BuildAttachment
    • Convert-String
    • Copy-BuildArtifact
    • Get-JavaDevelopmentKitPath
    • Get-MSBuildLocation
    • Get-ServiceEndpoint
    • Get-TaskVariable
    • Get-ToolPath
    • Get-VisualStudioPath
    • Get-VssConnection
    • Get-X509Certificate
    • Invoke-Ant
    • Invoke-BatchScript
    • Invoke-IndexSources
    • Invoke-Maven
    • Invoke-MSBuild
    • Invoke-PublishSymbols
    • Invoke-Tool
    • Invoke-VSTest
    • Publish-BuildArtifact
    • Register-XamarinLicense
    • Unregister-XamarinLicense
  • Microsoft.TeamFoundation.DistributedTask.Task.TestResults.dll
    • Invoke-ResultPublisher
    • Publish-TestResults

The Update 1 RC1 for TFS increased the agent to the version 1.89.0 and the RC2 incremented it to version 1.89.1. As of the time I’m writing this blog post, the RC2 is the latest version available, I’ll list and show the differences with a plain TFS 2015 build agent.

Three new modules are added:

  • Microsoft.TeamFoundation.DistributedTask.Task.CodeCoverage
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Internal
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.RemoteDeployment

Following the list of all importable functions and cmdlets in the those modules.

  • Microsoft.TeamFoundation.DistributedTask.Task.CodeCoverage
    • Enable-CodeCoverage
    • Publish-CodeCoverage
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Internal
    • Get-OperationLogs
    • Get-ResourceCredentials
    • Get-ResourceFQDNTagKey
    • Get-ResourceHttpsTagKey
    • Get-ResourceHttpTagKey
    • Get-ResourceOperationLogs
    • Get-SkipCACheckTagKey
    • Get-SqlPackageCommandArguments
    • Import-DevtestLabsCommomDll
    • Write-ResponseLogs
  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.RemoteDeployment
    • Invoke-RemoteDeployment

Furthermore some cmdlets are added to the existing modules:

  • Microsoft.TeamFoundation.DistributedTask.Task.Deployment.Azure.psm1
    • Set-CurrentAzureRMSubscription
    • Set-CurrentAzureSubscription
  • Microsoft.TeamFoundation.DistributedTask.Task.DevTestLabs.dll
    • Get-ExternalIpAddress
    • Get-ParsedSessionVariables
  • Microsoft.TeamFoundation.DistributedTask.Task.Internal.dll
    • Get-IndexedSourceFilePaths
    • Get-TfsClientCredentials

It also seems that in the Microsoft.TeamFoundation.DistributedTask.Task.Internal.dll, Invoke-IndexSources cmdlet is not available any more.

You can reference these modules from your custom Build tasks by calling a standard import module cmdlet.

import-module "Microsoft.TeamFoundation.DistributedTask.Task.Common"

If you are interested about the agent version, you can check it by executing the vsoagent on your build server, with the following parameter:

VsoAgent.exe /version

As soon as an RTM version of the Update 1 becomes available, I’ll update this post.

In case you are interested in a short description of these functions, you can their source code for further info