Automate setting ‘Retain indefinitely’ flag on your releases

Overview

As a common practice, after a successful release to production, often there is a need to retain the involved artifact and relevant release information for a certain amount of time. In order to avoid that a retention policy removes this information, you will mark that release with a Retain indefinitely flag, by choosing that option from the VSTS UI.

As this is a manual process that I would like to automate, I will use the Retain indefinitely current release task available in VSTS Marketplace.

After installing the extension, in the list of available tasks, check the Utility category and you’ll find the above-mentioned task.

Only a single parameter is presented by the task in a form of a checkbox labelled Mark the current release to be retained indefinitely. By default is set to true. If checked, it will mark the current release with Retain indefinitely flag. Otherwise, it will take the Retain indefinitely flag off the current release.

I will add it only to my production environment process and I will place it as the last task. E.g.

This is because I do not want this task to execute if any of the previous tasks in the process do fail. It should execute only if I managed to deploy my application in production. That is the criteria for retaining this release for a longer time.
The only requirement for this task to run is that the account on which the build agent is running has sufficient privileges to set Retain indefinitely flag.

That’s all. Now you do not need to remember to set the Retain indefinitely flag after every successful production release.

UPDATE:

After extensive testing, I noticed that it’s mostly the case that additional rights do need to be granted for the build agent identity in order for this task to succeed.
In case the permissions are missing, a similar error will be visible in the log:

##[error]VS402904: Access denied: User Project Collection Build Service (mummy) does not have manage releases permission. Contact your release manager.

Edit the security settings for that particular release or for all the releases and set the following to the needed account.

Make sure ‘Manage Releases’ permission is granted for the indicated user.

Another note is that also the cross-platform agents are supported starting from the version 2.x of the extension. If you check on GitHub you’ll see that the task has been re-written and it is now based on node provider with the code written in TypeScript.

Uploading build/release tasks to VSTS

Let’s start with why? Why would someone upload a task directly to TFS/VSTS? You can just install or update the extension that added those tasks, no?!?! So why?
Obviously there are many reasons, and aside the development of the tasks themselves, often is a case when you find and fix a bug in a task that you need to get in production ASAP. Notifying third party and waiting for a new version of the extension is often not acceptable.
But there is already a tool that Microsoft made for handling tasks! Does TFS-CLI tells you nothing? Sure, tool that works very well and which I mentioned already in several occasions on my blog. Still, getting it requires Java Runtime Environment, NodeJs, tool itself, configuration. These are often not available out of the box and if you are doing this things occasionally, a better way may be a simple script.

Following is a script I do use:

param(
   [Parameter(Mandatory=$true)][string]$TaskPath,
   [Parameter(Mandatory=$true)][string]$VstsUrl,
   [Parameter(Mandatory=$true)][string]$Pat
)

# Load task definition from the JSON file
$taskDefinition = (Get-Content $taskPath\task.json) -join "`n" | ConvertFrom-Json
$taskFolder = Get-Item $TaskPath

# Zip the task content
Write-Output "Zipping task content"
$taskZip = ("{0}\..\{1}.zip" -f $taskFolder, $taskDefinition.id)
if (Test-Path $taskZip) { Remove-Item $taskZip }

Add-Type -AssemblyName "System.IO.Compression.FileSystem"
[IO.Compression.ZipFile]::CreateFromDirectory($taskFolder, $taskZip)

# Prepare to upload the task
Write-Output "Uploading task content to $VstsUrl"

$taskZipItem = Get-Item $taskZip
$encodedCredentials = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$Pat"))

$headers = @{ Authorization = "Basic $encodedCredentials"; "Content-Range" = "bytes 0-$($taskZipItem.Length - 1)/$($taskZipItem.Length)"}

$url = "{0}/_apis/distributedtask/tasks/{1}?api-version=2.0-preview" -f $VstsUrl, $taskDefinition.id

try
{
	Invoke-RestMethod -Uri $url -Headers $headers -ContentType application/octet-stream -Method Put -InFile $taskZipItem
	Write-Host "Task uploaded successfully" -ForegroundColor Green
}
finally
{
    Remove-Item -LiteralPath $taskZip -Force
}

To invoke this, it is sufficient to provide the path to the folder containing the task, url towards your VSTS account (in case of TFS, path to the collection) and your personal access token. E.g.

.\TaskUploader.ps1 .\task\ https://myaccount.visualstudio.com jx3sa4h3cb56ehftgrjitj6kctei3pp7usuyxkoim5vpeqcgn7eq

That’s all.

Managing VSTS/TFS Release Definition Variables from PowerShell

Couple of days ago I was trying to provision my Release Definitions variables in my VSTS/TFS projects via PowerShell. As this turned out not to be a trivial Web-Request and as some of the calls I discovered are not yet documented, I decided to share my findings with you.

In the following lines I’ll show you a couple of cmdlets that will allow you to manipulate all of the variables in your Release Definition, those on the definition level, environment specific ones and also variable groups.

For the purpose of adding Release definition, Environment level variables and relating Variable Groups I wrote the following cmdlet:

function Add-EnvironmentVariable()
{
    [CmdletBinding()]
    param
    (
        [string][parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)][Alias("name")]$VariableName,
        [string][parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)][Alias("value")]$VariableValue,
        [string][parameter(ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)][Alias("env")]$EnvironmentName,
        [bool][parameter(ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)]$Secret,
        [int[]]$VariableGroups,
        [string][parameter(Mandatory = $true)]$ProjectUrl,
        [int][parameter(Mandatory = $true)]$DefinitionId,
        [string]$Comment,
        [switch]$Reset
    )
    BEGIN
    {
        Write-Verbose "Entering script $($MyInvocation.MyCommand.Name)"
        Write-Verbose "Parameter Values"
        $PSBoundParameters.Keys | ForEach-Object { if ($Secret -and $_ -eq "VariableValue") { Write-Verbose "VariableValue = *******" } else { Write-Verbose "$_ = '$($PSBoundParameters[$_])'" }}

        $ProjectUrl = $ProjectUrl.TrimEnd("/")

        $url = "$($ProjectUrl)/_apis/release/definitions/$($DefinitionId)?expand=Environments?api-version=3.0-preview"
        $definition = Invoke-RestMethod $url -UseDefaultCredentials

        if ($Reset)
        {
            foreach($environment in $definition.environments)
            {
                foreach($prop in $environment.variables.PSObject.Properties.Where{$_.MemberType -eq "NoteProperty"})
                {
                    $environment.variables.PSObject.Properties.Remove($prop.Name)
                }
            }

            foreach($prop in $definition.variables.PSObject.Properties.Where{$_.MemberType -eq "NoteProperty"})
            {
                $definition.variables.PSObject.Properties.Remove($prop.Name)
            }

            $definition.variableGroups = @()
        }
    }
    PROCESS
    {
        $value = @{value=$VariableValue}
    
        if ($Secret)
        {
            $value.Add("isSecret", $true)
        }

        if ($EnvironmentName)
        {
            $environment = $definition.environments.Where{$_.name -like $EnvironmentName}

            if($environment)
            {
                $environment.variables | Add-Member -Name $VariableName -MemberType NoteProperty -Value $value -Force
            }
            else
            {
                Write-Warning "Environment '$($environment.name)' not found in the given release"
            }
        }
        else
        {
            $definition.variables | Add-Member -Name $VariableName -MemberType NoteProperty -Value $value -Force
        }
    }
    END
    {
        $definition.source = "restApi"

        if ($Comment)
        {
            $definition | Add-Member -Name "comment" -MemberType NoteProperty -Value $Comment
        }
        
        if ($VariableGroups)
        {
            foreach($variable in $VariableGroups)
            {
                if ($definition.variableGroups -notcontains $variable)
                {
                    $definition.variableGroups += $variable
                }
            }
        }

        $body = $definition | ConvertTo-Json -Depth 10 -Compress

        Invoke-RestMethod "$($ProjectUrl)/_apis/release/definitions?api-version=3.0-preview" -Method Put -Body $body -ContentType 'application/json' -UseDefaultCredentials | Out-Null
    }
}

Don’t get scared by the number of parameters, or apparent complexity of the cmdlet. I’ll quickly explain those parameters, usage and the expected result.

Let’s start with some why’s. As you can see, in the BEGIN block of my cmdlet (which is triggered once per a pipeline invocation) I retrieve the given build definition, in the PROCESS block I add the desired variables (hopefully from the pipeline) then in the END block I persist all of the changes.

If you are unfamiliar with Windows PowerShell Cmdlet Lifecycle, please consult the following article Windows PowerShell: The Advanced Function Lifecycle.

This is intentional, as I want to have a single call to the API for all of the added variables. In this way in the history of the build definition there will be a single entry for all of the variables we added, no matter the number of them. Otherwise, we would persist the changes for each of the variables and our history would be messy.

If structured differently, we may see a history entry on per each variable that we do add. This obviously applies only if you are trying to add multiple variable in one go.

Following would be a simple invocation to add a single variable into one of our environments defined in a release template:

$Project = "https://my.tfs.local:8080/tfs/DefaultCollection/MyProject"
$DefinitionId = 23

Add-EnvironmentVariable -VariableName "Mario2" -VariableValue "1" -EnvironmentName "DEV" -Secret $true -ProjectUrl $Project -DefinitionId $DefinitionId -VariableGroups 25 -Comment "Added by PSScript"

The above command will add a variable named Mario2 with a value 1 in the DEV environment, defined in the definition with id 23. It will also reference the variable group that has id 25.

Following would be the result:

In case you would like to add multiple variables in one go, create an array of PSCustomObject with the following properties:

$variables = @()
$variables += [PSCustomObject]@{ name="var1"; value="123"; secret=$false; env="" }
$variables += [PSCustomObject]@{ name="var2"; value="sdfd"; secret=$false; env="" }
$variables += [PSCustomObject]@{ name="var3"; value="5678"; secret=$false; env="DEV" }
$variables += [PSCustomObject]@{ name="var4"; value="ghjk"; secret=$true; env="DEV" }

$variables | Add-EnvironmentVariable -ProjectUrl $Project -DefinitionId $DefinitionId

This will add two variables to the environment called DEV in your Release Definition and two more variables on the Release Definition level. As you can guess, if we omit the environment name, the variables will be added on the Release Definition level. The last variable, var4, is also marked as secret, meaning that once added will not be visible to the user. Also in this case, we will have only a single entry in the change history as a single call to the REST API will be made.

Other options you can specify are:

  • Reset – By setting this switch only the variables that are not passed in the invocation, but are present on the Release definition, will be removed.
  • Comment – In case you want a custom message to be visualized in the history for this change, you can specify it here.
  • VariableGroups – An integer array indicating id’s of the variable groups you wish to link to the Release definition

In case you are using variable groups you can create those via following cmdlet:

function Add-VariableGroupVariable()
{
    [CmdletBinding()]
    param
    (
        [string][parameter(Mandatory = $true)]$ProjectUrl,
        [string][parameter(Mandatory = $true)]$VariableGroupName,
        [string]$VariableGroupDescription,
        [string][parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)][Alias("name")]$VariableName,
        [string][parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)][Alias("value")]$VariableValue,
        [bool][parameter(ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)]$Secret,
        [switch]$Reset,
        [switch]$Force
    )
    BEGIN
    {
        Write-Verbose "Entering script $($MyInvocation.MyCommand.Name)"
        Write-Verbose "Parameter Values"

        $PSBoundParameters.Keys | ForEach-Object { Write-Verbose "$_ = '$($PSBoundParameters[$_])'" }
        $method = "Post"
        $variableGroup = Get-VariableGroup $ProjectUrl $VariableGroupName

        if($variableGroup)
        {
            Write-Verbose "Variable group $VariableGroupName exists."

            if ($Reset)
            {
                Write-Verbose "Reset = $Reset : remove all variables."
                foreach($prop in $variableGroup.variables.PSObject.Properties.Where{$_.MemberType -eq "NoteProperty"})
                {
                    $variableGroup.variables.PSObject.Properties.Remove($prop.Name)
                }
            }

            $id = $variableGroup.id
            $restApi = "$($ProjectUrl)/_apis/distributedtask/variablegroups/$id"
            $method = "Put"
        }
        else
        {
            Write-Verbose "Variable group $VariableGroupName not found."
            if ($Force)
            {
                Write-Verbose "Create variable group $VariableGroupName."
                $variableGroup = @{name=$VariableGroupName;description=$VariableGroupDescription;variables=New-Object PSObject;}
                $restApi = "$($ProjectUrl)/_apis/distributedtask/variablegroups?api-version=3.2-preview.1"
            }
            else
            {
                throw "Cannot add variable to nonexisting variable group $VariableGroupName; use the -Force switch to create the variable group."
            }
        }
    }
    PROCESS
    {
        Write-Verbose "Adding $VariableName with value $VariableValue..."
        $variableGroup.variables | Add-Member -Name $VariableName -MemberType NoteProperty -Value @{value=$VariableValue;isSecret=$Secret} -Force
    }
    END
    {
        Write-Verbose "Persist variable group $VariableGroupName."
        $body = $variableGroup | ConvertTo-Json -Depth 10 -Compress
        $response = Invoke-RestMethod $restApi -Method $method -Body $body -ContentType 'application/json' -Header @{"Accept" = "application/json;api-version=3.2-preview.1"}  -UseDefaultCredentials
        
        return $response.id
    }
}

function Get-VariableGroup()
{
    [CmdletBinding()]
    param
    (
        [string][parameter(Mandatory = $true)]$ProjectUrl,
        [string][parameter(Mandatory = $true)]$Name
    )
    BEGIN
    {
        Write-Verbose "Entering script $($MyInvocation.MyCommand.Name)"
        Write-Verbose "Parameter Values"
        $PSBoundParameters.Keys | ForEach-Object { Write-Verbose "$_ = '$($PSBoundParameters[$_])'" }
    }
    PROCESS
    {
        $ProjectUrl = $ProjectUrl.TrimEnd("/")
        $url = "$($ProjectUrl)/_apis/distributedtask/variablegroups"
        $variableGroups = Invoke-RestMethod $url -UseDefaultCredentials
        
        foreach($variableGroup in $variableGroups.value){
            if ($variableGroup.name -like $Name){
                Write-Verbose "Variable group $Name found."
                return $variableGroup
            }
        }
        Write-Verbose "Variable group $Name not found."
        return $null
    }
    END { }
}

This cmdlet will look for the given group and if it exists it will update it with the values you pass in. In case the variable group (matched by name) doesn’t exist, and if the -Force switch is selected, it will create a new group. Working principle is the same as for Add-EnvironmentVariable cmdlet. At the end, it will return the Variable Group Id that you can use later for Add-EnvironmentVariable cmdlet and reference it.

Following an example of invocation:

$ProjectUrl = "https://my.tfs.local:8080/tfs/DefaultCollection/MyProject"

Add-VariableGroupVariable -ProjectUrl $ProjectUrl -VariableGroupName "Mario Test" -Force -VariableName "var1" -VariableValue "1234"

That’s all folks! You now have 2 new cmdlets that will allow you to automate the management of the Release Definition variables. Use these wisely 🙂

Happy coding!

P.S.
A thank you goes to Ted van Haalen who, on my input, actually wrote and tested Add-VariableGroupVariable cmdlet (as you already may have noticed because of the different coding style).