Uploading XL Deploy DAR package via PowerShell

Introduction

Recently at my customer’s site, I started using XebiaLabs product called XL Deploy. It is a software solution that helps automating the deployment process. Many operations are exposed via a REST API. Uploading of packages is one of them and is done following the Multipart/form-data standard. In case you need to do so, via PowerShell, you may be surprised about the difficulty of something that appears to be a trivial task. As you may know, PowerShell doesn’t play well with Multipart/form-data requests. I will show you a cmdlet that I wrote that can be handy in accomplishing this task.

Welcome Send-Package cmdlet

Before showing you the actual cmdlet that will send a package to XL Deploy I need to mention that the core of this cmdlet is Invoke-MultipartFormDataUpload cmdlet about which I blogged earlier.

First of all two small helper cmdlets. As the file name needs to be specified in the URL, we need to encode it. This may be done with some simpler code for this particular case, however I will show you a cmdlet that I’m using also for other XL Deploy calls where this approach is necessary.

<############################################################################################ 
    Encodes each part of the path separately.
############################################################################################>
function Get-EncodedPathParts()
{
    [CmdletBinding()]
    param
    (
        [string][parameter(Mandatory = $true)]$PartialPath
    )
    BEGIN { }
    PROCESS
    {
        return ($PartialPath -split "/" | %{ [System.Uri]::EscapeDataString($_) }) -join "/"
    }
    END { }
}

The other one will make sure that the URL passed in is formatted correctly for XL Deploy and that is a valid URL.

<############################################################################################ 
    Verifies that the endpoint URL is well formatted and a valid URL.
############################################################################################>
function Test-EndpointBaseUrl()
{
	[CmdletBinding()]
	param
	(
		[Uri][parameter(Mandatory = $true)]$Endpoint
	)
	BEGIN
	{
		Write-Verbose "Endpoint = $Endpoint"
	}
	PROCESS 
	{
		#$xldServer = $serviceEndpoint.Url.AbsoluteUri.TrimEnd('/')
		$xldServer = $Endpoint.AbsoluteUri.TrimEnd('/')

		if (-not $xldServer.EndsWith("deployit", "InvariantCultureIgnoreCase"))
		{
			$xldServer = "$xldServer/deployit"
		}

		# takes in consideration both http and https protocol
		if (-not $xldServer.StartsWith("http", "InvariantCultureIgnoreCase"))
		{
			$xldServer = "http://$xldServer"
		}

		$uri = $xldServer -as [System.Uri] 
		if (-not ($uri.AbsoluteURI -ne $null -and $uri.Scheme -match '[http|https]'))
		{
			throw "Provided endpoint address is not a valid URL."
		}

		return $uri
	}
	END { }
}

This cmdlets and the one I described in the previous post are required to be available for our new upload cmdlet. After having assured that they are available we can write a cmdlet I named Send-Package.

<############################################################################################ 
    Uploads the given package to XL Deploy server.
############################################################################################>
function Send-Package()
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$PackagePath,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$EndpointUrl,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $PackagePath))
        {
            $errorMessage = ("Package {0} missing or unable to read." -f $PackagePath)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'SendPackage', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $PackagePath
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        $EndpointUrl = Test-EndpointBaseUrl $EndpointUrl
    }
    PROCESS
    {
        $fileName = Split-Path $packagePath -leaf
        $fileName = Get-EncodedPathParts($fileName) 

        $uri = "$EndpointUrl/package/upload/$fileName"

        $response = Invoke-MultipartFormDataUpload -InFile $PackagePath -Uri $uri -Credential $credentials

        return ([xml]$response).'udm.DeploymentPackage'.id
    }
    END { }
}

You can now invoke this cmdlet and pass the requested parameters. If the call succeeds you will get back the package id.

At example:

$uri = "http://xld.majcica.com:4516"
$filePath = "C:\Users\majcicam\Desktop\package.dar"
$credentials = (Get-Credential)

$packageId = Send-Package $filePath $uri $credentials

Write-Output "Package was uploaded successfully with the following id: '$packageId'"

In case you do not want to provide credentials interactively, the following cmdlet may help you:

<############################################################################################ 
    Given the username and password strings, create a valid PSCredential object.
############################################################################################>
function New-PSCredential()
{
	[CmdletBinding()]
	param
	(
		[string][parameter(Mandatory = $true)]$Username,
		[string][parameter(Mandatory = $true)]$Password
	)
	BEGIN
	{
		Write-Verbose "Username = $Username"
	}
	PROCESS
	{
		$securePassword = ConvertTo-SecureString –String $Password -asPlainText -Force
		$credential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $Username, $securePassword

		return $credential
	}
	END { }
}

That’s all folks! I tested this scripts with XL Deploy 5.1.0 and 5.1.3 and had not encountered any issues. If any, do not hesitate to comment.

Happy deploying!

PowerShell tips and tricks – Multipart/form-data requests

Introduction

If you ever came across a need to invoke a request that needs to oblige to Multipart/form-data standard in PowerShell, you probably got to know quite quickly that none of commonly used cmdlets do support it.
Both, Invoke-WebRequest and Invoke-RestMethod are unaware on how to format the request body in order to comply to Multipart/form-data standard. In order to succeed in your intent, you probably Googled this out, and the best you could find are a couple of questions on StackOverflow that are showing on how to manually set the message body and then invoke your call with one of above mentioned cmdlets. I did it too. Still I was not satisfied by the answers I found. Although what proposed could be written in a slightly better way, still it was not optimal. First of all the memory footprint that it has. In case you are transmitting a large amount of data, all of the objects are loaded into memory.
After trying this approach, I dug into the .Net framework and found out that all that we need is right there. True, it is only available on .Net 4.5, and if that is not an obstacle, I would advise you to follow that approach. HttpClient and connected classes do have all of the necessary to support Multipart/form-data standard. Initially I created a prototype in a form of a small C# application and once succeed I translated all of that into PowerShell.

In this post I will show you both, PowerShell approach by formatting the request body and transmitting via Invoke-WebRequest cmdlet, as using an approach that is less resource intensive and is based on HttpClient .Net class.

Lets start.

Throw everything in

Multipart/form-data standard requires the message body to follow a certain structure. You can read more about it in the RFC 2388.
Aside the body structure, there is a concept of a boundary. Boundary is nothing else than an unique string that will be used for delimitation purposes inside the message body. It will be specified in the request header and used inside the message body to circumvent files that we intend to transmit.

As I mentioned earlier, this approach constructs the body string in a variable and once ready invokes the Invoke-WebRequest cmdlet to execute the call.

function Invoke-MultipartFormDataUpload
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
        [string]$ContentType,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $InFile))
        {
            $errorMessage = ("File {0} missing or unable to read." -f $InFile)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        if (-not $ContentType)
        {
            Add-Type -AssemblyName System.Web

            $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)
            
            if ($mimeType)
            {
                $ContentType = $mimeType
            }
            else
            {
                $ContentType = "application/octet-stream"
            }
        }
    }
    PROCESS
    {
		$fileName = Split-Path $InFile -leaf
		$boundary = [guid]::NewGuid().ToString()
		
    	$fileBin = [System.IO.File]::ReadAllBytes($InFile)
	    $enc = [System.Text.Encoding]::GetEncoding("iso-8859-1")

	    $template = @'
--{0}
Content-Disposition: form-data; name="fileData"; filename="{1}"
Content-Type: {2}

{3}
--{0}--

'@

        $body = $template -f $boundary, $fileName, $ContentType, $enc.GetString($fileBin)

		try
		{
			return Invoke-WebRequest -Uri $Uri `
									 -Method Post `
									 -ContentType "multipart/form-data; boundary=$boundary" `
									 -Body $body `
									 -Credential $Credential
		}
		catch [Exception]
		{
			$PSCmdlet.ThrowTerminatingError($_)
		}
    }
    END { }
}

As you can see, we are preparing “by hand” the correct message body and executing Invoke-WebRequest in order to send our file. It is not efficient at all as it encodes the whole file as a string and keeps it in memory. Consider transmitting large files and make yourself an idea on the memory footprint of this approach. Aside that, it is just not elegant!

Wait a moment, what a waste

Previously shown method is as said, not elegant and waists resources. A better approach is to let the HttpClient class handle everything. It is way more efficient as it uses streams in order to read our file and transfers it to http StreamContent. It is less resource intensive and a more elegant solution.

Let’s see now how our refactored Invoke-MultipartFormDataUpload cmdlet looks like:

function Invoke-MultipartFormDataUpload
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
        [string]$ContentType,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $InFile))
        {
            $errorMessage = ("File {0} missing or unable to read." -f $InFile)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        if (-not $ContentType)
        {
            Add-Type -AssemblyName System.Web

            $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)
            
            if ($mimeType)
            {
                $ContentType = $mimeType
            }
            else
            {
                $ContentType = "application/octet-stream"
            }
        }
    }
    PROCESS
    {
        Add-Type -AssemblyName System.Net.Http

		$httpClientHandler = New-Object System.Net.Http.HttpClientHandler

        if ($Credential)
        {
		    $networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password)
		    $httpClientHandler.Credentials = $networkCredential
        }

        $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler

        $packageFileStream = New-Object System.IO.FileStream @($InFile, [System.IO.FileMode]::Open)
        
		$contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue "form-data"
	    $contentDispositionHeaderValue.Name = "fileData"
		$contentDispositionHeaderValue.FileName = (Split-Path $InFile -leaf)

        $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
        $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
        $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $ContentType
        
        $content = New-Object System.Net.Http.MultipartFormDataContent
        $content.Add($streamContent)

        try
        {
			$response = $httpClient.PostAsync($Uri, $content).Result

			if (!$response.IsSuccessStatusCode)
			{
				$responseBody = $response.Content.ReadAsStringAsync().Result
				$errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody

				throw [System.Net.Http.HttpRequestException] $errorMessage
			}

			return $response.Content.ReadAsStringAsync().Result
        }
        catch [Exception]
        {
			$PSCmdlet.ThrowTerminatingError($_)
        }
        finally
        {
            if($null -ne $httpClient)
            {
                $httpClient.Dispose()
            }

            if($null -ne $response)
            {
                $response.Dispose()
            }
        }
    }
    END { }
}

Conclusion

There is still space for improvement, however in my case it is good enough and I will not continue extending this cmdlet. If you think this is not enough or good enough, please extend it and let me know. As an idea, you may make possible passing multiple files via the pipeline and execute the upload only once, in the END step. In that way it may be more efficient and flexible. Also supporting some other common parameters as Proxy would be beneficial to others. If any let me know, I would be happy to hear from you in comments.

Tough life behind a proxy

In many enterprises when it comes to Internet access you will find yourself behind a proxy server. In such conditions many application do require a specific setup or they will not work at all. Recently I had to install a npm package from the official public npm repository and the “usual” command doesn’t work. Thus in case of npm client behind a proxy server, you need to pass the following argument --proxy.

My command from:

npm install -g tfx-cli

became

npm --proxy http://my.proxy.com:8080 install -g tfx-cli

Also, as I am actively using Visual Studio Code, I had an unpleasant surprise. If you do tend to install a Visual Studio Code extension (plugin) and you are behind a proxy, at the moment you are doomed. There is no way, in the latest version at the time of writing, to install Visual Studio Code extension if behind a proxy.

When it comes to .NET software development, all of the commonly used classes as HttpClient do cope well with proxy, the only problems is that developers rarely think about it and do not implement adequate support. In this particular case specifying a proxy in HttpClientHandler is the solution, which then needs to be passed to the HttpClient constructor.

So fellow developers please have mercy for all of the people behind a proxy, it ain’t an easy life!