Uploading XL Deploy DAR package via TypeScript

Another day, another language, another challenge

In past I already wrote about multipart/form-data requests that are used to upload files. It was about PowerShell and leveraging .Net libraries to achieve this task. Now it is the turn of TypeScript.

I had a need to implement the file upload to XL Deploy which requires multipart/form-data standard in order to do so. I wrote about the same thing in the past, using PowerShell to upload DAR package to XL deploy. This time, however, I needed to use NodeJs/TypeScript.

This is not meant to be a guide about the TypeScript, I do suppose you already have some knowledge about it. I just would like to show you how did I achieve it, hoping to help others not to go through the same discovery process that brought me to result.

First of all, I will be making my HTTP calls via typed-rest-client. This library is again based on the plain NodeJs http.ClientRequest class. This also means that if you do not plan to use this library, you can still follow the method I’m suggesting.
Creating the multipartform-data message structure and adding the necessary headers will be done with form-data package. It is one of the most popular libraries used to create “multipart/form-data” streams. Yes, I just mentioned the keyword, streams, and yes, we are going to use streams to achieve this and allow us not to saturate resources on our client host in case of large files upload.

Enough talking now, let’s see some code.

Sending multipartform-data messages in Typescript

Before we start, make sure that you install the following packages:

npm install typed-rest-client
npm install form-data
npm install @types/form-data

This is all we need. Note I also installed typings for the form-data library so that we can comfortably use it in TypeScript and make sure that “typed-rest-client” library is at least of version 1.0.11.

Code wise, first of all, we need to create an instance of our client.

import { BasicCredentialHandler } from "typed-rest-client/Handlers";
import { RestClient } from "typed-rest-client/RestClient";

async function run() {
    const requestOptions = { ignoreSslError: true };
    const authHandler = [new BasicCredentialHandler("user", "password")];

    const baseUrl: string = "https://myXLServer:4516";

    const client = new RestClient("myRestClient", baseUrl, authHandler, requestOptions);
}

I will skip commenting on the necessary imports and quickly analyze the remaining code.

I need to create request options and set the ignoreSslError property. This is so to allow my self-signed certificate to be accepted.
Then I do create a basic authentication handler and pass in the requested username and password. Once I have all of the necessary, I create an instance of the RestClient.

You spotted well, it is a RestClient and above I talked about the HttpClient. Do not wary, it is a wrapper around it, helping me to deserialize the response body, verify the status code, etc.

Let’s now prepare our form data.

...
import FormData from "form-data";
import fs from "fs";

async function run() {
	...
	const formData = new FormData();
	formData.append("fileData", fs.createReadStream("C:\\path\\to\\myfile.dar"));
}

We need a couple of extra imports and once that is sorted out, we just do create an instance of the FormData class. Once we have it, we will call the append method, pass in the file name and the stream that points to my file of choice. In order to get my file that is on the disk, I’m using createReadStream function from fs library which is a very common way to setup a stream.

At this point, we are ready to make our HTTP call.

async function run() {
	...
	const response = await client.uploadStream(
		"POST", `deployit/package/upload/myfile.dar`,
		formData,
		{ additionalHeaders: formData.getHeaders() });

	console.log(response.result.id);
}

As you can see, we are invoking the upload stream method from the rest client and passing in the following parameters.
HTTP method to use, POST in our case (XL Deploy), second, rest resource URL that needs to be triggered. Bear in mind that actual URL will be composed with the base you passed in the constructor of the RestClient. Then, the stream containing the body. This is going to be the instance of our FormData class, which is of type stream, and as the fourth parameter, we need to pass the additional headers. The additional headers we are specifying are overriding the content-type as for multipart/form-data it needs to be set to multipart/form-data and contains the correct boundary value. That’s what getHeaders will do, return the necessary content-type header with the necessary correct boundary value.

Once the call has been made, the upload of the file will start. As the response from XL Deploy on a successful import we will receive a message in form of JSON where one of the fields do report the ID of the package, and that’s what I’m printing in the console on my last line.

This may be specific for XL Deploy, however, you can easily adapt this code for any other service where multipart/form-data upload is necessary.

Following the complete code sample.

import FormData from "form-data";
import fs from "fs";
import { BasicCredentialHandler } from "typed-rest-client/Handlers";
import { RestClient } from "typed-rest-client/RestClient";

async function run() {
    const requestOptions = { ignoreSslError: true };
    const authHandler = [new BasicCredentialHandler("user", "password")];

    const baseUrl: string = "https://myXLServer:4516";

    const client = new RestClient("myRestClient", baseUrl, authHandler, requestOptions);

    const formData = new FormData();
    formData.append("fileData", fs.createReadStream("C:\\path\\to\\myfile.dar"));

    const response = await client.uploadStream<any>(
        "POST", `deployit/package/upload/myfile.dar`,
        formData,
        { additionalHeaders: formData.getHeaders() });

    // tslint:disable-next-line:no-console
    console.log(response.result.id);
}

run();

Good luck!

Uploading XL Deploy DAR package via PowerShell

Introduction

Recently at my customer’s site, I started using XebiaLabs product called XL Deploy. It is a software solution that helps automating the deployment process. Many operations are exposed via a REST API. Uploading of packages is one of them and is done following the Multipart/form-data standard. In case you need to do so, via PowerShell, you may be surprised about the difficulty of something that appears to be a trivial task. As you may know, PowerShell doesn’t play well with Multipart/form-data requests. I will show you a cmdlet that I wrote that can be handy in accomplishing this task.

Welcome Send-Package cmdlet

Before showing you the actual cmdlet that will send a package to XL Deploy I need to mention that the core of this cmdlet is Invoke-MultipartFormDataUpload cmdlet about which I blogged earlier.

First of all two small helper cmdlets. As the file name needs to be specified in the URL, we need to encode it. This may be done with some simpler code for this particular case, however I will show you a cmdlet that I’m using also for other XL Deploy calls where this approach is necessary.

<############################################################################################ 
    Encodes each part of the path separately.
############################################################################################>
function Get-EncodedPathParts()
{
    [CmdletBinding()]
    param
    (
        [string][parameter(Mandatory = $true)]$PartialPath
    )
    BEGIN { }
    PROCESS
    {
        return ($PartialPath -split "/" | %{ [System.Uri]::EscapeDataString($_) }) -join "/"
    }
    END { }
}

The other one will make sure that the URL passed in is formatted correctly for XL Deploy and that is a valid URL.

<############################################################################################ 
    Verifies that the endpoint URL is well formatted and a valid URL.
############################################################################################>
function Test-EndpointBaseUrl()
{
	[CmdletBinding()]
	param
	(
		[Uri][parameter(Mandatory = $true)]$Endpoint
	)
	BEGIN
	{
		Write-Verbose "Endpoint = $Endpoint"
	}
	PROCESS 
	{
		#$xldServer = $serviceEndpoint.Url.AbsoluteUri.TrimEnd('/')
		$xldServer = $Endpoint.AbsoluteUri.TrimEnd('/')

		if (-not $xldServer.EndsWith("deployit", "InvariantCultureIgnoreCase"))
		{
			$xldServer = "$xldServer/deployit"
		}

		# takes in consideration both http and https protocol
		if (-not $xldServer.StartsWith("http", "InvariantCultureIgnoreCase"))
		{
			$xldServer = "http://$xldServer"
		}

		$uri = $xldServer -as [System.Uri] 
		if (-not ($uri.AbsoluteURI -ne $null -and $uri.Scheme -match '[http|https]'))
		{
			throw "Provided endpoint address is not a valid URL."
		}

		return $uri
	}
	END { }
}

This cmdlets and the one I described in the previous post are required to be available for our new upload cmdlet. After having assured that they are available we can write a cmdlet I named Send-Package.

<############################################################################################ 
    Uploads the given package to XL Deploy server.
############################################################################################>
function Send-Package()
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$PackagePath,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$EndpointUrl,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $PackagePath))
        {
            $errorMessage = ("Package {0} missing or unable to read." -f $PackagePath)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'SendPackage', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $PackagePath
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        $EndpointUrl = Test-EndpointBaseUrl $EndpointUrl
    }
    PROCESS
    {
        $fileName = Split-Path $packagePath -leaf
        $fileName = Get-EncodedPathParts($fileName) 

        $uri = "$EndpointUrl/package/upload/$fileName"

        $response = Invoke-MultipartFormDataUpload -InFile $PackagePath -Uri $uri -Credential $credentials

        return ([xml]$response).'udm.DeploymentPackage'.id
    }
    END { }
}

You can now invoke this cmdlet and pass the requested parameters. If the call succeeds you will get back the package id.

At example:

$uri = "http://xld.majcica.com:4516"
$filePath = "C:\Users\majcicam\Desktop\package.dar"
$credentials = (Get-Credential)

$packageId = Send-Package $filePath $uri $credentials

Write-Output "Package was uploaded successfully with the following id: '$packageId'"

In case you do not want to provide credentials interactively, the following cmdlet may help you:

<############################################################################################ 
    Given the username and password strings, create a valid PSCredential object.
############################################################################################>
function New-PSCredential()
{
	[CmdletBinding()]
	param
	(
		[string][parameter(Mandatory = $true)]$Username,
		[string][parameter(Mandatory = $true)]$Password
	)
	BEGIN
	{
		Write-Verbose "Username = $Username"
	}
	PROCESS
	{
		$securePassword = ConvertTo-SecureString –String $Password -asPlainText -Force
		$credential = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $Username, $securePassword

		return $credential
	}
	END { }
}

That’s all folks! I tested this scripts with XL Deploy 5.1.0 and 5.1.3 and had not encountered any issues. If any, do not hesitate to comment.

Happy deploying!

PowerShell tips and tricks – Multipart/form-data requests

Introduction

If you ever came across a need to invoke a request that needs to oblige to Multipart/form-data standard in PowerShell, you probably got to know quite quickly that none of commonly used cmdlets do support it.
Both, Invoke-WebRequest and Invoke-RestMethod are unaware on how to format the request body in order to comply to Multipart/form-data standard. In order to succeed in your intent, you probably Googled this out, and the best you could find are a couple of questions on StackOverflow that are showing on how to manually set the message body and then invoke your call with one of above mentioned cmdlets. I did it too. Still I was not satisfied by the answers I found. Although what proposed could be written in a slightly better way, still it was not optimal. First of all the memory footprint that it has. In case you are transmitting a large amount of data, all of the objects are loaded into memory.
After trying this approach, I dug into the .Net framework and found out that all that we need is right there. True, it is only available on .Net 4.5, and if that is not an obstacle, I would advise you to follow that approach. HttpClient and connected classes do have all of the necessary to support Multipart/form-data standard. Initially I created a prototype in a form of a small C# application and once succeed I translated all of that into PowerShell.

In this post I will show you both, PowerShell approach by formatting the request body and transmitting via Invoke-WebRequest cmdlet, as using an approach that is less resource intensive and is based on HttpClient .Net class.

Lets start.

Throw everything in

Multipart/form-data standard requires the message body to follow a certain structure. You can read more about it in the RFC 2388.
Aside the body structure, there is a concept of a boundary. Boundary is nothing else than an unique string that will be used for delimitation purposes inside the message body. It will be specified in the request header and used inside the message body to circumvent files that we intend to transmit.

As I mentioned earlier, this approach constructs the body string in a variable and once ready invokes the Invoke-WebRequest cmdlet to execute the call.

function Invoke-MultipartFormDataUpload
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
        [string]$ContentType,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $InFile))
        {
            $errorMessage = ("File {0} missing or unable to read." -f $InFile)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        if (-not $ContentType)
        {
            Add-Type -AssemblyName System.Web

            $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)
            
            if ($mimeType)
            {
                $ContentType = $mimeType
            }
            else
            {
                $ContentType = "application/octet-stream"
            }
        }
    }
    PROCESS
    {
		$fileName = Split-Path $InFile -leaf
		$boundary = [guid]::NewGuid().ToString()
		
    	$fileBin = [System.IO.File]::ReadAllBytes($InFile)
	    $enc = [System.Text.Encoding]::GetEncoding("iso-8859-1")

	    $template = @'
--{0}
Content-Disposition: form-data; name="fileData"; filename="{1}"
Content-Type: {2}

{3}
--{0}--

'@

        $body = $template -f $boundary, $fileName, $ContentType, $enc.GetString($fileBin)

		try
		{
			return Invoke-WebRequest -Uri $Uri `
									 -Method Post `
									 -ContentType "multipart/form-data; boundary=$boundary" `
									 -Body $body `
									 -Credential $Credential
		}
		catch [Exception]
		{
			$PSCmdlet.ThrowTerminatingError($_)
		}
    }
    END { }
}

As you can see, we are preparing “by hand” the correct message body and executing Invoke-WebRequest in order to send our file. It is not efficient at all as it encodes the whole file as a string and keeps it in memory. Consider transmitting large files and make yourself an idea on the memory footprint of this approach. Aside that, it is just not elegant!

Wait a moment, what a waste

Previously shown method is as said, not elegant and waists resources. A better approach is to let the HttpClient class handle everything. It is way more efficient as it uses streams in order to read our file and transfers it to http StreamContent. It is less resource intensive and a more elegant solution.

Let’s see now how our refactored Invoke-MultipartFormDataUpload cmdlet looks like:

function Invoke-MultipartFormDataUpload
{
    [CmdletBinding()]
    PARAM
    (
        [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
        [string]$ContentType,
        [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
        [System.Management.Automation.PSCredential]$Credential
    )
    BEGIN
    {
        if (-not (Test-Path $InFile))
        {
            $errorMessage = ("File {0} missing or unable to read." -f $InFile)
            $exception =  New-Object System.Exception $errorMessage
			$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
			$PSCmdlet.ThrowTerminatingError($errorRecord)
        }

        if (-not $ContentType)
        {
            Add-Type -AssemblyName System.Web

            $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)
            
            if ($mimeType)
            {
                $ContentType = $mimeType
            }
            else
            {
                $ContentType = "application/octet-stream"
            }
        }
    }
    PROCESS
    {
        Add-Type -AssemblyName System.Net.Http

		$httpClientHandler = New-Object System.Net.Http.HttpClientHandler

        if ($Credential)
        {
		    $networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password)
		    $httpClientHandler.Credentials = $networkCredential
        }

        $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler

        $packageFileStream = New-Object System.IO.FileStream @($InFile, [System.IO.FileMode]::Open)
        
		$contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue "form-data"
	    $contentDispositionHeaderValue.Name = "fileData"
		$contentDispositionHeaderValue.FileName = (Split-Path $InFile -leaf)

        $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
        $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
        $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $ContentType
        
        $content = New-Object System.Net.Http.MultipartFormDataContent
        $content.Add($streamContent)

        try
        {
			$response = $httpClient.PostAsync($Uri, $content).Result

			if (!$response.IsSuccessStatusCode)
			{
				$responseBody = $response.Content.ReadAsStringAsync().Result
				$errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody

				throw [System.Net.Http.HttpRequestException] $errorMessage
			}

			return $response.Content.ReadAsStringAsync().Result
        }
        catch [Exception]
        {
			$PSCmdlet.ThrowTerminatingError($_)
        }
        finally
        {
            if($null -ne $httpClient)
            {
                $httpClient.Dispose()
            }

            if($null -ne $response)
            {
                $response.Dispose()
            }
        }
    }
    END { }
}

Conclusion

There is still space for improvement, however in my case it is good enough and I will not continue extending this cmdlet. If you think this is not enough or good enough, please extend it and let me know. As an idea, you may make possible passing multiple files via the pipeline and execute the upload only once, in the END step. In that way it may be more efficient and flexible. Also supporting some other common parameters as Proxy would be beneficial to others. If any let me know, I would be happy to hear from you in comments.