Using Windows Machine File Copy (WinRM) VSTS extension

Mario Majcica December 1, 2017 3 Comments

Implementation of the original Windows Machine File Copy task is based on Net Use Command and Robocopy. This command makes use of the SMB (server message block) and the Netbios protocol on port 139 or 445. Although by default this should be always supported in Intranets, it may be that due to the network restrictions or security policies it is not possible to setup such a connection or you are running a copy on a machine that is out of your local network. Recently I faced an issue copying files with Windows Machine File Copy task due to the SMB restrictions. This pushed me to recreate the same task as the original Windows Machine File Copy task, however with the transfer based on WinRM protocols. I shared my work in a form of an extension on Visual Studio Team Services – Visual Studio Marketplace. You can find my extension here WinRm File Copy.

Sources are available on GitHub in the repository called mmajcica/win-rm-file-copy, meanwhile the original implementation is part of the Microsoft/vsts-tasks repository.

In this post I will not go into the implementation details, just illustrate the usage of the task itself.

Usage wise, there are no differences with the original Microsoft task and this was also my main goal. Here is a screenshot of the task:

As you can see, all of the parameters are almost the same as for the original task.

Requirements wise, PowerShell V5 is required both on the build server as on the destination machine. And that is the only requirement, given for granted that WinRM is correctly setup.

Let’s quickly see how to set up a file copy. As for the Microsoft task, you need to specify the following parameters:

• Source: The source of the files. As described above using pre-defined system variables like $(Build.Repository.LocalPath) make it easy to specify the location of the build on the Build Automation Agent machine. The variables resolve to the working folder on the agent machine, when the task is run on it. Wild cards like **/*.zip are not supported. Probably you are going to copy something from your artifacts folder that was generated in previous steps of your build/release, at example $(System.ArtifactsDirectory)\Something
• Machines: Specify comma separated list of machine FQDNs/ip addresses along with port(optional). For example dbserver.fabrikam.com, dbserver_int.fabrikam.com:5988,192.168.34:5989.
• Admin Login: Domain/Local administrator of the target host. Format: \ < Admin User>.
• Password: Password for the admin login. It can accept variable defined in Build/Release definitions as ‘$(passwordVariable)’. You may mark variable type as ‘secret’ to secure it.
• Destination Folder: The folder in the Windows machines where the files will be copied to. An example of the destination folder is C:\FabrikamFibre\Web.
• Use SSL: In case you are using secure WinRM, HTTPS for transport, this is the setting you will need to flag.
• Clean Target: Checking this option will clean the destination folder prior to copying the files to it.
• Copy Files in Parallel: Checking this option will copy files to all the target machines in parallel, which can speed up the copying process.

There is not much more to say. If you need to copy a file or a folder, from your build agent, in a target folder on a remote machine, using WinRm as a transfer media, this is the way to go.

Happy coping!

XL Deploy in VSTS/TFS build/release pipelines

Mario Majcica July 19, 2017 0 Comments

Introduction

Circa a year ago I showed you how to use XebiaLabs Team Foundation Server 2015 XL Deploy plugin to create, import and deploy, XL Deploy Deployment Archives, on TFS and VSTS. With a new version of the plugin, that is now renamed into VSTS extension for XL Deploy, XebiaLabs delivers two more build/release tasks. This tasks are supported both by build and release pipelines in VSTS and Team Foundation Server. This will allow us to be more granular and chose to perform only certain operations at each stage, in our pipelines. We can at example choose to create our XL Deploy Deployment Archive during the build and later on, in our release definition, deploy the same artifact in different environments, using XL Deploy.
In the following post, I’ll show you an example on how to achieve this by using our two new tasks, respectively, Package with XL Deploy build task and Deploy with XL Deploy build task.

Building the artifact

As in my previous post, we will keep our Visual Studio Build, however, instead of using the XL Deploy build task, we will add Package With XL Deploy task which you can find in the Package category in the Task Catalog

Once added, we need to point it to the right manifest path, chose if we want the task to set the version of the created package based on the build number and, in our case, Publish the artifact to TFS. In this way once the package is created, it will be uploaded to TFS and associated as an artifact to the current build. At the end the result should be similar to the following.

You can now try your build and at the end, if successful, you should see our DAR package under the build artifacts.

If you can see your artifact uploaded, we can move to the next step.

Deploying from the VSTS Release Management

We are going now to create a new release definition and start with an empty template. As artifact source we are going to indicate a build and select from the relevant drop-down, build definition that we created in our previous step. You also may choose to select a continuous deployment option and a specific build queue for this build.

Once the release definition is created, for your initial Environment, choose Add task and select Deploy with XL Deploy from the task catalog.

This task will allow us to import the DAR package into XL Deploy and trigger the deployment for the desired environment. Bare in mind that it will check if the package is already in XL Deploy and, if so, will skip the import. This means that if you are using it multiple times for different environment definitions, it will behave as efficiently as possible.

Now we need to select the XL Deploy Endpoint, which indicates the instance of XL Deploy we are using for this deployment and fill in the necessary indications about the artifact location and the name of our target environment. For what concerns the XL Deploy Endpoint you can get further information in Add an XL Deploy endpoint in Visual Studio Team Services document. Artifact can be ‘picked up’ from the Server, it means that it will be made available by the release itself, or it can be retrieved from the file share. In our case it will be delivered by the release which will gets it from the associated build.

That’s all! Now we can create a new release and let our task delegate the deployment to XL Deploy.

Conclusion

We saw how can we leverage the new build/release tasks to interact with XL Deploy in different phases of our Build/Release process.

You can find more VSTS specific information at the following link or more information about the VSTS extension for XL Deploy here.

VSTS/TFS Agents behind a proxy

Mario Majcica April 9, 2017 0 Comments

Since I wrote this post things regarding the agents and proxies changed drastically, for good. Now setting up an agent behind the proxy is a documented procedure and can be done directly from the initial configuration of the agent. You can read more about the procedure itself here Run a self-hosted agent behind a web proxy.

It often happens that I do write about my love affair with a web proxy systems. In the past years I shared many of these challenges through my blog. Following is another one that I came lately across and that is precisely, allowing the build agent tasks to gain the access to internet via a proxy.
Till now I have never noticed any necessity for setting explicitly the access via a proxy server on any of our build agents. We were using Azure App Service Deployment and other build/release tasks that did accessed an external web address and everything always worked without the need for any particular intervention. However, after the TFS 2017 Update 1, we noticed that our Azure App Service Deployment task started failing with the following error:

Error: connect ETIMEDOUT 104.45.0.17:443

Clearly the call towards the Azure services was failing. Couldn’t be anything else then our lovely proxy. Strange thing was that it was all working till before the upgrade. At this point I was curious why this was happening. Here is what I found.

With the TFS 2017 Update 1, a new version of the Azure App Service Deployment build/release task has shipped. This new task implementation is now based on Node.js handler and no Node.js applications were able to access the web. So obviously I went to setup the necessary to enable Node.js applications to get the access to the web. This was however not the right solution, also because build agent doesn’t use the Node.js engine installed on that machine, but executes everything via a version of Node.js that is shipped together with the agent itself. In order to get a bit more insight in this, I had to peak in the couple of MS Github repositories and luckily I found a reference in which I see that the Node.js HttpClient wrapper sets the proxy for all calls based on the environment variables called HTTP_PROXY and HTTPS_PROXY. Now, I tried adding those two variables and voila my build task started working again (after the build agent service restart).

So, in order to have your Node.js based tasks that do need to access Internet from a build agent behind a proxy, add two new environment variables called HTTP_PROXY and HTTPS_PROXY and set the value of those variables pointing to the proxy itself, like http://my.proxy.local:8080.

Do not forget to restart your agent and give it a go.
I hope this solves it.

Happy releasing!