Working with TFS in IntelliJ IDEA via VSTS Plugin

Mario Majcica January 2, 2017 0 Comments

Recently Microsoft released a plugin for JetBrains IntelliJ IDEA in order to integrate this IDE with VSTS and TFS. Although Microsoft did a good job describing on how this plugin should be installed and described some basic settings, there are many non covered questions by the documentation provided. This is especially true when it comes to TFS and HTTPS. I will show you what is necessary in order to setup a connection for both Git and TFVC repositories on TFS.

Installing the plugin

In order to install the plugin, in the main screen of IDEA, choose Plugins from Configure menu:

Once the plugin window opens, choose Browse repositories

In browse repositories search for Visual Studio

If the Visual Studio Team Services plugin is not found, your connection may not be setup correctly. In case you are, as I am, behind a proxy, you need to click on HTTP Proxy settings button in the same screen (bottom left) and you will be presented with the following dialog

Here you need to setup the necessary parameters (Auto detect proxy settings worked for me) and test your connection by clicking on Check connection button. Once done you will be prompted to enter an address that is behind the proxy so that internet connection can be verified. I used http://www.google.com/ for my test.

After confirming, you should see the following message

Now, back to Browse repositories screen, the Visual Studio Team Services plugin now should be found. Click on install and after the procedure finishes you will be asked to restart IntelliJ IDEA. Do so, restart the IDE.

The plugin is now installed. In case this is not sufficiently clear, you can also follow the Microsoft guide Visual Studio Team Services Plugin for IntelliJ IDEA and Android Studio.

Be also sure that you have at least version v1.111.0 installed as I encountered issues with TFVC and previous versions of the plugin.

Importing the certificate

In case you are not accessing TFS through SSL connection you can skip this part. For Visual Studio Team Services plugin to connect to the TFS via https, that is using a self-signed certificate, no matter if you do intend using Git or TFVC, you need to import the certificate in the IntelliJ IDEA certificate store. To do so, export your certificate in the Base-64 encoded X.509 format. You can read about this in one of mine previous posts, Installing self-signed certificates into Git cert store.
Once you exported your certificate and let’s say saved it in C:\temp folder under the name tfs.cer, you need to open the command prompt and position yourself under the folder containing keytool.exe application. You can find it in the IntelliJ IDEA install folder, which in my case is C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.3.1\jre\jre\bin.

Once there you will then execute the following command

keytool -keystore "C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.3.1\jre\jre\lib\security\cacerts" -importcert -alias TFS -file "C:\temp\tfs.cer"

You will be then prompted for the keystore password. If you haven’t changed it, the default password is

changeit

After typing the password you will see the details about the certificate you are trying to import, and again you will be prompted to confirm that you trust this certificate. Type yes and you are done. Your screen should look like this

Visual Studio Team Services plugin and Git

Before we even start, we need to be sure that a Git client is installed on our machine. From the Welcome screen choose Configure then Settings.

In the settings window move to Version control -> Git pane and test the path to your Git client

In case all is good you should get back the version of your Git client

If this is not the case, or the path is wrong (change it and try again) or the Git client is not present on your machine. You can install Git for windows and you will find the necessary here.

In case you are using the https connection with TFS and it is based on a self-signed certificate do not forget to add that certificate into the Git cert store. This is something different then adding it to the Java cert store that I described in the previous chapter. You will need to do both of these steps. As I already described this procedure in detail, you can follow my other blog post Installing self-signed certificates into Git cert store.

Once the Git client is installed and certificate is imported, we can continue setting up the Visual Studio Team Services plugin. Get back to the welcome scree of IntelliJ IDEA and in the version control drop down choose Team Services Git

At this point a new dialog will be presented to you.

Move to the Team Foundation Server tab and specify the address of your TFS server, then click connect. You will now be prompted for the credentials and if everything is ok, you will be show the list of available repositories

You are now able to clone the repository of your choice and start working with it directly from IntelliJ IDEA.

Visual Studio Team Services plugin and TFVC

Before starting with TFVC, as for Git, we need an external tool. The tool in question is TF command line tool. It ships with the Microsoft Team Explorer Everywhere 2015 and you can download it here.
The file we are interested in is TEE-CLC-14.0.3.zip. Download it and unzip it in a folder of your choice. You should end up with something similar to this.

Now, open the command prompt, move into the folder where you have extracted the TF command line tool and run the following:

tf eula /accept

If command succeeded and you haven’t received any error, you are good to go.

Now back to IDEA. Open the settings panel.

and move to Version Control -> TFVC pane. In the select path to executable field, enter the exact path to the tf.cmd command file located in TF command line tool folder.

Once done, press the test button and you should see the following message

Confirm all of the open windows and get back to the IDEA welcome page. Now you are ready to choose Team Services TFVC (Preview) version control.

At this point, same as for Git, you will be prompted about the connection towards your TFS. The following dialog will be shown.

Move to the Team Foundation Server tab and specify the address of your TFS server, then click connect. You will now be prompted for the credentials and if everything is ok, you will be shown the list of available TFVC repositories.

You can now create a new workspace directly from IDEA and start working with your TFVC repositories.

Troubleshooting

There are a couple of common issues you may encounter in following what I just described. I will tell you about the most common ones and how to overcome those.

In case you see the message in the following screenshot

You are probably facing some issues with the certificate. Make sure that you exported/imported the certificate correctly.

During the test of the TF command line utility, you may encounter the following exception

In may case it turned out to be a problem with allocating the heap memory from TF process. I could clearly see in the log file the following:

2016-12-22 13:14:08,949 [ 17642] INFO - ugin.external.commands.Command - 167680800(ns) - elapsed time for add -noprompt -?
2016-12-22 13:15:46,592 [ 115285] INFO - lugin.external.ToolRunnerCache - getRunningToolRunner: toolLocation={0}
2016-12-22 13:15:46,592 [ 115285] INFO - lugin.external.ToolRunnerCache - getRunningToolRunner: slow version - null
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - ToolRunner.start: toolLocation = C:\Utils\TEE-CLC-14.0.3\tf.cmd
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - ToolRunner.start: workingDirectory = null
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - arguments: add -noprompt -?
2016-12-22 13:15:46,716 [ 115409] INFO - ugin.external.commands.Command - CMD: Error occurred during initialization of VM
2016-12-22 13:15:46,717 [ 115410] INFO - ugin.external.commands.Command - CMD: Could not reserve enough space for 2097152KB object heap
2016-12-22 13:15:46,731 [ 115424] WARN - ugin.external.commands.Command - CMD: parsing output failed
com.microsoft.alm.plugin.external.exceptions.ToolBadExitCodeException: KEY_TF_BAD_EXIT_CODE
at com.microsoft.alm.plugin.external.tools.TfTool.throwBadExitCode(TfTool.java:109)
at com.microsoft.alm.plugin.external.commands.Command$1.completed(Command.java:155)
at com.microsoft.alm.plugin.external.ToolRunner$ListenerProxy.completed(ToolRunner.java:289)
at com.microsoft.alm.plugin.external.ToolRunner$ProcessWaiter.run(ToolRunner.java:327)

A workaround for this issue is to modify the tf.cmd file by specifying a lower -Xmx parameter. By default it is set to 2014MB however a 1024MB also worked well for me.

Another issue with this version of Visual Studio Team Services plugin is in changing in between the Team Services TFVC (Preview) and Team Services Git, or viceversa. You can read more about it here, TfsAuthenticator hangs the IDE.

No matter the issue, you can always find more information about it in the log file. The log file is located in my case in C:\Users\majcicam\.IdeaIC2016.3\system\log. Adapt this path to your case. The log file name is idea.log.
Visual Studio Team Services Plugin settings can be found in vsts_settings.xml file under C:\Users\majcicam\.IdeaIC2016.3\config\options folder.

Useful links

On visualstudio.com you can read more about this topic and see some useful how-to video.

1. Installing Visual Studio Team Services Plugin for IntelliJ IDEA and Android Studio
2. Using Visual Studio Team Services Plugin for IntelliJ

With all of these information I do hope you can get on going with the plugin and boost your productivity.

Happy coding.

Installing self-signed certificates into Git cert store

Mario Majcica December 27, 2016 2 Comments

Introduction

Since it’s introduction, Git repositories in TFS became quite a popular choice. Most of early adopters used the integrated Visual Studio tooling to interact with their repositories. It is all straight forward, simple and easy, clone your repository are you are ready to go. Now, if you ever tried to use the command line Git client or another IDE as Visual Studio Code (which relies on the command line tool), and communication with your Git repositories is based on SSL connection (https), you may have noticed that things do not work out of the box. Visual Studio will take care of certain things for us, as authentication and certificates (Windows cert store), and make it transparent (in case certificates are distributed via domain). If we intend to use the Git client we need to set a couple of things up. I will illustrate here how to retrieve your TFS certificate and install it in the Git certificate store.

Certificates

Often, in the enterprise environments, access to Team Foundation Server is made possible only through Transport Layer Security cryptographic protocol. This means that the client will need to validate a certificate before establishing the connection. Often the certificate is a self-signed and if you try to clone a repository you are going to receive the following error:

SSL certificate problem: unable to get local issuer certificate

This is due to the fact that the root certificate which vouches for the authenticity of your SSL certificate is private to your organization. That root certificate is distributed to all domain-joined machines in your organization via group policy, and it is stored in the Windows certificate store for your machine.

Any application written to use the Windows crypto APIs will have access to that root certificate, and will consider your TFS deployment to be trusted. Applications using the Windows certificate store include Internet Explorer, Google Chrome, Visual Studio and others. However, Git for Windows (git.exe) uses OpenSSL for its crypto stack, and the Git for Windows distribution includes a set of trusted root certificates in a simple text file. Your organization’s root certificate is not in this list, and if you try to use git.exe to perform network operations against your TFS server, you’ll get the error specified above.

In order to solve this problem we need to include our self signed certificate in the list of certificates used by Git.

Retrieve the TFS root certificate

In order to get the certificate I will use IE 11. However you can achieve the same result in multiple ways, following different steps.

First open your TFS portal in IE and once opened, click on the lock icon in the address bar:

Choose to view the certificate by clicking on the View certificates button. A new window will open showing the certificate details. Move to the Certification path tab as show here:

Make sure that the top level certificate is selected, same as in this screenshot and click on View certificate button. Another certificate details window will now open. In it, choose the Details tab:

Now, choose Copy to file option and follow the wizard that you will be presented with. You will need to export the certificate as Base64 encoded:

Save the certificate somewhere on your disk, name it lets say tfs.crt and close all of the open windows. Now we have the certificate in a format that we need, next step is adding it to the certificate store used by git.

Add TFS certificate to Git certificate store

On most of modern computers since the Git for Windows version 2.5, the certificate store is located in the following directory:
C:\Program Files\Git\mingw64\ssl\certs

Note that in some cases the folder may be located here:
C:\Users\\AppData\Local\Programs\Git\mingw64\ssl\certs

In case that you are using an older version this can differ. In that case an upgrade is advised.
Open the above mentioned directory and you should find a file called ca-bundle.crt.

Now, first open our certificate file, tfs.crt, with a text editor of your choice, select all content and copy it.

Then open the ca-bundle.crt file with the same text editor and position yourself at the end of the file. Now paste the previously copied content, save and close the all files.

Try again to clone a repository in TFS via git.exe. You should not receive the error message anymore and you should be prompted about credentials. By entering correct credentials the operation should succeed.

SonarQube on Windows and MS SQL

Mario Majcica April 28, 2016 10 Comments

Introduction

In the following post we will see what is necessary to install and configure SonarQube 5.4. We will also see how to setup some basic security concerns by making our SonarQube part of our LDAP infrastructure and map security groups to roles.
I’m sure that there are plenty of guides out there, but what I found most annoying meanwhile reading some of them, is that all of them do give several things for granted. Also the information is segmented and not easy to find. I will try in this post to cover even the basic steps that can save you hours of struggling. I’m going to install SonarQube on Windows platform using MS SQL as my database of choice, you can also try Couchbase. Both of these services in my case are going to reside on the same machine, but nothing limits you to use multiple machines for your setup.

Prerequisites

Java runtime is the main prerequisite. Although it works with Java 7, my advice is to install and use JDK 8. At the moment of writing the latest version for my platform is jdk-8u77-windows-x64.exe.
For what concerns MS SQL versions, 2008, 2012 and 2014 are supported. Also the SQL Express is supported. Your SQL server needs to support case-sensitive (CS) and accent-sensitive (AS) collation.

Installing the database

After you installed your MS SQL version of choice, you need to create a database. Add a new database and name it SonarQube.

Now the important step. In the Options page you need to specify the right collation. It needs to be one of the case-sensitive (CS) and accent-sensitive (AS) collations. In my case I will go for SQL_Latin1_General_CP1_CS_AS.

Once that is set, click OK and create the new database.

After the database is created, we need to make sure that the TCP/IP protocol is enabled for our SQL instance. Open the Sql Server Configuration Manager and, in the console pane, expand SQL Server Network Configuration. Choose the Protocols for your instance. In the details pane, right-click TCP/IP, and then click Enable. Once done, restart the service. A detailed guide is available on Technet at Enable TCP/IP Network Protocol for SQL Server.

Last but not least, make sure that SQL Server Browser service is running. Often it is disabled by default, however for the JDBC driver to work, it needs to be enabled and running. Open the Services management console and find the Service called SQL Server Browser. If disabled, enable it and start the service.

That’s all for now for what database concerns.

Installing SonarQube

Before we start, make sure that the latest JDK is installed, then download the SonarQube installation file from SonarQube website. For this demo I will be using the latest available version of SonarQube at the moment of writing and that is 5.4. After I downloaded sonarqube-5.4.zip I will extract it’s content in a folder of my choice and that is D:\SonarQube.

There is another important file we need to get and set before we can continue configuring SonarQube and that is Microsoft JDBC driver. Go to Download the Microsoft JDBC Driver 6.0 (Preview), 4.2, 4.1, or 4.0 for SQL Server and download sqljdbc_4.2.6420.100_enu.tar.gz file. Once done, open the just downloaded file with compression tool of your choice and extract all of it’s content in a temporary folder. Get into sqljdbc_4.2\enu\auth\x64 folder and copy the only file present in that path, sqljdbc_auth.dll and paste it into your System32 directory, usually C:\Windows\System32.

Now we are ready to start the configuration. Open the main configuration file of SonarQube called sonar.properties. You can find it in the conf folder in your SonarQube installation path. Open it with the editor of your choice and search for the line reporting ‘Microsoft SQLServer 2008/2012/2014 and SQL Azure’. Under that line you should see a the following configuration item that is commented out:

#sonar.jdbc.url=jdbc:sqlserver://localhost;databaseName=sonar;integratedSecurity=true

We need to uncomment this line by removing the hash sign in front of it and change the connection string to point towards our SQL database instance (the one we create earlier).
Following, an example of the connection string using a name instance of SQL:

sonar.jdbc.url=jdbc:sqlserver://localhost;databaseName=SonarQube;instanceName=DEV_01;integratedSecurity=true

If you are using the default instance, you can simply omit the instanceName=DEV_01 from your connection string.

Also you can see I’ve set to use the integrated security. If you want to use SQL Authentication, remove the integratedSecurity=true part and specify the credentials as separate configuration items under your connection string (also create users in SQL accordingly and map the newly create user to dbo schema).

sonar.jdbc.url=jdbc:sqlserver://localhost;databaseName=SonarQube
sonar.jdbc.username=sonarqube
sonar.jdbc.password=mypassword

Once the connection string is set, save the configuration file and try starting SonarQube. Open the command prompt and move to ...\bin\windows-x86-64 folder and execute StartSonar.bat

If everything is set right, you should see a message in the console INFO app[o.s.p.m.Monitor] Process[web] is up.

Now you can open the web browser of your choice and head to http://localhost:9000. A welcome page on SonarQube should be shown.

If the page loaded, congratulations, SonarQube is running correctly on your machine.

What is left to do is to create a service that will run SonarQube. Stop the current execution with a CTRL+C and terminate the batch job. In the same bin folder where StartSonar.bat is located, you will find InstallNTService.bat. Execute the just mentioned batch file and you should receive the wrapper | SonarQube installed. message. This means that a new service is created. Check your services management console and you should find a service called SonarQube:

As you can see from the picture, service is created but not started.
By default, the “Local System” account is used to execute the SonarQube service. If this account doesn’t have the required permission to create some directories/files in the SonarQube installation directory (which is the case by default on recent Windows versions), the execution of the SonarQube service will fail. In such case, the SonarQube service must be configured to run in the context of a suitable account.
Right click on the SonarQube service and choose properties then move to Log On tab choose “This account”, and select an account that can read/write the folder in which SonarQube is installed. Hopefully you will have a specific service account created for this purpose.

Now, you can start the service manually or by launching StartNTService.bat.

Services configuration

SonarQube is the only web application running on my server, so I will move it from the port 9000 to the default 80. To do so, edit the sonar.properties configuration file and find the #sonar.web.port=9000 comment line. Uncomment it and change port value to 80, sonar.web.port=80.

After this change you need to restart your SonarQube service and try to reach your localhost in the browser. If all went fine you will not need to specify the port at the end of the address.

SonarQube behind a proxy

I wrote in the past time numerous post about running services and applications behind a proxy. SonarQube will not be an exception to that practice. You may wonder why SonarQube should have access to internet and my answer is, plug-ins. Plug-ins are essential to SonarQube and installing and updating them is easiest done via Update Center, a functionality integrated in the administrative portal. In order for it to work, SonarQube needs to be able to access the internet. In case you are behind a proxy, you need to modify again sonar.properties configuration file.

Search for #sonar.web.javaAdditionalOpts= configuration line and modify it by specifying http, https proxy host and port:

sonar.web.javaAdditionalOpts=-Dhttp.proxyHost=swg.myProxy.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=swg.myProxy.com -Dhttps.proxyPort=8080

Restart the service and try the Update Center. Open SonarQube web page and log in with the default admin user (password is also admin). Click on the administration menu item and then, in the sub-menu, choose System -> Update Center. Check if the updates are retrieved and try to update one of the plug-ins installed by default, like C#. If all goes well you will see the following screen

Once the plug-in is installed you will see a button in the notification message that offers to restart the server for you. In my case it never worked and after choosing this option my server stopped replaying. In order to get it back online, you need to manually restart the service.

This problem is addressed in SONAR-7422 and it is a recognized bug. It should be fixed in the SonarQube version 5.6.

If you where able to install or update plug-in correctly, then your proxy settings where picked up fine.

Securing SonarQube connection

You can setup SonarQube to run on a https secure connection. It natively supports the SSL certificates however it is not advised to configure it. Using a reverse proxy infrastructure is the recommended way to set up your SonarQube installation on production environments which need to be highly secured. This allows to fully master all the security parameters that you want. I will not dig into details on how to set up IIS to leverage the reverse proxy setup. If interested in this, you can read the following blog post on Jesse Houwing’s blog, Configure SSL for SonarQube on Windows. It will guide you in setting up IIS that will act as a proxy for the secure calls towards your SonarQube server.

Security configuration

My desire is to integrate the authentication of the SonarQube server with my LDAP (Active Directory domain services). In order to do that, we need to install LDAP plug-in. Locate the LDAP plug-in in update center under available plug-ins and install it.

Before you restart your SonarQube service, open the sonar.properties configuration file and add the following section:

#----------------------------------------------------------------------
# LDAP

sonar.security.realm=LDAP
sonar.forceAuthentication=true
sonar.authenticator.downcase=true

This are the only necessary settings if you are part of the Active Directory domain. Restart the SonarQube service and open the portal. If all went well, SSO kicked in, and you should be logged in with your domain account. Now comes the fun part. Log out, then, log in again as administrator and go to Administration -> Security -> Users screen. You should see in the list the domain account you logged in with. Update groups for this account and assign it to sonar-administrators group.

Now close the browser and reopen it. Surprise, surprise, you are logged in again via your user profile but you do not see Administration option in your menu, as you would expect. Once the LDAP is configured, on each login, the membership information will be retrieved and local settings will be overwritten. Thus no group membership we assigned will be persisted. In this case, LDAP/AD becomes the one and only place to manage group membership. In order to do so, we need to create a security group in AD and map it in the SonarQube Security Groups.

Before we create a new group in the SonarQube Security Groups we need to get the groups precise name. Group names are case sensitive and do require the domain to be specified. This is not something we can guess but we can extract it from our log file.

Add your user to the AD security group of choice. Edit sonar.properties configuration file again and set the logging level to a higher setting. In order to do so, find the #sonar.log.level=INFO line, uncomment it and change the level from INFO to DEBUG. You line should now look like sonar.log.level=DEBUG. Restart the service and open the portal.

If you are successfully logged in, open the log file. In the SonarQube directory there is a folder called logs, in my case it is, sonarqube-5.4\logs. Inside you will find a file called sonar.log. Open it with your editor of choice and search for your domain username. Next to your username (probably at the bottom of the log file) you will find a couple of log lines made by web[o.s.p.l.w.WindowsAuthenticationHelper] and in one of those lines you will find written Groups for the user YOURDOMAIN\YOU and a list of security groups you are part of. Find the correct one and copy it, in my case this is sonar@maiolocal. Now log in as admin and open the Groups screen. Create new group by clicking to the Create Group button in top right corner and set the name to your group of choice, in my case sonar@maiolocal.

Once the group is created, move to Global Permissions screen (always in the Security menu), and assign the desired permissions to just created group. Let’s suppose that this group will list all of the administrators, under Administer System permission, click on groups and select the newly created group.

Now if you close your browser and reopen it pointing to your SonarQube portal, you will get logged in via SSO and you should be able to see the Administration button in the menu. Same can be done for the users.

Conclusion

This is roughly it. There are some details you would probably like to set as SMTP/Email settings and Source Control Manager settings, however all of this is quite trivial as you can find all of the necessary settings in the UI under General Settings. For more details check Notifications – Administration page in SonarQube documentation site, as SCM support page.

Your SonarQube server should now be correctly installed and configured to access LDAP. Ahhh, I almost forgot it, get the logging level back to INFO, otherwise you are risking quite a large log files on your disk.

UPDATE

After I published my post I realized that there is a better way of forcing the authentication. As Nicolas Bontoux pointed out this setting should be set in Administration – General Settings – Security pane. Otherwise you do risk encountering a problem during the upgrade of your SonarQube instance to a newer version.

Set the Force user authentication in previously mentioned pane.

Once done, comment or remove the sonar.forceAuthentication=true line from the sonar.properties configuration file and restart your service.
In this way you will not be bothered during your SonarQube updates.

UPDATE 2

In regard to the proxy settings, since SonarQube 5.5 specifying your proxy address via javaAdditionalOpts is not necessary anymore (and it is not advisable). Now it is sufficient to search in the sonar.properties configuration file the following line #http.proxyHost= and set the proxy parameters as shown here:

http.proxyHost=swg.myproxy.com
http.proxyPort=8080
https.proxyHost=swg.myproxy.com
https.proxyPort=8080

Save your settings and restart the service. Your Update Center should still be working correctly.