Publish your VSTS extension using Team Build and Release Management

While I’ve released my VSTS Extension Tasks to GitHub and the Visual Studio Marketplace some time ago, I was reminded that I’ve never really blogged about it yet. You may wonder what it does.

The VSTS Extension Tasks are a set of tasks for Team Build and Release Management to package, publish, share and install extensions to your Visual Studio Team Services account or TFS 2015 update 2 server.

There are quite a few people using the extension to package and publish their Visual Studio Team Services and TFS extensions.

There’s a few places with more information on how to use them:

I’m using these tasks to package and publish my other build tasks:
A few other MVPs/Rangers that are using these tasks include Richard Fennel, Rene van Osnabrugge, Peter Groenewegen and Utkarsh Shigihalli and others.

Publish your extension to a local TFS Update 2 server

With the availability of extensions for TFS I’ve been looking for an easy way to publish extensions and their updates to the local marketplace. While I’m sure that Microsoft will at some point integrate the two, for now you need to manually sync the extensions between the VSTS Marketplace and your local TFS Marketplace.

Turns out that the tools used to publish to the VSTS Marketplace work for the TFS one as well. to publish an extension you do need to pass in the service url manually and this took a little bit of fiddling to figure out what to put in there. Turns out that it takes the server root,


If your server isn’t configured with Basic Authentication enabled, you can use the Fiddler hack to authenticate over NTLM. As you can see by the –proxy option int he command above, that’s what I’m doing at the moment.

It should be relatively easy to build a PowerShell script that uses the –json option to list all extensions on the local TFS Marketplace and then check the online marketplace for a newer version to automatically sync extensions which have already been installed. Stick that in a Build Definition on a Schedule and your local marketplace will always be up to date with the latest versions. That’s something for a future blogpost.

Define a custom Authentication Scheme for your VSTS extension

The standard Authentication Schemed in VSTS include Basic (username/password), Token, Service Credential (for Azure) and AzureCertificate Authentication. If you simply need a username password or a Personal Access token you can derive your own Credential Type from one of the exiting Authentication Schemes.

You can see an example of this technique in my VSTS Extension tasks

But what if you need to extend beyond the basics? There are ways to extend the UI with additional textboxes, as done in some of the other extensions you’ll find out on GitHub.

NOTE: I’ve been asked by Microsoft to explain that even though the below code will currently create your custom auth scheme, the product still assumes that the standard list of authentication schemes is fixed. You will run into unforeseen issues until this fully opens up at some point in time.

I’m leaving the below piece for reference, as it’s a nice way to understand how the extensibility works and how items are defined, but please do not use it at the moment to create custom auth schemes. While your extension will likely pass all validations, it may cause all kinds of issues for your consumers.

Instead, expand the Basic or Token auth scheme with additional parameters and change the Name/Description to make the UI reflect your needs.

If you want to go beyond that and want to define your own UI with its own fields, the next step is to define your own endpoint-auth-scheme. The documentation on this contributionpoint is still scant, but if you dig into a TFS 2015 update 2 installation you’ll find some very useful examples.

Navigate to C:Program FilesMicrosoft Team Foundation Server 14.0ToolsDeployTfsServicingFilesExtensions to find the standard extensions that define the basic behavior of a standard installation. In there you’ll find 4 vsix files



If you dig a little deeper you’ll find extension.vsomanifest which contains the Json snippet you’re after. The example from the Service Credential is one of the most extensive:

You’ll be able to stick these in your own extension manifests in order to create a full custom endpoint credential type with a complete custom authentication scheme. The sample below can be adapted to do exactly that:

Automatically populate your Source branch when publishing to Sonar

When you enable your build definition to trigger on multiple branches, you may not want them all to publish into the same Sonar project. 
Sonar has the concept of Branches, they’re simply appended to the Project Key with a -. This prevents an old Hotfix build from messing up the trend information of your current development. It also helps you configure different baselines for older versions (branches).
When building from the Visual Studio Team Services Build tasks you can configure the branch automatically by including the Build.SourceBranchName variable in your Project Key:

Some background on the values assigned to this variable can be found on the Visual Studio Team Build Variables documention:

Build.SourceBranchName
BUILD_SOURCEBRANCHNAME
The name of the branch the build was queued for.
  • For Git this is the last path segment in the ref. For example in the name is .
  • For TFVC this will last path segment in the root server path for the workspace. For example in   the name is 

Configure SSL for SonarQube on Windows

The documentation for SonarQube explains how to configure SSL when you’re running on Linux and how to use the native Tomcat functionality for a simple test environment, yet they recommend not to use this functionality in any production like setting.

Instead SonarQube recommends the use of a Reverse Proxy to secure you sonar installation. With the help of IIS and the Url Rewrite module, that’s a piece of cake to setup.

What you’ll need:

After installing and enabling these features, I had to do an IIS Reset and re-open the IIS Manager for all features to start working.
Next we’ll go through the steps to configure the reverse proxy.
First step is to create a IIS website which will act as the reverse proxy.
Unless you’re required to do Kerberos authentication, you don’t need to configure any form of authentication on your Reverse Proxy. It should forward the challenge from SonarQube if you’ve configured Active Directory integration there. 
Configure the binding to use SSL and setup the correct hostnames and the certificate. I’m cheating a little by using the IIS Express Development Certificate installed on my machine:
Next we’ll open the URL Rewrite settings to configure reverse proxy:
Click Add Rule to create a new rule
And pick “Reverse Proxy” from the list of templates:
Enter the destination server URL (can be https://xebia.com/blog:9000, or even a remote server) and click OK to create the rule:

You’re back in the URL Rewrite screen where we’ll need to add an extra server variable which we’ll send along with the request to the other server in order to tell SonarQube it’s actually behind a Reverse Proxy that’s doing the SSL offloading for it:
Click “Add…” to create the server variable:
Add the server variable “X_FORWARDED_PROTO” to allow the Rewrite Module to manipulate this header:
You should now have the variable listed in the Variable list. Click “Go back to Rules” to move back to the rules list:
Edit the URL Rewrite rule you’ve just created:
Expand the Server variables section of the rule definition:
Add the “X_FORWARDED_PROTO” header you’ve allowed in the previous step and give it the value “https“:
Apply the changes:
And now you should be able to access SonarQube over SSL. You may want to configure the original SonarQube instance to only accept traffic from your reverse proxy or only accept traffic from localhost through the Windows Firewall.

Reference specific line of a file in the Code hub

One of the features of GitHub I use quite often is the ability to right-click any line number to create a link to that specific file on the exact line. It can eb useful when submitting issues or when providing a link to a piece of sample code.

While not so easy to find in VsTeam, the feature is there. To link to a specific line in the code hub first navigate to the file you want to link to, then copy the current location from the browser and append &line={number}.

That’s all!

Access all available build variables from your VSTS Build task

NOTE: This cmdlet works with the Powershell host that shipped with TFS 2015 RTM and which will be replaced with a new task runner. Whether this functionality will remain, is uncertain.

If you want to know which variables are available during your build, or if you want people to create variables with a specific prefix and want to do anything with that, then you need access to the list of all available variables.

Though it’s easy to access a specific variable by name using the Get-Variable cmdlet, it’s not so easy to access all variables. Plus, the Get-Variable command only works when you know the full name beforehand.

Using the below function you receive a dictionary with the name and current value of each variable. You can use it to build tasks that use convention over configuration and retrie a list of all variables that start with a prefix you expect:

Use te cmdlet below to fetch all variables and do with them what you like:

I used this little snippet to create a new build task which can expand the value of nested variables. You can find it here:

Binary promotion of Visual Studio Team Service Extension

When creating a Visual Studio Team Services extension (such as a collection of build tasks), it makes sense to publish it as a private extension first. You can do this by overwriting the extensionid from the commandline, but there is no direct commandline switch to override the visibility of an extension. I used to patch the extension manifest, but today finally uncovered the correct way of doing it.

It turns out that the TFS cross platform commandline (tfx) has a –override option which you can pass a piece of json to which will override any json in the extension.

So if your standard publish command looks like this:

tfx extension publish –extension-id vsts-tfvc-tasks –publisher jessehouwing –vsix package.vsix

Then the command to publish the private test version would look like this:

tfx extension publish –extension-id vsts-tfvc-tasks-TEST –publisher jessehouwing –override “{“”gallerySpecs””: [ “”Private””, “”Preview””], “”public””: false }” –share-with jessehouwing

Instead of passing the json on the commandline, you can also specify an –overridesFile.

Or, on the safer side, make the extension-manifest you commit to your source control repository always do a private preview release, and then use the –override option to merge the real publish settings.

Note: ensure you escape every double quote in the JSON by doubling it up.

Now I can setup a Visual Studio Release Management template to publish the tasks, test against my personal account and then promote that vsix unmodified to the marketplace. But that’s for the another blog.

Upgrading from XAML to Build 2015 with minimal changes

This series is currently a work in progress

XAML builds on Visual Studio Team Services will be deprecated in September 2016. While support for XAML builds seems to stay around on-premise for a little while longer, it’s time to upgrade to Build 2015.

You may have invested quite a bit in the XAML build system in the past, but as long as your changes have been mostly limited to custom MsBuild tasks and targets, Powershell or batch scripts, then upgrading to Build 2015 is (almost) a piece of cake.

In this series I’ll go through the basic settings in the XAML Default Template and I’ll explain how to transfer each setting over to the new system. There are still a few things which are not possible in the new system, such as configuring Gated Checkins for TFVC, but almost every other setting can be converted over.

Before going over each setting, I’ll assume that you’ve installed at least one build agent, or are using Visual Studio Team Services with the Hosted Build option.

Other tips and tricks

Manipulating your TFVC repository from Build 2015

Though generally considered a bad practice I crafted two build tasks to manipulate your TFVC repository from the build. You can now edit, delete and add files and check those changes in when you’re ready.

Before I explain how they work and how you can use them, I need to get the following out of the way:

I do not think it’s a good idea to change your sources as part of a build.
Unless that’s the only thing your build definition does.
In these cases it should not build your sources and produce your shipping artifacts.

The reason why I’m reluctant, is that changing your sources during the build has impact on other features of the Visual Studio Team Services platform. 
  1. As your code is being built, certain artifacts are generated, such as your debug symbols, test results, code coverage etc. And these artifacts are associated with the Changeset number of the build. This may break or cause problems with Remote debugging, Source Server support, Intellitrace and Test Impact Analysis among other features. Other tools that may be impacted are obfuscators.
  2. Since your scripts change files, features such as Incremental builds won’t work. They’ll always have to rebuild the projects, since there are always modified files. This is especially so when you use tasks to patch the AssemblyVersion attribute. This will slow down your builds slow down the feedback cycle as a result.
So, now I hope you understand this is a bad idea and you may wonder why I chose to build these tasks anyway.
The primary reason I built these, is to help clients migrate from customized XAML builds to the new build system with minimal impact. I’d rather have them fix their mistakes in the new system, than delay their move by fixing it in the XAML workflows.
The secondary reason I built these, is to provide ways to schedule changes to sources automatically. You may have a manual build trigger which updates all the version numbers and checks in the changed files, which in turn triggers a CI build to build the code. By using this two-stage approach, your build and symbols will be associated with the correct changeset and you won’t have the issues described before.
I’m not terribly happy about the quality of the code, yet, but that will improve now that they do the thing they’re required to do. I admit I’ve learned way too much about Powershell/.NET interop in the past days…
You may ask why I didn’t simply wrap tf.exe, but I assure you, I’ve tried. It fails to perform the check-in as it doesn’t pick-up the security token authentication used by the new Build system. You’ll be greeted by a cryptic security error when you try to check in the files:
#[error]TF30063: You are not authorized to access https://account.visualstudio.com/DefaultCollection.
#[error]Process completed with exit code 100 and had 1 error(s) written to the error stream.

Introducing the TFCV Build tasks

So with those warnings and a bit of guidance I present to you the VSTS-TFVC-Tasks project on Github. Which consists of two tasks for the new 2015 build system:

TFVC Check-in changes

The first task checks in any changes and deletes you’ve made to your $(Build.SourcesDirectory), optionally scoped to a specific path. It’s able to by-pass your checkin policies and can optionally suppress a CI build by adding a specific comment.

Since Build 2015 uses Local workspaces any edit or delete is automatically detected.

Consent
  • I understand – if your build system dived into an endless loop, uses up all your build minutes or destroys your source tree by accident. You’ve signaled you understand this risk, the task won’t work without this consent.
Configuration
  • Files to check in – specify one or more paths, separated by ; or newline. While wildcards (?, *) are supported, they’re not the same as the minimatch patterns used by other build tasks. 
  • Recursion – controls whether you’ll be checking in just the current folder/file (None), one level below the selected path (One Level) or all levels below the selected path (Full).
  • Check-in comment The comment which will show up in the source control history for this changeset.
Notes
  • Notes – specify one or more checkin notes using the following format Name=Value separated by ; or newline.
Policy Override
  • Bypass CI on check in – adds a special string to the check-in comment to ensure that no other build is triggered.
  • Override policies – to override any check-in policies or changeset-notes that may be required by your administrator.
  • Override reason – when enabling Override Policies, you’re required to specify a reason. This reason is stored with the changeset.

TFVC Add new files

Unlike edits and deletes, additions aren’t automatically checked in, they need to be signaled before they’ll be picked up by the Check-in task. Adding them is simple, just specify the paths you want included and they’ll be added to the list.
Configuration
  • Files to add – specify one or more paths, separated by ; or newline. While wildcards (?, *) are supported, they’re not the same as the minimatch patterns used by other build tasks. 
  • Recursive – controls whether you’ll be checking in just the current folder/file (unechecked), or all levels below the selected path (checked).
  • Apply Localitem Exclusions – will ignore files that are specified in your .tfignore file if your repository has one.

Installation

To install these tasks you’ll need to install Node.js and the tfs-cli node package using the instructions described here. Then configure your Project Collection uri using tfx login. After that you can either run the publish.ps1 . script in the repository or issue the tfx build tasks upload command yourself for each task you’re interested in.
To install these tasks to a TFS 2015 server, you’ll need to configure Basic Authentication on your application tier or use this trick to let Fiddler handle the authentication for you.