Running Large(r) Startup Scripts On Azure Windows VMs

Recently I was challenged with running a large startup script on a Windows Virtual Machine with limited internet connectivity. In this blog I’ll show you how to compress a startup script and run large(r) startup scripts.

VM Startup scripts

Startup scripts are used to configure or initialize a Virtual Machine at boot or provisioning time. The script is usually provided as user data and executed using cloud-init. See AWS, Azure and GCP.

Azure Windows VM Startup scripts

Azure provides different options to configure a startup script. The easiest solution is the CustomScriptExtension. This extension accepts your script and executes it.

resource "azurerm_virtual_machine_extension" "vm_initialize" {
  virtual_machine_id = var.virtual_machine_id
  name               = "My VM Init Script"

  publisher                  = "Microsoft.Compute"
  type                       = "CustomScriptExtension"
  type_handler_version       = "1.10"

  # NOTE: Script is executed from a cmd-shell, therefore escape " as ".
  #       Second, since value is json-encoded, escape " as \".
  settings = <<SETTINGS
    {
      "commandToExecute": "powershell -Command Write-Output \"Hello CustomScriptExtension!\""
    }
SETTINGS
}

Running Large Startup scripts

You can only deploy the CustomScriptExtension once for your VM. Therefore you quickly end up combining scripts into a large script. As a result you’re faced with the command line string limitations: a single command cannot exceed 8191 characters.

We decrease the number of characters by compressing the script using Terraform’s base64gzip-method. Next we update the PowerShell-command to decode and invoke the script:

$InputStream = [IO.MemoryStream]::New([System.Convert]::FromBase64String("base64gzipped string..."));
$GzipStream = [IO.Compression.GzipStream]::New($InputStream, [IO.Compression.CompressionMode]::Decompress);
$ScriptReader=[IO.StreamReader]::New($GzipStream, [System.Text.Encoding]::UTF8);
Set-Content "C:WindowsTempstartup-script.ps1" $ScriptReader.ReadToEnd();
$ScriptReader.Close();
."C:WindowsTempstartup-script.ps1" -parameterX "someValue"

Finally, the command is integrated into the Terraform configuration:

resource "azurerm_virtual_machine_extension" "vm_initialize" {
  virtual_machine_id = var.virtual_machine_id
  name               = "My Large VM Init Script"

  publisher                  = "Microsoft.Compute"
  type                       = "CustomScriptExtension"
  type_handler_version       = "1.10"

  # NOTE 1: Script is executed from a cmd-shell, therefore escape " as ".
  #         Second, since value is json-encoded, escape " as \".
  # NOTE 2: A Windows command is limited to 8191 characters. Therefore 
  #         the script is gzipped and base64 encoded. Source:
  #         https://docs.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation#more-information
  settings = <<SETTINGS
    {
      "commandToExecute": "powershell -ExecutionPolicy Unrestricted -Command $is=[IO.MemoryStream]::New([System.Convert]::FromBase64String(\"${base64gzip(file("${path.module}/startup-script.ps1"))}\")); $gs=[IO.Compression.GzipStream]::New($is, [IO.Compression.CompressionMode]::Decompress); $r=[IO.StreamReader]::New($gs, [System.Text.Encoding]::UTF8); Set-Content \"C:\Windows\Temp\startup-script.ps1\" $r.ReadToEnd(); $r.Close(); .\"C:\Windows\Temp\startup-script.ps1\" -parameterX \"${var.some_value}\""
    }
SETTINGS
}

Discussion

The CustomScriptExtension is no perfect fit for startup scripts. Ideally a startup script runs at provisioning time, and stops the VM on failure. Custom data is the Azure offering for this. This requires you to build a custom image to run your script. Building a custom image, however, defeats the purpose of the startup script, because you’ll just include the software and configuration the custom image. I’d love to see Azure improve this using fixed metadata keys like GCP. In GCP, running a startup script is as easy as assigning the script to the VM metadata using key windows-startup-script-ps1.

The applied compression will not scale indefinitely. At some point in time the script is too large to inline. The Azure provided alternative is to host the script on a storage account and use the system assigned managed identity to access the storage account using private network connectivity. Managing this additional infrastructure is challenging, because you need to add the file to the storage account. This requires data plane access which involves adding checks to deal with IAM, Private endpoint or Firewall rule propagation delays. Therefore, I’d recommend hosting the script on a trusted artifact registry. That way, you can still pull the script, check the integrity and run even larger startup scripts.

Conclusion

Running startup scripts should be easy. Regardless of the environmental constraints. By compressing the startup script you are able to run larger scripts with the CustomScriptExtension.

Photo by Tam Ming from Pexels