Many SSL certificate authorities (CAs) do not natively support .PFX format certificates which means that if you plan on installing them on something like an Azure App Service, you may encounter issues. Today, let’s figure out how to convert a CRT SSL certificate chain to PFX format.

First, let’s generate a private key and certificate signing request. Run the following command, and answer the questions as accurately as possible. The private key file (domain.key) should be kept secret and protected.

openssl req \
        -newkey rsa:2048 -nodes -keyout domain.key \
        -out domain.csr

Next, take the contents of domain.csr (it is just a plaintext file with your answers and some other non-secret information base64-encoded; it can be opened in any text editor) and request your certificate through your CA. This process varies per certificate authority, and so is out of scope for this article.

[Time passes]

Now, your CA provides you with a .ZIP file with the following files.

your_domain_com.crt
AAACertificateServices.crt
DomainValidationSecureServerCA.crt
USERTrustRSAAAACA.crt

^{(where your_domain_com.crt is the actual certificate file and the other .CRT files represent the various certificates that will allow a browser to chain up to the root; while the filenames and number of files will almost certainly be different for each certificate authority, the point here is to illustrate that there will be some number of .CRT files and that they are all important)}

Extract those files into the same folder that you have the domain.key file from earlier in.

Finally, let’s take our certificate and combine them with the rest of the chain to create a single .PFX file by running the following command. Your site’s certificate should be specified in the -in parameter, and for each of the chain certificates, adding another -certfile entry.

openssl pkcs12 -export -out certificate.pfx \
        -inkey domain.key \
        -in your_domain_com.crt \
        -certfile AAACertificateServices.crt \
        -certfile DomainValidationSecureServerCA.crt \
        -certfile USERTrustRSAAAACA.crt

NOTE: Azure App Services and Azure Key Vaults require a password-protected .PFX file, so ensure that you enter one when prompted. When you go to upload the certificate and you are required to select the .PFX file and a password, the password you created here is the one it’s referring to.

And you’re done! You now have a file in that folder (certificate.pfx) that you can upload/install and ensure your site is protected against MITM attacks.

The problem

We’ve recently begun attempting to scale our Azure App Services up and down for our test environments; scaling them up to match production performance levels (SKU: PremiumV2) during the day and then back down to minimal (SKU: Basic) at the end of the working day to save on costs. Just in the last month or two however, we’ve started to get the error “Cannot use the SKU Basic with File Change Audit for site XXX-XXX-XXX-XXX”.

Initially, we thought it had to do with the fact that we had a Diagnostic setting that was tracking AppServiceFileAuditLogs, but even after removing that Diagnostic setting before attempting the scale down, the issue persisted.

After banging our head against the walls with no progress being made, we opened a low-severity ticket with Azure Support to understand what was going on. They suggested we remove the following App Configuration settings:

• DIAGNOSTICS_AZUREBLOBCONTAINERSASURL
• DIAGNOSTICS_AZUREBLOBRETENTIONINDAYS
• DiagnosticServices_EXTENSION_VERSION
• WEBSITE_HTTPLOGGING_CONTAINER_URL
• WEBSITE_HTTPLOGGING_RETENTION_DAYS

Again, these did not have any effect.

It was at this time that I was in the portal browsing around for something else and happened to notice the “JSON View” option at the top right of the App Service so I checked it out and grep’d for audit just to see what I’d find.

Bingo: fileChangeAuditEnabled

Not so bingo: fileChangeAuditEnabled: null

But seeing that setting got me to thinking. What if there’s a bug in what JSON View is showing. The error we’re receiving is clearly saying it’s enabled, but the website is showing null; what if there’s some kind of type-mismatch going on behind the portal that is showing null but actually has a setting? What if we could use a different mechanism to test that theory?

Well, it just so happens that Azure PowerShell has a Get-AzResource function that can do just that and this blog post shows us how to do that.

The solution

First, let’s get the resource:

$appServiceRG = "example-resource-group"
$appServiceName = "example-app-service-name"
$config = Get-AzResource -ResourceGroupName $appServiceRG `
    -ResourceType "Microsoft.Web/sites/config" `
    -ResourceName "$($appServiceName)/web" `
    -apiVersion 2016-08-01

We now have an object in $config that we can now check the properties of by doing:

$config.Properties

And there it is:

fileChangeAuditEnabled                 : True

Now all we need to do is configure it to false (and also unset a property called ReservedInstanceCount which is a duplicate of preWarmedInstanceCount but cannot be included when we try and reset the other setting due to what I assume is Azure just keeping it around for legacy reasons):

$config.Properties.fileChangeAuditEnabled = "false"
$config.Properties.PSObject.Properties.Remove('ReservedInstanceCount')

Next, as per the suggestion from Parameswaran in the comments (thank you!), create a new Array (since existing arrays are of fixed size and cannot be modified) while removing AppServiceFileAuditLogs from the list of azureMonitorLogCategories

$newCategories = @()

ForEach ($entry in $config.Properties.azureMonitorLogCategories) {
    If ($entry -ne "AppServiceFileAuditLogs") {
        $newCategories += $entry
    }
}

$config.Properties.azureMonitorLogCategories = $newCategories

And finally, let’s set the setting:

$config | Set-AzResource -Force

Next, for any Deployment Slots you have on this resource, repeat these steps again, but using the following resource retrieval code:

$config = Get-AzResource -ResourceGroupName $appServiceRG `
    -ResourceType "Microsoft.Web/sites/slots" `
    -ResourceName "$($appServiceName)" `
    -apiVersion 2016-08-01

Now, when you try to scale down from a PremiumV2 SKU to a Basic SKU, you will no longer receive the error of “Cannot use the SKU Basic with File Change Audit for site XXX-XXX-XXX-XXX”.

As many of you have probably gathered, over the past few weeks, I’ve been working on building a process for deploying an Azure App Service from scratch, including DNS and TLS in a single Terraform module.

Today, I write this post with success in my heart, and at the bottom, I provide copies of the necessary files for your own usage.

One of the biggest hurdles I faced was trying to integrate Cloudflare’s CDN services with Azure’s Custom Domain verification. Typically, I’ll rely on the options available in the GUI as the inclusive list of “things I can do” so up until now, if we wanted to stand up a multi-region App Service, we had to do the following:

1. Build and deploy the App Service, using the azurewebsites.net hostname for HTTPS for each region (R1 and R2)
   
   e.g. example-app-eastus.azurewebsites.net (R1), example-app-westus.azurewebsites.net (R2)
   
2. Create the CNAME record for the service at Cloudflare pointing at R1, turning off proxying (orange cloud off)
   
   e.g. example-app.domain.com -> example-app-eastus.azurewebsites.net
   
3. Add the Custom Domain on R1, using the CNAME verification method
   
4. Once the hostname is verified, go back to Cloudflare and update the CNAME record for the service to point to R2
   
   e.g. example-app.domain.com -> example-app-westus.azurewebsites.net
   
5. Add the Custom Domain on R2, using the CNAME verification method
   
6. Once the hostname is verified, go back to Cloudflare and update the CNAME record for the service to point to the Traffic Manager, and also turn on proxying (orange cloud on)

While this eventually accomplishes the task, the failure mode it introduces is that if you ever want to add a third (or fourth or fifth…) region, you temporarily have to not only direct all traffic to your brand new single instance momentarily to verify the domain, but you also have to turn off proxying, exposing the fact that you are using Azure (bad OPSEC).

After doing some digging however, I came across a Microsoft document that explains that there is a way to add a TXT record which you can use to verify ownership of the domain without a bunch of messing around with the original record you’re dealing with.

This is great because we can just add new awverify records for each region and Azure will trust we own them, but Terraform introduces a new wrinkle in that it creates the record at Cloudflare so fast that Cloudflare’s infrastructure often doesn’t have time to replicate the new entry across their fleet before you attempt the verification, which means that the lookup will fail and Terraform will die.

To get around this, we added a null_resource that just executes a 30 second sleep to allow time for the record to propagate through Cloudflare’s network before attempting the lookup.

I’ve put together a copy of our Terraform modules for your perusal and usage:

• Create an ha-app-service
• ha-app-service Providers
• ha-app-service Module
• ha-app-service/variables.tf

Using this module will allow you to easily deploy all of your micro-services in a Highly Available configuration by utilizing multiple regions.

For the last two days, I’ve been trying to deploy some new microservices using a certificate stored in Key Vault in an Azure App Service. By now, you’ve probably figured out that we love them around here. I’ve also been slamming my head against the wall because of some not-well-documented functionality about granting permissions to the Key Vault.

As a quick primer, here’s the basics of what I was trying to do:

resource "azurerm_app_service" "centralus-app-service" {
   name                = "${var.service-name}-centralus-app-service-${var.environment_name}"
   location            = "${azurerm_resource_group.centralus-rg.location}"
   resource_group_name = "${azurerm_resource_group.centralus-rg.name}"
   app_service_plan_id = "${azurerm_app_service_plan.centralus-app-service-plan.id}"

   identity {
     type = "SystemAssigned"
   }
 }

data "azurerm_key_vault" "cert" {
   name                = "${var.key-vault-name}"
   resource_group_name = "${var.key-vault-rg}"
 }

resource "azurerm_key_vault_access_policy" "centralus" {
   key_vault_id = "${data.azurerm_key_vault.cert.id}"
   tenant_id = "${azurerm_app_service.centralus-app-service.identity.0.tenant_id}"
   object_id = "${azurerm_app_service.centralus-app-service.identity.0.principal_id}"
   secret_permissions = [
     "get"
   ]
   certificate_permissions = [
     "get"
   ]
 }

resource "azurerm_app_service_certificate" "centralus" {
   name                = "${local.full_service_name}-cert"
   resource_group_name = "${azurerm_resource_group.centralus-rg.name}"
   location            = "${azurerm_resource_group.centralus-rg.location}"
   key_vault_secret_id = "${var.key-vault-secret-id}"
   depends_on          = [azurerm_key_vault_access_policy.centralus]
 }

and these are the relevant values I was passing into the module:

  key-vault-secret-id       = "https://example-keyvault.vault.azure.net/secrets/cert/0d599f0ec05c3bda8c3b8a68c32a1b47"
  key-vault-rg              = "example-keyvault"
  key-vault-name            = "example-keyvault"

But no matter what I did, I kept bumping up against this error:

Error: Error creating/updating App Service Certificate "example-app-dev-cert" (Resource Group "example-app-centralus-rg-dev"): web.CertificatesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation." Details=[{"Message":"The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"59716","Message":"The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","MessageTemplate":"The service does not have access to '{0}' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","Parameters":["/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault"]}}]

I checked and re-checked and triple-checked and had colleagues check, but no matter what I did, it kept puking with this permissions issue. I confirmed that the App Service’s identity was being provided and saved, but nothing seemed to work.

Then I found this blog post from 2016 talking about a magic Service Principal (or more specifically, a Resource Principal) that requires access to the Key Vault too. All I did was add the following resource with the magic SP, and everything worked perfectly.

resource "azurerm_key_vault_access_policy" "azure-app-service" {
   key_vault_id = "${data.azurerm_key_vault.cert.id}"
   tenant_id = "${azurerm_app_service.centralus-app-service.identity.0.tenant_id}"

   # This object is the Microsoft Azure Web App Service magic SP 
   # as per https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html
   object_id = "abfa0a7c-a6b6-4736-8310-5855508787cd" 

   secret_permissions = [
     "get"
   ]

   certificate_permissions = [
     "get"
   ]
 }

It’s frustrating that Microsoft hasn’t documented this piece (at least officially), but hopefully with this knowledge, you’ll be able to automate using a certificate stored in Key Vault in your next Azure App Service.

We have several micro-services that we run as App Services within Azure. In the past few weeks, multiple times we’ve experienced problems where a single instance has decided to go crazy but not crazy enough that Azure knows to take it out of rotation. Being able to diagnose these per-instance issues is imperative when it comes to offering a functioning App Service.

The first thing we’ve done is setup Alerts to monitor each App Service not just as a whole, but also per-instance. (NOTE: These are services we have not migrated to Terraform yet, so we have created these alerts manually, just like the services were as well. The alert definitions have been built into our Terraform stack and automatically get deployed as we move these micro-services over to the new Terraform-managed stack.) To get notified of broken single instances:

1. Open the App Service in the Portal
2. Under the ‘Monitoring’ section of the blade, select ‘Alerts’
3. Select ‘New Alert Rule’
4. Under ‘Condition’, choose ‘Add’
5. Choose the ‘Http Server Errors’ signal
6. Place a check in the box next to the ‘Instance’ dimension
7. Set your Threshold settings according to your preference (for the record, we are using Dynamic / Medium / 5 minutes / Every 5 Minutes)
8. Click ‘Done’
9. Set up your Action Group accordingly
10. Save the alert

Within 10 minutes, it will begin monitoring each instance which should give you better insight not just into the health of your application, but how each of the pieces that comprise your app are operating.

This is important because you may have 10 instances running a single App Service but the failure of a single instance may not create enough failures to trip an alert, depending on your routing scheme or traffic levels.

I personally believe that more visibility is always a positive, and so being able to detect per-instance issues in your Azure App Service can result in shorter outages and ultimately happier customers.

Last week, we were attempting to deploy a new Terraform-owned resource but every time we ran terraform plan or terraform apply, we got the error Error: insufficient items for attribute "sku"; must have at least 1. We keep our Terraform code in a Azure DevOps project, with approvals being required for any new commits even into our dev environment, so we were flummoxed.

Our first thought was that we had upgraded the Terraform azurerm provider from 1.28.0 to 1.32.0 and we knew for a fact that the azurerm_key_vault resource had been changed from accepting a sku {} block to simply requiring a sku_name property. We tried every combination of having either, both, and none of them defined, and we still received the error. We even tried downgrading back to 1.28.0 as a fallback, but it made no change. At this point we were relatively confident that it wasn’t the provider.

The next thing we looked for was any other resources that had a sku {} block defined. This included our azurerm_app_service_plans, our azure_virtual_machines, and our azurerm_vpn_gateway. We searched for and commented out all of the respective declarations from our .tf files, but still we received the error.

Now we were starting to get nervous. Nothing we tried would solve the problem, and we were starting to get a backlog of requests for new resources that we couldn’t deploy because no matter what we did, whether adding or removing potentially broken code, we couldn’t deploy any new changes. To say the tension on our team was palpable would be the understatement of the year.

At this point we needed to take a step back and analyze the problem logically, so we all took a break from Terraform to clear our minds and de-stress a bit. We started to suspect something in the state file was causing the problem, but we weren’t really sure what. We decided to take the sledgehammer approach and using terraform state rm, we removed every instance of those commented out resources we found above.

This worked. Now we could run terraform plan and terraform apply without issue, but we still weren’t sure why. That didn’t bode well if the problem re-occured; we couldn’t just keep taking a sledgehammer to the environment, it’s just too disruptive. We needed to figure out the root cause.

We opened an issue on the provider’s GitHub page for further investigation, and after some digging by other community members and Terraform employees themselves, it seems that Microsoft’s API returns a different response for App Service Plans than any other resource when it is found to be missing. An assumption was being made that it would be the same for all resources, but it turned out that this was a bad assumption to make.

This turned out to be the key for us. Someone had deleted several App Service Plans from the Azure portal (thinking they were not being used) and so our assumption is that when the provider is checking for the status of a missing App Service Plan, the broken response makes Terraform think it actually exists, even though there’s no sku {} data in it, causing Terraform to think that that specific data was missing.

Knowing the core problem, the error message Error: insufficient items for attribute "sku"; must have at least 1 kind of makes sense now: the sku attribute is missing at least 1 item, it just doesn’t make clear that the “insufficient items” are on the Azure side, not the Terraform / .tf side.

They’ve added a workaround in the provider until Microsoft updates the API to respond like all of the other resources.

Have you seen this error before? What did you do to solve it?

Yesterday, it was discovered that our developers had built a Docker container that was encrypted with a password that resided in a single location: the Azure App Service’s Application Settings (aka: an environment variable). Of course we discovered this when they pushed out a deployment of the container, something broke during the deployment, the Application Setting with the password disappeared, and no one knew what the password was.

It took nearly 30 minutes to rebuild the container with a new encryption password, which is entirely too long for a core piece of our company’s booking system to be unavailable, so until we have a proper password management solution in place, we wanted to stand up something.

Azure Key Vault to the rescue!

Azure Key Vault is a tool for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. A vault is logical group of secrets.

https://docs.microsoft.com/en-ca/azure/key-vault/key-vault-whatis

We created a new Key Vault, and set up secrets for each of our App Services because as it turns out, several – although not all – of our microservices have similar encryption setups. We then dumped in the encryption passwords for each service so that if in the future the password disappears, it’s as simple as grabbing it out of the Key Vault and re-creating it as the Application Setting.

In the future, the developers will be updating their applications to reference the Key Vault directly, but for the time being and until they can change how their apps are architected.