I just recently started a new job and the codebase that I inherited is…not great. One of the main problems is that it was written for Terraform 0.11 and was never fully upgraded to 0.12 and the goodies that HCL2 and 0.13 brought with them. Of course this means that I needed to find all of the old-style interpolations in our Terraform code base and replace them with the new format. Luckily with VS Code and a little regex, this is a super easy task. This regex is PCRE so you can also use it with sed/awk too if you prefer the command line.

1. Open the Find and Replace panel
2. Next to the Find box, click the box that looks like: .*
3. Search for: "\${([a-zA-Z0-9_.]+)}" (you can visit https://regex101.com/r/K8E5xc/2 for an explanation of how it works along with some examples)
4. In the Replace box, enter: $1
5. Now when you actually run the search and replace, it’ll pull the interpolated data out of the old-style interpolation by removing the opening "${ and closing }"

Hopefully this helps some of you who might not be as comfortable with regex clean up your code base and find those old-style interpolations in your Terraform code, since I believe 0.14 will start throwing actual errors and not just warnings on these issues.

A lot of the time when I’m having to debug Terraform, I find that I want to quickly look at what Terraform may be storing inside a variable so that I can understand where I’ve made a mistake passing data around. After literal years of using the old TF_LOG=debug method – which can get way too unwieldy with any decently sized environment, I realized I could write a simple module to dump out what I was looking for. Hopefully this will help you too.

resource "null_resource" "terraform-debug" {
  provisioner "local-exec" {
    command = "echo $VARIABLE1 >> debug.txt ; echo $VARIABLE2 >> debug.txt"

    environment = {
        VARIABLE1 = jsonencode(var.your_variable_name)
        VARIABLE2 = jsonencode(local.piece_of_data)
    }
  }
}

Just replace var.your_variable_name_goes_here with your actual variable reference and then run your terraform apply. A file should be spit out in the same directory that the .tf file that you inserted this resource lives in, which you can then open up and understand what might be going on.

I don’t know why it took me so long to figure out something so simple, but coming from PHP/Laravel – where you can just dd() or dump() the contents of a variable to the console – the idea of using the local-exec provisioner to debug terraform just never occurred to me.

EDIT [Jan 6, 2020]: Wrapped variable data in jsonencode() to avoid issues with echoing an empty string to a file. Also show how to dump multiple variables to the same file.

As many of you have probably gathered, over the past few weeks, I’ve been working on building a process for deploying an Azure App Service from scratch, including DNS and TLS in a single Terraform module.

Today, I write this post with success in my heart, and at the bottom, I provide copies of the necessary files for your own usage.

One of the biggest hurdles I faced was trying to integrate Cloudflare’s CDN services with Azure’s Custom Domain verification. Typically, I’ll rely on the options available in the GUI as the inclusive list of “things I can do” so up until now, if we wanted to stand up a multi-region App Service, we had to do the following:

1. Build and deploy the App Service, using the azurewebsites.net hostname for HTTPS for each region (R1 and R2)
   
   e.g. example-app-eastus.azurewebsites.net (R1), example-app-westus.azurewebsites.net (R2)
   
2. Create the CNAME record for the service at Cloudflare pointing at R1, turning off proxying (orange cloud off)
   
   e.g. example-app.domain.com -> example-app-eastus.azurewebsites.net
   
3. Add the Custom Domain on R1, using the CNAME verification method
   
4. Once the hostname is verified, go back to Cloudflare and update the CNAME record for the service to point to R2
   
   e.g. example-app.domain.com -> example-app-westus.azurewebsites.net
   
5. Add the Custom Domain on R2, using the CNAME verification method
   
6. Once the hostname is verified, go back to Cloudflare and update the CNAME record for the service to point to the Traffic Manager, and also turn on proxying (orange cloud on)

While this eventually accomplishes the task, the failure mode it introduces is that if you ever want to add a third (or fourth or fifth…) region, you temporarily have to not only direct all traffic to your brand new single instance momentarily to verify the domain, but you also have to turn off proxying, exposing the fact that you are using Azure (bad OPSEC).

After doing some digging however, I came across a Microsoft document that explains that there is a way to add a TXT record which you can use to verify ownership of the domain without a bunch of messing around with the original record you’re dealing with.

This is great because we can just add new awverify records for each region and Azure will trust we own them, but Terraform introduces a new wrinkle in that it creates the record at Cloudflare so fast that Cloudflare’s infrastructure often doesn’t have time to replicate the new entry across their fleet before you attempt the verification, which means that the lookup will fail and Terraform will die.

To get around this, we added a null_resource that just executes a 30 second sleep to allow time for the record to propagate through Cloudflare’s network before attempting the lookup.

I’ve put together a copy of our Terraform modules for your perusal and usage:

• Create an ha-app-service
• ha-app-service Providers
• ha-app-service Module
• ha-app-service/variables.tf

Using this module will allow you to easily deploy all of your micro-services in a Highly Available configuration by utilizing multiple regions.

For the last two days, I’ve been trying to deploy some new microservices using a certificate stored in Key Vault in an Azure App Service. By now, you’ve probably figured out that we love them around here. I’ve also been slamming my head against the wall because of some not-well-documented functionality about granting permissions to the Key Vault.

As a quick primer, here’s the basics of what I was trying to do:

resource "azurerm_app_service" "centralus-app-service" {
   name                = "${var.service-name}-centralus-app-service-${var.environment_name}"
   location            = "${azurerm_resource_group.centralus-rg.location}"
   resource_group_name = "${azurerm_resource_group.centralus-rg.name}"
   app_service_plan_id = "${azurerm_app_service_plan.centralus-app-service-plan.id}"

   identity {
     type = "SystemAssigned"
   }
 }

data "azurerm_key_vault" "cert" {
   name                = "${var.key-vault-name}"
   resource_group_name = "${var.key-vault-rg}"
 }

resource "azurerm_key_vault_access_policy" "centralus" {
   key_vault_id = "${data.azurerm_key_vault.cert.id}"
   tenant_id = "${azurerm_app_service.centralus-app-service.identity.0.tenant_id}"
   object_id = "${azurerm_app_service.centralus-app-service.identity.0.principal_id}"
   secret_permissions = [
     "get"
   ]
   certificate_permissions = [
     "get"
   ]
 }

resource "azurerm_app_service_certificate" "centralus" {
   name                = "${local.full_service_name}-cert"
   resource_group_name = "${azurerm_resource_group.centralus-rg.name}"
   location            = "${azurerm_resource_group.centralus-rg.location}"
   key_vault_secret_id = "${var.key-vault-secret-id}"
   depends_on          = [azurerm_key_vault_access_policy.centralus]
 }

and these are the relevant values I was passing into the module:

  key-vault-secret-id       = "https://example-keyvault.vault.azure.net/secrets/cert/0d599f0ec05c3bda8c3b8a68c32a1b47"
  key-vault-rg              = "example-keyvault"
  key-vault-name            = "example-keyvault"

But no matter what I did, I kept bumping up against this error:

Error: Error creating/updating App Service Certificate "example-app-dev-cert" (Resource Group "example-app-centralus-rg-dev"): web.CertificatesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation." Details=[{"Message":"The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"59716","Message":"The service does not have access to '/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","MessageTemplate":"The service does not have access to '{0}' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.","Parameters":["/subscriptions/[SUBSCRIPTIONID]/resourcegroups/example-keyvault/providers/microsoft.keyvault/vaults/example-keyvault"]}}]

I checked and re-checked and triple-checked and had colleagues check, but no matter what I did, it kept puking with this permissions issue. I confirmed that the App Service’s identity was being provided and saved, but nothing seemed to work.

Then I found this blog post from 2016 talking about a magic Service Principal (or more specifically, a Resource Principal) that requires access to the Key Vault too. All I did was add the following resource with the magic SP, and everything worked perfectly.

resource "azurerm_key_vault_access_policy" "azure-app-service" {
   key_vault_id = "${data.azurerm_key_vault.cert.id}"
   tenant_id = "${azurerm_app_service.centralus-app-service.identity.0.tenant_id}"

   # This object is the Microsoft Azure Web App Service magic SP 
   # as per https://azure.github.io/AppService/2016/05/24/Deploying-Azure-Web-App-Certificate-through-Key-Vault.html
   object_id = "abfa0a7c-a6b6-4736-8310-5855508787cd" 

   secret_permissions = [
     "get"
   ]

   certificate_permissions = [
     "get"
   ]
 }

It’s frustrating that Microsoft hasn’t documented this piece (at least officially), but hopefully with this knowledge, you’ll be able to automate using a certificate stored in Key Vault in your next Azure App Service.

You may find yourself in a position where a resource already exists in your cloud environment but was created in the respective provider’s GUI rather than in Terraform. You may feel a bit overwhelmed at first, but there are a few ways to generate Terraform files for existing resources, and we’re going to talk about the various ways today. This is also not an exhaustive list; if you have any other suggestions, please leave a comment and I’ll be sure to update this post.

Method 1 – Manual

Be warned, the manual method takes a little more time, but is not restricted to certain resource types. I prefer this method because it means that you’ll be able to see every setting that is already set on your resource with your own two eyes, which is good for sanity checking.

First, you’re going to want to create a .tf file with just the outline of the resource type you’re trying to import or generate.

For example, if I wanted to create the Terraform for a resource group called example-resource-group that had several tags attached to it, I would do:

resource "azurerm_resource_group" "example-resource-group" {
}

and then save it.

Next, I would go to the Azure GUI, find and open the resource group, and then open the ‘Properties’ section from the blade.

I would look for the Resource ID, for example /subscriptions/54ba8d50-7332-4f23-88fe-f88221f75bb3/resourceGroups/example-resource-group and copy it.

I would then open up a command prompt / terminal and import the state by running: terraform import azurerm_resource_group.example-resource-group /subscriptions/54ba8d50-7332-4f23-88fe-f88221f75bb3/resourceGroups/example-resource-group

Finally, and this is the crucial part, I would immediately run terraform plan. There may be required fields that you will need to fill out before this comamnd works, but in general, this will compare the existing state that you just imported to the blank resource in the .tf file, and show you all of the differences which you can then copy into your new Terraform file, and be confident that you have imported all of the settings.

Example:

# azurerm_resource_group.example-resource-group will be updated in-place
   ~ resource "azurerm_resource_group" "example-resource-group" {
         id       = "/subscriptions/54ba8d50-7332-4f23-88fe-f88221f75bb3/resourceGroups/example-resource-group"
         location = "centralus"
         name     = "example-resource-group"
       ~ tags     = {
           ~ "environment" = "dev" -> null
           ~ "owner"       = "example.person" -> null
           ~ "product"     = "internal" -> null
         }
     }

A shortcut I’ve found is to just copy the entire resource section, and then replace all of the tildes (~) with spaces, and then find and remove all instances of -> null.

Method 2 – Az2tf (Azure only)

Andy Thomas (Microsoft employee) put together a tool called Az2tf which iterates over your entire subscription, and generates .tf files for most of the common types of resources, and he’s adding more all the time. Requesting a specific resource type is as simple as opening an issue and explaining which resource is missing. In my experience, he’s responded within a few hours with a solution.

Method 3 – Terraforming (AWS only)

Daisuke Fujita put together a tool called Terraforming that with a little bit of scripting can generate Terraform files for all of your AWS resources.

Method 4 – cf-terraforming (Cloudflare only)

Cloudflare put together a fantastic tool called cf-terraforming which rips through your Cloudflare tenant and generates .tf files for everything Cloudflare related. The great thing about cf-terraforming is that because it’s written by the vendor of the original product, they treat it as a first class citizen and keep it very up-to-date with any new resources they themselves add to their product. I wish all vendors would do this.

To sum things up, there are plenty of ways to generate Terraform files for existing resources. Some are more time consuming than others, but they all have the goal of making your environment less brittle and your processes more repeatable, which will save time, money, and most importantly stress, when an inevitable incident takes place.

Do you know of any other tools for these or other providers that can assist in bringing previously unmanaged resources under Terraform management? Leave a comment and we’ll add them to this page as soon as possible!

Last week, we were attempting to deploy a new Terraform-owned resource but every time we ran terraform plan or terraform apply, we got the error Error: insufficient items for attribute "sku"; must have at least 1. We keep our Terraform code in a Azure DevOps project, with approvals being required for any new commits even into our dev environment, so we were flummoxed.

Our first thought was that we had upgraded the Terraform azurerm provider from 1.28.0 to 1.32.0 and we knew for a fact that the azurerm_key_vault resource had been changed from accepting a sku {} block to simply requiring a sku_name property. We tried every combination of having either, both, and none of them defined, and we still received the error. We even tried downgrading back to 1.28.0 as a fallback, but it made no change. At this point we were relatively confident that it wasn’t the provider.

The next thing we looked for was any other resources that had a sku {} block defined. This included our azurerm_app_service_plans, our azure_virtual_machines, and our azurerm_vpn_gateway. We searched for and commented out all of the respective declarations from our .tf files, but still we received the error.

Now we were starting to get nervous. Nothing we tried would solve the problem, and we were starting to get a backlog of requests for new resources that we couldn’t deploy because no matter what we did, whether adding or removing potentially broken code, we couldn’t deploy any new changes. To say the tension on our team was palpable would be the understatement of the year.

At this point we needed to take a step back and analyze the problem logically, so we all took a break from Terraform to clear our minds and de-stress a bit. We started to suspect something in the state file was causing the problem, but we weren’t really sure what. We decided to take the sledgehammer approach and using terraform state rm, we removed every instance of those commented out resources we found above.

This worked. Now we could run terraform plan and terraform apply without issue, but we still weren’t sure why. That didn’t bode well if the problem re-occured; we couldn’t just keep taking a sledgehammer to the environment, it’s just too disruptive. We needed to figure out the root cause.

We opened an issue on the provider’s GitHub page for further investigation, and after some digging by other community members and Terraform employees themselves, it seems that Microsoft’s API returns a different response for App Service Plans than any other resource when it is found to be missing. An assumption was being made that it would be the same for all resources, but it turned out that this was a bad assumption to make.

This turned out to be the key for us. Someone had deleted several App Service Plans from the Azure portal (thinking they were not being used) and so our assumption is that when the provider is checking for the status of a missing App Service Plan, the broken response makes Terraform think it actually exists, even though there’s no sku {} data in it, causing Terraform to think that that specific data was missing.

Knowing the core problem, the error message Error: insufficient items for attribute "sku"; must have at least 1 kind of makes sense now: the sku attribute is missing at least 1 item, it just doesn’t make clear that the “insufficient items” are on the Azure side, not the Terraform / .tf side.

They’ve added a workaround in the provider until Microsoft updates the API to respond like all of the other resources.

Have you seen this error before? What did you do to solve it?

Step 1 – Find a cat facts API

https://catfact.ninja/

Well that was easy.

Step 2 – Build a serverless, Azure Logic App using Terraform that will connect to the API and spam your co-workers with a new fact every 5 minutes

https://github.com/nexxai/cat-facts/

Ok that part was easy too, but come on, it’s gotta be at least a little difficu–

Step 3 – Create an Office 365 connection that your Logic App can use

Open the Azure Logic Apps blade

You have 60 seconds to manually add a step that connects your Office 365 account to this app. ‘Get Calendars’ requires the least configuration.

Step 4 – Wait for your co-workers’ email clients to play their New Email alert sound

Start laughing, and keep laughing every 5 minutes from now until forever, asserting your feline dominance over your team.

“But that was only 4 steps, where’s number fi

Step 5 – Have Senior PM of Microsoft Azure Functions see your stupid app and tweet about it

Sure, no prob–wait, what?

EDIT: Updated on July 10, 2019; modified second- and third-last paragraphs to show the correct process of retrieving the AWS_SECRET_ACCESS_KEY from the Key Vault and setting it as a protected environment variable

Our primary cloud is in Azure which makes building DevOps pipelines with automation scoped to a particular subscription very easy, but what happens when we want to deploy something in AWS, since storing keys in source control is A Very Bad Idea?

Simple, we use Azure Key Vault.

First, we created a Key Vault specifically for this purpose called company-terraform which will specifically be used to store the various secrets for Terraform-based deployments. When you tie a subscription from Azure DevOps to an Azure subscription, it creates an “application” in the Azure Enterprise Applications list, so give that application Get and List permissions to this vault.

Next, we created a secret called AmazonAPISecretKey and then set the secret’s content to the actual API key you are presented when you enable programmatic access to an account in the AWS IAM console.

In our Azure DevOps Terraform build and release pipelines, we then added an Azure Key Vault step, selecting the appropriate subscription and Key Vault. Once selected, we added a Secrets filter AmazonAPISecretKey meaning that it will only ever fetch that secret on run; if you will be adding multiple secrets which will all be used in this particular pipeline, add them to this filter list.

Finally, we can now use the string $(AmazonAPISecretKey) in any shellexec or other pipeline task to authenticate against AWS, while never having to commit the actual key to a viewable source.

Since one of the methods the Terraform AWS provider can use to authenticate is by using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, we will set them up so that DevOps can use them in its various tasks.

First, open your Build or Release pipeline and select the Variables tab. Create a new variable called AWS_ACCESS_KEY_ID and set the value to your access key ID (usually something like AK49FKF4034F42DZV2VRMD). Then create a second variable called AWS_SECRET_ACCESS_KEY which you can leave blank, but click the padlock icon next to it, to tell DevOps that its contents are secret and shouldn’t be shared.

Now create a shellexec task and add the following command to it, which will set the AWS_SECRET_ACCESS_KEY environment variable to the contents of the Key Vault entry we created earlier:

echo "##vso[task.setvariable variable=AWS_SECRET_ACCESS_KEY;]$(AmazonAPISecretKey)"

And there you have it! You can now reference your AWS accounts from within your Terraform structure without ever actually exposing your keys to prying eyes!

In the first part of this series, we built a Terraform system that uses a single set of files to maintain 3 separate environments, the next step is to automate as much of this process as possible.

To do that, we’re going to leverage Azure DevOps to create a build pipeline for each environment, which will kick off when we merge code into the respective environment’s git branch.

Step 1: Create a git repo to store the .tf files

First you’ll need to sign into Azure DevOps. Once signed in, you’ll want to either select an existing organization, or create a new one, and then select an existing project, or create a new one.

Once you’ve got a project open, the + symbol at the top of the left-hand panel and select ‘New repository’. Create a repository using the naming convention of your company (or if you don’t have a particular convention, I prefer Terraform.DevOps.IT-CompanyName)

Finally, if you have existing .tf files that already exist, create a branch called dev (git checkout -b dev), and then commit and push the files up to the repo.

Step 2: Use Azure Blob Storage as the backend provider to store your state files

Since the alternative is hosting a state file on a single location (like your laptop), which may die/be stolen/other bad things, we want to tell Terraform to keep its state files in the cloud for easy and shared access.

Open up the Azure portal and browse to the Storage Accounts section. Create a new Storage Account and use the default options. Unless your Terraform environment is absolutely massive (e.g. greater than 1 gigabyte of .tf files), this account will use next to no storage, certainly less than the threshold to be charged $0.01 per month. Once the the Storage Account is created, you’ll want to add the appropriate backend type to your Terraform library as defined here, and then run terraform init which will initialize the Storage Account to host your state file.

Step 3: Begin creating the Build pipeline

Now that we have our state stored in Azure Blob Storage, we can begin working on the actual pipeline. This is where the automation actually happens and you’ll see how much easier it makes things.

1. Go back to Azure DevOps, under the Pipelines section on the left-hand panel, select ‘Builds’, and then click the ‘New pipeline’ button.
2. When the window opens, choose the Classic editor link near the bottom.
3. Select the correct project, repository, and branch (dev)
4. Choose to start with an Empty job

Now that you’ve got a skeleton of a build, here’s where we actually start adding the steps to create a fully functional Terraform setup.

Next to ‘Agent job 1’, click the + sign. In the search field, enter Terraform, select the entry called ‘Terraform Build & Release Tasks’ by Charles Zipp, and then select ‘Get it free’. Follow the prompts and install it on DevOps (it doesn’t cost anything). Once it’s installed, just refresh the build page and click the + sign and search for Terraform again.

For each pipeline we build, we’ll first need to install the Terraform tools, so add a ‘Terraform Installer’ task. Select the new task and ensure that the version of Terraform being installed is exactly the same as the version you’re using on your local machine, so edit the ‘Version’ field and replace it with the correct value.

Add another task, so search for Terraform again, but this time select ‘Terraform CLI’. It will probably default to ‘terraform validate’ so select it so that we can update a few things. Since the first thing we need to do is initialize the environment by downloading all of the necessary providers, change the Display Name to something more descriptive like terraform init and change the Command dropdown to ‘init’. Leave everything else the same.

Add another Terraform CLI task, this time for the validation step. This will make sure that before we try to do anything with our .tf files that they’re valid and the syntax is correct. Since validate is the default option for the Terraform CLI task, we’re done with this one.

Add another task, but this time, since the Terraform add-on doesn’t support the workspace command (as of writing), we need to add a shellexec task, so search for that string and add it. Change the Display Name and Code to both read terraform workspace select dev.

You guessed it, add another Terraform CLI task for the plan. Update the Display Name to read terraform plan and then select plan from the Command dropdown. Here’s where we get to define the environment, so in the ‘Command Options’ text field, enter -var-file=environments/dev.tfvars -out=TFPlan -input=False This tells Terraform to use the variables that you’ve specified for the DEV environment, write the output to a file called TFPlan and don’t expect any input.

Almost there.

Add another task, this time searching for copy and selecting Microsoft’s Copy Files task. For the Source Folder, enter $(Build.SourcesDirectory) and for the Target Folder, enter $(Build.ArtifactStagingDirectory)

For the final task, search for publish and select Microsoft’s Publish Build Artifacts. The defaults here are fine.

All you have to do to finish up is click the dropdown arrow next to ‘Save and queue’ and select ‘Save’, accept the defaults, and click ‘Save’ again.

You now have a working DevOps Build Pipeline!

In the next part, we’ll take the artifacts published from this Build pipeline and use them to create a Release pipeline that will actually tell Terraform to make your changes.

We are a fairly small company (~350 employees) and a very small cloud team (myself and one other guy), so making use of automation where it’s cheap or free is imperative if we don’t want to get overwhelmed with the amount of work being thrown our way. One major challenge we faced was that for compliance reasons, we needed to have separate environments for development, QA, and production, but at the same time minimize the amount of time it takes to promote successful projects from that same development environment to QA, and then eventually from QA to prod.

This is the path we took.

First, we created Terraform workspaces for each one:

$ terraform workspace create dev
$ terraform workspace create qa
$ terraform workspace create prod

Next, we created Azure subscriptions for each environment in PowerShell:

Install-Module Az.Subscription -AllowPrerelease
Get-AzEnrollmentAccount
New-AzSubscription -OfferType MS-AZR-0148P -Name "IT.TechServices.DEV" -EnrollmentAccountObjectId <enrollmentAccountObjectId>
New-AzSubscription -OfferType MS-AZR-0148P -Name "IT.TechServices.QA" -EnrollmentAccountObjectId <enrollmentAccountObjectId>
New-AzSubscription -OfferType MS-AZR-0017P -Name "IT.TechServices.PRD" -EnrollmentAccountObjectId <enrollmentAccountObjectId>

Note: The OfferType property is different between the DEV/QA subscriptions and the PRD subscription. This is because as part of our Enterprise Agreement with Microsoft, we have access to separate Dev/Test subscription pricing on the condition that we don’t run any production workloads in it. I am not sure if these values are universal, so if they don’t work for you, please check with your MS rep for the correct ones.

Then we created a sub-folder called environments, and in that folder we created Terraform variable files for each respective environment (dev.tfvars, qa.tfvars, and prod.tfvars), containing the appropriate Azure subscription ID in a variable called subscription_id and the name of the environment in a variable called environment_name. We then created a main.tf file with the contents variable subscription_id {} and variable environment_name {}. So, for example, in dev.tfvars we would have subscription_id = "109b6c11-e163-477e-8453-7613249447c" and environment_name = "dev" and in qa.tfvars we would have subscription_id = "95958666-8ab6-3980-828a-23f7382b9c5a" and environment_name = "qa"

/terraform    
 |    
 +--+ /environments    
 |         |    
 |         +--+ dev.tfvars
 |         +--+ qa.tfvars    
 |         +--+ prod.tfvars    
 | 
 +--+ main.tf

OK, let’s say we wanted to create some service called example-service and which required a resource group to start placing components in. If we wanted to do that in the Azure Canada Central region with some descriptive tags for sorting and billing, we would do the following:

resource "azurerm_resource_group" "example-service" {
   name     = "example-service-${var.environment_name}"
   location = "canadacentral"
   tags = {
     environment = "${var.environment_name}"
     owner       = "justin.smith"
     product     = "example-product"
     department  = "tech.services"
   }
 }

We’re using the ${var.environment_name} placeholder which means that we only have to create a single .tf file for each resource, and it will be named according to the environment we specify.

Finally, we have created a git repo for our entire Terraform collection, and have created 3 branches within it: dev, qa, and prod, with each one having successively more restrictions on committing to it than the previous. We’ve also setup Azure DevOps build and release pipelines which are triggered each time code is committed to the respective branch.

For example, anyone on our team can deploy changes to the dev branch because we don’t really care what happens in it. With the qa branch, it requires at least one of either myself or my co-worker to approve the commit. We want to make sure that people aren’t just adding unnecessary resources to QA, but it’s still not “live” so restrictions are relaxed somewhat. The prod branch requires the change ticket number from our ticketing system to be present in the Pull Request before being approved, ensuring that it has gone through the appropriate Change Approval Process before it becomes part of our daily operational management routine.

Now we’re ready to deploy! (Please note that the steps below simply replicate our pipelines in a manual way and will work for this demo. It is considered best practice to automate these steps once you’re comfortable with the process.)

First, we ensure we’re in the dev workspace:

$ terraform workspace select dev

Then, to test to make sure that we’re not complete idiots, we run the Terraform plan, including the appropriate environment’s variable file:

$ terraform plan -var-file=environments/dev.tfvars

And assuming nothing is screaming, we apply it:

$ terraform apply -var-file=environments/dev.tfvars

We now have a resource group in our Azure DEV subscription that we can use to deploy resources into, and named example-service-dev!

Now let’s say we perform the various development tasks like creating the other resources via Terraform and we’ve applied them using the same method as above (using the ${var.environment_name} placeholder at the end of each resource name), and we’re happy with how things look in the development environment. All we have to do is switch to the qa workspace and plan/apply it, using the QA variable file:

$ terraform workspace select qa
$ terraform plan -var-file=environments/qa.tfvars
$ terraform apply -var-file=environments/qa.tfvars

Now you’ve got example-service-qa all set up and ready to go.

And finally, once your QA team has validated and tested the service, you just run it one more time, this time using the prod environment settings:

$ terraform workspace select prod
$ terraform plan -var-file=environments/prod.tfvars
$ terraform apply -var-file=environments/prod.tfvars

And assuming that between each promotion (dev-to-qa, and qa-to-prod), the Terraform files were committed and promoted correctly, you should now have 3 fully functional environments (example-service-dev, example-service-qa, and example-service-prod) with only one set of Terraform files!

In the next part, we’ll walk through building an Azure DevOps Build pipeline to begin automating the deployment so that we don’t need to be so hands-on every time a Terraform change is made!