The API-driven run workflow

HCP Terraform has three workflows for managing Terraform runs.

The UI/VCS-driven run workflow, which is the primary mode of operation.
The API-driven run workflow described below, which is more flexible but requires you to create some tooling.
The CLI-driven run workflow, which uses Terraform's standard CLI tools to execute runs in HCP Terraform.

Summary

In the API-driven workflow, workspaces are not directly associated with a VCS repo, and runs are not driven by webhooks on your VCS provider.

Instead, one of your organization's other tools is in charge of deciding when configuration has changed and a run should occur. Usually this is something like a CI system, or something else capable of monitoring changes to your Terraform code and performing actions in response.

Once your other tooling has decided a run should occur, it must make a series of calls to HCP Terraform's runs and configuration-versions APIs to upload configuration files and perform a run with them. For the exact series of API calls, see the pushing a new configuration version section.

The most significant difference in this workflow is that HCP Terraform does not fetch configuration files from version control. Instead, your own tooling must upload the configurations as a .tar.gz file. This allows you to work with configurations from unsupported version control systems, automatically generate Terraform configurations from some other source of data, or build a variety of other integrations.

Important: The script below is provided to illustrate the run process, and is not intended for production use. If you want to drive HCP Terraform runs from the command line, please see the CLI-driven run workflow.

Pushing a New Configuration Version

Pushing a new configuration to an existing workspace is a multi-step process. This section walks through each step in detail, using an example bash script to illustrate.

You need queue plans permission to create new configuration versions for the workspace. Refer to the permissions documentation for more details.

1. Define Variables

To perform an upload, a few user parameters must be set:

path_to_content_directory is the folder with the terraform configuration. There must be at least one .tf file in the root of this path.
organization is the organization name (not ID) for your HCP Terraform organization.
workspace is the workspace name (not ID) in the HCP Terraform organization.
$TOKEN is the API Token used for authenticating with the HCP Terraform API.

This script extracts the path_to_content_directory, organization, and workspace from command line arguments, and expects the $TOKEN as an environment variable.

#!/bin/bash

if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Usage: $0 <path_to_content_directory> <organization>/<workspace>"
  exit 0
fi

CONTENT_DIRECTORY="$1"
ORG_NAME="$(cut -d'/' -f1 <<<"$2")"
WORKSPACE_NAME="$(cut -d'/' -f2 <<<"$2")"

2. Create the File for Upload

The configuration version API requires a tar.gz file to use the configuration version for a run, so you must package the directory containing the Terraform configuration into a tar.gz file.

Important: The configuration directory must be the root of the tar file, with no intermediate directories. In other words, when the tar file is extracted the result must be paths like ./main.tf rather than ./terraform-appserver/main.tf.

UPLOAD_FILE_NAME="./content-$(date +%s).tar.gz"
tar -zcvf "$UPLOAD_FILE_NAME" -C "$CONTENT_DIRECTORY" .

3. Look Up the Workspace ID

The first step identified the organization name and the workspace name; however, the configuration version API expects the workspace ID. As such, the ID has to be looked up. If the workspace ID is already known, this step can be skipped. This step uses the jq tool to parse the JSON output and extract the ID value into the WORKSPACE_ID variable.

WORKSPACE_ID=($(curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  https://app.terraform.io/api/v2/organizations/$ORG_NAME/workspaces/$WORKSPACE_NAME \
  | jq -r '.data.id'))

4. Create a New Configuration Version

Before uploading the configuration files, you must create a configuration-version to associate uploaded content with the workspace. This API call performs two tasks: it creates the new configuration version and it extracts the upload URL to be used in the next step.

echo '{"data":{"type":"configuration-versions"}}' > ./create_config_version.json

UPLOAD_URL=($(curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data @create_config_version.json \
  https://app.terraform.io/api/v2/workspaces/$WORKSPACE_ID/configuration-versions \
  | jq -r '.data.attributes."upload-url"'))

5. Upload the Configuration Content File

Next, upload the configuration version tar.gz file to the upload URL extracted from the previous step. If a file is not uploaded, the configuration version will not be usable, since it will have no Terraform configuration files.

HCP Terraform automatically creates a new run with a plan once the new file is uploaded. If the workspace is configured to auto-apply, it will also apply if the plan succeeds; otherwise, an apply can be triggered via the Run Apply API. If the API token used for the upload lacks permission to apply runs for the workspace, the run can't be auto-applied. (More about permissions.)

curl \
  --header "Content-Type: application/octet-stream" \
  --request PUT \
  --data-binary @"$UPLOAD_FILE_NAME" \
  $UPLOAD_URL

6. Delete Temporary Files

In the previous steps a few files were created; they are no longer needed, so they should be deleted.

rm "$UPLOAD_FILE_NAME"
rm ./create_config_version.json

Complete Script

Combine all of the code blocks into a single file, ./terraform-enterprise-push.sh and give execution permission to create a combined bash script to perform all of the operations.

chmod +x ./terraform-enterprise-push.sh
./terraform-enterprise-push.sh ./content my-organization/my-workspace

Note: This script does not have error handling, so for a more robust script consider adding error checking.

./terraform-enterprise-push.sh:

#!/bin/bash

# Complete script for API-driven runs.

# 1. Define Variables

if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Usage: $0 <path_to_content_directory> <organization>/<workspace>"
  exit 0
fi

CONTENT_DIRECTORY="$1"
ORG_NAME="$(cut -d'/' -f1 <<<"$2")"
WORKSPACE_NAME="$(cut -d'/' -f2 <<<"$2")"

# 2. Create the File for Upload

UPLOAD_FILE_NAME="./content-$(date +%s).tar.gz"
tar -zcvf "$UPLOAD_FILE_NAME" -C "$CONTENT_DIRECTORY" .

# 3. Look Up the Workspace ID

WORKSPACE_ID=($(curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  https://app.terraform.io/api/v2/organizations/$ORG_NAME/workspaces/$WORKSPACE_NAME \
  | jq -r '.data.id'))

# 4. Create a New Configuration Version

echo '{"data":{"type":"configuration-versions"}}' > ./create_config_version.json

UPLOAD_URL=($(curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data @create_config_version.json \
  https://app.terraform.io/api/v2/workspaces/$WORKSPACE_ID/configuration-versions \
  | jq -r '.data.attributes."upload-url"'))

# 5. Upload the Configuration Content File

curl \
  --header "Content-Type: application/octet-stream" \
  --request PUT \
  --data-binary @"$UPLOAD_FILE_NAME" \
  $UPLOAD_URL

# 6. Delete Temporary Files

rm "$UPLOAD_FILE_NAME"
rm ./create_config_version.json

Advanced Use Cases

For advanced use cases refer to the Terraform Enterprise Automation Script repository for automating interactions with HCP Terraform, including the creation of a workspace, uploading code, setting variables, and triggering the plan and apply operations.

In addition to uploading configurations and starting runs, you can use HCP Terraform's APIs to create and modify workspaces, edit variable values, and more. See the API documentation for more details.