Skip to main content

Azure how-to guides

These guides cover the tasks of running Polaris on Azure, from a first deployment of your own workload through monitoring, upgrades and teardown. The Azure reference has the sizes, permissions and config keys they mention.

Run every command from the top folder of the Polaris tooling, and always from the same copy of it. That copy holds each deployment's Terraform state, tokens and pinned host keys, which status, promote, watch, destroy and the SSH commands below rely on.

Some tasks that exist on AWS are not available on Azure yet. Each section says so where that is the case.

Deploy your own workload​

Your workload fits Polaris when it follows the workload contract: one container built for linux/amd64, serving plain HTTP on one port and keeping its state outside the VM.

1. Sign in and choose a subscription​

Sign in with the Azure CLI. deploy stops with Azure CLI not authenticated. Run: az login otherwise, and it switches the CLI to cloud.azure.subscription_id for the rest of the run.

az login

The identity needs the rights in The identity that deploys, including creating role assignments.

The machine that runs deploy also needs Docker running, Terraform 1.6 or newer, openssl, and an SSH key pair. The public key defaults to ~/.ssh/id_rsa.pub.

2. Choose a region and sizes​

Pick a region that offers the size you want. For SEV-SNP, the defaults are Standard_DC2as_v6 for both VMs, and Standard_EC2as_v6 gives the workload more memory. For Intel TDX, set tee.cpu: tdx and the defaults become Standard_DC2es_v6.

cloud:
  provider: azure
  azure:
    subscription_id: "<subscription id>"
    location: eastus

tee:
  cpu: sev-snp

compute:
  cvm_machine_type: Standard_DC4as_v6

Keep metadata.name to 12 characters or fewer, since the Key Vault name is built from it.

3. Point the config at your image​

The VMs log in to the deployment's own container registry, so the simplest choice is to let the tooling build your image and push it there:

workload:
  dir: my-workload
  port: 8080

workload.dir is a folder with a Dockerfile, relative to the top folder of the tooling. To use an image you built yourself, put it in the deployment's registry and name it with workload.image, pinned by digest. An image in any other registry has to be public, since the VMs log in to no other.

Polaris checks the running container against the exact image it deployed, so ship each change under a new tag or digest.

4. Allow inbound clients and outbound destinations​

By default the endpoint on port 443 admits only the public address of the machine that ran deploy. Name your clients:

network:
  workload_ingress_cidrs:
    - "203.0.113.0/24"

The module refuses 0.0.0.0/0 and any other /0 or /1 range, so name your clients or the range of a CDN in front of the endpoint.

Outbound, both VMs can reach any internet address on ports 443 and 80. For other ports, list the ranges, which then open on every port:

network:
  allowed_egress_cidrs:
    - "198.51.100.10/32"

The runtime layer also learns where the workload connects, and under enforcement a connection to an address it has not seen counts as unknown behavior.

5. Turn on runtime and kernel evidence​

The Sentinel agent collects the runtime and kernel evidence, and on Azure it runs only when features.ebpf is true. The tooling's default is false.

features:
  ebpf: true

6. Set configuration​

Pass settings as environment variables:

workload:
  env:
    LOG_LEVEL: info

The values are written into the VM's custom data. The module keeps its own tokens out of custom data for that reason, so keep secrets out of workload.env too. Polaris has no route yet for the workload to decrypt a secret with the gated key on Azure, as the next section explains.

Set workload.resources.memory_mb and workload.resources.cpus as well, so the workload runs within limits and leaves room for Polaris's own components.

7. Open SSH for yourself​

cloud:
  azure:
    operator_ssh_cidrs:
      - "203.0.113.4/32"

With it set, deploy reads each VM's SSH host keys from its boot diagnostics console and pins them, then captures the Policy Manager's identity record through the workload VM. Without it, deploy skips both and warns that Polaris Desktop has no record to check the Policy Manager against. status and promote reach the Policy Manager through the same SSH route, so they need it too.

8. Deploy and verify​

./scripts/deploy.sh deploy --config <file>

The VMs read their tokens from Key Vault at boot and retry for up to 10 minutes while their new role assignments take effect, so a first deploy can take several minutes after Terraform finishes. Then check the deployment as in Verifying a Deployment, and watch learning with watch.

Use the gated key from your workload​

This is not available on Azure yet.

The Policy Manager does gate the workload key. While the workload VM passes, the VM's managed identity holds Key Vault Crypto User on the key, and the Policy Manager removes it when a layer you enforce fails. But nothing on the VM offers the workload a way to use it:

  • The proxy serves the key routes /polaris-container/key/encrypt and /polaris-container/key/decrypt only for HashiCorp Vault and AWS KMS. The Azure deployment passes the proxy the vault's URL and the key's name, and the proxy does not read them yet.
  • The workload container cannot reach the instance metadata service, so it cannot get a token for the VM's identity. Only the proxy is let through.

Until a key route lands for Azure, do not plan on decrypting data inside the workload with the deployment's key.

Move a layer to enforce​

A layer in advise reports findings and leaves key access alone, while a layer in enforce removes the key's role assignment when it fails. The baseline the tooling writes for Azure sets l2, l3 and l31 to advise, and the platform check always enforces.

promote reaches the Policy Manager over SSH through the workload VM, with the host keys deploy pinned for both VMs. Before you start, check that cloud.azure.operator_ssh_cidrs includes your address and that terraform/deploy/azure/terraform-<name>.known_hosts exists. A deploy with operator_ssh_cidrs set writes it. Without the pinned keys, promote stops rather than trust whatever host answers.

  1. Wait until the deployment has learned the workload:

    ./scripts/deploy.sh status --config <file>

    Overall and every layer should show pass, with Alerts at 0, and the autolearn section should show no learner still active. status prints one snapshot, so run it again until they do.

    The Policy Manager also refuses to enforce a layer until it has been clean under advise for the promotion window, a day by default, so that the VM's hourly and daily jobs have run at least once. To change the window, pass --promotion-window, such as 12h or 0, to deploy.

  2. Move one or more layers:

    ./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3

    The layers are l2 (image), l3 (runtime), l31 (kernel) and ima_boot (boot-time measurements). The runtime layer also has three parts that you can set on their own, l3_syscalls, l3_binaries and l3_network. promote prints the settings the deployment has before it changes anything, refuses to enforce a layer whose allowlist is still empty, and reads the settings back to confirm they took.

    The Azure deployment leaves ima_boot unset, and an unset ima_boot follows l2. Name it whenever you name l2, so that promote checks its allowlist.

To keep network checks advisory while the rest of the runtime layer enforces:

./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3_syscalls,l3_binaries
./scripts/deploy.sh promote --config <file> --layers l3_network --mode advise

To move a layer back, run promote with --mode advise.

The Policy Manager keeps these settings, and everything it learned, on its state disk, so they survive a restart and a replacement of the Policy Manager VM. A deploy that changes the deployment's baseline, such as a new workload image digest or a new VM image, returns the settings to advise and starts learning again, and you run promote again once the layers are ready.

Under l3 enforce, commands run in an SSH session on a VM can count as unknown behavior and remove key access, so plan hands-on maintenance for a time when a short withdrawal is acceptable.

The Marketplace template asks for an enforcement mode at deploy time instead. enforce there makes a failed image check or a live kernel change remove key access, while the runtime layer stays in advise.

Monitor a deployment​

The proxy serves three status routes on the deployment's endpoint. They stay reachable while the proxy is holding workload traffic.

RouteReturns
/polaris-container/health{"status":"ok"} while the proxy runs
/polaris-container/readyready and gate, with 503 when not ready
/polaris-container/attestation-statusThe latest verdict, with status, a status for each layer and checked_at. With the read-only token it adds the failure reason and alerts

Read them together like this:

  • The proxy has just started. On Azure the proxy has no start-up grace period, so it holds workload traffic from the moment it starts until the first passing verdict. attestation-status returns 503 with no appraisal yet until then.
  • A layer in advise found something. attestation-status shows alert, and traffic still flows.
  • The Policy Manager has stopped answering. attestation-status keeps showing the last verdict, with a checked_at that keeps getting older. Alert when it is more than a few report intervals old.
  • Your workload is down. The status routes look healthy, and requests to your workload fail.

status lists the VMs and the endpoint, and reads each workload VM's verdict, the license state and how far learning has got from the Policy Manager itself, over the same SSH route promote uses:

./scripts/deploy.sh status --config <file>

Without operator_ssh_cidrs and the pinned host keys, status lists the VMs and says why it did not read the Policy Manager.

For learning progress through the endpoint, with no SSH, run watch with the read-only token:

./scripts/deploy.sh watch --endpoint https://<address> \
  --token "$(cat terraform/modules/azure/terraform-<name>.readonly.token)"

Azure has no equivalent of the AWS grant retirer. If the Policy Manager stops or hangs, the workload VM's role assignment on the key stays until the Policy Manager runs again. Watch checked_at for that reason.

To send the Policy Manager's health metrics to your own collector, pass --otel-endpoint to deploy, and --otel-egress-cidrs when the collector is outside the virtual network.

Diagnose a withdrawal of access​

When requests start returning 503 with attestation pending, the Policy Manager's records say why, and you can read them through the endpoint with the read-only token.

  1. Keep the token at hand:

    TOKEN="$(cat terraform/modules/azure/terraform-<name>.readonly.token)"

  2. Find the workload VM's ID in the cvm_id field:

    curl -sk -H "Authorization: Bearer $TOKEN" https://<address>/management/status

  3. Read the latest incident as a plain-text report:

    curl -sk -H "Authorization: Bearer $TOKEN" \
      https://<address>/management/incidents/<cvm_id>/report

  4. For more context, /management/checks/<cvm_id> lists the latest checks, and /management/history/<cvm_id> returns recent verdicts.

On the Azure side, the Policy Manager's grant is a role assignment of Key Vault Crypto User on the workload key, described as polaris:cvm=<cvm_id>. Listing the role assignments on the key shows whether it is in place.

Read the workload's logs​

Container logs stay on the workload VM, and you read them over SSH. The pinned host keys are filed under the VM's vmUniqueId, so pass it as the host key alias:

cd terraform/deploy/azure
terraform output -state=terraform-<name>.tfstate -json cvm_vm_unique_ids
cd ../../..

ssh -i <private key> \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=terraform/deploy/azure/terraform-<name>.known_hosts \
  -o HostKeyAlias=<vmUniqueId> \
  azureuser@<address> 'sudo docker logs --tail 200 workload'

The workload container is named workload. The proxy is polaris-proxy and the Sentinel polaris-sentinel. SSH needs your address in cloud.azure.operator_ssh_cidrs. Commands you run over SSH can show up as runtime findings.

Change a deployment​

Edit the config and run deploy again.

  • A change to the workload VM's configuration, such as a new workload image or new environment variables, replaces it. The new VM has a new managed identity, which the Policy Manager's own configuration names, so the Policy Manager is replaced too.
  • The workload VM keeps its static public address across a replacement, since the address is a separate resource.
  • The replacement Policy Manager attaches the same state disk. It keeps what it learned and your promote settings unless the run changed the deployment's baseline, which names the workload image's digest, the workload port and the VM image. A new digest or a new VM image therefore returns every layer to advise and starts learning again.
  • Unless you set compute.cvm_image_id, the workload VM boots the version of Canonical's Ubuntu confidential VM image that latest names when deploy runs. When Canonical publishes a new version, the next deploy replaces the workload VM, and with it the Policy Manager, even if you changed nothing, and the new image in the baseline starts learning again.
  • A run that changes nothing, at a time Canonical has published nothing new, replaces nothing. --force-replace replaces the workload VM anyway, and with it the Policy Manager.

set-workload changes the workload image without a deploy. It needs the endpoint, the management token or the workload-swap token from Files the tooling keeps, and an image the workload VM can pull:

./scripts/deploy.sh set-workload --endpoint https://<address> \
  --token "$(cat terraform/modules/azure/terraform-<name>.token)" \
  --image <registry>/<repository>@sha256:<digest>

The Policy Manager replaces the image allowlist and resets its learner, and the workload VM pulls the image and restarts the container within about 30 seconds. Update workload.image in the config as well, so the next deploy names the same image.

To move a deployment to another region, tear it down and deploy it again with the new cloud.azure.location.

Take a security fix​

The VMs boot Canonical's Ubuntu 22.04 confidential VM image, and install Docker and its dependencies from the Ubuntu archive at first boot. A VM picks up newer packages when it is recreated. The workload VM also switches off automatic package updates at boot, since an update would change the files Polaris measures.

  • The workload VM boots the image version latest names when deploy runs, unless you set compute.cvm_image_id. When Canonical publishes a new version, a plain deploy replaces the workload VM, and the Policy Manager with it.
  • The Policy Manager VM boots whatever latest names when it is created, and a new version alone does not replace it.

To recreate both, whether or not a new image is out:

./scripts/deploy.sh deploy --config <file> --force-replace

This is a short outage. A new VM image changes the baseline, so Polaris learns the new VMs from the start, and you run promote again once learning has finished.

Upgrade Polaris​

A Polaris release brings a new version of the tooling, whose manifest names the exact Polaris images to run. Switch to the new tooling and run deploy. The new image references change both VMs' boot configuration, so both are replaced and both move to the new versions together. The Policy Manager's state disk carries over, so it keeps what it learned and your promote settings unless the run also changed the baseline.

To roll back, deploy again with the previous tooling.

Prepare for production​

  • Give the deployment a subscription of its own, and keep the list of people who can create role assignments in it short. Anyone who can change role assignments on the key, the vault or anything above them can grant access to the key, or widen the Policy Manager's role. The Policy Manager itself can assign Key Vault Crypto User on the key only to this deployment's workload VM identities.
  • Know that the deploying identity keeps Key Vault Crypto Officer on the vault after deploy, which lets it use the key.
  • Set features.ebpf: true for runtime and kernel evidence.
  • Pin your workload image by digest.
  • Set network.workload_ingress_cidrs to your clients, and keep allowed_egress_cidrs to what your workload needs.
  • Keep operator_ssh_cidrs as narrow as you can, and set it before you rely on the Policy Manager identity record, status or promote, since all three go over SSH.
  • Enforce at least l2 and l3 with promote once learning has finished, with l3_network advisory if your workload's destinations change often. Name ima_boot along with l2.
  • Treat the tooling folder as a secret. The state, the tfvars file and the token files carry the deployment's tokens, and the tfvars file the registry password.
  • Monitor checked_at on attestation-status, since a stopped Policy Manager leaves the key's role assignment in place.
  • Plan for the key's lifetime. The deployment creates the key, and destroy deletes it along with the vault.
  • When Microsoft rolls a new paravisor signing authority, deploy stops and prints the new digest, when it can reach the Policy Manager over SSH. Append it to cloud.azure.id_key_digests once you have checked the rollout.

Troubleshoot​

deploy says Azure requires metadata.name to be at most 12 characters. The Key Vault name is built from it. Shorten the name.

deploy says Azure CLI not authenticated or Cannot access subscription. Run az login with an identity that can see cloud.azure.subscription_id.

deploy says the SSH public key was not found. Create a key pair, or point cloud.azure.ssh_public_key_file at an existing .pub file.

deploy reports unknown config keys. A key is misspelled or does not exist for this tooling. Check it against the Azure reference.

Terraform says the size does not match the TEE. tee.cpu: sev-snp needs an AMD size, such as Standard_DC2as_v6, and tee.cpu: tdx an Intel one, such as Standard_DC2es_v6, for both VMs.

deploy says tee.gpu nvidia-h100 requires --local-build. The release manifest pins no GPU build of the proxy. A GPU deployment also needs compute.cvm_image_id.

Terraform cannot write the key or the secrets. The vault's firewall admits the public address api.ipify.org reports for the machine that runs deploy. Run from a machine whose outbound address is stable and is the one Key Vault sees.

You cannot reach the endpoint. With network.workload_ingress_cidrs empty, the endpoint admits only the address that ran deploy. Set your clients' ranges and run deploy again.

The VMs take a long time to start. Each VM retries reading its tokens from Key Vault for up to 10 minutes while its new role assignment takes effect.

deploy says the PM's paravisor is signed by an ID key deploy does not accept. Microsoft may have rolled a new paravisor signing authority. Append the printed digest to cloud.azure.id_key_digests once you have checked the rollout, and run deploy again.

deploy warns there is no SSH route to the Policy Manager. cloud.azure.operator_ssh_cidrs is empty or the private key file is missing. The deployment works, while the host keys and the Policy Manager identity record are skipped.

deploy says the console never printed an SSH host key block. Cloud-init had not finished on that VM within 5 minutes. Run deploy again, or read the VM's boot log in the portal.

promote or status says there are no pinned host keys. deploy pins both VMs' SSH host keys only when cloud.azure.operator_ssh_cidrs is set. Add your address and run deploy again.

promote cannot reach the Policy Manager. cloud.azure.operator_ssh_cidrs no longer includes your address, or the private key is missing. Update it and run deploy, and run promote from the copy of the tooling that deployed, since it needs that copy's state, token and pinned host keys.

promote refuses a layer. Either that layer's allowlist is still empty, so learning is still running, or the layer has not yet been clean under advise for the promotion window, a day by default. promote prints the reasons. Wait and try again.

Requests return 503 with attestation pending. The proxy is waiting for a verdict or holding traffic after a failure. Read the status routes and the latest incident, as in Diagnose a withdrawal of access.

watch cannot reach the Policy Manager. It may still be starting. Otherwise check that you are using the workload VM's endpoint and a token for this deployment.

Learned allowlists are gone and layers are back in advise. A deploy changed the deployment's baseline, for example with a new workload image digest or a new VM image from Canonical. A restart or replacement of the Policy Manager alone keeps them. Run promote again once learning has finished.

The Policy Manager does not start after you changed attestation.trust_domain or attestation.deployment_id. Its state disk holds a record written under the old values, and it refuses to start against a record with another deployment ID. Set them back.

destroy says there is no state file. Run it from the copy of the tooling that deployed.

Tear down a deployment​

./scripts/deploy.sh destroy --config <file>

destroy runs terraform destroy, which removes the resource group and everything in it: the VMs and their disks, the Policy Manager's state disk with what it learned, the network, the custom role, the role assignments Terraform made, and the Key Vault with the workload key and the secrets. It then deletes the token files. The role assignment the Policy Manager made on the key is not in Terraform's state.

The key goes with the deployment

On Azure the deployment creates the workload key, and destroy deletes it. Azure keeps the deleted vault and key in a soft-deleted state for the retention window, 90 days by default, and purge protection stops anyone purging them sooner. Data encrypted under the key can be decrypted again only if the vault is recovered within that window.

The container registry and its resource group remain, since other deployments in the subscription may share them. So do the tooling folder's state, tfvars, known hosts and identity record files, listed in Files the tooling keeps. Delete them once you no longer need them, since the state and tfvars carry the deployment's tokens.

Tear down with destroy rather than by deleting the resource group in the portal, so the tooling's state stays in step.