Skip to main content

GCP how-to guides

These guides cover the tasks of running Polaris on Google Cloud, from a first deployment of your own workload through monitoring, upgrades and teardown. The GCP reference has the machine types, permissions and config keys they mention, and Polaris on Google Cloud explains why GCP works the way it does.

Run every command from the top folder of the Polaris tooling, and always from the same copy of it. That copy holds each deployment's Terraform state and tokens, which status, promote and destroy rely on.

Deploy your own workload​

Your workload fits Polaris when it follows the workload contract: one container built for linux/amd64, serving plain HTTP on one port and keeping its state outside the VM.

1. Prepare your machine and project​

The machine that runs deploy needs gcloud, docker with a running daemon, terraform, openssl, python3 and bash 4 or newer. Sign in with gcloud auth login, and give Docker read access to the Polaris images on ghcr.io/fr0ntierx, which deploy copies into your project.

In the project, enable the APIs listed in The identity that deploys and grant your account the two roles deploy checks for, as direct bindings:

gcloud projects add-iam-policy-binding <project> \
  --member=user:<you> --role=roles/iam.workloadIdentityPoolAdmin
gcloud projects add-iam-policy-binding <project> \
  --member=user:<you> --role=roles/iam.roleAdmin

Give each deployment a project of its own where you can. The deploying identity can act with every permission in the project, and the service accounts the deployment creates hold project-level roles.

2. Make the Polaris image available​

Both VMs boot from a Polaris image in a Compute Engine image family, polaris-cvm in the deployment's project unless you set compute.cvm_image_family and compute.cvm_image_project. deploy stops before it creates anything if the family has no image, and names the Packer template that builds one, infrastructure/packer/polaris-cvm.pkr.hcl. GPU deployments use the polaris-cvm-gpu family, built from polaris-cvm-gpu.pkr.hcl.

3. Choose the TEE, region and zone​

Set tee.cpu to sev-snp-direct, sev-snp-svsm or tdx, and pick a region and zone that offer the matching machine type with confidential computing. Regions, zones and machine types lists the defaults.

apiVersion: polaris/v1
kind: DeploymentConfig

metadata:
  name: <deployment name>

cloud:
  provider: gcp
  gcp:
    project_id: <project>
    region: us-central1
    zone: us-central1-a

tee:
  cpu: sev-snp-direct

features:
  attestation_mode: full
  ebpf: true
  ima: true

features.ebpf and features.ima default to false. Without features.ebpf, a full deployment collects no runtime or kernel evidence.

The deployment creates its own Cloud KMS key, <name>-key, in the region. It keeps that key when you tear the deployment down, and a later deployment with the same name gates the same key, so choose the name with that in mind.

4. Point the config at your image​

If your image is in Artifact Registry in the deployment's own project, name it:

workload:
  image: "<region>-docker.pkg.dev/<project>/<repository>/<image>@sha256:<digest>"
  port: 8080

The workload VM's service account can read Artifact Registry in the deployment's project, and the VM reaches it through Google's private API addresses. For an image in another registry, add that registry's address ranges to network.allowed_egress_cidrs, although copying the image into Artifact Registry in the project is usually simpler. To have the tooling build the image for you, put a folder with a Dockerfile inside the tooling's top folder and use workload.dir in place of workload.image.

Pin images by digest, or ship each change under a new tag. Polaris checks the running container against the exact image it deployed.

The tooling copies the Polaris images into the Artifact Registry repository polaris-docker in the region, which every deployment in the project and region shares unless you name its own:

polaris:
  docker_repo: <deployment name>

5. Allow outbound destinations​

Outbound traffic from both VMs is denied unless a rule allows it. Google APIs and Artifact Registry are reachable through Google's private API addresses. Everything else goes over HTTPS on port 443 to the address ranges you list:

network:
  allowed_egress_cidrs:
    - "198.51.100.0/24"

The list takes address ranges and applies to the Policy Manager as well as the workload VM. A service behind a CDN whose addresses change works best when it publishes its ranges. Ports other than 443 are not available.

The runtime layer also learns where the workload connects, and under enforcement a connection to an address it has not seen counts as unknown behavior. If your workload connects to destinations that change often, keep network checks advisory while the rest of the runtime layer enforces. Move a layer to enforce shows how.

6. Set configuration​

Pass settings as environment variables:

workload:
  env:
    LOG_LEVEL: info

The values become part of the workload VM's instance metadata, which anyone who can view the instance in the project can read. Keep secrets out of workload.env.

7. Size the VMs​

Set compute.cvm_machine_type for your workload. The Policy Manager is light, and its default n2d-standard-2 suits most deployments. It must stay an AMD SEV-SNP machine type.

Set workload.resources.memory_mb and workload.resources.cpus as well, so the workload runs within limits and leaves room for Polaris's own components.

8. Deploy and verify​

./scripts/deploy.sh deploy --config <file>

deploy runs Terraform, waits up to 15 minutes for the workload container and up to 20 for the Policy Manager to answer, then reads the Policy Manager's measurement. On a first deploy it installs that value on the workload VM as the pin and prints the config to pin it before boot from then on. It prints the endpoint at the end, and status prints it again later:

./scripts/deploy.sh status --config <file>

The workload VM's address is reserved for the deployment, so it stays the same when the VM is replaced.

A deployment made with deploy also sends one request through the proxy to your workload every 60 seconds, from the workload VM itself, so that Polaris learns a working workload rather than an idle one. Let learning finish before you move a layer to enforce, and check the deployment the way a client will, as described in Verifying a Deployment.

Use the gated key from your workload​

Not available on GCP yet. The proxy serves the key routes on AWS and on-prem only, and on GCP the workload's container cannot reach the metadata server, so it has no credential to call Cloud KMS with. The Policy Manager still adds and removes the workload VM's service account on the key as the verdict changes, and you can watch that in the key's IAM policy.

Pin the Policy Manager's measurement​

Pinning makes the workload VM refuse any Policy Manager whose launch measurement or privilege level differs from the one you recorded, from the workload VM's first boot. Without a pin, the workload VM trusts the first Policy Manager that answers until deploy installs the value it captured.

The measurement covers Google's firmware and the Policy Manager's vCPU shape, not the software the Policy Manager runs. A pin therefore holds the workload VM to an SEV-SNP VM of that shape on that firmware, and every such VM in any project matches it. See What the hardware report proves.

  1. Deploy without pm_measurement_hex. deploy reads the Policy Manager's report over IAP, writes it to pm-snp-baseline-<name>.yaml beside the Terraform state, installs it on the workload VM and restarts the proxy. It then prints the lines to add to your config.

  2. Add them under cloud.gcp:

    cloud:
      gcp:
        pm_measurement_hex: "<96 hex characters>"
        pm_required_vmpl: 0

  3. Deploy again. The workload VM boots with the pin already in place, and deploy skips the install. It reads the Policy Manager's report back and confirms it matches the pin. If it does not match, the run fails and prints both values.

The measurement changes when Google rolls out new firmware or when you change compute.pm_machine_type, and the workload VM then refuses its Policy Manager, so no check passes and the workload has no access to the key. To capture the new value, remove pm_measurement_hex, deploy, and start again from step 1.

The pin leaves out the field that names the physical chip, so the Policy Manager still matches after host maintenance moves it.

deploy reads the reported value as plain JSON from the Policy Manager's own report, without checking AMD's signature on that copy. The proxy checks the signature on every connection. Compare a new value with the previous one, and find out why it changed before you copy it into the pin.

For sev-snp-svsm, you can also pin the SVSM firmware the workload VM runs, with tee.svsm_measurement, the firmware's SHA-384 as 96 hex characters. Unset, the platform check still requires a genuine SEV-SNP report from VMPL 2, without checking which firmware booted. The value differs by platform, and a wrong one fails the platform check and withdraws access to your key, as does a firmware update by Google until you change the value. So set it only once you know the value for your platform.

Move a layer to enforce​

A layer in advise reports findings and leaves your key alone, while a layer in enforce withdraws access to your key when it fails. The platform check always enforces.

  1. Wait until the deployment has learned the workload:

    ./scripts/deploy.sh status --config <file>

    Overall and every layer should show pass, with Alerts at 0, and the autolearn section should show no learner still active. status prints one snapshot, so run it again until they do.

    The Policy Manager also refuses to enforce a layer until it has been clean under advise for the promotion window, a day by default, so that the VM's hourly and daily jobs have run at least once. To change the window, pass --promotion-window, such as 12h or 0, to deploy.

  2. Move one or more layers:

    ./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3

    The layers are l2 (image), l3 (runtime), l31 (kernel) and ima_boot (boot-time measurements). The runtime layer also has three parts that you can set on their own, l3_syscalls, l3_binaries and l3_network. promote prints the settings the deployment has before it changes anything, refuses to enforce a layer whose allowlist is still empty, and reads the settings back to confirm they took.

    The GCP deployment leaves ima_boot unset, and an unset ima_boot follows l2. Name it whenever you name l2, so that promote checks its allowlist.

To keep network checks advisory while the rest of the runtime layer enforces:

./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3_syscalls,l3_binaries
./scripts/deploy.sh promote --config <file> --layers l3_network --mode advise

To move a layer back, run promote with --mode advise.

promote and status reach the Policy Manager over IAP, running their requests on the Policy Manager VM, so they touch nothing on the workload VM.

Every deploy resets the settings

The Policy Manager keeps what it learned, and the settings promote makes, on its state disk, so they survive a restart of its container or VM. They last only while the deployment's baseline file stays the same, and on GCP every deploy writes the baseline under a new name that carries the time of the run. So every deploy returns every layer to advise and starts learning again. Run promote again once learning has finished.

Under l3 enforce, commands run in an SSH session on the workload VM can count as unknown behavior and withdraw access to your key, so plan hands-on maintenance for a time when a short withdrawal is acceptable.

Monitor a deployment​

The proxy serves three status routes on the deployment's endpoint. They stay reachable while the proxy is holding workload traffic, so your monitoring can tell why traffic stopped.

RouteReturns
/polaris-container/health{"status":"ok"} while the proxy runs
/polaris-container/readyWhether the proxy is serving workload traffic, and the state of its gate, with 503 when not ready
/polaris-container/attestation-statusThe latest verdict, with a status for each layer and when it was checked. With the read-only token it adds the reason for a failure and the alerts

Read them together like this:

  • An enforcing layer failed. ready is false, and attestation-status shows fail. Ask it again with the read-only token to see the reason.

  • The proxy has just started. On GCP the proxy holds workload traffic from the moment it starts until the Policy Manager's first verdict that is not a failure, with no grace period.

  • The Policy Manager has stopped answering. The proxy holds traffic after five reports in a row go unanswered. attestation-status still shows the last verdict, so alert when its check time is more than a few report intervals old.

  • A layer in advise found something. attestation-status shows alert, while traffic flows, because only layers you enforce hold traffic.

  • Polaris is still learning. /management/profiles/learn, /management/ima/learn and /management/kernel/learn, read with the read-only token, show each learner's progress. watch reads the same routes:

    ./scripts/deploy.sh watch --endpoint https://<address> \
      --token "$(cat terraform/modules/gcp/terraform-<name>.readonly.token)"

Any route your workload serves, including a metrics endpoint, sits behind the same check as the rest of your traffic, so scrape your workload's metrics with the status routes alongside.

Watch the Policy Manager VM itself as well. GCP has no separate component that ends the workload's key access when the Policy Manager stops, so a stopped or hung Policy Manager leaves the workload VM's service account on the key until it comes back. The Policy Manager can export its own health, such as verdict counts, latencies and key gate events, over OTLP. Pass --otel-endpoint to deploy, with --otel-egress-cidrs for a collector outside the deployment's network. ./scripts/deploy.sh --help lists the other telemetry flags.

Diagnose a withdrawal of access​

When requests start returning 503, the Policy Manager's records say why, and you can read them through the endpoint with the read-only token.

  1. Keep the token at hand:

    TOKEN="$(cat terraform/modules/gcp/terraform-<name>.readonly.token)"

  2. Find the workload VM's ID in the Policy Manager's status list, in the cvm_id field:

    curl -sk -H "Authorization: Bearer $TOKEN" https://<address>/management/status

  3. Read the latest incident as a plain-text report:

    curl -sk -H "Authorization: Bearer $TOKEN" \
      https://<address>/management/incidents/<cvm_id>/report

  4. For more context, /management/checks/<cvm_id> lists the latest checks one by one, and /management/history/<cvm_id> returns recent verdicts.

On GCP a failure also locks the workload VM's service account out of the key. The lock lifts by itself when the VM that failed passes again. If it does not lift, for example because the workload VM was replaced, list the locks and lift one with the management token:

curl -sk -H "Authorization: Bearer $TOKEN" https://<address>/management/kms/locks
curl -sk -X DELETE \
  -H "Authorization: Bearer $(cat terraform/modules/gcp/terraform-<name>.token)" \
  https://<address>/management/kms/locks/<service account>

Once you know the cause, fix it in the workload, or learn the new behavior and move the layer back to enforce.

Read the workload's logs​

Container logs stay on the workload VM, and you read them over SSH through IAP:

gcloud compute ssh <name>-cvm --zone <zone> --project <project> \
  --tunnel-through-iap --command 'sudo docker logs --tail 200 client-workload'

The containers also log to the system journal, which the VM copies to its serial console, so you can read recent output without SSH:

gcloud compute instances get-serial-port-output <name>-cvm --zone <zone> --project <project>

Under l3 enforce an SSH session on the workload VM can withdraw access to your key, so keep log reading for a time when that is acceptable, or keep the runtime layer advisory while you investigate. The same commands with <name>-pm read the Policy Manager's logs, from its polaris-policy-manager and polaris-proxy containers.

Change a deployment​

Edit the config and run deploy again. Plan every run as a short outage and a fresh round of learning:

  • Each run gives the deployment's baseline a new name, which is part of the Policy Manager's boot configuration, so each run replaces the Policy Manager VM. The new VM attaches the same state disk, so it keeps its keys and its record of the workload VMs. The baseline changed, though, so it drops what it learned and every promote setting, and layers return to advise.
  • Both VMs are replaced when a newer image is in the family they boot from. Each run looks up the family's newest image and boots from that.
  • The workload VM is replaced when its own boot configuration changes, such as a new workload image, new environment variables or a new port. A sev-snp-svsm workload VM is deleted and created again on every run.
  • The workload VM keeps its reserved public address, so clients need no change.
  • deploy reads the new Policy Manager's measurement. With a pin, it checks the value against the pin. Without one, it installs the value on the workload VM.

Run promote again once learning has finished. To move a deployment to another region or zone, tear it down and deploy it again.

Take a security fix​

The Polaris image keeps the packages it was built with, and automatic package updates are switched off on the workload VM. A security fix arrives as a new image.

  1. Publish the new Polaris image to the image family the deployment boots from.
  2. Run deploy. It finds the newer image in the family and replaces both VMs so that they boot from it. --force-replace replaces both VMs even when the family has no newer image.
  3. Polaris learns the new VMs from the start. Run promote again once learning has finished.

You decide when to rebuild, and a deployment left alone keeps the packages its image was built with, so set a schedule that fits your patching policy.

Upgrade Polaris​

A Polaris release brings a new version of the tooling, whose manifest names the exact Polaris component images to run. Upgrade by switching to the new tooling and running deploy, which copies the new images into your registry by digest. A new image changes the boot configuration of each VM that runs it, so deploy replaces those VMs, the Policy Manager among them, and Polaris learns again.

To roll back, deploy again with the previous tooling.

Prepare for production​

  • Use a dedicated project for each deployment, with a short list of people who can change IAM in it or act as its service accounts.
  • Choose metadata.name with care, since it names the Cloud KMS key the deployment gates and keeps.
  • Set features.ebpf: true and features.ima: true for full attestation.
  • Pin your workload image's digest.
  • Enforce at least l2 and l3 once learning has finished, with l3_network advisory if your workload's destinations change often. Name ima_boot along with l2.
  • Pin the Policy Manager's measurement in cloud.gcp.pm_measurement_hex, as in Pin the Policy Manager's measurement, and follow Google's firmware notices, since a firmware change moves it.
  • Remember that every deploy returns every layer to advise, and run promote again afterwards.
  • Monitor the Policy Manager VM, since nothing outside it ends the workload's key access when it stops.
  • Keep allowed_egress_cidrs to the ranges your workload needs.
  • Treat the tooling folder as a secret. It holds the deployment's Terraform state, its .tfvars file and its tokens, and the state and .tfvars file carry the tokens in plain text, so keep it encrypted, backed up and access-controlled. Limit who can view the instances' metadata too, since the tokens are in it.
  • Monitor the status routes from Monitor a deployment, and keep the diagnosis steps in your on-call runbook.

Troubleshoot​

The tooling says it needs bash 4 or newer. macOS ships bash 3.2. Install a newer one, for example with brew install bash, and run again.

deploy says the deployer lacks required roles. Grant roles/iam.workloadIdentityPoolAdmin and roles/iam.roleAdmin to your own account directly on the project, as in step 1. The check looks for those exact roles on your account, so a broader role or a group membership does not count.

deploy says no CVM base image was found in the family. The image family named by compute.cvm_image_family, polaris-cvm by default, has no image in the project named by compute.cvm_image_project, the deployment's project by default. Build or copy one there, as in step 2.

deploy stops at failed to pull released image ghcr.io/.... Docker on your machine cannot read the Polaris images on ghcr.io. Log in and run deploy again.

deploy reports unknown config keys. A key is misspelled or does not exist, and the tooling stops so that nothing is silently ignored. Check the key against the GCP reference.

Terraform rejects the deployment name. metadata.name must start with a lowercase letter and have 2 to 25 lowercase letters, digits and hyphens.

Terraform refuses a GPU deployment's machine type. compute.cvm_machine_type names a size other than a3-highgpu-1g. Set that size or leave the field empty, and deploy picks it. GPU deployments also need tee.cpu: tdx and --local-build.

deploy warns that it is using a non-production CVM image family. compute.cvm_image_family names a family other than polaris-cvm. Use the production family for anything you hand to others.

deploy waits a long time for the Policy Manager, then stops. The Policy Manager has no external address and pulls its images through Cloud NAT, retrying while it waits. The error prints two commands, one to list the containers on the VM and one to read its serial console, which shows each pull attempt.

deploy warns that the workload container is not healthy. The container is crash-looping, and deploy prints its last log lines. A crash-looping workload also keeps learning from finishing.

The workload never starts. An image outside Artifact Registry in the deployment's project needs its registry in network.allowed_egress_cidrs, or the pull waits at boot.

Requests return 503 with attestation pending. The proxy is waiting for its first passing verdict or holding traffic after a failure. Read the status routes and the latest incident, as in Diagnose a withdrawal of access.

Access to the key does not come back after the workload passes again. A failure locked the workload VM's service account. Lift the lock as in Diagnose a withdrawal of access.

status or promote cannot reach the Policy Manager. Check that you can SSH to <name>-pm through IAP, and run the command from the copy of the tooling that deployed, since it needs that copy's token.

promote refuses a layer. Either that layer's allowlist is still empty, so learning is still running, or the layer has not yet been clean under advise for the promotion window, a day by default. promote prints the reasons. Wait and try again.

Layers are back in advise. A deploy ran, and each one writes the baseline under a new name, which drops what the Policy Manager learned and every promote setting. A restart alone keeps them. Run promote again once learning has finished.

deploy says the pinned measurement is not the one the Policy Manager reports. Google rolled out new firmware, or the Policy Manager's machine type changed. See Pin the Policy Manager's measurement.

The Policy Manager does not start after you changed attestation.trust_domain or attestation.deployment_id. Its state disk holds a record written under the old values, and it refuses to start against a record with another deployment ID. Set them back.

destroy says there is nothing to destroy. It found no Terraform state. Run it from the copy of the tooling that deployed.

Tear down a deployment​

./scripts/deploy.sh destroy --config <file>

destroy removes the VMs and their disks, the Policy Manager's state disk <name>-pm-state with what it learned, its keys and its locks, the network and its firewall rules, the DNS zones, the service accounts, the custom role, the federation pool and the reserved addresses. It deletes the management and read-only token files.

These remain:

  • The Cloud KMS key ring and key. A later deployment with the same name uses them again.
  • The IAM bindings the Policy Manager added to the key for the workload VM's service account. destroy does not remove them.
  • The Artifact Registry repository. Delete it only if it carries the deployment's own name, since the default polaris-docker may hold other deployments' images.
  • The tooling folder's other files for the deployment, listed in Files the tooling keeps. Run status before deleting them, not after, because status writes new token files.

After destroy, check the key's IAM policy and remove what is left from this deployment:

gcloud kms keys get-iam-policy <name>-key \
  --keyring <name>-keyring --location <region> --project <project>
gcloud kms keys remove-iam-policy-binding <name>-key \
  --keyring <name>-keyring --location <region> --project <project> \
  --member <member> --role roles/cloudkms.cryptoKeyEncrypterDecrypter
gcloud kms keys remove-iam-policy-binding <name>-key \
  --keyring <name>-keyring --location <region> --project <project> \
  --member <member> --role roles/cloudkms.publicKeyViewer

Use the member exactly as get-iam-policy prints it for the workload VM's service account, <name>-cvm.

Tear down with destroy rather than with Terraform directly. destroy deletes a sev-snp-svsm workload VM, which Terraform does not manage, and keeps the key out of Terraform's reach.