Skip to main content

AWS how-to guides

These guides cover the tasks of running Polaris on AWS, from a first deployment of your own workload through monitoring, upgrades and teardown. They assume you have done the tutorial or know the tooling, and the AWS reference has the regions, permissions and config keys they mention.

Run every command from the top folder of the Polaris tooling, and always from the same copy of it. That copy holds each deployment's Terraform state, tokens and pinned host keys, which promote, watch and destroy rely on.

Deploy your own workload​

Your workload fits Polaris when it follows the workload contract: one container built for linux/amd64, serving plain HTTP on one port and keeping its state outside the VM.

1. Choose a region​

On shared tenancy, use US East (Ohio) us-east-2 or Europe (Ireland) eu-west-1, the two regions where AWS offers SEV-SNP to VMs that share servers. For any other commercial region, such as US West (Oregon) us-west-2, run the deployment on a Dedicated Host as described in Run on a Dedicated Host, which is in preview.

If your organization limits regions with a service control policy, ask your AWS Organizations administrator to allow EC2 in the region you choose. Region limits usually exempt IAM and KMS, so a blocked region tends to show up first when the instances launch.

2. Create the KMS key​

Give each deployment its own key in the deployment's region:

aws kms create-key --region <region> \
  --description 'Polaris key for <deployment name>' \
  --query KeyMetadata.Arn --output text
aws kms create-alias --region <region> \
  --alias-name alias/polaris-<deployment name> --target-key-id <key id>

The key must be customer-managed, enabled, symmetric, for ENCRYPT_DECRYPT, and in the deployment's region, and the config names it by its full ARN. deploy checks each of these and names any that is wrong. Keep the key for as long as your data lives, since your data is encrypted under it and it outlives every deployment.

Each grant the Policy Manager makes is limited to its deployment's encryption context, so deployments can share a key without reading each other's data. A key of its own still keeps a deployment's key policy and CloudTrail apart. To limit the grants the Policy Manager can make, add the statements in Your KMS key's policy. Most can be added before the first deploy.

3. Point the config at your image​

If your image is in ECR in the deployment's own account and region, name it:

workload:
  image: "<account>.dkr.ecr.<region>.amazonaws.com/<repository>@sha256:<digest>"
  port: 8080

The workload VM pulls it through a private endpoint in the VPC. For an image in another registry, add that registry's address ranges to network.allowed_egress_cidrs, although copying the image into ECR in the region is usually simpler. To have the tooling build the image for you, put a folder with a Dockerfile inside the tooling's top folder and use workload.dir in place of workload.image.

Pin images by digest, or ship each change under a new tag. Polaris checks the running container against the exact image it deployed.

The tooling copies the Polaris images into ECR repositories under polaris-docker/, which every deployment in the account and region shares unless you name its own. In an account that runs more than one deployment, set the prefix to the deployment's name, so that its registries can be removed without touching anyone else's:

polaris:
  docker_repo: <deployment name>

4. Allow outbound destinations​

The workload reaches ECR, S3, KMS, Secrets Manager and STS through endpoints in the VPC, and reaches everything else, other AWS services included, over HTTPS on port 443 to the address ranges you list:

network:
  allowed_egress_cidrs:
    - "198.51.100.0/24"

The list takes address ranges, so a service behind a CDN whose addresses change works best when it publishes its ranges or when you reach it through an endpoint in your VPC. The deployment builds its own VPC, 10.0.0.0/16, and connections to other VPCs or to ports other than 443 are not available yet.

The runtime layer also learns where the workload connects, and under enforcement a connection to an address it has not seen counts as unknown behavior. If your workload connects to destinations that change often, such as webhooks to customer URLs, keep network checks advisory while the rest of the runtime layer enforces. Move a layer to enforce shows how.

5. Set configuration​

Pass settings as environment variables:

workload:
  env:
    LOG_LEVEL: info

The values become part of the VM's launch configuration, which anyone who can describe the instance can read, and which processes on the VM can read from the instance metadata service. Keep secrets out of workload.env.

For a secret, encrypt it with the deployment's gated key and keep the ciphertext, for example in your own storage or in workload.env. Your workload decrypts it at run time, as Use the gated key from your workload shows, and can do so only while the deployment passes.

6. Size the VMs​

Set compute.cvm_machine_type for your workload, choosing from the sizes in Instance sizes. The Policy Manager is light, and its default m6a.large suits most deployments.

Set workload.resources.memory_mb and workload.resources.cpus as well, so the workload runs within limits and leaves room for Polaris's own components.

7. Deploy and verify​

./scripts/deploy.sh deploy --config <file>

Then check the deployment the way a client will, as in step 8 of the tutorial, and let learning finish before you move a layer to enforce. On AWS that SDK check does not pass yet, for the reason step 8 gives.

Use the gated key from your workload​

Your workload never holds an AWS credential. It asks the Polaris proxy in its own VM, on the proxy's internal listener, which only the workload's network reaches. The proxy calls KMS under the workload VM's role, which only the grant lets use your key, so the calls fail once the Policy Manager retires the grant.

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/encrypt \
  -H 'Content-Type: application/json' -d '{"plaintext":"aGVsbG8="}'
# {"ciphertext":"polaris:aws-kms:v1:..."}

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/decrypt \
  -H 'Content-Type: application/json' -d '{"ciphertext":"polaris:aws-kms:v1:..."}'
# {"plaintext":"aGVsbG8="}
  • plaintext is base64, up to a little under 768 KiB.
  • Treat the ciphertext as opaque. The proxy uses envelope encryption: a fresh data key from KMS for each encrypt, and the wrapped key kept inside the ciphertext.
  • A 400 means the ciphertext was not made by this deployment, including one made by another deployment on the same key. A 503 means there is no passing verdict yet, or KMS refused the call.
  • Nothing is cached, so every decrypt is a fresh KMS call. A plaintext your workload keeps in memory outlives a revoke, so decrypt per operation if a revoke has to stop it.
  • A retired grant can keep working for about 30 seconds while KMS applies the change, and AWS documents that this can take several minutes in some cases.

Sign and check an HMAC, such as an HS256 JWT​

Give the deployment an HMAC key of its own as cloud.aws.hmac_key_arn, and the proxy also serves two MAC routes, on the same terms as encrypt and decrypt. The Policy Manager grants the workload VM's role GenerateMac and VerifyMac on the key while the VM passes, so the key never leaves KMS. A revoke ends signing on the next call, and the grant retirer and destroy end this grant along with the data key's.

# Sign. For an HS256 JWT the message is "<header>.<payload>", and the MAC is the signature.
curl -s -X POST http://polaris-proxy:3000/polaris-container/key/mac \
  -H 'Content-Type: application/json' -d '{"message":"aGVhZGVyLnBheWxvYWQ="}'
# {"mac":"..."}

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/verify-mac \
  -H 'Content-Type: application/json' -d '{"message":"aGVhZGVyLnBheWxvYWQ=","mac":"..."}'
# {"valid":true}
  • Both take base64. The MAC comes back in standard base64, and a JWT uses base64url.
  • The message can be up to 4096 bytes, which is KMS's limit.
  • A MAC that does not match answers {"valid":false}.
  • Use one HMAC key per deployment, and never share it. KMS refuses an encryption-context constraint on a grant for an HMAC key, so unlike the data key, nothing limits a grant to one deployment. Dev and prod each get their own. deploy checks that it is a customer-managed, enabled HMAC_256 key in the region and not the data key.
  • Add the VPC statement to the HMAC key's policy too.

To keep tokens you already issued verifying, import your current secret into the key rather than letting KMS generate one. The secret has to be at least 32 bytes. A shorter one is refused at import, and then you need a new secret, which invalidates the tokens signed with the old one.

aws kms create-key --key-spec HMAC_256 --key-usage GENERATE_VERIFY_MAC --origin EXTERNAL \
  --description 'Polaris prod HMAC key' --query KeyMetadata.Arn --output text
aws kms get-parameters-for-import --key-id <arn> \
  --wrapping-algorithm RSAES_OAEP_SHA_256 --wrapping-key-spec RSA_4096 > params.json
# Decode PublicKey to wrapping.der and ImportToken to token.bin from params.json, then:
openssl pkeyutl -encrypt -in secret.bin -out wrapped.bin -inkey wrapping.der -keyform DER -pubin \
  -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256
aws kms import-key-material --key-id <arn> --encrypted-key-material fileb://wrapped.bin \
  --import-token fileb://token.bin --expiration-model KEY_MATERIAL_DOES_NOT_EXPIRE

The secret still exists wherever you kept it before. Only a later switch to a KMS-generated key makes the attested service the only thing that can sign, and that switch drops the tokens issued before it.

Run on a Dedicated Host​

Preview

Dedicated Host support is released and has yet to complete a full validation run on real hosts. Try it in a non-production account first, and tell Fr0ntierX what you see.

A Dedicated Host brings SEV-SNP to every commercial region. AWS bills the whole host by the hour from the moment you allocate it, whether or not instances run on it, so release it when you are done.

  1. Check your Dedicated Host quota for the instance family in the region, in Service Quotas under EC2, and request an increase if it is zero.

  2. Allocate a host with SEV-SNP enabled, in the Availability Zone you want:

    aws ec2 allocate-hosts --region <region> --availability-zone <zone> \
      --instance-family m6a --cpu-options AmdSevSnp=enabled --quantity 1

    It prints the host's ID, which starts with h-. SEV-SNP is a setting of the host itself, so allocate it this way rather than from Terraform, which has no option for it. AWS prepares a new host before it accepts instances, which can take up to a couple of hours, and the host is billed during that time.

  3. Set the host and the region in your config, with both machine types from the host's family:

    cloud:
      aws:
        region: us-west-2
        dedicated_host_id: h-0123456789abcdef0

    compute:
      cvm_machine_type: m6a.xlarge
      pm_machine_type: m6a.large

  4. Deploy as usual. deploy places both VMs on the host and both subnets in the host's zone, and it stops before Terraform creates anything if a machine type belongs to another family. By then it has already created the deployment's registries and identity key, which a later deploy reuses.

The SDK does not yet accept a Dedicated Host's report. It expects every AWS report to be signed with a VLEK, while a Dedicated Host signs with the chip's VCEK, so checking the deployment the way a client will, as in step 8 of the tutorial, prints valid: false. The Policy Manager and the workload VM check each other with the VCEK and are not affected.

Both VMs share the one host, so the host is a single point of failure for the deployment. AWS applies new firmware to a host when it is allocated, so moving to newer firmware means allocating a new host and deploying onto it, and capturing the Policy Manager's measurement again if you pinned it.

The tooling carries AWS's instance identity certificates for us-east-2, eu-west-1, us-west-1 and us-west-2. For another region, add AWS's RSA-2048 certificate for that region, from AWS's instance identity document certificates, to the tooling's terraform/modules/aws/aws-iid-certs/ folder as <region>.pem, and deploy picks it up. Take the one on the RSA-2048 tab. The Policy Manager refuses to start on a smaller key.

After destroy, release the host:

aws ec2 release-hosts --region <region> --host-ids h-0123456789abcdef0

Use the Polaris image from AWS Marketplace​

A Marketplace image belongs to AWS Marketplace and has its own ID in each region, so you look the ID up and set it in your config.

Draft

The product ID and version below come from the Polaris listing, which is being finalized together with the private offer.

  1. Accept the private offer in AWS Marketplace and subscribe to the product, using the AWS account you deploy into.

  2. Look up the image ID for your region and the Polaris version you deploy:

    aws ssm get-parameter --region <region> \
      --name /aws/service/marketplace/prod-<product id>/<version> \
      --query Parameter.Value --output text

    It prints an ID that starts with ami-.

  3. Set it under cloud.aws:

    cloud:
      aws:
        ami_id: ami-0123456789abcdef0

Name an exact version here and change it deliberately. Polaris learns the image it runs, so a version that moved on its own would make a healthy machine look changed.

Licensing is not enforced yet. Licensing describes how it will work.

Pin the Policy Manager's measurement​

Pinning makes the workload VM refuse any Policy Manager whose launch measurement differs from the one you recorded. Without a pin, the workload VM learns the measurement from the first Policy Manager it verifies.

  1. Deploy without pm_measurement_hex. At the end of the run, deploy prints a block starting Nothing is pinned on this deployment with the two lines to add to your config. Each printed line starts with ▸, which you leave out when you copy. With --json, the block is not printed.

  2. Add the two lines under cloud.aws:

    cloud:
      aws:
        pm_measurement_hex: "<96 hex characters>"
        pm_required_vmpl: 0

  3. Deploy again. deploy reads the Policy Manager's report back and confirms it matches the pin, and if it does not match, the run fails and prints both values in full.

The measurement changes when AWS updates its firmware or when you change compute.pm_machine_type, and the workload VM then refuses its Policy Manager. To capture the new value, remove pm_measurement_hex, deploy, and start again from step 1.

deploy reads the reported value back over SSH, outside the AMD-signed path, so copy a new value into the pin once you know why it changed.

Move a layer to enforce​

A layer in advise reports findings and leaves your key alone, while a layer in enforce withdraws access to your key when it fails.

  1. Wait until the deployment has learned the workload:

    ./scripts/deploy.sh watch --endpoint https://<address> \
      --token "$(cat terraform/modules/aws/terraform-<name>.readonly.token)"

    Overall and every layer should show pass, with Alerts at 0. watch prints one snapshot, so run it again until they do, since enforcing a layer that is still learning makes access to your key come and go.

    The Policy Manager also refuses to enforce a layer until it has been clean under advise for a day, with at least 90% of that day's report slots covered, so the VM's hourly and daily jobs have run at least once. /management/status, read with the read-only token, shows each layer's promotion_readiness. A day covers only what ran in it, so a weekly job can still trip a layer the first time it runs after you enforce. To change the window, pass --promotion-window, such as 12h or 0, to deploy.

  2. Move one or more layers:

    ./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3

    The layers are l2 (image), l3 (runtime), l31 (kernel) and ima_boot (boot-time measurements). The runtime layer also has three parts that you can set on their own, l3_syscalls, l3_binaries and l3_network. promote refuses to enforce a layer whose allowlist is still empty, and it reads the settings back from the Policy Manager to confirm they took. If the upload fails, it stops with an error and the deployment keeps its previous settings. If only the read-back fails, the new settings may already be live, so run the same promote again, which prints the settings the deployment has before it changes anything.

    The AWS deployment leaves ima_boot unset, and an unset ima_boot follows l2, so enforcing l2 enforces the boot-time measurements as well. promote checks the boot-time allowlist only when you name ima_boot, so name it whenever you name l2.

To keep network checks advisory while the rest of the runtime layer enforces:

./scripts/deploy.sh promote --config <file> --layers l2,ima_boot,l3_syscalls,l3_binaries
./scripts/deploy.sh promote --config <file> --layers l3_network --mode advise

To move a layer back, run promote with --mode advise.

promote reaches the Policy Manager over SSH through the workload VM, so cloud.aws.ssh_allowed_cidrs has to include your address while you run it.

The Policy Manager keeps these settings, and everything it learned, across a restart. A deploy that changes the deployment's baseline returns the settings to their deployed values and starts learning again, and you run promote again once the layers are ready.

Under l3 enforce, commands run in an SSH session on a VM can count as unknown behavior and withdraw access to your key for one report cycle, so plan hands-on maintenance for a time when a short withdrawal is acceptable.

Monitor a deployment​

The proxy serves three status routes on the deployment's endpoint. They stay reachable while the proxy is holding workload traffic, so your monitoring can tell why traffic stopped.

RouteReturns
/polaris-container/health{"status":"ok"} while the proxy runs
/polaris-container/ready{"ready": true or false, "gate": "open" or "blocked"}, with 503 when not ready
/polaris-container/attestation-statusThe latest verdict, with status, a status for each layer and checked_at. With the read-only token it adds fail_reason and alerts

The fail reason and alerts name programs, destinations and file paths from your workload, so the route serves them only to a caller with the read-only or management token.

Read them together like this:

  • An enforcing layer failed. ready is false, and attestation-status shows fail. Ask it again with the read-only token to see the reason.
  • The Policy Manager has stopped answering. ready shows blocked while attestation-status still shows the last verdict, with a checked_at that keeps getting older, so alert when checked_at is more than a few report intervals old.
  • The proxy has just started. ready is true while gate reads blocked, for up to two minutes, and attestation-status returns 503 with no appraisal yet until the first verdict.
  • A layer in advise found something. attestation-status shows alert, while ready stays true and traffic flows, because only layers you enforce hold traffic.
  • Polaris is still learning. /management/profiles/learn, /management/ima/learn and /management/kernel/learn, read with the read-only token, list a session per learner whose status reads learning until it converges. watch reads the same routes.
  • Your workload is down. The status routes look healthy, and requests to your workload return 502.

Any route your workload serves, including a metrics endpoint, sits behind the same check as the rest of your traffic, so scrape your workload's metrics with the status routes alongside.

If the Policy Manager stops or hangs, the deployment's grant retirer ends the grant on your key. A stopped instance triggers it within seconds. A Policy Manager that hangs on a running instance stops refreshing its heartbeat, and the retirer catches that within about 10 minutes: the 300-second heartbeat limit plus one 5-minute check. The deployment also creates CloudWatch alarms for the Policy Manager's status check, the retirer's errors and each time it retires grants. They notify nobody until you add a target, such as an SNS topic, so add one.

Diagnose a withdrawal of access​

When requests start returning 503, the Policy Manager's records say why, and you can read them through the endpoint with the read-only token, with no SSH needed.

  1. Keep the token at hand:

    TOKEN="$(cat terraform/modules/aws/terraform-<name>.readonly.token)"

  2. Find the workload VM's ID in the Policy Manager's status list, in the cvm_id field:

    curl -sk -H "Authorization: Bearer $TOKEN" https://<address>/management/status

    The ID changes whenever the workload VM's proxy restarts, so look it up fresh each time.

  3. Read the latest incident as a plain-text report:

    curl -sk -H "Authorization: Bearer $TOKEN" \
      https://<address>/management/incidents/<cvm_id>/report

    An incident records what opened it, the status of each layer, the reason, each distinct alert with when it was first and last seen, and the grant and revoke events on your key. A runtime alert names the program and, for a network finding, the destination it connected to.

  4. For more context, /management/checks/<cvm_id> lists the latest checks one by one, and /management/history/<cvm_id> returns the last 100 verdicts.

Once you know the cause, fix it in the workload, or learn the new behavior and move the layer back to enforce.

Read the workload's logs​

Container logs stay on the workload VM, where Docker keeps them, and you read them over SSH:

ssh -i <private key> \
  -o StrictHostKeyChecking=yes \
  -o UserKnownHostsFile=terraform/modules/aws/terraform-<name>.known_hosts \
  ubuntu@<address> 'sudo docker logs --tail 200 client-workload'

Under l3 enforce an SSH session can withdraw access to your key for one report cycle, so keep log reading for a time when that is acceptable, or keep the runtime layer advisory while you investigate. Shipping logs to CloudWatch or another service from the workload VM is not available yet.

Change a deployment​

Edit the config and run deploy again.

Each run replaces both VMs, because each run issues the Policy Manager a new identity key that both VMs receive at launch. Plan every change, including a new version of your workload, as a short outage.

  • The workload VM gets a new public address, so update DNS or client configuration once deploy prints the new endpoint.
  • The Policy Manager's state is on its own volume and survives.
  • When the deploy changes the baseline, as a new version of your workload or a new image does, layers return to their deployed settings and Polaris learns the workload again, which takes at least 10 minutes. Run promote again once learning has finished.
  • When it does not, as with a new instance type, layers you enforce stay enforced. If ima_boot is enforced, as it is whenever l2 is and ima_boot is unset, the new workload VM fails l2 and has no access to your key while Polaris learns its boot measurements, about 10 minutes with the default learning window. It then passes with no promote, and what it measured at boot is in the allowlist from then on. Polaris learns whatever the new VM boots and then gives it the key, trusting it because it booted from the image the baseline names. The 10 minutes are a delay, not a check.

Changing compute.pm_machine_type also changes the Policy Manager's measurement, so capture it again if you pinned it. To move a deployment to another region, tear it down and deploy it again with the new region.

Take a security fix​

The Polaris image keeps the packages it was built with, and a security fix arrives as a new image.

  1. Get the new image ID for your region, as in Use the Polaris image from AWS Marketplace, with the new version.

    Draft

    How Fr0ntierX announces a new Polaris image version will be confirmed with the AWS Marketplace offer.

  2. Set it as cloud.aws.ami_id and run deploy, which replaces both VMs.

  3. The baseline names the image, so a new image returns layers to their deployed settings while Polaris learns what it boots. Run promote again once learning has finished. A workload VM that boots from any other image fails L1 platform and gets no access to your key.

You decide when to rebuild, and a deployment left alone keeps the packages its image was built with, so set a schedule that fits your patching policy.

Upgrade Polaris​

A Polaris release brings a new version of the tooling, whose manifest names the exact Polaris components to run, and usually a new image. Upgrade by switching to the new tooling and image and running deploy, which replaces both VMs as in any other change. Both VMs always move to the new versions together.

To roll back, deploy again with the previous tooling and image. Running an older Policy Manager against a state volume written by a newer one has not been tested, so snapshot the state volume before you upgrade and keep the snapshot until the new version has run cleanly.

Plan for AWS maintenance​

AWS gives notice through AWS Health before maintenance on the host under an SEV-SNP instance, and asks you to stop and start the instance yourself. The route we have tested for both VMs is a fresh deploy in a maintenance window, with the same outage and relearning as any other change.

A stop and start keeps the instance and its disks. The workload VM comes back with a new public address, and the grant on your key returns once the workload VM reports and passes again. Stopping the Policy Manager ends the workload's access to your key, because the grant retirer retires the grant as soon as the instance stops. When the Policy Manager is back, it grants access again on the workload VM's next passing report, so plan a Policy Manager stop as a short outage of key access. If you pinned the Policy Manager's measurement and the new host runs newer AWS firmware, capture the measurement again.

Prepare for production​

  • Use a dedicated AWS account for each deployment, with a short list of people who hold iam:PassRole on the Polaris roles.
  • Add the statements from Your KMS key's policy to your key, including the one that keeps the workload role's calls inside the VPC. Each grant is limited to its deployment's encryption context, so deployments can share a key, and a key of its own still keeps a deployment's key policy and CloudTrail apart from the others.
  • Add a notification target to the deployment's CloudWatch alarms, so a stopped Policy Manager or a failing grant retirer reaches someone.
  • Alert on changes to the Polaris roles' trust policies and on grant changes to your key, from CloudTrail.
  • Pin the Policy Manager's measurement and your workload image's digest.
  • Enforce at least l2 and l3 once learning has finished, with l3_network advisory if your workload's destinations change often. Enforcing l2 enforces ima_boot too, so name both when you promote.
  • Keep ssh_allowed_cidrs as narrow as you can. Changing it takes a deploy, which replaces both VMs and starts learning again, so choose it before you enforce rather than opening and closing it as you go.
  • Keep allowed_egress_cidrs to the ranges your workload needs.
  • Treat the tooling folder as a secret. It holds the deployment's Terraform state, its config file, its tokens and its pinned host keys, and the state and config file carry the tokens in plain text, so keep it encrypted, backed up and access-controlled.
  • Grant the deploying identity kms:RevokeGrant on your key, so teardown can clean up its grants.
  • Snapshot the Policy Manager's state volume before anything destructive. Its ID is the Terraform output pm_state_volume_id.
  • Monitor the status routes from Monitor a deployment, and keep the diagnosis steps in your on-call runbook.
  • Follow AWS's firmware and maintenance notices, since a firmware change moves the Policy Manager's measurement.

Troubleshoot​

The tooling says it needs bash 4 or newer. macOS ships bash 3.2. Install a newer one, for example with brew install bash, and run again.

deploy stops at failed to pull released image ghcr.io/.... Docker on your machine has no login to ghcr.io with read access to the Polaris images. Log in and run deploy again. It reuses the ECR repositories the first run created.

Terraform stops with No valid credential sources found. The AWS CLI works but Terraform cannot read your sign-in, which happens with aws login. Add the profile from step 1 of the tutorial, set AWS_PROFILE, and run deploy again.

The SDK check fails with self-signed certificate. Pass allowSelfSigned: true to verifyProxy, as in step 8 of the tutorial.

The SDK check fails with certificate table is missing the ARK certificate. The SDK is older than polaris-sdk#3, which adds AMD's certificates above the VLEK that AWS leaves out. Ask Fr0ntierX for a newer SDK, as step 8 of the tutorial explains. The Policy Manager's own hardware check is not affected.

deploy reports unknown config keys. A key is misspelled or does not exist, and the tooling stops so that nothing is silently ignored. Check the key against the AWS reference.

deploy stops at the KMS key check. The message names what is wrong: the value is an alias or key ID rather than the key's full ARN, the key is in a different region from the deployment, the key cannot be read, or the key is disabled, AWS-managed, not symmetric, or not for ENCRYPT_DECRYPT. Fix the key or the ARN.

Terraform rejects the deployment name. metadata.name must start with a lowercase letter and have 2 to 25 lowercase letters, digits and hyphens.

deploy says no Polaris image was found. Set cloud.aws.ami_id. A Marketplace image always needs it, as described in Use the Polaris image from AWS Marketplace.

Terraform refuses the region or the instance size. On shared tenancy SEV-SNP runs in us-east-2 and eu-west-1, on the sizes in Instance sizes, and other regions need cloud.aws.dedicated_host_id. This check runs late, after images are pushed, so fix the config and run deploy again.

Terraform says a machine type cannot run on the Dedicated Host. Both compute.cvm_machine_type and compute.pm_machine_type have to belong to the host's instance family, or match its instance type when the host was allocated for one type.

Terraform says there is no instance identity certificate for the region. Add the region's RSA-2048 certificate to the tooling, as described at the end of Run on a Dedicated Host.

Instance launch fails with an explicit deny in a service control policy. A rule in your AWS organization blocks EC2 in this region or account, and your Organizations administrator can allow it.

deploy warns there is no SSH route to the Policy Manager. cloud.aws.ssh_allowed_cidrs is empty or the private key file is missing. The deployment works, while deploy skips the measurement check and host-key pinning. promote needs both, so add your address and run deploy again, which replaces both VMs and starts learning again.

The workload never starts, or the deployment never reports. An image outside ECR in the deployment's region needs its registry in network.allowed_egress_cidrs, or the pull waits at boot.

Requests return 503 with attestation pending. The proxy is waiting for a verdict or holding traffic after a failure. Read the status routes and the latest incident, as in Diagnose a withdrawal of access.

watch cannot reach the Policy Manager. It may still be starting, so wait a minute. Otherwise check that you are using the workload VM's endpoint and a token for this deployment.

promote cannot reach the Policy Manager. cloud.aws.ssh_allowed_cidrs is empty or no longer includes your address. Update it and run deploy, and run promote from the copy of the tooling that deployed, since it needs that copy's state, token and pinned host keys.

promote refuses a layer. Either that layer's allowlist is still empty, so learning is still running, or the layer has not yet been clean under advise for the promotion window, a day by default. promote prints the Policy Manager's reasons. Wait and try again.

Access to the key stops while the deployment passes. The Policy Manager stopped or stopped refreshing its heartbeat, and the grant retirer ended the grant. Check the Policy Manager instance and its CloudWatch alarms. Access returns on the workload VM's next passing report once the Policy Manager is back.

SSH refuses to connect because of the host keys. The tooling pins each VM's host keys when it deploys and refuses to connect if they differ. Run deploy to pin them again. If the VMs have run for a long time, their boot messages may have aged out of the serial console, and a new deploy replaces them with fresh ones.

deploy says the pinned measurement does not match. AWS updated its firmware, or the Policy Manager's size changed. See Pin the Policy Manager's measurement.

Access to the key drops for one report cycle and comes back. An enforcing layer saw something it had not learned, and the Policy Manager records the cause in an incident. If you were running commands over SSH at the time, that was the cause. If it happened at 17 minutes past the hour, the VM's hourly scheduled jobs touched a file for the first time since learning, such as /etc/pam.d/cron.

Layers are back in advise. A deploy changed the deployment's baseline, as a new version of your workload or a new image does. Run promote again once learning has finished.

The workload VM fails l2 for about 10 minutes after a deploy. The deploy replaced the workload VM without changing the baseline, as a new instance type or --force-replace does, so ima_boot stayed enforced while Polaris learned the new VM's boot measurements. ima_boot is enforced whenever l2 is and ima_boot is unset. It passes once learning finishes, with no promote.

destroy prints KMS commands to run. It could not revoke the deployment's grants, usually because the deploying identity lacks kms:RevokeGrant on your key. See the end of Tear down a deployment.

destroy says there is nothing to destroy. It found no Terraform state. Run it from the copy of the tooling that deployed.

Tear down a deployment​

./scripts/deploy.sh destroy --config <file>

destroy revokes the deployment's grants on your key and then removes the VMs and their disks, the network, the IAM roles, the secrets, the grant retirer and the Policy Manager's state volume. It also schedules the Policy Manager's identity key for deletion. Snapshot the state volume first if you need what it holds.

Your KMS key, the ECR repositories and the tooling folder remain, along with any Dedicated Host you allocated, which you release yourself. Delete the repositories only under the deployment's own polaris.docker_repo prefix. Those under the default polaris-docker/ may belong to other deployments in the account.

Once destroy has finished and the key's grant list shows none from this deployment, delete the deployment's files from the tooling folder, since they carry its tokens. They are listed in Files the tooling keeps. Run status before deleting them, not after, because status writes new token files.

When destroy could not revoke the grants, or one is still listed afterwards, finish by hand with an identity that holds kms:RevokeGrant on your key:

aws kms list-grants --key-id <key ARN> \
  --query "Grants[?starts_with(Name,'polaris-')].[GrantId,GranteePrincipal,Name]" \
  --output text
aws kms revoke-grant --key-id <key ARN> --grant-id <grant id>

The list shows every Polaris grant on the key. While the deployment's role exists, its grants show the grantee arn:aws:iam::<account>:role/<name>-cvm-role. Once destroy has deleted the role, KMS shows the role's unique ID instead, which starts with AROA, so the ARN no longer matches anything. To tell this deployment's grants apart on a key other deployments share, note the role's unique ID before you run destroy:

aws iam get-role --role-name <name>-cvm-role --query Role.RoleId --output text

Then revoke the grants whose grantee is that ID. On a key only this deployment used, revoke every grant the list shows.

Tear down with destroy rather than with Terraform directly, because destroy revokes the grants while the deployment's roles still exist.

Your data stays yours throughout. Anyone your key policy allows can decrypt it with your key directly, with no Polaris component involved, during a deployment's life and after it.