Deploying on your own hardware

Polaris on hardware you own needs no cloud attestation service, no cloud identity, and no outbound internet once the artifacts are in place. Read Deploying Polaris first for the choices common to every environment. This page covers what is specific to bare metal, including the air-gapped case.

On-premises deployments are delivered as engagements with our team, so you will not be doing this alone. This page is what we work through together, and it is written so your own operators can run it afterwards.

What this gives you, and what it does not

Worth being plain about the boundary before anything else, because it differs from the cloud deployments.

On your own hardware you get memory confidentiality for the workload against host software, since SEV-SNP encrypts guest memory with a key the hypervisor cannot read. You get integrity monitoring, because the sentinel watches syscalls, network destinations, kernel modules and the IMA log while the Policy Manager appraises what it sees against a baseline. And you get key gating, because once a layer is promoted from advisory to enforcing, a failed appraisal stops the workload being served.

What it does not give you is a defence against whoever owns the machine. For this release the host is inside the trust boundary. The person with root controls the key manager, the baseline and the artifacts the guests are launched from, so an operator who wants the keys can have them. Polaris on-premises is there to detect drift in your own estate and to gate keys on that, not to protect a workload from its own infrastructure owner. If your case has the workload owner and the host owner as different parties, tell us, because that needs a different design.

Before you start

You need an AMD EPYC server with SEV-SNP, Milan generation or later, and SEV-SNP enabled in firmware. On most vendors that is more than one setting, typically SEV, SEV-SNP and an SNP memory reservation, so consult your vendor's documentation rather than assuming a single toggle. Two guests is the minimum, one for the workload and one for the Policy Manager, so you need the ASIDs for both.

Confirm the host agrees before going further:

cat /sys/module/kvm_amd/parameters/sev_snp    # expect Y
ls -l /dev/sev                               # must exist
dmesg | grep -i 'SEV-SNP'                    # the firmware version the PSP reports

If sev_snp reports N or the parameter is absent, the fault is in firmware or the kernel and nothing below will work.

Record three numbers from the machine that will run the guests:

grep -E '^cpu family|^model[[:space:]]|^stepping' /proc/cpuinfo | head -3

These go into the configuration and they are required. CPUID 1 EAX appears in every VMSA and every VMSA is covered by the launch measurement, so a set of artifacts is valid for one CPU model. Launching them on a host whose family, model or stepping differs is refused rather than silently accepted.

The reference platform is the RHEL 10 family. Validated on Rocky Linux 10.2 with libvirt-daemon, qemu-kvm, virt-install, guestfs-tools, edk2-ovmf, python3 and cpio installed. rpm2cpio comes from rpm itself, so rpm-build is not needed.

SELinux

SELinux Enforcing is the RHEL default and is supported. The deployment tool labels what it stages, because the artifact directory is not a path the libvirt policy knows and the defaults there are not usable by the processes that need them:

• the guest firmware as virt_content_t, which is the type qemu may read. Without it a guest fails with qemu: could not load PC BIOS and no mention of SELinux.
• a serial/ directory for the guests' console logs as virt_log_t, root-owned and mode 0700. virtlogd has to create and rotate files there, which needs permission on the directory and not just the files.

Labels are applied with semanage fcontext where policycoreutils-python-utils is installed, so they survive a filesystem relabel, and with chcon otherwise. On an Enforcing host the tool stops if it cannot label at all, rather than leaving you a deployment that will not start.

If something still does not start, the host has already recorded why. ausearch -m AVC -ts recent, or grep 'avc: *denied' /var/log/audit/audit.log, names the process and the path. On a Permissive host the same lines appear with permissive=1, meaning the access was allowed but would have been blocked, so a Permissive host will tell you what Enforcing is going to break before you switch.

TEE options

AMD SEV-SNP, verified directly against AMD's published keys. There is no cloud attestation service in the path and no external attestation authority, which is the property that makes this configuration suitable for air-gapped and sovereign environments.

Recent AMD generations also support SVSM-based key isolation, which moves the guest's virtual TPM inside the trust boundary. Confidential GPUs are supported on-premises as they are in the clouds, with the caveat that NVIDIA's attestation service is the one outbound dependency Polaris cannot remove when they are in play.

How you deploy

Start from the shipped on-premises example configuration and read its comments, which are the reference for what each key changes. Five values have no default and the tool stops without them: an NTP server, the Policy Manager's address, and the three host CPU numbers above. There is no metadata service on bare metal, which is why the time source and the Policy Manager address have to be stated rather than discovered.

To see what a configuration produces without building anything:

./scripts/deploy.sh deploy --dry-run --cloud onprem --config onprem.yaml

Then bake the artifacts and launch:

./scripts/deploy.sh deploy --cloud onprem --config onprem.yaml \
  --image-tag <tag> --workload-dir ./workload
./scripts/deploy.sh launch --cloud onprem --config onprem.yaml

On this path the tool acquires no container images. It expects your Docker daemon to already hold them and exports exactly those references into the guest image, so if you build elsewhere then save, copy and load them on this host first.

The bake prints two launch measurements, one per guest, which differ because the vCPU counts differ. Both are computed from the same staged firmware. Container images are not part of a launch measurement, which covers the initial memory image rather than the disk, so rebuilding an image leaves both measurements unchanged. Catching a changed container is the job of the runtime layers and the IMA allowlist.

That has a consequence for the Policy Manager measurement pin the workload checks. The pin is the Policy Manager's launch measurement, so it accepts any guest booted from this firmware with this many vCPUs on this CPU, not only the Policy Manager you deployed. The vCPU count is the only thing that tells the two guests apart. With equal counts the pin would accept a workload guest as the Policy Manager, so the tool refuses a configuration that gives both guests the same count, a bake whose two measurements come out equal, and a launch from an artifact set that gives both guests the same count.

To take the guests down without discarding the artifacts, use destroy with the same configuration.

Verification path

Read the verdict from the Policy Manager through its own attested TLS front, using the read-only token the deployment generated:

curl -sk -H "Authorization: Bearer $READONLY_TOKEN" https://<pm_ip>:443/api/v1/status
curl -sk -H "Authorization: Bearer $READONLY_TOKEN" https://<pm_ip>:443/api/v1/checks

See Verifying a deployment for what each layer means.

Two things to expect. The first cycles will raise findings, because the learners converge over roughly a day and a converged deployment reports a passing status with no alerts. And the Policy Manager's alert list is the observer, not the guest's console log. A console log shows only what the containers chose to write, so when what you are checking is that something did not happen, read the alert list.

Baselines ship with every layer advisory, which means findings are raised and no key is ever withheld. Promoting a layer to enforcing is a deliberate step, and one you should take only once that layer reports clean, because enforcing a layer whose allowlist is still empty is either inert or immediately fatal depending on the layer. On-premises:

./scripts/deploy.sh promote --cloud onprem --config onprem.yaml \
  --layers l31 --mode enforce

--layers takes any of l2, l3, l31, ima_boot, l3_syscalls, l3_binaries and l3_network, comma-separated. The three l3_ dials govern one L3 category each and follow l3 while unset, so syscalls can enforce while network still follows an advising l3. --mode is enforce, advise or off. Add --dry-run to see the current dials and what the change would be without sending it, which also prints what each layer has actually learned so far.

Once a layer is enforcing, a failed appraisal stops the workload being served. There is no cloud key service to revoke on your own hardware, so what you observe instead is the proxy refusing traffic and its readiness route reporting unavailable, which takes the workload out of service behind any load balancer in front of it. The timing is a budget of failed reports rather than a single one, so expect a couple of minutes between attestation stopping and traffic being refused, and recovery within seconds of the cause being resolved.

Key management

Key release integrates with your own key manager rather than a cloud service. For HashiCorp Vault, Polaris uses the transit engine only and needs no enterprise features. No namespaces, no Vault Agent, and nothing installed: built-in auth and secret engines only, because a plugin is the thing a security review of an air-gapped site tends to stop on.

The key never leaves Vault. Polaris moves permission rather than key material, so a node that fails appraisal loses the ability to ask Vault to decrypt, rather than losing a copy it could have kept. The Policy Manager never holds the key either.

What Polaris writes into your Vault

At deploy time, with the provisioning token described below, all of it namespaced to the deployment:

• a certificate auth mount at auth/polaris-<deployment id>, with one trusted-certificate entry named nodes
• a transit key, polaris-workload unless you name another
• a policy for your nodes allowing encrypt and decrypt on that one key and nothing else, not even reading the key
• a policy for the Policy Manager covering six paths: read and update on the certificate entry, update on the mount's CRL, looking up a token by accessor, revoking one, revoking leases under its own mount, and renewing its own token. The lookup is there so the Policy Manager can check a token belongs to this deployment before revoking it, since Vault offers no way to scope a revoke by accessor to one mount
• one periodic token for the Policy Manager, carrying that policy

scripts/vault-provision.sh is the whole of it and is readable on its own, so you can review exactly what will be written before you run a deploy, or undo it afterwards by hand. Re-running a deploy re-applies the same objects rather than creating more.

The provisioning token comes from VAULT_TOKEN in your environment. It needs to create mounts and policies, which is far more than Polaris ever holds afterwards, and it is used for that one step and never stored. What Polaris keeps is the periodic token above, which carries only those six paths.

How the Policy Manager's token reaches it

The deploy never holds that token. It is minted response-wrapped, so what the deploy writes, and what travels in the cloud-init seed and sits under /etc/polaris/vault-token in the guest, is a single-use wrapping token. On its first start the Policy Manager unwraps it and seals the token on its state disk the same way it seals its issuer keys, to the vTPM or an SNP derived key. After that every copy outside the Policy Manager is spent.

That also makes a copied seed visible. Vault destroys a wrapping token the first time anyone uses it, so if something unwraps the seed before the Policy Manager does, the Policy Manager refuses to start and says so. Treat the token as exposed: revoke it and re-run the deploy.

The wrapping token is good for 24 hours, which has to cover the bake and the first launch. Set VAULT_PM_TOKEN_WRAP_TTL (a Vault duration such as 72h) if yours take longer. The seed records when it expires, so a Policy Manager launched too late says the seed expired rather than that it was stolen. What the deploy keeps is the token's accessor, in vault-token.accessor beside the seed, so the next deploy can revoke it.

The token is not a child of the VAULT_TOKEN you deploy with. Vault revokes a token's children with it, so a child would die when your login expires or when you revoke the root token after setup. Creating it this way needs root or sudo, which the deploy needs anyway to create a periodic token, and you are free to revoke the provisioning token as soon as the deploy finishes.

The token is periodic with a thirty day period, and the Policy Manager renews it at a third of each lease, so it does not run out while the Policy Manager runs. Its status shows vault_pm_token: how the token is kept, when it expires, and the last renewal error. If renewals fail for thirty days, for example because Vault was unreachable that long, the token lapses and the Policy Manager can no longer revoke a failed node's token or publish the CRL.

A deployment made before the token was wrapped has a plain token in all those places, and a policy without renew-self. It keeps working, with a warning, until the token's period runs out. Redeploying fixes both.

Getting a fresh token takes all three commands, in this order:

./scripts/deploy.sh destroy --cloud onprem --config onprem.yaml
./scripts/deploy.sh deploy --cloud onprem --config onprem.yaml \
  --image-tag <tag> --workload-dir ./workload
./scripts/deploy.sh launch --cloud onprem --config onprem.yaml

deploy mints the new token and revokes the previous one the moment it runs, and the guests only pick the new one up when they are relaunched. launch refuses to start while the old guests are up. So the destroy has to come first, and the deployment is down from there until the launch finishes. Worth planning for before the thirty days rather than after.

Set server_ca to the PEM for your Vault's own serving certificate. On a cloud you could lean on the public trust store, and here you usually cannot, because the certificate is issued by your own CA. Leaving it out means the provisioning token goes to whatever answers on that address.

Give address as an IPv4 address, not a hostname, for example https://10.20.0.5:8200. The workload CVM connects to Vault itself for every key operation, so the generated L3 allowlist names Vault, and that rule has to be an address. A deploy with a hostname there stops and says so. Vault's serving certificate then needs an IP SAN for that address, or the TLS check fails at the first login.

Both guests need a route to that address on that port: the Policy Manager, which revokes and publishes the CRL, and the workload CVM, whose proxy logs in and calls Transit. Opening it for the Policy Manager alone leaves the workload unable to decrypt, with every key operation answering 503.

How a node proves it should get the key

The Policy Manager issues a node a certificate each time it passes appraisal, signed by its own node-ca, over the key the node already attested with. The node presents that certificate to Vault over mTLS, and Vault accepts it because the certificate entry pins node-ca. There are no shared secrets, no AppRole, and no bearer token for a node to leak.

The entry starts with a placeholder CA that trusts nothing, because node-ca does not exist until the Policy Manager first starts. It replaces the placeholder itself on that first start, and again whenever its issuer keys rotate. If you inspect the entry before the Policy Manager has ever run, a certificate you do not recognise is expected.

How a workload uses the key

The workload never talks to Vault and never holds a Vault credential. It asks the Polaris proxy in its own CVM, and the proxy calls Transit under the node's login. When the Policy Manager stops passing the node, the next call fails.

Two routes, on the proxy's internal listener:

POST http://polaris-proxy:3000/polaris-container/key/encrypt
POST http://polaris-proxy:3000/polaris-container/key/decrypt

The bodies are Transit's. Plaintext goes in as standard base64, and the ciphertext comes back exactly as Vault writes it:

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/encrypt \
  -H 'Content-Type: application/json' -d '{"plaintext":"aGVsbG8="}'
# {"ciphertext":"vault:v1:..."}

curl -s -X POST http://polaris-proxy:3000/polaris-container/key/decrypt \
  -H 'Content-Type: application/json' -d '{"ciphertext":"vault:v1:..."}'
# {"plaintext":"aGVsbG8="}

Only the workload's own network reaches them. A request from the guest itself, through the proxy's published port or from anywhere else, gets 403. They are never on 443.

What each answer means:

Status	Meaning	What to do
200	Done.
400	The request was malformed, or Vault could not use the input, such as a ciphertext that is not one of its own.	Fix the input. Retrying will not help.
403	The request did not come from the workload network.	Call from the workload container.
413	The body is over 1 MiB.	Split it, or encrypt a data key and use that for the bulk.
503	No key operation right now: the node has no passing verdict yet, it has no node certificate yet, or Vault refused or could not be reached. Comes with Retry-After: 30.	Wait and retry. At startup this is normal until the first appraisal passes. Later it means the node lost its grant or Vault is down.

The error body says which case it is, and never repeats what you sent.

Call per operation if a revoke has to end decryption. A data key the workload decrypts once and keeps in memory outlives the revoke, as it would with envelope encryption on any key service. The proxy caches no results, so every call is a fresh Transit request under a token that is at most a minute old.

How long a withdrawal takes

When the Policy Manager decides a node should lose access, it revokes that node's Vault token straight away and publishes a CRL so the node cannot log in again. That is immediate.

The number to plan around is the case where the Policy Manager cannot reach Vault at all, so it can revoke nothing. Then access ends when the node's current certificate and token expire on their own, which is six minutes by default: a five minute node certificate plus a sixty second token. Both are deliberately short for this reason, and both are written by the deploy rather than left to Vault's defaults, which are far longer.

Two behaviours sit behind that number and are worth understanding rather than memorising. A token issued from a certificate login outlives the certificate that bought it, so the token lifetime is the window a withdrawn credential keeps working. And revoking a certificate stops the next login without touching tokens already issued. Polaris does both actions for that reason, and the six minutes is what remains when it can do neither.

Unsealing, and what happens after a power cut

Unsealing your Vault is yours and never ours. Use Shamir shares distributed to holders you choose, or PKCS#11 auto-unseal where you already run an HSM. Polaris never holds an unseal share, because a Polaris component that could unseal the key manager that gates Polaris keys would make the gate decorative.

The consequence is worth planning for before you meet it. After a site-wide restart, Vault comes back sealed unless you have configured auto-unseal, and nothing Polaris runs can change that. Until enough keyholders are available, your guests attest normally and produce verdicts normally, and no workload can decrypt anything. If that is not an acceptable recovery story for the site, PKCS#11 auto-unseal against an HSM you already run is the way out, and it is a Vault configuration rather than a Polaris one.

Two more things come back changed after a restart, neither of them Vault's doing. The learners start again, so a deployment that had converged needs its soak period again before its allowlists are full. And enforcement dials come back at whatever the stored baseline says, which is advisory unless you promoted them and the promotion was persisted. Check both before assuming a recovered site is back in the posture it was in.

Air-gapped deployments

Everything Polaris would otherwise fetch has a local alternative: the guest firmware package, the guest base image, the guest's own packages, the container images, and AMD's certificate and revocation endpoints, which are replaced by certificates supplied locally. Your own NTP server covers time. The generated on-premises baseline contains no cloud metadata destinations and no cloud identity fields.

Keep local copies of both pinned upstream files, the firmware package and the guest base image. Each is verified by digest, so a substituted file is refused rather than used, but a file that has been removed upstream cannot be fetched at all, and distributions do prune superseded versions. If you may need to reproduce a build months later, copy both into your own storage now and point the configuration at them. A build host with no egress needs this regardless.

Also keep a local cache of the guest's packages. Without one the build resolves them against the distribution archive, which needs egress and is not reproducible, so two builds weeks apart will produce different guest images.

Keeping the firmware current

Polaris does not build its own guest firmware. It takes your distribution's packaged EDK2 firmware, verifies both the package version and the digest of the firmware image inside it against values pinned in the tooling, and stages exactly those bytes. The digest is the contract, because the baseline's launch measurement was computed for those bytes, so a mismatch is refused rather than shipped.

So when EDK2 has a vulnerability, the path is that your distribution ships a patched package, Polaris updates the pinned version and digest, and you rebuild. The firmware bytes change, so both launch measurements change and so does the Policy Manager's measurement pin, which means a new set of artifacts rather than a package update inside a running guest. In an air-gapped site that is something delivered rather than downloaded. Your firmware therefore moves on your distribution's schedule for the patch plus a Polaris release for the pin, and it does not wait on us to build firmware.

On a Rocky 10.2 host you already have the firmware and can confirm you have the right bytes against the copy your own package installed:

sha256sum /usr/share/edk2/ovmf/OVMF.amdsev.fd

Notes

There is no SSH into the guests. They are measured and carry no authorized keys, so the serial consoles are the way in, written to serial/ inside the artifact directory and readable as root. virsh console <domain> works too.

If the guests boot, load their images, start every container and then fail attestation, check the platform setting. Bare-metal SEV-SNP has its own value, and a cloud platform value on bare metal produces evidence claiming to come from that cloud.

If every guest fails the launch measurement check, the artifacts were built for different firmware, a different vCPU count or a different host CPU. All three are covered by the measurement.

If the workload's proxy refuses the Policy Manager over a measurement mismatch, the Policy Manager measurement pin in the artifacts is from an earlier build. A pin is only valid for the firmware and guest sizing it was computed from.