# ADR-0009: Zero-trust identity: asymmetric service tokens, OIDC users, every hop authenticated

**Status:** accepted, 2026-10-06 (completes the capability design of [ADR-0008](0008-closed-loop-capabilities-and-alert-intake.md))

## Context
Phase 7 made the agent able to change production, gated by single-use capabilities. Its weakest link was identity:
- **One HS256 secret signed every token** (users, tool tokens, integration tokens, capabilities). Any holder could mint a capability: the gateway, the remediation adapter, and the console's dev login. Until the Phase 7 fix, that included every pod.
- **Hops that trusted the network.** The agent API and the observability MCP server accepted any caller that could reach them.
- **No real user identity.** Users came only from a dev login; there was no IdP and no roles.
- **Integration tokens could be minted offline** and couldn't be listed or revoked.
- **Checkpoints were isolated by thread key only**, not by the database.

## Decisions

### 1. The gateway is the only token issuer for services; ES256 with a published JWKS
- **Key ring:** P-256 keys loaded from `SENTINEL_SIGNING_KEYS`, a JWK set generated by `scripts/gen_signing_key.py` and stored in a secret manager.
  - Public halves are served at `/.well-known/jwks.json`.
  - Each token carries a `kid`, so rotation needs no coordination: add the key, make it active, retire the old one after its tokens expire.
  - Production refuses to start without configured keys (`requireConfigured`); dev uses an ephemeral key.
- **One minter:** `ServiceTokens` issues every service token (`iss`, `aud`, `jti`, short TTL):
  - tool tokens: `sentinel-mcp`, 15 min;
  - capabilities: `sentinel-remediation`, 5 min, single-use;
  - agent-call tokens: `sentinel-agent`, 60 s;
  - integration tokens: `sentinel-alerts`, ≤ 90 days, revocable.
- **Verifiers hold no secrets.** The agent, the observability MCP server and the remediation adapter check tokens against the JWKS, with ES256 only and issuer, audience and expiry required. They can verify but never mint.

### 2. Users come from OIDC; the gateway routes by issuer, with algorithms pinned
- `RoutingJwtDecoder` reads the unverified `iss` only to choose a verifier. Each verifier then checks the signature with its own key material and a fixed algorithm:
  - gateway: ES256, and the audience must be a service audience;
  - IdP: RS256 via its JWKS, and the audience must be `sentinel-api`;
  - dev: HS256, only if enabled, and never with a service audience.
- So no token can choose its own verifier: HS256 tokens claiming the gateway's issuer and `alg: none` tokens are rejected, and both cases are tested.
- **Per-path rules:**
  - `/mcp`: gateway-issued tool tokens;
  - the alerts webhook: gateway-issued integration tokens whose `jti` is active in `integration_tokens`;
  - `/api/v1/integrations`: users with the `tenant-admin` role;
  - everything else: users only.
- **Keycloak in dev:**
  - `tenant_id` is a *managed* user-profile attribute only admins can edit. With unmanaged attributes, users could move themselves into another tenant; that attempt was tested live and refused.
  - an audience mapper for `sentinel-api`;
  - realm roles mapped to `ROLE_*` authorities.
- The console runs authorization code + PKCE server-side. Tokens stay in httpOnly cookies, refresh happens in the BFF proxy, and logout ends the IdP session too.

### 3. Every hop is authenticated, and the tenant is bound
- **Gateway → agent:** a fresh 60-second token per call. The agent checks that the token's tenant equals the tenant in the URL or body, then binds it to the request.
- **Agent → observability:** the tool token, which now carries `telemetry:read`. The observability server rejects unauthenticated `/mcp` calls.
- **Decisions** also carry a tool token, used by the verification probes.
- **Checkpoints get row-level security:**
  - every checkpoint cursor sets `app.tenant_id` from the authenticated request's tenant;
  - the `langgraph` tables FORCE RLS, with policy `split_part(thread_id, ':', 1) = app.tenant_id`;
  - a code path with no tenant sees nothing;
  - RLS is lifted only while LangGraph's own `setup()` migrations run.

### 4. Integration tokens are issued, recorded and revocable
`POST /api/v1/integrations/alertmanager` (tenant admins) returns the token once and stores only its `jti` (V7). The webhook accepts a token only while its record is active, so revocation is immediate. Compose and `dev-local.sh` provision Alertmanager's token through this API.

### 5. Least privilege per pod, enforced in CI
Each chart component lists the secret keys it may receive.

| Pod | Secrets in production |
|---|---|
| gateway | signing keys, DB password, LLM key |
| agent | LLM key, checkpoint DSN |
| web | OIDC client secret |
| MCP servers and remediation adapter | none |

`scripts/check_chart_secrets.py` fails CI if signing material reaches any other pod, or if the dev secret appears in production values.

## Alternatives considered
- **Keep HS256 and give each service its own secret.** Every verifier would still be able to mint whatever its secret signs. Asymmetric keys separate the power to verify from the power to mint.
- **mTLS service mesh (Istio/Linkerd) instead of tokens.** It gives transport identity, but not *tenant* or *action* binding, which is what capabilities and agent-call tokens carry. A mesh is complementary and stays on the roadmap.
- **Tenant-scoped tokens from the IdP for the agent (token exchange, RFC 8693).** This is the standard approach, but it ties every service-to-service call to IdP availability and latency. The gateway already authenticates the user, so minting downscoped tokens itself is simpler and auditable.
- **A schema per tenant for checkpoints.** Stronger physical isolation, but migrations and pooling become per tenant. RLS matches the rest of the data model.

## Consequences
- + A compromised agent, MCP server, adapter or console holds no key that can mint a capability. Forgeries with the old shared secret are refused at every verifier (tested).
- + Real users, roles and logout. Revocable, auditable integration credentials.
- + Defense in depth for tenancy: JWT claim → gateway, then token-bound tenant → agent, then RLS → database.
- − The gateway is a single issuer: if it's down, no new service tokens can be minted. That's already true for every request path, so it adds no new dependency.
- − Each verifier fetches the JWKS on first use and on an unknown `kid`, so the gateway must be reachable from those pods (NetworkPolicy updated).
- − Keycloak in dev adds about 700 MB of image and around 15 s of startup. Kubernetes (kind) keeps dev user tokens instead of running an IdP; production values require a real IdP.