Sentinel
An agentic incident-response copilot that closes the loop: the alert opens the incident, an agent investigates and cites its evidence, a human approves, the fix runs exactly once, and recovery is verified.
🛡️ New in v2: Sentinel Guard, envelope protection for any AI agent
No agent changes your cluster unless policy, or a human, approved that exact change, and then only once.
Sentinel's safety core, extracted for coding agents with kubectl, CI and GitOps bots, and LangGraph or MCP agents. It needs no agent code changes, because it enforces at the Kubernetes API. Humans, CI and controllers are never sent to it, so a guard outage stops only agents.
Provable to others. kubectl sentinel audit export writes one evidence file: every request, its signed events, the policy version it was decided under, the signed chain head. Anyone can verify it offline, and an edited copy fails, naming the record. Every replica re-verifies the chain itself, and tested alert rules flag drift in each agent's behaviour. How this maps to the EU AI Act and the NIST AI RMF →
Try it in one command. From a clone, docker compose up starts the guard and a support-desk agent working behind it. Open localhost:8080 as Alice (token alice-demo): every refund waits for you in the approvals inbox, closing an account needs Alice and Bob, receipts go out on policy, and erasures are never allowed. --profile monitoring adds Prometheus with the drift alerts and a Grafana dashboard. Quickstart →
helm install sentinel-guard oci://ghcr.io/abdulmdev/charts/sentinel-guard --version 2.0.0 -n sentinel-system --create-namespace · then kubectl sentinel approve <id> · Sentinel Guard docs →
✈️ Why Sentinel exists: the cockpit idea
Airliners don't rely on a pilot's nerve at 3 a.m. On-call engineers shouldn't have to either.
When something fails in a modern airliner, Airbus's ECAM and Boeing's EICAS show the alert, the affected systems and the checklist to run, and Airbus's flight-envelope protection stops inputs outside safe limits. The pilot stays in command; the automation keeps the right next step in front of them, so stress, fear or fatigue don't turn a manageable failure into a reactive one.
Production incidents have the same shape: paged at night, dashboards screaming, the fast reactive move is tempting (roll back whatever deployed last, restart everything). Sentinel brings the cockpit discipline to on-call:
| In the cockpit | In Sentinel |
|---|---|
| Failure alert with the affected systems (ECAM / EICAS) | The alert opens an incident; the agent gathers deploys, logs, metrics and runbooks |
| The right checklist for this failure | One proposed action, every claim cited to evidence |
| Envelope protection: unsafe inputs are blocked | Policy interlocks block unsafe actions even when proposed |
| The pilot decides | Nothing runs until a human approves |
| Exactly the commanded input, nothing more | A single-use capability executes exactly the approved action, once |
| Instruments confirm the recovery | Real-traffic probes verify health before "resolved" |
| The flight data recorder | Audit trail, timeline and auto-written postmortem |
The goal isn't to replace the engineer. It's to make the calm, correct call the easy one, at any hour. Sentinel is not affiliated with Airbus or Boeing; the comparison describes a design philosophy.
Measured, not claimed
What happens when production breaks
- Detect. Prometheus fires; the incident opens itself, deduplicated and correlated per service.
- Investigate. A LangGraph agent calls MCP tools (deploys, logs, metrics, runbooks) within hard budgets.
- Diagnose. Ranked root causes, each cited to evidence; deterministic guardrails block unsafe actions.
- Approve. Nothing touches production until a human says yes.
- Execute once. Approval mints a 5-minute, single-use capability for exactly that action.
- Verify. Real-traffic probes decide “resolved”; time to mitigate and the postmortem are recorded.
Engineering highlights
Agentic AI, done safely
- LangGraph with a human-approval interrupt and durable Postgres checkpoints
- Tools over MCP; step and dollar budgets enforced by the graph
- PII redaction, injection removal, citation validation, policy interlocks, constrained decoding
- Claude, local gemma4 via Ollama, or rules
Zero-trust security
- Only the gateway mints credentials: ES256 + JWKS; verifiers can't forge
- Single-use capabilities; replay store fails closed
- OIDC (Keycloak, PKCE); every hop authenticated and tenant-bound
- Row-level security incl. agent checkpoints; least-privilege secrets checked in CI
Distributed systems
- Kafka ingestion, idempotent re-index, dead-letter topic
- Hybrid retrieval (pgvector + full-text, RRF) in one Postgres transaction
- Race-safe alert dedup with a per-service advisory lock
- Atomic Lua token bucket on Redis with local fallback
Rigor and operations
- Deterministic benchmark (112 incidents + held-out) as a CI gate
- OpenTelemetry trace per incident across Java → Python → MCP; Grafana
- Helm (hardened pods, NetworkPolicies), kind e2e, Terraform for AWS
- 9 ADRs; a 16-chapter handbook incl. every bug found
Architecture
Built with
How it was built
| Phase | Outcome |
|---|---|
| 1 · Foundation | Gateway ↔ LangGraph agent ↔ chaos-lab; human approval; tenancy from the JWT |
| 2 · Ingestion + RAG | Kafka ingestion; hybrid retrieval; row-level security by a non-superuser role |
| 3 · Agents | MCP tool servers, downscoped tokens, budgets, durable checkpoints |
| 4 · Evaluation | Replayed-incident benchmark as a CI gate; found the 16% harmful-rollback bug |
| 5 · Hardening | Guardrails + injection tier, Redis limits, semantic cache, tracing, load tests |
| 6 · Ship | Console, Helm on Kubernetes, Terraform for AWS, CI |
| 7 · Close the loop | Alerts open incidents; capability-gated execution; verification; time to mitigate |
| 8 · Zero trust | ES256 + JWKS, OIDC, every hop authenticated, checkpoint RLS |
| 9 · Publish and measure | Full local-LLM benchmark; runbook preconditions take gemma4 from 11.6% to 0% harmful actions |
| 10–13 · Sentinel Guard | The safety core for any agent: Kubernetes admission, SDKs and an MCP proxy, the copilot on it, SafeActionBench |
| 14 · Governance | Signed evidence export verifiable offline, policy versions, guardian alerts on agent drift, EU AI Act / NIST AI RMF mapping, one-click Codespaces demo |
| 15 · Try it, ship it | docker compose up with a live agent, an approvals inbox in the guard, a Grafana dashboard for the guard, signed charts and SDK packages on every release |
| 16 · Two people | Two-person approvals for the riskiest actions, approvers verified by OIDC at the guard, notifications to Slack; v2.0.0 released |
| 17 · The whole picture | An Overview of every agent's actions and a details view for every request, with its signed audit trail; incidents can no longer get stuck; released as v2.0.0, a stable API with upgrades tested |