PolicyStrata is a deterministic regression-testing framework for policy drift in LLM data-agent stacks.
It generates principals, requests, semantic plans, database states, lowered queries, and release decisions, then compares each layer against a canonical policy oracle. Failures are minimized into small reproducible witnesses.
System
- Canonical policy oracle kept independent from manifests, validators, compilers, and database behavior
- Deterministic support, finance, and ClickHouse-style benchmark domains
- Scanner mode for imported application traces and release-gate workflows
- Packaged Python CLI, PyPI distribution, GitHub Action, Docker fixtures, and artifact bundle
Runtime
policystrata runwrites traces, summaries, metadata, manifests, and minimized witnessespolicystrata scanchecks exported SQL/tool-call traces against a scanner config- Frozen benchmark manifests protect held-out suites from accidental detector drift
- Reproduction scripts rebuild paper-facing evidence tables from a clean checkout
Proof
- Paper artifact reports 1720/1720 injected non-clean cases with 0 false positives on 80 clean controls
- Failing runs minimize to witness JSON that localizes the violated layer and obligation
- Tests cover policy logic, compiler behavior, database assets, scanner config, evidence output, and benchmark freezing
- Public release includes SHA256-pinned paper and artifact archive