cfg-eval tests whether grammar-constrained decoding improves NL-to-SQL reliability for ClickHouse.
The project includes a live demo and an eval harness that grades model output against live ClickHouse reference runs rather than mocked expectations.
System
- Natural-language prompts mapped to a visible ClickHouse schema subset
- Lark grammar for syntactically valid SQL generation
- Hidden adversarial fields used to test schema adherence
- Result-set oracle with semantic comparison instead of string equality
Runtime
- Next.js demo surface with TypeScript test harness
- Live ClickHouse execution for reference and candidate queries
- Python Lark grammar checks plus LLGuidance token-mask validation
- Constrained and unconstrained model paths compared head to head
Proof
- Constrained path reaches full execution success on the adversarial slice
- Explicit whitespace threading blocks implicit whitespace injection
- Minimal reasoning effort reduced constrained-path latency in testing
- Refusal tests follow false-positive hardening guidance from prior NL-to-SQL eval work