Portfolio

Selected engineering proof: what I built, the systems underneath, and the tradeoffs that mattered. Raintree product pages live at raintree.technology.

Agent Safety & Evals

Context & Developer Tools

Data & Intelligence Systems

Product SaaS

Systems & Low-Level

Music & Creative Systems

SQL evals / personal case study

cfg-eval

grammar-constrained model tests

What I built

Grammar-constrained SQL evals scored against live ClickHouse results.

Stack

Next.js 16 / React 19 / OpenAI Responses API / Lark CFG / ClickHouse Cloud / Vitest / Zod

Runtime

`/api/query` validates input, checks an LRU cache, calls GPT-5 with a custom CFG tool, executes bounded ClickHouse SQL, and returns rows.
`/api/evals` runs one trial using the same grading contracts as Vitest so browser and test harness stay aligned.
Raindrop traces emit from Next `after()` so observability does not add user-facing latency.

Data model

NYC Taxi sample table: 20M trips from 2015-07-01 to 2015-09-30 with a 13-column exposed subset.
Eval cases carry question, reference SQL, difficulty, compare mode, distractors, and expected refusal behavior.
Run records include generated SQL, parser state, execution rows, latency, token usage, and verdict details.

Techniques

Lark grammar makes DDL, DML, joins, subqueries, comments, lat/long columns, lowercase keywords, and phantom fields structurally unrepresentable.
Whitespace is threaded explicitly instead of `%ignore WS` to match LLGuidance constraints.
Earley parser validates the grammar locally because explicit whitespace creates LALR conflicts.
Correctness oracle compares ClickHouse result sets rather than SQL strings; schema introspection catches invalid identifiers.

Verification

Constrained path reaches full execution success on adversarial slice where unconstrained generation is more likely to hallucinate invalid columns.
Whitespace and token threading are tested so grammar constraints do not create accidental injection or parse bypasses.
The eval records parser failures, execution failures, and wrong-result failures separately to avoid a single vague accuracy number.
Refusal tests harden false positives where a model declines answerable queries or fabricates policy blockers.

Portfolio

Selected engineering proof: what I built, the systems underneath, and the tradeoffs that mattered. Raintree product pages live at raintree.technology.

Agent Safety & Evals

Context & Developer Tools

Data & Intelligence Systems

Product SaaS

Systems & Low-Level

Music & Creative Systems

SQL evals / personal case study

cfg-eval

grammar-constrained model tests

What I built

Grammar-constrained SQL evals scored against live ClickHouse results.

Stack

Next.js 16 / React 19 / OpenAI Responses API / Lark CFG / ClickHouse Cloud / Vitest / Zod

Runtime

`/api/query` validates input, checks an LRU cache, calls GPT-5 with a custom CFG tool, executes bounded ClickHouse SQL, and returns rows.
`/api/evals` runs one trial using the same grading contracts as Vitest so browser and test harness stay aligned.
Raindrop traces emit from Next `after()` so observability does not add user-facing latency.

Data model

NYC Taxi sample table: 20M trips from 2015-07-01 to 2015-09-30 with a 13-column exposed subset.
Eval cases carry question, reference SQL, difficulty, compare mode, distractors, and expected refusal behavior.
Run records include generated SQL, parser state, execution rows, latency, token usage, and verdict details.

Techniques

Lark grammar makes DDL, DML, joins, subqueries, comments, lat/long columns, lowercase keywords, and phantom fields structurally unrepresentable.
Whitespace is threaded explicitly instead of `%ignore WS` to match LLGuidance constraints.
Earley parser validates the grammar locally because explicit whitespace creates LALR conflicts.
Correctness oracle compares ClickHouse result sets rather than SQL strings; schema introspection catches invalid identifiers.

Verification

Constrained path reaches full execution success on adversarial slice where unconstrained generation is more likely to hallucinate invalid columns.
Whitespace and token threading are tested so grammar constraints do not create accidental injection or parse bypasses.
The eval records parser failures, execution failures, and wrong-result failures separately to avoid a single vague accuracy number.
Refusal tests harden false positives where a model declines answerable queries or fabricates policy blockers.

Selected projects

What I built

Stack

Selected projects

What I built

Stack