Bring parsed document rows, media references, tables, logs, and model outputs into one pipeline. Use SQL for deterministic checks, explicit UDF stages for extraction, and model-assisted review where it belongs.
messy materials
outputs
Today that chain is stitched across OCR scripts, temp files, model calls, SQL jobs, and review tools.
⚠ scattered systems · glue code everywhere · lost source references · hard to debug · poor reproducibility
Start with parsed rows and source metadata, then add SQL rules, explicit UDF stages, and model-assisted review without splitting the workflow across separate jobs.
SQL ai_prompt returns a scalar result; alias it in the projection, then keep document IDs, rule hits, the model response, and source URIs together in the same review row.
Validate locally first, then switch runner and UDF backends when distribution helps. The relation shape stays stable; worker storage, dependencies, and credentials still matter.
Register a Python policy rule once, then call it beside ai_prompt in SQL. Business IDs and source references stay on every audit row.
import vane @vane.func(return_dtype="VARCHAR") def policy_check(text): return "missing_signature" if "missing signature" in text.lower() else None con = vane.connect() vane.attach_function( policy_check, parameters=["VARCHAR"], connection=con, ) findings = con.sql(""" SELECT claim_id, document_id, policy_check(text) AS rule_hit, ai_prompt( text, struct_pack( provider := 'openai', model := 'gpt-4o-mini', system_message := 'Find missing claim evidence.' ) ) AS ai_finding, source_uri FROM read_parquet('claims/*.parquet') """) findings.write_parquet("audit_findings.parquet")