Process multimodal datawith SQL and Python.

Combine file processing, model inference and relational queries in one pipeline. Develop locally and scale execution with Ray.

Quickstart →GitHub ↗

pip install vane-ai

quickstart.py

import vane


con = vane.connect()
images = con.sql("""
    SELECT image_file(
        'quickstart-images/large.png'
    ) AS source
""")


images.select("""
    source,
    image_file_metadata(source).width AS width,
    resize(decode_image_file(source, 'RGB'),
           224, 224) AS thumbnail
""").show()

Set up and run the example →

Data processing and model inference. One engine.

Multimodal inputs

Process media and business fields in one data model.

Text & documentsVARCHAR · FILE
ImagesIMAGEFILE · IMAGE
AudioAUDIOFILE · TENSOR
VideoVIDEOFILE · IMAGE

TablesLocal filesObject storage

Vane DataOne data model. One execution plan.

Relation pipeline

SQL / Python

Read
Filter
Prepare
Infer
Query & write

Python UDF

Custom logic and stateful processing

AI Functions

ai_prompt · ai_embed

Model providersvLLM · model APIs

Logical plan → execution graph

Unified execution

Local / Ray

I/O↔CPU↔GPU

Dynamic batching · Concurrency · Backpressure

State reuse · Retry & recovery

Queryable results

Use analysis results in filters, joins and aggregations.

Structured fields
Model output as a native STRUCT.
Embeddings
Typed vectors, ready for search.
Queryable results
Filter, join, aggregate and write.

Query or save the results

Use the results in the next query or write them to your data store.

One data model

Query multimodal files and business fields in the same data model.

Composable pipelines

Compose SQL, Python UDFs and AI Functions for processing and inference.

Unified execution

Coordinate batching, concurrency and compute resources.

Read the architecture walkthrough →

Build multimodal pipelines with SQL and Python.

Follow an equipment inspection pipeline from native media types and Python UDFs to AI extraction, embeddings and business queries.

file.sql

SELECT file AS source_file
FROM list_files($file_glob)
WHERE file_path(file) LIKE '%.pdf'
  AND file_size(file) < 20*1024*1024;

Setup and parameters

import vane
con = vane.connect()
root = "s3://manufacturing/factory-a"

Both interfaces share input relations, instructions, model configuration and output schemas. Bind SQL parameters through con.sql(..., params=...). The complete example includes setup and indexes.

Equipment inspection example

Filter files before reading their content.

source_file: FILE
source_url: equipment/A-017/manual.pdf
size: 2.4 MB

Filter object metadata first. Read the candidate files when a later operation needs their content.

Example results. Configure a model that supports each input type.View the complete example

Coordinate I/O, CPU and GPU in one pipeline.

Overlap reading, preprocessing and inference across batches, with backpressure to control in-flight data.

Batch timeline: read C, prepare B and infer A overlap. Each batch advances from reading to preprocessing, inference and output. One model instance is reused. Bounded submission makes upstream wait when downstream slows.

Batch executionIllustration

Read C, preprocess B and run inference on A concurrently.

I/O read

ABC

CPU prepare

ABC

GPU infer

ABC

Write

ABC

Initialize and reuse the model within the Actor instance.

UpstreamBounded queueInference

↶ Upstream waits for capacity when downstream slows.

Overlap pipeline stages

Read the next batch, prepare the current one and infer the previous one at the same time.

Backpressure and bounded execution

Batch and submission limits keep faster upstream work from overwhelming slower stages.

Reuse model and client instances

A stateful UDF keeps its model or client across batches within the same execution instance.

Execution diagram. Stages overlap; error handling and retries follow the relevant execution settings.

Explore the execution architecture →

From local development to distributed execution.

Reuse your processing logic across local and Ray execution.

The same SQL or Python processing definition runs on a local machine with subprocess workers, or through a Ray scheduler and CPU workers / GPU Actors. All distributed workers need access to the inputs, dependencies and result storage.

Same definition: read → prepare / UDF → AI → query & write

One machine · Local Runner

Application / Driverimport vane

Local worker / subprocessCPU execution

Inputs and output storage

Local files or accessible object storage.

Develop and validate locally

Use the local runner for development, validation and on-demand processing, without starting Ray.

local_job.py

import vane


vane.configure(runner="local")
con = vane.connect()
result = con.sql(pipeline_sql)
result.show()

Same definition: read → prepare / UDF → AI → query & write

Ray Runtime / Cluster

Driverimport vane

Ray scheduler · Tasks & resources

CPU Worker

GPU ActorModel instance reuse

Inputs and output storage

All workers need access to inputs, models, dependencies and output storage.

Scale execution with Ray

Run Ray on one machine or a cluster, scheduling CPU workers and GPU actors according to declared resource needs.

job.py

import vane


# Ray is the default runner.
con = vane.connect()
result = con.sql(pipeline_sql)
result.show()

Both examples use the same pipeline_sql. Configure credentials, dependencies and data access for your environment. Ray runs locally or on a cluster and handles GPU resource scheduling.

Deployment guide →

Multimodal pipeline benchmarks.

Explore execution times alongside the workloads, hardware and settings behind them.

Vane DataRay DataDaftElapsed time in seconds. Lower is better.

Documents32.2% less time than Ray Data

Vane Data

86.09 s

Ray Data

127 s

Daft

413 s

Workload and configuration

Document extraction and processing.

Batch sizes：Vane 2560, Ray 320, Daft 20.

Workload definitions and scripts →

Images35.1% less time than Ray Data

Vane Data

1147 s

Ray Data

1767.11 s

Daft

OOM

Workload and configuration

Image processing pipeline.

Batch sizes：Vane 100, Ray 100, Daft OOM.

Workload definitions and scripts →

Audio2.2% less time than Ray Data

Vane Data

2312 s

Ray Data

2363.08 s

Daft

OOM

Workload and configuration

Audio processing pipeline.

Batch sizes：Vane 128, Ray 128, Daft OOM.

Workload definitions and scripts →

Video9.8% more time than Ray Data

Vane Data

7603 s

Ray Data

6922 s

Daft

8322 s

Workload and configuration

Video processing pipeline.

Batch sizes：Vane 32, Ray 32, Daft not set.

Workload definitions and scripts →

36 CPU / 64 GB RAM / 2080 Ti, 22 GB modified VRAM. Single-node, batch-size-tuned results.

This single-node evaluation follows the workload design of the Ray Data multimodal AI benchmark. Results apply to the documented hardware, data and settings. Daft ran out of memory on the image and audio workloads. See the benchmark page for methodology and scripts.

View methodology, scripts and results →

Put multimodal pipelines to work.

Vane Data

Multimodal training data

Prepare images, video and audio for training: filter, transform, label and export reproducible datasets with SQL and Python.

Vane Data

Enterprise data pipelines

Turn enterprise files and media into structured, searchable records. Keep business identifiers and source references for your retrieval and agent applications.

Run your first multimodal query.

Install and run an image processing example, with no model credentials required.

Prepare the same three images for either example

Save as prepare_images.py and run python3 prepare_images.py once in your working directory.

prepare_images.py

from pathlib import Path
from PIL import Image


folder = Path("quickstart-images")
folder.mkdir(exist_ok=True)
for name, size in [("large", (640, 480)),
                   ("medium", (320, 240)), ("small", (64, 64))]:
    Image.new("RGB", size, "#546354").save(folder / f"{name}.png")

quickstart.sql

WITH images AS (
  SELECT image_file(file) AS source
  FROM list_files('quickstart-images/*.png')
), metadata AS (
  SELECT source, image_file_metadata(source) AS info FROM images
), thumbnails AS (
  SELECT info.width AS source_width, info.height AS source_height,
         resize(decode_image_file(source, 'RGB'), 224, 224) AS thumbnail
  FROM metadata WHERE info.width >= 300
)
SELECT source_width, source_height,
       image_width(thumbnail) AS width,
       image_height(thumbnail) AS height
FROM thumbnails ORDER BY source_width DESC;

Save the query above, then run:

terminal

python3 -c 'import sys, vane; vane.configure(runner="local"); vane.connect().sql(sys.stdin.read()).show()' < quickstart.sql

The same result, with either interface

source_width	source_height	width	height
640	480	224	224
320	240	224	224

The 64 × 64 image is filtered out before pixel decoding. Selected images become native IMAGE values, resized to 224 × 224.

Install and run

install.sh

python3 -m pip install 'vane-ai[image]'

Install in a virtual environment, create the sample images, then run the same example using SQL or Python.

Quickstart →

AI Functions →Python UDF →Ray →