Use Case · Multimodal Model Training

From raw multimodal data to training-ready dataset releases.

Use native file, image, audio, and video types to prepare training data with SQL and Python. Add model labeling, embedding, quality filters, and deduplication, then write to files or lake formats and scale from local execution to Ray clusters.

Try the quickstart →Request a demo →

raw · multimodal

dataset release

Execution timeline

Overlaps heterogeneous resources.

Traditional pipelines create stage barriers and pipeline bubbles. Vane overlaps CPU, GPU, and I/O workloads through streaming execution and dynamic batching.

Same pipelineIllustrative timing for the same stages; actual performance depends on the data, models, and hardware.

Camera frame at an urban intersection — camera frame · ts 00:14.280

inputs

IMAGEFILEAUDIOFILEVIDEOFILEFILE

prepare

decode + resizeresample + frames

infer

model labelingembedding

select

filter + dedupe

write

Lance / IcebergParquet files

Traditional Pipelinestage barriers · longer critical path

CPUdecoding

GPUwaitinginference

I/Owrite

Stage barriers serialize CPU decoding, GPU inference, and I/O, leaving the GPU waiting for decoded batches.

Vane Pipelineoverlapped streaming · shorter critical path

CPUdecoding

GPUinference

I/Ostreaming

Dynamic batching overlaps CPU decoding, GPU inference, and I/O streaming, completing the same pipeline earlier.

Why Vane

Faster pipelines, in far less code.

Performance — Overlap CPU, GPU, and I/O work

From data preparation to embedding, pipeline efficiency matters. Vane coordinates processing and data movement across heterogeneous resources.

Efficient heterogeneous executionOverlap CPU processing, GPU inference, data movement, and I/O asynchronously, enabling heterogeneous resources to work concurrently instead of waiting on each other.
Streaming execution with backpressure & dynamic batchingContinuously process large-scale media and sensor data with adaptive batching and flow control, with backpressure to manage memory usage.
Ray-Native distributed scalingScale data preparation from a local runtime to a Ray cluster, with CPU, GPU, and memory requirements declared per processing stage.

See the benchmarks →

Simplicity — Native types, familiar interfaces

Multimodal data workflows often require multiple systems and layers of orchestration. Vane unifies data processing, AI inference, and dataset preparation into a single execution graph.

Native multimodal processingFILE, IMAGEFILE, AUDIOFILE, and VIDEOFILE carry file references. Built-in operations inspect metadata, decode and resize images, resample audio, and extract video frames.
SQL and Python togetherCombine SQL quality filters and deduplication with your Python UDFs or AI functions for labeling and embedding, including GPU workloads.
Connect training data storageRead and write Lance or Iceberg through separately installed providers, or export Parquet files as in the example below.

Read the code →

Representative code

Prepare uniform image samples with native types.

Use the three sample images created in the quickstart. Inspect their dimensions, keep images at least 300 pixels wide, and export two 224 × 224 PNG samples as bytes in Parquet. No model credentials or GPU are needed.

Create the sample images →

Storage providers: Lance · Iceberg

IMAGEFILEMetadata filterDecode + resizePNG → Parquet

prepare_training_images.py

from pathlib import Path


import vane


con = vane.connect()
assets = Path("quickstart-images").resolve()
files = vane.from_files(
    [str(assets / f"{name}.png") for name in ("large", "medium", "small")],
    connection=con,
)
images = con.sql("SELECT image_file(file) AS source FROM files")
metadata = con.sql("""
    SELECT source, image_file_metadata(source) AS info
    FROM images
""")
samples = con.sql("""
    SELECT source.url AS source_uri,
           info.width AS source_width,
           info.height AS source_height,
           encode_image(
               resize(decode_image_file(source, 'RGB'), 224, 224),
               'PNG'
           ) AS image_bytes
    FROM metadata
    WHERE info.width >= 300
""")
samples.write_parquet("training-image-samples.parquet")

Build a reproducible multimodal training-data pipeline.

Become a design partner →Explore a training-data tutorial