Multimodal Data Lake
This page follows the four canonical media and model scripts in the Vane repository:
- examples/querying_images.py
- examples/image_generation.py
- examples/voice_ai_analytics.py
- examples/multimodal_structured_outputs.py
They demonstrate image decoding and analysis, image generation, audio transcription and embedding, and vision-language evaluation with structured output.
Create a clean environment
After installing uv as described in Installation, sparsely clone only the example scripts into a new directory:
git clone --depth 1 --filter=blob:none --sparse \ https://github.com/AstroVela/vane.git vane-examples cd vane-examples git sparse-checkout set examples uv venv --python 3.12 source .venv/bin/activate uv pip install vane-ai numpy pyarrow Pillow openai pydantic uv pip install torch uv pip install sentence-transformers transformers
The sparse checkout excludes the vane/ and duckdb/ source directories, so the Python process and local Ray workers use the installed wheel. This environment runs every default sample on the page. For GPU acceleration, install a CUDA-enabled PyTorch build compatible with the operating system, GPU, and driver; use PyTorch Start Locally when the default package is not the right build. Confirm that python -c "import torch; print(torch.cuda.is_available())" prints True; Vane's Transformers provider then requests one Ray GPU automatically. The first voice example downloads sentence-transformers/all-MiniLM-L6-v2 unless it is already cached. The structured-output example also needs provider credentials because synthetic input removes the dataset dependency, not the model call.
If PyPI downloads are slow, append the Tsinghua mirror to an install command:
uv pip install torch -i https://pypi.tuna.tsinghua.edu.cn/simpleThe mirror changes where packages are downloaded from; it does not choose a compatible CUDA build for the machine.
Run the local media samples
These three commands use generated input and do not need storage credentials, an external dataset, or a hosted model provider:
python examples/querying_images.py --source sample --limit 5 python examples/image_generation.py --source sample --limit 4 python examples/voice_ai_analytics.py --source sample --limit 3
With the current built-in samples, the scripts report 5 analyzed images, 4 generated placeholder images, 3 transcripts, and 6 embedded subtitle rows.
Image querying
querying_images.py generates or loads image bytes, decodes them in AnalyzeRedRegionsBatch, counts red pixels, and writes ranked previews. Its default output directory contains:
- images/ with decoded PNG previews
- masks/ with red-region masks
- top_red_images.csv with ranked metadata
To analyze local files:
python examples/querying_images.py \ --source glob \ --image-glob '/path/to/images/*' \ --limit 100 \ --top-k 20
The --source open-images mode reads a bounded list of public OpenImages objects and therefore needs network access.
Image generation
The default placeholder backend creates deterministic PNGs using only the base package. It is intended to validate relation, UDF, batching, and output behavior before downloading a diffusion model. Outputs are written to examples/output/image_generation/ with a metadata.csv manifest.
For real Stable Diffusion generation, install the model runtime and request a Ray GPU resource explicitly:
uv pip install diffusers transformers accelerate torch Pillow python examples/image_generation.py \ --source sample \ --backend diffusers \ --device cuda \ --dtype float16 \ --gpus 1 \ --limit 4
The model is downloaded on first use unless --local-files-only is set or --model-id points to a local model directory.
Voice AI analytics
The default voice workflow generates short WAV samples, uses deterministic placeholder transcripts, creates local summaries and subtitle rows, and embeds every subtitle segment. It writes summaries.csv, subtitles.csv, and segment_embeddings.csv under examples/output/voice_ai_analytics/.
For real transcription:
uv pip install faster-whisper av python examples/voice_ai_analytics.py \ --source glob \ --audio-glob '/path/to/audio/*.wav' \ --transcription-backend faster-whisper \ --device cuda \ --compute-type float16 \ --gpus 1 \ --limit 10
This mode still needs the Transformers extra for subtitle embeddings. Add --summary-backend openai only when the openai package, provider credentials, and the desired model configuration are available.
Multimodal structured output
The structured-output script asks the same multiple-choice question with and without an image, compares accuracy, classifies each row into an evaluation quadrant, and optionally sends failure cases through a judge pass.
Using the default Hugging Face router:
export HF_TOKEN=your_hugging_face_token python examples/multimodal_structured_outputs.py \ --source synthetic \ --limit 1 \ --skip-judge
Using another OpenAI-compatible endpoint:
export MODEL_API_KEY=your_api_key python examples/multimodal_structured_outputs.py \ --source synthetic \ --limit 1 \ --api-key-env MODEL_API_KEY \ --base-url https://provider.example/v1 \ --model provider/model-name \ --structured-output-mode prompt \ --skip-judge
Remove --skip-judge to enable the failure-analysis call. Use real AI2D samples only after installing the dataset dependencies:
uv pip install datasets Pillow python examples/multimodal_structured_outputs.py \ --source hf-ai2d \ --limit 4
Keep API keys in environment variables, not in command history, source files, or committed configuration.
Default Ray execution
All four scripts rely on Vane's default Ray runner. They do not set VANE_RUNNER, call vane.configure(...), or choose a task or actor backend. --gpus 1 only adds a Ray GPU resource request to the model UDF; it does not select the runner.
Set RAY_ADDRESS before starting a script to use an existing cluster. Every worker must have the matching dependencies and access to the same files, model cache, and credentials.
Promote outputs safely
- Validate row counts and stable IDs before joining model output to source metadata.
- Keep raw bytes only when downstream consumers require them.
- Record model IDs, prompts, and generation parameters with produced data.
- Test provider and model failures on bounded inputs.
- Let the destination system own catalog commits and snapshot semantics after Vane writes the curated files.