Vane Data / Reference

vane.func.batch

Use vane.func.batch to process data in batches inside select(). Expression inputs and Python literals are materialized as Arrow columns, and the function must return one Arrow column of the same length.

Signature

text

vane.func.batch(
    *,
    return_dtype: Any,
    name: str | None = None,
    batch_size: int | None = None,
    unnest: bool = False,
    gpus: float | None = None,
) -> Callable[[_PythonFunction], VaneBatchFunction]

Parameters

Name	Type	Description	Default
return_dtype	SQL type string, Vane DuckDBPyType, or supported pyarrow.DataType	Output column type	Required
name	Non-empty str or None	UDF name; defaults to the function __qualname__	None
batch_size	Positive integer or None	Maximum rows processed by each call	None
unnest	bool	Expands the fields of a Struct result into columns	False
gpus	Finite non-negative number or None	GPU resource per Task; positive values require Ray	None

Returns and errors

When called in select(), the output has the same number of rows as the input. With unnest=False, the query produces one return_dtype column. With unnest=True, it expands the fields of a Struct return type into separate columns.

A direct call accepts only pyarrow.Array and pyarrow.ChunkedArray inputs. Every input must have the same length. It returns an Array when the normalized result has one chunk and a ChunkedArray otherwise.

Function errors, non-Arrow inputs or results, row-count mismatches, and values that cannot be converted to return_dtype raise during a direct call or when query results are fetched. Distributed backends may retry batches, so external effects must be idempotent.

Example

example.py

import vane




@vane.func.batch(return_dtype="BIGINT")
def add(a, b):
    import pyarrow.compute as pc


    return pc.add(a, b)




source = vane.sql("SELECT * FROM (VALUES (1, 4), (2, 5), (3, 6)) AS t(a, b)")
result = source.select(add(vane.col("a"), vane.col("b")).alias("total"))


print(result.order("total").fetchall())
vane.close()

Output: