Vane Data / Reference

Relation.flat_map

Relation.flat_map calls a synchronous Python callable for each input row. One row may produce zero, one, or many rows, and schema declares the complete output layout.

Signature

text

Relation.flat_map(
    function: Callable[..., typing.Any],
    schema: dict[str, sqltypes.DuckDBPyType] | None = None,
    *,
    batch_size: int | None = None,
    output_batch_size: int | None = None,
    min_task_batch_size: int | None = None,
    preserve_compute_batch_boundaries: bool | None = None,
    cpus: float | None = None,
    gpus: float | None = None,
    memory_bytes: int | None = None,
    execution_backend: typing.Literal["subprocess_task", "subprocess_actor", "ray_task", "ray_actor"] | None = None,
    actor_number: int | None = None,
    target_max_batch_bytes: int | None = None,
    task_input_max_bytes: int | None = None,
    output_target_max_bytes: int | None = None,
) -> DuckDBPyRelation

Parameters

Name	Type	Description	Default
function	Synchronous function, bound method, or zero-argument callable class	Receives a row dict and returns a row dict, synchronous iterable, or None	Required
schema	Non-empty dict[str, DuckDBPyType]	Complete output names, order, and types. The current runtime requires an explicit value	Required
batch_size	Positive integer or None	Maximum rows per compute batch; the callable still receives one dict at a time	None
output_batch_size	Positive integer or None	Target rows per output Arrow block	None
min_task_batch_size	Positive integer or None	Soft Task-input floor; requires batch_size and cannot be smaller	None
preserve_compute_batch_boundaries	bool or None	Accepted, but the row-output path has no distinct boundary-flush behavior	None
cpus	Finite non-negative number or None	CPU resource per Task or Actor	None
gpus	Finite non-negative number or None	GPU resource per Task or Actor; positive values require Ray	None
memory_bytes	Positive integer or None	Memory resource per Task or Actor; available only with Ray backends	None
execution_backend	subprocess_task, subprocess_actor, ray_task, ray_actor, or None	Execution backend; defaults from the runner and callable shape	None
actor_number	Positive integer or None	Actor instance count; required for Actor backends and invalid for Task backends	None
target_max_batch_bytes	Positive integer or None	Common byte target for Task input and output blocks	None
task_input_max_bytes	Positive integer or None	Input-byte target for one Task or Actor call	None
output_target_max_bytes	Positive integer or None	Output-block byte target	None

Returns and errors

flat_map() returns a new Relation and leaves the input unchanged. The result contains only the columns declared in schema.

Invalid functions, schemas, or execution options raise when flat_map() is called. Function errors, unsupported return values, and values that do not match schema raise when results are fetched.

Task and Actor backends may retry calls, so external effects must be idempotent. A callable class runs in independent, ephemeral Actors with no work affinity or global ordering; Actor reconstruction resets local state.

Example

example.py

import vane




def split_words(row):
    for word in row["text"].split():
        yield {"word": word.lower()}




source = vane.sql("SELECT 'Vane Data' AS text")
result = source.flat_map(
    split_words,
    schema={"word": vane.sqltypes.VARCHAR},
)


print(result.fetchall())
vane.close()

Output: