Vane Data / Concepts

UDFs

A Vane Data UDF can be described along four related dimensions. The dimensions answer different questions, but the API constrains which combinations are valid:

API surface: SQL Expression, Python Expression, or Relation.
Invocation shape: scalar values, one row, or an Arrow batch; represented by Relation methods map, flat_map, and map_batches.
Cardinality: whether the stage preserves or changes row count.
Lifecycle: stateless function or stateful callable class.

1. API Surface

Choose the API surface from the output contract and any execution controls the stage requires.

API surface	How it is called	Output contract	Use when
SQL Expression UDF	Attach a callable with vane.attach_function, then call its alias in SELECT	One typed projection column; row-preserving in v1	One result belongs beside each input row and the surrounding pipeline is SQL
Python Expression UDF	Build an Expression with vane.func, vane.func.batch, vane.cls, or vane.cls.batch	One typed projection column; normally row-preserving	The same projection is easier to construct with Python objects
Relation UDF	Call rel.map, rel.flat_map, or rel.map_batches	A Relation; map appends value, while flat_map and map_batches define the complete output schema	The callable owns table shape or row count, or the stage needs Relation-only controls such as an explicit backend, actor pool, or side_effects

Python batch Expression has one narrow exception: with row_preserving=False, it may change row count only when it is the sole top-level projection. Relation UDFs are currently Python methods; Vane does not expose them as SQL table functions.

Case: SQL Expression, Python Expression, and Relation

This program runs the same normalization through SQL and Python Expression APIs, then uses a Relation UDF for a multi-column result:

example.py

import pyarrow as pa
import vane


con = vane.connect()
source = con.values(
    (
        vane.lit(1).alias("document_id"),
        vane.lit(" Refund requested ").alias("text"),
    ),
    (vane.lit(2), vane.lit(" Shipment delayed ")),
)




def normalize_value(value):
    return str(value).strip().lower()




@vane.func(return_dtype="VARCHAR")
def normalize_text(value):
    return normalize_value(value)




# SQL Expression UDF.
vane.attach_function(
    normalize_text,
    alias="normalize_text_sql",
    connection=con,
    parameters=["VARCHAR"],
)
try:
    sql_result = con.sql("""
        SELECT
            document_id,
            text,
            normalize_text_sql(text) AS normalized_text
        FROM source
        ORDER BY document_id
    """)
    sql_result.show()
finally:
    vane.detach_function("normalize_text_sql", connection=con)




# Python Expression UDF.
python_result = source.select(
    vane.col("document_id"),
    vane.col("text"),
    normalize_text(vane.col("text")).alias("normalized_text"),
).order("document_id")
python_result.show()




# Relation UDF: the returned Arrow table owns the complete output shape.
def enrich_batch(table):
    document_ids = table.column("document_id").to_pylist()
    values = table.column("text").to_pylist()
    normalized = [normalize_value(value) for value in values]
    return pa.table({
        "document_id": document_ids,
        "normalized_text": normalized,
        "text_length": [len(value) for value in normalized],
    })




relation_result = source.map_batches(
    enrich_batch,
    schema={
        "document_id": "INTEGER",
        "normalized_text": "VARCHAR",
        "text_length": "INTEGER",
    },
).order("document_id")
relation_result.show()

Keep a SQL alias attached until show() or another terminal operation materializes the relation. Python Expression calls build lazy projection objects and need no alias. The Relation callable must return every column that should survive its stage.

2. Invocation Shape: map, flat_map, and map_batches

These are literal Relation method names. Their closest Expression forms have similar callable shapes but retain the one-column Expression contract.

Operation	Invocation shape	Cardinality	Relation output	Expression counterpart
map	Scalar column values in, one typed value out	Preserves row count	Preserves input columns and appends value	vane.func in Python or an attached scalar alias in SQL
flat_map	One row dict in, zero or more row dicts out	May change row count	Returns exactly the declared schema	Python vane.func.batch(..., row_preserving=False) is a restricted one-column alternative; SQL has no counterpart in v1
map_batches	pyarrow.Table in, pyarrow.Table out	May preserve or change row count	Returns exactly the declared schema	vane.func.batch in Python or an attached batch alias in SQL; the Expression schema has exactly one column

Case: Compose All Three Relation Operations

This pipeline uses map, flat_map, and map_batches in sequence:

example.py

import pyarrow as pa
import vane


con = vane.connect()
source = con.values(
    vane.lit(1).alias("document_id"),
    vane.lit("Refund requested").alias("text"),
)




# map: append one scalar value to every input row.
def text_length(document_id, text):
    return len(text)




mapped = source.map(
    text_length,
    return_type=vane.sqltypes.INTEGER,
).project("document_id, text, value AS text_length")




# flat_map: turn one document into zero or more word rows.
def split_words(row):
    for word in row["text"].split():
        yield {
            "document_id": row["document_id"],
            "text_length": row["text_length"],
            "word": word.lower(),
        }




words = mapped.flat_map(
    split_words,
    schema={
        "document_id": "INTEGER",
        "text_length": "INTEGER",
        "word": "VARCHAR",
    },
)




# map_batches: own the complete output table for each Arrow batch.
def add_word_length(table):
    document_ids = table.column("document_id").to_pylist()
    text_lengths = table.column("text_length").to_pylist()
    word_values = table.column("word").to_pylist()
    return pa.table({
        "document_id": document_ids,
        "text_length": text_lengths,
        "word": word_values,
        "word_length": [len(word) for word in word_values],
    })




result = words.map_batches(
    add_word_length,
    schema={
        "document_id": "INTEGER",
        "text_length": "INTEGER",
        "word": "VARCHAR",
        "word_length": "INTEGER",
    },
)
result.show()

map automatically keeps upstream columns. flat_map and map_batches return exactly the columns declared by schema, so include any upstream columns needed later.

3. Cardinality

Cardinality is separate from invocation shape. Scalar Expression UDFs, registered SQL aliases, and Relation map preserve one output row per input row. Relation flat_map explicitly emits zero or more rows for each input row. Relation map_batches may return the same or a different number of rows.

Python batch Expressions declare this contract with row_preserving. With row_preserving=True, the result can be projected beside ordinary columns. With row_preserving=False, it may change row count but must be the sole top-level projection. SQL registration accepts only row-preserving batch wrappers in v1.

4. Stateless and Stateful

State describes callable lifetime, not output shape. Both Expression and Relation UDFs can be stateless or stateful.

Lifecycle	Expression form	Relation form	Runtime behavior
Stateless	vane.func or vane.func.batch; the wrapper may also be attached as a SQL alias	Pass a function to a task backend	No callable instance is promised across calls; batches can be processed independently
Stateful	Instantiate vane.cls or vane.cls.batch; call it in Python or attach the instance as a SQL alias	Pass a callable class to subprocess_actor or ray_actor	One instance per actor; each actor owns independent, execution-scoped state

Use stateless functions for deterministic work that needs no reusable setup. Use stateful classes to reuse a model, tokenizer, client, lookup table, or cache. In v1 a stateful Expression uses one actor per query. A Relation actor pool may contain several independent instances.

Case: Stateless Function and Stateful Actors

This example compares stateless and stateful Expression UDFs through Python and SQL, then shows actor reuse for a stateful Relation UDF:

example.py

import pyarrow as pa
import vane


con = vane.connect()
source = con.values(
    (
        vane.lit(1).alias("document_id"),
        vane.lit(" Refund requested ").alias("text"),
    ),
    (vane.lit(2), vane.lit(" Shipment delayed ")),
    (vane.lit(3), vane.lit(" Refund requested ")),
)




def normalize_value(value):
    return str(value).strip().lower()




@vane.func(return_dtype="VARCHAR")
def stateless_normalize(value):
    return normalize_value(value)




@vane.cls(actor_number=1, return_dtype="VARCHAR")
class CachedNormalizer:
    def __init__(self):
        self.cache = {}


    def __call__(self, value):
        if value not in self.cache:
            self.cache[value] = normalize_value(value)
        return self.cache[value]




cached_normalize = CachedNormalizer()


# Python Expression: a task-backed function beside one stateful actor.
python_result = source.select(
    vane.col("document_id"),
    stateless_normalize(vane.col("text")).alias("stateless_result"),
    cached_normalize(vane.col("text")).alias("stateful_result"),
).order("document_id")
python_result.show()


# SQL Expression: attach the stateless wrapper and the instantiated stateful
# wrapper, not the decorated class itself.
vane.attach_function(
    stateless_normalize,
    alias="stateless_normalize_sql",
    connection=con,
    parameters=["VARCHAR"],
)
vane.attach_function(
    cached_normalize,
    alias="cached_normalize_sql",
    connection=con,
    parameters=["VARCHAR"],
)
try:
    sql_result = con.sql("""
        SELECT
            document_id,
            stateless_normalize_sql(text) AS stateless_result,
            cached_normalize_sql(text) AS stateful_result
        FROM source
        ORDER BY document_id
    """)
    sql_result.show()
finally:
    vane.detach_function("cached_normalize_sql", connection=con)
    vane.detach_function("stateless_normalize_sql", connection=con)




# Relation actor reuse: each actor constructs and owns one class instance.
class CachedNormalizerBatch:
    def __init__(self):
        self.cache = {}


    def __call__(self, table):
        document_ids = table.column("document_id").to_pylist()
        values = table.column("text").to_pylist()
        normalized = []
        for value in values:
            if value not in self.cache:
                self.cache[value] = normalize_value(value)
            normalized.append(self.cache[value])
        return pa.table({
            "document_id": document_ids,
            "normalized_text": normalized,
        })




actor_result = source.map_batches(
    CachedNormalizerBatch,
    schema={"document_id": "INTEGER", "normalized_text": "VARCHAR"},
    execution_backend="ray_actor",
    actor_number=1,
    batch_size=1,
    gpus=0.0,
).order("document_id")
actor_result.show()

batch_size=1 forces this small source through several actor calls so the actor can reuse its cache; it is not a throughput recommendation. Registering a stateful SQL alias does not extend its state across queries.

State is not durable, checkpointed, global, keyed, transactional, or exactly-once. Multiple actors do not share memory, distributed input has no stable global row order, and retries may repeat external side effects. Keep state out of result semantics when possible.

Combine the Dimensions

Every UDF can be summarized by answering each question while respecting the valid combinations above:

Example	API surface	Invocation shape	Cardinality	Lifecycle
SQL text normalization alias	SQL Expression	Scalar values	Row-preserving	Stateless
Python cached normalization	Python Expression	Scalar values	Row-preserving	Stateful
Token expansion	Relation	One row (flat_map)	Row-changing	Stateless
Actor-backed model inference	Relation	Arrow batch (map_batches)	Declared by the returned table	Stateful

Edit this page