Training Data Pipeline

This page follows the three canonical text-processing scripts in the Vane repository:

• examples/common_crawl.py
• examples/minhash_dedupe.py
• examples/llms_red_pajamas.py

Together they cover web-text extraction and chunking, near-duplicate removal, embeddings, and semantic matching.

Create a clean environment

After installing uv as described in Installation, sparsely clone only the example scripts into a new directory:

shell

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/AstroVela/vane.git vane-examples
cd vane-examples
git sparse-checkout set examples


uv venv --python 3.12
source .venv/bin/activate
uv pip install vane-ai numpy pyarrow
uv pip install torch
uv pip install sentence-transformers transformers

The sparse checkout excludes the vane/ and duckdb/ source directories, so the Python process and local Ray workers use the installed wheel. The model packages are needed by the Common Crawl and Red Pajamas embedding stages. minhash_dedupe.py does not need them. For GPU acceleration, install a CUDA-enabled PyTorch build compatible with the operating system, GPU, and driver; use PyTorch Start Locally when the default package is not the right build. Confirm that python -c "import torch; print(torch.cuda.is_available())" prints True; Vane's Transformers provider then requests one Ray GPU automatically. The first embedding run downloads sentence-transformers/all-MiniLM-L6-v2; use --local-files-only only after that model is available in the local cache or at the model path you pass.

If PyPI downloads are slow, append the Tsinghua mirror to an install command:

shell

uv pip install torch -i https://pypi.tuna.tsinghua.edu.cn/simple

The mirror changes where packages are downloaded from; it does not choose a compatible CUDA build for the machine.

Run all three samples

Run the scripts from the repository root:

shell

python examples/common_crawl.py --source sample --limit 5
python examples/minhash_dedupe.py --source sample --limit 10
python examples/llms_red_pajamas.py --source sample --limit 6

With the current built-in samples, the scripts report:

Script	Verification result
common_crawl.py	5 source records, 3 decoded English pages, 7 text chunks, 384-dimensional embeddings
minhash_dedupe.py	10 input rows and 5 retained representatives
llms_red_pajamas.py	6 embedded questions and 3 semantic matches

The exact terminal table formatting can change, but the scripts validate row counts before combining source rows with model output.

Common Crawl workflow

common_crawl.py performs these stages:

1. Load built-in WARC-shaped rows, a local WET/WARC file, or a WET URL.
2. Keep WARC-Type = conversion records.
3. Decode content and parse language metadata with DecodeWarcBatch.
4. Keep the requested language and split text with ChunkTextBatch.
5. Call vane.ai.embed_text with the Transformers provider.
6. Write filtered pages, chunks, and embeddings.

Default output files are written under examples/output/common_crawl/:

• filtered_pages.csv
• chunks.csv
• chunk_embeddings.csv
• chunk_embeddings.parquet

You can validate extraction and chunking without installing a model runtime:

shell

uv pip install vane-ai numpy pyarrow
python examples/common_crawl.py \
  --source sample \
  --limit 5 \
  --skip-embeddings

Use a real local or remote WET source with one of these commands:

shell

python examples/common_crawl.py \
  --source wet-file \
  --wet-path /path/to/sample.warc.wet.gz \
  --limit 100


python examples/common_crawl.py \
  --source wet-url \
  --wet-url https://example.org/sample.warc.wet.gz \
  --limit 100

MinHash deduplication workflow

minhash_dedupe.py normalizes each text block, computes deterministic MinHash signatures, creates LSH candidate pairs, verifies candidates with exact Jaccard similarity by default, builds connected components, and keeps one representative per component.

Its default output directory, examples/output/minhash_dedupe/, contains annotated rows, retained rows, duplicates, clusters, candidate pairs, and colliding LSH buckets.

For a CSV input:

shell

python examples/minhash_dedupe.py \
  --source csv \
  --csv-path /path/to/documents.csv \
  --text-column text \
  --id-column document_id \
  --limit 1000

For local HTML files:

shell

python examples/minhash_dedupe.py \
  --source html-glob \
  --html-glob '/path/to/pages/*.html' \
  --limit 1000

Red Pajamas semantic matching workflow

llms_red_pajamas.py embeds StackExchange-style question text, separates low-score queries from higher-score candidates, computes cosine similarity, and prints the top matches. To persist the result:

shell

python examples/llms_red_pajamas.py \
  --source sample \
  --limit 6 \
  --output-csv examples/output/red_pajamas_matches.csv

The real source mode reads the Red Pajamas StackExchange JSONL path configured by the script:

shell

python examples/llms_red_pajamas.py \
  --source redpajama \
  --limit 1000

Default Ray execution

The scripts create Vane relations without calling vane.configure(...) and without passing execution_backend. Ray is already the default runner. Built-in sample rows are represented as DuckDB VALUES relations so the plan can travel to Ray workers; they are not captured as process-local Arrow scans.

Set RAY_ADDRESS before launching a script to use an existing cluster. Every worker must have the same Vane and Transformers packages and access to the same model cache or model path. No code change is needed.

Production adaptation

• Keep source loading separate from transformation stages.
• Keep every batch UDF schema explicit.
• Preserve stable source IDs through chunking and deduplication.
• Validate row counts before appending embedding or prompt output.
• Use bounded inputs first, then increase --limit and tune the existing --batch-size options.
• Treat the files under examples/ as the implementation contract; do not copy older backend or runner flags from external snippets.