Process files, images, audio, and video with native types, SQL, and Python UDFs. Preserve business IDs and source references, then write text and embeddings to your retrieval and analytics systems.
messy materials
outputs
Today that chain is stitched across OCR scripts, temp files, model calls, SQL jobs, and review tools.
⚠ scattered systems · glue code everywhere · lost source references · hard to debug · poor reproducibility
Start with FILE, IMAGEFILE, AUDIOFILE, or VIDEOFILE references. Combine media operators, SQL, and Python UDFs; parsing, OCR, and transcription use the libraries and models you configure.
Select business IDs, source URIs, and segment timestamps alongside text and embeddings. Keep model classifications separate from review status so downstream users can check the source.
Use distributed upsert sinks for Milvus and Qdrant, or Arrow Stream Load writes to Doris. Configure the destination schema, credentials, and embedding dimensions for your system.
Adapted from the tutorial, this SQL constructs FILE and AUDIOFILE references, calls the tutorial’s transcription UDF, and embeds timestamped segments. The full tutorial supplies Python UDF registration, model setup, candidate classification, and DorisStreamLoadSink ingestion.
-- Adapted excerpt: the tutorial registers transcribe_audio in prepare.py. WITH recordings AS ( SELECT regexp_extract(f.url, '([^/]+)[.]wav$', 1) AS ticket_id, audio_file(file(f.url, NULL, NULL, NULL, NULL)) AS audio FROM list_files('./recordings/*.wav') AS f ), transcripts AS ( SELECT ticket_id, audio.url AS source_uri, transcribe_audio(audio) AS segments_json FROM recordings ), segments AS ( SELECT ticket_id, source_uri, unnest(from_json(segments_json, '[{"segment_id":"INTEGER","text":"VARCHAR","start_ms":"BIGINT","end_ms":"BIGINT"}]' )) AS segment FROM transcripts ) SELECT ticket_id, segment.segment_id AS segment_id, segment.text AS text, source_uri, segment.start_ms AS start_ms, segment.end_ms AS end_ms, ai_embed( segment.text, provider := 'transformers', model := 'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2', dimensions := 384, on_error := 'raise', options := struct_pack(normalize := true, actor_number := 1, batch_size := 16) ) AS embedding FROM segments WHERE length(trim(segment.text)) > 0;