This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
The VLM Run Gateway is an
OpenAI-compatible API in front of vision and speech models: OCR and
document parsing, visual question answering, segmentation, pose
estimation, multimodal embeddings and speech-to-text. Pixeltable’s
vlmrun_gateway module turns each of these into a computed column, so
images, PDFs and audio in your tables are processed as rows are
inserted.
Gateway usage is billed per token, image, video or audio second, with
a minimum of $0.001 per call on most models (not embeddings or the
partner Gemini, Kimi, MiniMax and Muse models)
(pricing).
chat_completions() reports each call’s cost in usage.cost, as does
transcriptions() in its JSON formats.
Be mindful of sensitive data and consider security measures when
integrating with external services.
First you’ll need to install the required libraries and enter your VLM
Run API key.
%pip install -qU pixeltable openai
import getpassimport osif 'VLMRUN_API_KEY' not in os.environ: os.environ['VLMRUN_API_KEY'] = getpass.getpass( 'Enter your VLM Run API key:' )
Now let’s create a Pixeltable directory to hold the tables for our demo.
import jsonimport pixeltable as pxtfrom pixeltable.functions import vlmrun_gateway# Remove the 'vlmrun_gateway_demo' directory and its contents, if it existspxt.drop_dir('vlmrun_gateway_demo', force=True)pxt.create_dir('vlmrun_gateway_demo')SAMPLES = ( 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples')
Created directory ‘vlmrun_gateway_demo’.
Read PDFs with OCR
ocr() sends a document to an OCR model and returns the Gateway’s JSON,
whose pages list has one record per page. The default model,
paddlepaddle/paddleocr-vl-1.6, reads each page as Markdown. Use
pages to read part of a long document. Pages are rasterized at 96 DPI
by default; dense tables, like the line items on this invoice, can come
back garbled at that resolution, so this example raises dpi to 150.
docs = pxt.create_table('vlmrun_gateway_demo/docs', {'doc': pxt.Document})docs.add_computed_column(pages=vlmrun_gateway.ocr(docs.doc, dpi=150))# The first page's text, as Markdowndocs.add_computed_column( first_page=docs.pages.pages[0].content['items'][0].text)
Created table ‘docs’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.
Inserted 1 row with 0 errors in 2.68 s (0.37 rows/s)
To get text lines with bounding boxes instead, pick a model and method
that return regions, such as model='paddleocr/pp-ocrv6', method='ocr'.
Each model’s methods are listed in the Gateway
docs.
Extract structured data from images
chat_completions() works like the OpenAI endpoint of the same name.
Image columns can be referenced directly in messages. With
response_format set to a JSON schema, the model’s reply matches your
schema.
receipts = pxt.create_table( 'vlmrun_gateway_demo/receipts', {'image': pxt.Image})messages = [ { 'role': 'user', 'content': [ { 'type': 'text', 'text': 'Extract the merchant and the total.', }, {'type': 'image_url', 'image_url': receipts.image}, ], }]receipt_schema = { 'type': 'object', 'properties': { 'merchant': {'type': 'string'}, 'total': {'type': 'number'}, }, 'required': ['merchant', 'total'],}receipts.add_computed_column( response=vlmrun_gateway.chat_completions( messages, model='qwen/qwen3.8-27b', model_kwargs={ 'response_format': { 'type': 'json_schema', 'json_schema': { 'name': 'receipt', 'schema': receipt_schema, }, } }, ))# The reply is a JSON string; parse it into a JSON column@pxt.udfdef parse_json(s: str) -> pxt.Json: return json.loads(s)receipts.add_computed_column( receipt=parse_json(receipts.response.choices[0].message.content))
Created table ‘receipts’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
No rows affected.
Inserted 1 row with 0 errors in 2.62 s (0.38 rows/s)
Segment objects
analyze() runs detection, segmentation and pose models. Here
facebook/sam3.1 finds every object that matches a text prompt. Boxes
are normalized to the range 0 to 1, and 'mask_format': 'none' in
method_params skips the masks. The bridge photo has no cars, so its
list is empty.
Created table ‘photos’.
Added 0 column values with 0 errors in 0.01 s
Inserted 2 rows with 0 errors in 0.80 s (2.51 rows/s)
Search images with text
embeddings() puts text and images in one vector space, so an embedding
index on an image column can be searched with a text query.
photos.add_embedding_index( photos.image, embedding=vlmrun_gateway.embeddings.using( model='qwen/qwen3-vl-embedding-2b' ),)sim = photos.image.similarity(string='a bridge over the water')photos.order_by(sim, asc=False).limit(2).select( photos.image, sim).collect()
Transcribe audio
transcriptions() turns speech in an audio column into text, with
nvidia/parakeet-tdt-0.6b-v3 by default. The sample here is a 10-second
clip, so the transcript ends mid-sentence. For timestamps, pass
model_kwargs={'response_format': 'verbose_json', 'timestamp_granularities': ['segment', 'word']}.
Created table ‘audio’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 0.73 s (1.38 rows/s)
Rate limits
The Gateway allows 240 requests per minute and 10,000 per hour per
account, below Pixeltable’s default of 600 RPM. To stay under both, set
rate_limit = 160 in the vlmrun_gateway section of your Pixeltable
config, or set the VLMRUN_GATEWAY_RATE_LIMIT environment variable.