This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Pixeltable’s VLM Run integration lets you use VLM Run’s multimodal AI
capabilities directly from tables: chat completions, image generation,
image editing, annotation, and video generation, without leaving your
data workflows.
What is VLM Run?
VLM Run is a multimodal AI platform with an OpenAI-compatible chat
completions API and specialized toolsets for:
Understanding: Analyze images, documents, and video with natural
language
Image generation: Create new images from text prompts
Image editing: Edit existing images with natural language
instructions
Visualization: Annotate images with bounding boxes, keypoints, and
segmentation masks
Important: VLM Run API calls consume credits based on your plan, so
monitor your usage to avoid unexpected charges.
%pip install -qU pixeltable vlmrun
import getpassimport osif 'VLMRUN_API_KEY' not in os.environ: os.environ['VLMRUN_API_KEY'] = getpass.getpass('VLM Run API Key: ')
To read more about working with API keys in Pixeltable, see
Configuration.
Setup
import pixeltable as pxtfrom pixeltable.functions import vlmrunMODEL = 'vlmrun-orion-2:auto'pxt.create_dir('vlmrun_demo', if_exists='replace_force')
Chat completions
The vlmrun.chat_completions UDF provides direct access to VLM Run’s
chat completions API. It accepts an OpenAI-compatible messages list
and returns the raw API response as a dictionary.
Text-only messages
t = pxt.create_table( 'vlmrun_demo.text_chat', {'prompt': pxt.String}, if_exists='replace_force',)messages = [{'role': 'user', 'content': t.prompt}]t.add_computed_column( response=vlmrun.chat_completions(messages, model=MODEL))t.add_computed_column( answer=t.response['choices'][0]['message']['content'])t.insert(prompt='What is structured data extraction?')t.select(t.prompt, t.answer).collect()
Created table ‘text_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 17.14 s (0.06 rows/s)
prompt
0 What is structured data extraction? answer
0 \n\nStructured data extraction is the proc…
Image understanding
Reference image columns directly in message content. Referenced media is
uploaded to VLM Run automatically before each API call.
Created table ‘image_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 15.18 s (0.07 rows/s)
Document understanding
Process PDFs and other documents by referencing a document column via
file_url, combined with the document toolset.
Created table ‘doc_chat’.
Added 0 column values with 0 errors in 0.01 s
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 18.07 s (0.06 rows/s)
Models
VLM Run offers three Orion 2 tiers:See the VLM Run documentation for the full list
of models.
Image generation
Use vlmrun.generate_image to create images from text prompts. It
returns a PIL.Image.Image that Pixeltable stores as a native
pxt.Image column.
t = pxt.create_table( 'vlmrun_demo.image_gen', {'prompt': pxt.String}, if_exists='replace_force',)t.add_computed_column(image=vlmrun.generate_image(t.prompt, model=MODEL))t.insert(prompt='A futuristic cityscape at sunset with flying cars')t.select(t.prompt, t.image).collect()
Created table ‘image_gen’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 138.98 s (0.01 rows/s)
Image editing
Pass an image column via the image parameter to generate_image to
edit an existing image according to your prompt.
EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg't = pxt.create_table( 'vlmrun_demo.image_edit', {'image': pxt.Image}, if_exists='replace_force',)t.add_computed_column( blurred=vlmrun.generate_image( 'Blur all the faces in this image', image=t.image, model=MODEL ))t.insert(image=EXAMPLE_IMAGE)t.select(t.image, t.blurred).collect()
Created table ‘image_edit’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 86.00 s (0.01 rows/s)
Image annotation (visualization)
Use vlmrun.annotate_image to overlay bounding boxes, keypoints, or
segmentation masks on an image using VLM Run’s viz toolset. Unlike
generate_image, the input image is required, since you always need an
image to annotate.
EXAMPLE_IMAGE = 'https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg't = pxt.create_table( 'vlmrun_demo.annotation', {'image': pxt.Image}, if_exists='replace_force',)t.add_computed_column( annotated=vlmrun.annotate_image( 'Draw bounding boxes around all people', t.image, model=MODEL ))t.insert(image=EXAMPLE_IMAGE)t.select(t.image, t.annotated).collect()
Created table ‘annotation’.
Added 0 column values with 0 errors in 0.01 s
Inserted 1 row with 0 errors in 368.13 s (0.00 rows/s)
Video generation
Use vlmrun.generate_video to create videos from text prompts. It
returns a pxt.Video stored natively in Pixeltable.Note: Video generation can take a minute or more, and document
generation over ten minutes. Each VLM Run call waits up to 1800 seconds;
to change that, set VLMRUN_TIMEOUT, or timeout in the vlmrun
section of the config file.
t = pxt.create_table( 'vlmrun_demo.video_gen', {'prompt': pxt.String}, if_exists='replace_force',)t.add_computed_column(video=vlmrun.generate_video(t.prompt, model=MODEL))t.insert( prompt='Create a 3 second long video of a red dot bouncing across the screen')t.select(t.prompt, t.video).collect()
Document transformation (redaction)
Use vlmrun.generate_document to transform a document and get the
edited PDF back as a pxt.Document, for example, redacting PII at
scale. Being explicit that you want the rendered PDF (not a text
summary) produces the best results.Note: very long documents can exceed the model’s input-token limit;
for multi-page PDFs, consider splitting into pages first.
SAMPLE_PDF = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/tests/data/documents/Vector_database.pdf't = pxt.create_table( 'vlmrun_demo.redaction', {'document': pxt.Document}, if_exists='replace_force',)t.add_computed_column( redacted=vlmrun.generate_document( 'Produce a redacted version of this PDF: draw black redaction boxes over all person names, ' 'keeping the rest visually identical. Return the rendered PDF file.', t.document, model=MODEL, ))t.insert(document=SAMPLE_PDF)t.select(t.document, t.redacted).collect()
UDF reference
All UDFs require model and accept model_kwargs for additional API
parameters.