This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Generate vector embeddings for text data to enable semantic search and
similarity matching.This notebook uses pxt.create_table() so you can run cells without a
project. An app puts the same columns on a TableModel in app.py and
creates the tables with pxt schema update.
Problem
You need to convert text into vector embeddings for:
Semantic search (find similar documents)
RAG pipelines (retrieve relevant context)
Clustering and classification
Solution
What’s in this recipe:
Generate embeddings with OpenAI’s models
Store embeddings as computed columns
Use embeddings for similarity queries
You add an embedding column that automatically generates vectors for new
rows. The embeddings are cached and only recomputed when the source text
changes.
Setup
%pip install -qU pixeltable openai
import getpassimport osif 'OPENAI_API_KEY' not in os.environ: os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key: ')
import pixeltable as pxtfrom pixeltable.functions.openai import embeddings# Create a fresh directorypxt.drop_dir('embed_demo', force=True)pxt.create_dir('embed_demo')
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory ‘embed_demo’.
<pixeltable.catalog.dir.Dir at 0x14ee4fcd0>
# Add embedding column using OpenAI's text-embedding-3-smalldocs.add_computed_column( embedding=embeddings(docs.content, model='text-embedding-3-small'))
Added 0 column values with 0 errors.
No rows affected.
Insert documents
# Insert sample documentssample_docs = [ { 'title': 'Python Basics', 'content': 'Python is a high-level programming language known for its clear syntax and readability.', }, { 'title': 'Machine Learning', 'content': 'Machine learning is a subset of AI that enables systems to learn from data.', }, { 'title': 'Web Development', 'content': 'Web development involves building websites and web applications using HTML, CSS, and JavaScript.', }, { 'title': 'Data Science', 'content': 'Data science combines statistics, programming, and domain expertise to extract insights from data.', }, { 'title': 'Cloud Computing', 'content': 'Cloud computing provides on-demand computing resources over the internet.', },]docs.insert(sample_docs)