This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Create a searchable knowledge base that finds content by meaning, not
just keywords.
Problem
You have a collection of text content (articles, notes, documentation)
and need to find relevant items based on meaning.Keyword search fails when users phrase queries differently from the
source text:
Solution
What’s in this recipe:
Create a text table with embeddings
Search by semantic similarity
Combine with metadata filters
You add an embedding index to your text column. Pixeltable automatically
generates embeddings for each row and enables similarity search.
Setup
%pip install -qU pixeltable sentence-transformers
Create knowledge base
import pixeltable as pxtfrom pixeltable.functions.huggingface import sentence_transformer# Create a fresh directorypxt.drop_dir('search_demo', force=True)pxt.create_dir('search_demo')
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory ‘search_demo’.
<pixeltable.catalog.dir.Dir at 0x14208ca10>
# Create table with content and metadatakb = pxt.create_table( 'search_demo/articles', {'title': pxt.String, 'content': pxt.String, 'category': pxt.String},)
Created table ‘articles’.
# Insert sample contentkb.insert( [ { 'title': 'Debugging best practices', 'content': 'Use logging, breakpoints, and unit tests to identify and fix issues in your code.', 'category': 'engineering', }, { 'title': 'Machine learning model optimization', 'content': 'Improve training efficiency with batch normalization, learning rate schedules, and early stopping.', 'category': 'ml', }, { 'title': 'Production infrastructure setup', 'content': 'Deploy applications using containers, load balancers, and automated scaling.', 'category': 'devops', }, { 'title': 'API design principles', 'content': 'Create RESTful endpoints with proper versioning, authentication, and error handling.', 'category': 'engineering', }, ])