A pxt.Document column takes a URL as readily as a local path. Insert the URL and Pixeltable
fetches and parses the page, so there is no separate scraping step to keep in sync with the table.
Insert the URL
Chunk it, two ways
The chunking strategy decides what a retrieval hit looks like, so it is worth seeing the difference
before committing to one. A view per strategy, over the same table:
On that one page: 152 paragraph chunks against 117 fixed-size chunks.
paragraph follows the document’s own structure, so a chunk is a complete thought and a retrieval
hit reads as prose. Chunk length is whatever the author wrote, which on a page with one-line
paragraphs produces chunks too small to carry context. char_limit gives you predictable sizes and
predictable embedding cost, and cuts mid-sentence.
Valid separators are heading, paragraph, sentence, token_limit, char_limit, and page.
sentence needs spacy and its default English model: run pip install spacy, then
python -m spacy download en_core_web_sm. token_limit needs pip install tiktoken.
Combine separators with a comma, most structural first:
That splits on headings, then splits any section still over 300 tokens, which keeps chunks inside a
model’s context window without cutting across two topics.
Make it searchable
Declare an index on the chunk text and query it with similarity():
The index loads with the chunks already there and updates as you insert more URLs.
Notes
- Insert several URLs and every view and index below the table extends to them. There is no
per-page bookkeeping.
sentence_transformer needs pip install sentence-transformers, and downloads the model on
first use.pxt.create_table(), create_view(), and add_embedding_index() are the notebook and test
form. An application declares the tables and views on a TableModel in app.py, puts the index
in __indexes__, and creates them with pxt schema update.
Last modified on September 23, 2026
