This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Break PDFs and documents into searchable chunks for retrieval-augmented
generation (RAG) pipelines.
Problem
You have PDF documents or text files that you want to use for
retrieval-augmented generation (RAG). Before you can search them, you
need to:
Split documents into smaller chunks
Generate embeddings for each chunk
Store everything in a searchable index
Solution
What’s in this recipe:
Split PDFs into sentences with token limits
Control chunk size with token limits
Add embeddings for semantic search
You create a view with a document_splitter iterator that automatically
breaks documents into chunks. Then you add an embedding index for
semantic search.
Create a view that splits each document into sentences with a token
limit:
# Create a view that splits documents into chunkschunks = pxt.create_view( 'rag_demo/chunks', docs, iterator=document_splitter( docs.document, separators='sentence,token_limit', # Split by sentence with token limit limit=300, # Max 300 tokens per chunk ),)
Inserting rows into `chunks`: 217 rows [00:00, 42111.88 rows/s]
# View the chunkschunks.select(chunks.text).head(5)
Add semantic search
Create an embedding index on the chunks for similarity search:
# Add embedding index for semantic searchchunks.add_embedding_index( column='text', string_embed=sentence_transformer.using(model_id='all-MiniLM-L6-v2'),)