This documentation page is also available as an interactive notebook. You can launch the notebook in
Kaggle or Colab, or download it for use with an IDE or local Jupyter installation, by clicking one of the
above links.
Load columnar data from Parquet files into Pixeltable tables for
processing and analysis.
Problem
You have data stored in Parquet format—a common format for analytics,
data lakes, and ML pipelines. You need to load this data for processing
with AI models or combining with other data sources.
Solution
What’s in this recipe:
Import Parquet files directly into tables
Export tables to Parquet for external tools
Handle schema type overrides
You use pxt.create_table() with a source parameter to create a table
from a Parquet file. Pixeltable infers column types from the Parquet
schema automatically.
Setup
%pip install -qU pixeltable pyarrow pandas
Create sample Parquet file
First, create a sample Parquet file to demonstrate the import process:
Use create_table with the source parameter to create a table
directly from the Parquet file:
# Create a fresh directorypxt.drop_dir('parquet_demo', force=True)pxt.create_dir('parquet_demo')
Connected to Pixeltable database at: postgresql+psycopg://postgres:@/pixeltable?host=/Users/pjlb/.pixeltable/pgdata
Created directory ‘parquet_demo’.
<pixeltable.catalog.dir.Dir at 0x17f0ca920>
# Import Parquet file into a new tableproducts = pxt.create_table( 'parquet_demo/products', source=str(parquet_path))
Created table ‘products’.Inserting rows into `products`: 0 rows [00:00, ? rows/s]
Inserting rows into `products`: 5 rows [00:00, 653.18 rows/s]
Inserted 5 rows with 0 errors.
# View imported dataproducts.collect()
Add computed columns
Once imported, you can add computed columns like any other Pixeltable
table:
# Add a computed column for discounted priceproducts.add_computed_column(sale_price=products.price * 0.9)