pandas
The essential Python library for data manipulation — DataFrames, filtering, grouping, merging, cleaning. pandas turns messy data into structured tables you can analyze. The foundation of Python data science. Used by analysts, data scientists, and engineers.
- Data analysis and cleaning.
- CSV/Excel/SQL data manipulation.
- Python data pipelines.
- Prototyping data workflows.
- Big data (>RAM — use Spark/Dask).
- Production pipelines (use Polars/Dask).
- Non-tabular data.
Definition
pandas provides DataFrame (a table) and Series (a column) data structures. Load data from CSV, JSON, SQL, Excel. Filter, group, merge, pivot, clean (missing data, types), and transform. Integrates with NumPy, Matplotlib, scikit-learn.
Why it exists
Raw Python (lists, dicts) is tedious for tabular data. pandas provides a DataFrame — a familiar table interface with powerful operations for data cleaning and analysis.
How it works
pd.read_csv('data.csv') loads a DataFrame. df['col'] selects a column. df[df['age'] > 25] filters. df.groupby('cat').mean() aggregates. df.merge() joins. df.to_csv() saves. Chaining operations transforms data.
Scaling limits
Memory-bound (loads data into RAM). Slow on very large datasets (use Dask/Polars for scale). Not for big data (>RAM). Verbose for complex operations.
Cost shape
Free and open source.
Alternatives
- Polars — Faster, Rust-based, less mature ecosystem.
- Dask — Pandas API, parallel, for big data.
- SQL — For databases, not in-memory analysis.
Failure modes
- Memory limits.
- Slow on big data.
- Chained assignment bugs.
- Verbose API.