
dynamicio¶
Keep business logic independent of I/O. dynamicio is the seam between your pandas code and the places your data
lives. Declare where a dataset is in YAML, declare what it looks like as a Pandera
model, and your code calls read() and write(). The same code then runs on a laptop against sample files and in
production against S3, PostgreSQL, Athena or Kafka.
pandas only
Dataframe execution supports pandas. Polars is not supported.
The point¶
Developers should not have to think about connection strings, parquet engines, JSON orientations or which bucket a task reads this week. That code is not the logic they are paid to get right, and when it is mixed in with the logic, neither can change without the other.
flowchart LR
B[Business logic<br/>DataFrame in, DataFrame out] --> I[IO class<br/>the seam<br/>read / write + schema]
I --> L[LOCAL<br/>sample files]
I --> D[DEV<br/>S3, Postgres, Athena]
I --> P[PROD<br/>S3, Postgres, Kafka]
Switching LOCAL to DEV to PROD is a change of one identifier, not of code. Read Why dynamicio
for the full argument.
What you get¶
- Logic that does not know about storage. One call site per dataset, whatever the storage or format.
- Fast, safe feedback. Run the whole pipeline locally on sample files, with no credentials and no mocks of external services, then check the outputs against expected files.
- A loop that suits coding agents. An agent that changes a pipeline needs a place to run it that cannot hurt
production, and a check that says whether the change was right. A
LOCALenvironment and the test harness provide both. - Data contracts. Columns, dtypes, validations and metrics live in one reviewed Pandera model. A bad frame fails at the boundary with every violation listed.
- Environments as configuration. The same task runs locally, in development and in production.
- Metrics in your logs. Distributions are logged as
METRICJSON lines.
Supported sources and destinations¶
| Source | Read | Write | Formats |
|---|---|---|---|
Local files (local, local_batch) |
yes | yes | parquet, hdf, json, csv |
S3 object (s3) and prefix (s3_path_prefix) |
yes | yes | parquet, hdf, json, csv |
| PostgreSQL | yes | yes | tables |
| AWS Athena | yes | no | SQL query results |
| Kafka | no | yes | messages |
Details are in Supported sources and formats.
Install¶
Python 3.10 to 3.14 is supported. See Installation.
A minimal example¶
The quickstart builds this end to end and runs without any cloud credentials:
from dynamicio import UnifiedIO
from dynamicio.config import IOConfig
class OrdersIO(UnifiedIO):
schema = Orders # a pandera.DataFrameModel
config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)
orders.write(df)
df = orders.read()
Where next¶
- Why dynamicio, Quickstart and Concepts
- Guides: reading, writing, schemas and validation, error handling
- API reference
- Changelog, source code, PyPI, issues
The original logo was illustrated by Nick Loucas; the refreshed logo keeps its circular I/O motif.