Skip to content

API reference

Generated from the docstrings of the dynamicio package by mkdocstrings. The reference documents the public classes and functions; names beginning with an underscore are left out.

Core

core

Implements the DynamicDataIO class which provides functionality for data: loading; sinking, and; schema validation.

DynamicDataIO

Given a src.utils.dynamicio.config.IOConfig object, it generates an object with access to a series of methods for cloud I/O operations and data validations.

Example

input_sources_config = IOConfig( "path_to/input.yaml", os.getenv("ENVIRONMENT",default="LOCAL") )

class IO(WithS3File, WithLocal, DynamicDataIO): schema = S

my_dataset_local_mapping = input_config.get(source_key="MY_DATASET") my_dataset_io = IO(my_dataset_local_mapping) my_dataset_df = my_dataset_io.read()

async_read async
async_read()

Allows the use of asyncio to concurrently read files in memory.

Returns:

Type Description
DataFrame | Iterable

A pandas dataframe or an iterable.

read
read() -> Optional[pd.DataFrame]

Reads data source and returns a schema validated dataframe (by means of _apply_schema).

Returns:

Type Description
Optional[DataFrame]

A pandas dataframe or an iterable. None if the source does not exist and missing_ok is True.

async_write async
async_write(df: DataFrame)

Allows the use of asyncio to concurrently write files out.

Parameters:

Name Type Description Default
df DataFrame

The data to be written

required
write
write(df: DataFrame)

Sink data to a given source based on the sources_config.

Parameters:

Name Type Description Default
df DataFrame

The data to be written

required
validate_from_schema
validate_from_schema(df: DataFrame) -> DynamicDataIO

Validates a dataframe based on the validations present in its schema definition.

All validations are checked and if any of them fails, a SchemaValidationError is raised.

Parameters:

Name Type Description Default
df DataFrame

A dataframe to be validated.

required

Returns:

Type Description
DynamicDataIO

self (to allow for method chaining).

Raises:

Type Description
SchemaValidationError

if any of the validations failed. The message attribute of the exception object is a List[str], where each element is the name of a validation that failed.

log_metrics_from_schema
log_metrics_from_schema(df: DataFrame) -> DynamicDataIO

Calculates and logs metrics based on the metrics present in its schema definition.

Parameters:

Name Type Description Default
df DataFrame

A dataframe for which metrics are generated and logged

required

Returns:

Type Description
DynamicDataIO

self (to allow for method chaining).

Resource definitions

io_config

Implements the IOConfig class, generating objects used as a configuration parameter for the instantiation ofsrc.utils.dynamicio.dataio.DynamicDataIO objects.

The IOConfig object, essentially parses a yaml file that contains a set of input sources that will be processed by a task, converting filtering and converting them into dictionaries.

For example, suppose an input.yaml file, containing:

READ_FROM_S3_CSV:
  LOCAL:
    type: "local"
    local:
      file_path: "[[ TEST_RESOURCES ]]/data/input/some_csv_to_read.csv"
      file_type: "csv"
  CLOUD:
    type: "s3"
    s3:
      bucket: "[[ MOCK_BUCKET ]]"
      file_path: "[[ MOCK_KEY ]]"
      file_type: "csv"

would be loaded with:

input_sources_config = IOConfig(
        "path_to/input.yaml",
        env_identifier="CLOUD",
        dynamic_vars=config_module
    )

and:

input_sources_config.config

would return:

    {
        "READ_FROM_S3_CSV": {
            "LOCAL": {
                "type": "local",
                "local": {
                    "file_path": f"{test_global_vars.TEST_RESOURCES}/data/input/some_csv_to_read.csv",
                    "file_type": "csv",
                },
            },
            "CLOUD": {
                "type": "s3",
                "s3": {
                    "bucket": "mock-bucket",
                    "file_path": "mock-key",
                    "file_type": "csv"
                }
            },
        }
    }

SafeDynamicResourceLoader

Bases: SafeLoader

Implements a dynamic yaml loader that parses yaml files and replaces strings that map to [[ DYNAMIC_VAR ]].

Dynamic variables defined in a provided module object.

with_module classmethod
with_module(module: ModuleType)

Creates a dynamic subclass of SafeDynamicLoader with the data_module attribute set to module.

Parameters:

Name Type Description Default
module ModuleType

A global vars module with all the dynamic values defined in it.

required

Returns:

Type Description
type

The new loader subclass.

dyn_str_constructor
dyn_str_constructor(node: ScalarNode) -> str

Responsible for the switching of one or more "[[ DYNAMIC_VAR ]]" strings with the respective attributes value in a given module.

Parameters:

Name Type Description Default
node ScalarNode

Parsed item whose dynamic values that map to the "[[ DYNAMIC_VAR ]]" convention are replaced with the respective attributes in te provided module.

required

Returns:

Type Description
str

Constructed str or numerical.

IOConfig

Generates an object that returns a sub-dictionary of the elements of that yaml file.

The file serves as a config for setting up DynamicDataIO objects. Requires a resources yaml file, an ENVIRONMENT value {CLOUD or LOCAL} and a vars module.

Example

input_sources_config = IOConfig( "path_to/input.yaml", env_identifier="CLOUD", dynamic_vars=config_module )

default_environment property
default_environment: str

The first environment the resource definitions list (conventionally the local one), independent of the env_identifier of the process.

sources property
sources: List[str]

Class property for easy access to a list of sources.

Returns:

Type Description
List[str]

All top level names of the available resources for the used resources yaml config.

instances classmethod
instances() -> List[IOConfig]

Every IOConfig that is alive, so a test harness can select an environment for all of them at once.

get
get(source_key: str) -> IOEnvironment

A getter.

Parameters:

Name Type Description Default
source_key str

The name of the resource for which we want to create a config.

required

Returns:

Type Description
IOEnvironment

A dictionary with the necessary fields for loading the data from a source.

Example:

Given:

    VOYAGE_DATA:
      LOCAL:
        type: "local"
        local:
          file_path: "[[ TEST_RESOURCES ]]/data/processed/voyage_data.parquet"
          file_type: "parquet"
      CLOUD:
        type: "kafka"
        KAFKA:
          KAFKA_SERVER: "[[ KAFKA_SERVER ]]"
          KAFKA_TOPIC: "[[ KAFKA_TOPIC ]]"

If you do:

    input_sources_config = IOConfig(
        "path_to/input.yaml",
        env_identifier="CLOUD",
        dynamic_vars=globals
    )
    voyage_data_cloud_mapping = input_config.get(source_key="VOYAGE_DATA")

then `voyage_data_cloud_mapping` is:

    "KAFKA": {
        "KAFKA_SERVER": "mock-kafka-server",
        "KAFKA_TOPIC": "mock-kafka-topic"
    }

Backends

S3

with_s3

This module provides mixins that are providing S3 I/O support.

InMemStore

Bases: HDFStore

A subclass of pandas HDFStore that does not manage the pytables File object.

is_open property
is_open

Check if the in-memory table is open.

open
open(*_args, **_kwargs)

Open the in-memory table.

close
close(*_args, **_kwargs)

Close the in-memory table.

HdfIO

Provides in-memory stream support for reading and writing HDF5 tables.

Uses PyTables to create in-memory file handles, enabling read/write operations on HDF content without persisting to disk.

create_file
create_file(label: str, mode: str, data: Optional[bytes] = None) -> Generator[tables.File, None, None]

Create an in-memory HDF5 file using PyTables with optional preloaded data.

Parameters:

Name Type Description Default
label str

A label used for naming the temporary in-memory file.

required
mode str

File access mode ('r' for read, 'w' for write).

required
data Optional[bytes]

Raw file data to preload when opening for reading.

None

Yields:

Type Description
File

A PyTables file object representing the HDF5 structure.

load
load(fobj: IO[bytes], label: str = 'unknown_file.h5', options: Optional[Dict] = None) -> Union[DataFrame, Series]

Load a DataFrame or Series from an in-memory HDF5 file-like object.

Parameters:

Name Type Description Default
fobj IO[bytes]

A file-like object containing the HDF5 data.

required
label str

A logical name for the file (used in metadata).

'unknown_file.h5'
options Optional[dict]

Optional keyword arguments to pass to pd.read_hdf.

None

Returns:

Type Description
Union[DataFrame, Series]

The object read from the HDF file.

save
save(df: DataFrame, fobj: IO[bytes], label: str = 'unknown_file.h5', options: Optional[Dict] = None) -> None

Save a DataFrame to a file-like object as an HDF5 structure.

Parameters:

Name Type Description Default
df DataFrame

The DataFrame to store.

required
fobj IO[bytes]

The target file-like object.

required
label str

A logical name used for the in-memory file.

'unknown_file.h5'
options Optional[dict]

Optional keyword arguments to pass to HDFStore.put. You can also include a key (defaults to 'df').

None
Notes

Data is first written to an in-memory PyTables structure, then streamed into the provided file-like object.

S3Keys

List and delete the objects under an S3 prefix, for the overwrite and overwrite_partitions modes of a path prefix write.

list staticmethod
list(source_url: str) -> List[str]

Return the keys of every object under an S3 prefix, in listing (lexicographic key) order.

delete staticmethod
delete(bucket: str, keys: List[str])

Delete objects of a bucket, at most 1000 to a request.

Raises:

Type Description
RuntimeError

If S3 could not delete some of the objects.

S3TransferHandle dataclass

A dataclass used to track an ongoing data download from the s3.

WithS3PathPrefix

Bases: WithLocal

Handles I/O operations for AWS S3 path prefixes (reads, and partitioned parquet writes), using boto3 transfers.

This mixin assumes that the directories it reads from will only contain a single file-type.

Parquet prefixes are read one object at a time, in memory, on the download threads, and concatenated in listing order. Prefixes accumulate objects over years, so an object may predate a column that was added to the schema later: such an object is read with the columns it has, the missing column is null-filled on concatenation, and a warning counts the objects that lacked it. A column that no object has still raises a ValueError. Other file types are unchanged.

boto3_client property
boto3_client

Per-process shared S3 client.

WithS3File

Handles I/O operations for AWS S3 using in-memory streaming for CSV, JSON, Parquet, and HDF files.

For CSV, JSON, and Parquet, AWS Data Wrangler is used for efficient direct reads from S3. For HDF files, content is streamed into memory using boto3 and then loaded via PyTables.

s3_sync_down

s3_sync_down(source_url: str, dest_dir: str, include_pattern: Optional[str] = None, max_concurrency: int = _SYNC_MAX_CONCURRENCY)

Download every object under an S3 prefix into dest_dir, preserving the relative key layout.

One GetObject per file (no HeadObject), issued from a thread pool while the listing is still paginating.

Parameters:

Name Type Description Default
source_url str

s3://bucket/prefix to download from.

required
dest_dir str

Local directory to download into.

required
include_pattern Optional[str]

Optional fnmatch pattern matched against each key relative to the prefix.

None
max_concurrency int

Number of concurrent download threads.

_SYNC_MAX_CONCURRENCY

Raises:

Type Description
BotoCoreError

Whatever boto3 raises if listing or any download fails (a ClientError for a service error).

s3_read_down

s3_read_down(source_url: str, reader: Callable[[BytesIO], Any], include_pattern: Optional[str] = None, max_concurrency: int = _SYNC_MAX_CONCURRENCY) -> List[Any]

Fetch every object under an S3 prefix into memory and parse it with reader, overlapping network and parsing.

Unlike s3_sync_down there is no temp-dir round trip: each worker thread does a GetObject and parses the bytes straight away, so parsing of early files overlaps with the download of later ones.

Parameters:

Name Type Description Default
source_url str

s3://bucket/prefix to read from.

required
reader Callable[[BytesIO], Any]

Called with an in-memory file object for each object.

required
include_pattern Optional[str]

Optional fnmatch pattern matched against each key relative to the prefix.

None
max_concurrency int

Number of concurrent worker threads.

_SYNC_MAX_CONCURRENCY

Returns:

Type Description
List[Any]

The reader results, in listing (lexicographic key) order.

s3_sync_up

s3_sync_up(source_dir: str, dest_url: str, acl: str = 'bucket-owner-full-control', max_concurrency: int = _SYNC_MAX_CONCURRENCY) -> List[str]

Upload every file under source_dir to an S3 prefix, preserving the relative layout.

Parameters:

Name Type Description Default
source_dir str

Local directory to upload.

required
dest_url str

s3://bucket/prefix to upload into.

required
acl str

Canned ACL applied to every uploaded object.

'bucket-owner-full-control'
max_concurrency int

Number of concurrent upload threads.

_SYNC_MAX_CONCURRENCY

Returns:

Type Description
List[str]

The keys of the uploaded objects.

Postgres

with_postgres

This module provides mixins that are providing Postgres I/O support.

WithPostgres

Handles I/O operations for Postgres.

Write options (passed as keyword arguments to write, or set under options in the resource definition):

  • truncate_and_append: bool: If True, truncates the table with CASCADE and appends new rows (faster but destructive). If False (default), safely deletes all existing rows and then appends new rows. Use truncate only if you are certain views and FK constraints can be dropped or tolerated.
  • create_table: bool: If True, creates the table from the schema before writing when it does not exist. The table has no primary key, because the schema does not say which columns form it.
split_table_name staticmethod
split_table_name(table_name: str) -> Tuple[Optional[str], str]

The Postgres schema (None for the default one) and the table of a __table_name__: sales.orders is table orders of schema sales.

session_for

session_for(connection_string: str) -> Generator[SqlAlchemySession, None, None]

Connect to a database using connection_string and returns an active session to that connection.

Parameters:

Name Type Description Default
connection_string str
required

Yields:

Type Description
Session

Active session

Athena

with_athena

This module provides mixins that support AWS Athena I/O.

WithAthena

Handles I/O operations for AWS Athena using AWS Wrangler.

Note

The __abstractmethods__ = frozenset() is used to silence false positives from pylint, which might wrongly assume this class has abstract methods due to NotImplementedError. This class is not an abstract base class and does not use abc.ABC.

Kafka

with_kafka

This module provides mixins that are providing Kafka I/O support.

WithKafka

Handles I/O operations for Kafka.

Options (keyword arguments of write, or options in the resource definition):

  • Standard: keyword arguments passed to the producer constructor (see KafkaProducer.DEFAULT_CONFIG.keys()).
  • key_generator: Callable[[Any, Mapping], T]: defines the keying policy to be used for sending keyed-messages to Kafka. It is a Callable that takes a tuple(idx, row) and returns a string that will serve as the message's key, invoked prior to serialising the key. It defaults to the dataframe's index (which may not be composed of unique values or string type keys). It goes hand in hand with the default key-serialiser, which assumes that the keys are strings and encodes them as such.
  • key_serializer: Callable[T, bytes]: Custom key serialiser; if not provided, a default key-serializer will be used, applied on a string-key (unless key is None). A custom key-generator that generates a non-string key is best provided alongside a custom key-serializer suited to the custom key type.
  • document_transformer: Callable[[Mapping[Any, Any]], Mapping[Any, Any]]: Manipulates the messages/rows sent to Kafka as values. It is a Callable taking a Mapping as its only argument and returning a Mapping, invoked prior to serializing each document. This can be used, for example, to add metadata to each document that will be written to the target Kafka topic.
  • value_serializer: Callable[Mapping, bytes]: Custom value serialiser; if not provided, a default value-serializer will be used, applied on a Mapping.
  • flush_timeout: float: The most seconds that a write waits for Kafka with no progress. A write raises TimeoutError if a full local queue has no room for a message in that time, or if the final flush() delivers no message in that time. Progress is a smaller local queue: an error event of the producer, such as a refused connection, is not progress. If not provided, a write waits until every message is delivered. It is not a deadline for the whole write: a write that keeps delivering messages can take longer.
Example
Given

keyed_test_df = pd.DataFrame.from_records( [ ["key-01", "cm_1", "id_1", 1000, "ABC"], ["key-02", "cm_2", "id_2", 1000, "ABC"], ["key-03", "cm_3", "id_3", 1000, "ABC"], ], columns=["key", "id", "foo", "bar", "baz"], ).set_index("key")

kafka_cloud_config = IOConfig( path_to_source_yaml=(os.path.join(constants.TEST_RESOURCES, "processed.yaml")), env_identifier="CLOUD", dynamic_vars=constants, ).get(source_key="WRITE_TO_KAFKA_JSON")

write_kafka_io = WriteKafkaIO(kafka_cloud_config, key_generator=lambda key, _: key, document_transformer=lambda doc: doc["new_field"]="new_value")

When

with patch.object(mixins, "KafkaProducer") as mock__kafka_producer: mock__kafka_producer.DEFAULT_CONFIG = KafkaProducer.DEFAULT_CONFIG mock_producer = MockKafkaProducer() mock__kafka_producer.return_value = mock_producer write_kafka_io.write(keyed_test_df)

Then

assert mock_producer.my_stream == [ {"key": "key-01", "value": {"bar": 1000, "baz": "ABC", "foo": "id_1", "id": "cm_1", "new_field": "new_value"}}, {"key": "key-02", "value": {"bar": 1000, "baz": "ABC", "foo": "id_2", "id": "cm_2", "new_field": "new_value"}}, {"key": "key-03", "value": {"bar": 1000, "baz": "ABC", "foo": "id_3", "id": "cm_3", "new_field": "new_value"}}, ]

Note

The __abstractmethods__ = frozenset() is used to silence false positives from pylint, which might wrongly assume this class has abstract methods due to NotImplementedError. This class is not an abstract base class and does not use abc.ABC.

populate_cls_attributes
populate_cls_attributes()

Set the dynamicio options (key_generator, document_transformer, key_serializer, value_serializer) from the options, or their defaults.

The options stay in self.options: _write_to_kafka keeps them out of the producer configuration.

Local filesystem

with_local

This module provides mixins that are providing Local FS I/O support.

WithLocal

Handles local I/O operations.

WithLocalBatch

Bases: WithLocal

Responsible for batch reading local files.

Examples:

>>> READ_FROM_BATCH_LOCAL_PARQUET:
>>>   LOCAL:
>>>     type: "local_batch"
>>>     local:
>>>       path_prefix: "[[ TEST_RESOURCES ]]/data/input/batch/parquet/"
>>>       file_type: "parquet"
>>>
>>> READ_FROM_BATCH_LOCAL_TEMPLATED_PARQUET:
>>>   LOCAL:
>>>     type: "local_batch"
>>>     local:
>>>       path_prefix: "[[ TEST_RESOURCES ]]/data/input/{templated}/parquet/"
>>>       file_type: "parquet"
>>>
>>> READ_DYNAMIC_FROM_BATCH_LOCAL_PARQUET:
>>>   LOCAL:
>>>     type: "local_batch"
>>>     local:
>>>       dynamic_file_path: "[[ TEST_RESOURCES ]]/data/input/batch/parquet/dynamic/**/part_{runner_id}.parquet"
>>>       file_type: "parquet"

Utilities

utils

Mixin utility functions.

allow_options

allow_options(options: Union[Iterable[str], Callable])

Decorator to filter **kwargs passed to a function, allowing only valid ones.

Options are filtered per reader or writer: a key that is not in options is dropped with a warning only. A new dynamicio option (one that the target library does not accept, e.g. truncate_and_append) must be added to the list of each reader or writer that reads it, as in [*args_of(pd.HDFStore.put), "pickle_protocol"].

Parameters:

Name Type Description Default
options Union[Iterable[str], Callable]

A list of valid options or a callable that returns a list of valid options.

required

Returns:

Type Description
Callable

A decorator that filters **kwargs passed to the function.

args_of

args_of(*funcs: Callable) -> set[str]

Retrieve a set of accepted keyword arguments from one or more functions.

Parameters:

Name Type Description Default
*funcs Callable

The functions to inspect.

()

Returns:

Type Description
set[str]

A set of accepted keyword arguments.

get_string_template_field_names

get_string_template_field_names(s: str) -> Collection[str]

Given a string s, it parses the string to identify any template fields and returns the names of those fields.

If s is not a string template, the returned Collection is empty.

Parameters:

Name Type Description Default
s str

A string which is either a template, e.g. /path/to/file/{replace_me}.h5 or just a path /path/to/file/dont_replace_me.h5

required

Returns:

Type Description
Collection[str]

Collection[str]

Example

get_string_template_field_names("abc{def}{efg}") ["def", "efg"] get_string_template_field_names("{0}-{1}") ["0", "1"] get_string_template_field_names("hello world") []

resolve_template

resolve_template(path: str, options: MutableMapping[str, Any]) -> str

Given a string path, it attempts to replace all templates fields with values provided in options.

If path is not a string template, path is returned.

Parameters:

Name Type Description Default
path str

A string which is either a template, e.g. /path/to/file/{replace_me}.h5 or just a path /path/to/file/dont_replace_me.h5

required
options MutableMapping[str, Any]

A dynamic name for the "replace_me" field in the templated string. e.g. {"replace_me": "name_of_file"}

required

Returns:

Type Description
str

Returns a static path replaced with the value in the options mapping.

Raises:

Type Description
ValueError

if any template fields in s are not named using valid Python identifiers or if a given template field cannot be resolved in options

pickle_protocol

pickle_protocol(protocol: Optional[int])

Downgrade to the provided pickle protocol within the context manager.

The lock serializes the callers of this context manager only. pickle.HIGHEST_PROTOCOL is process-wide, so another thread that pickles while the context is open also gets the downgraded protocol. Lock order: take this lock before with_local.hdf_lock, never the other way, or two writers deadlock.

Parameters:

Name Type Description Default
protocol Optional[int]

The number of the protocol HIGHEST_PROTOCOL to downgrade to. Defaults to 4, which covers python 3.4 and higher.

required

get_file_type_value

get_file_type_value(file_type: Union[str, Enum]) -> str

Get the value of the file type.

Parameters:

Name Type Description Default
file_type Union[str, Enum]

The file type, which can be a string or an Enum.

required

Returns:

Type Description
str

The value of the file type.

Errors

errors

Hosts exception implementations for different errors.

DynamicIOError

Bases: Exception

Base class for DynamicIO errors.

message property
message: Optional[Any]

Easy access for optional message argument.

Returns:

Type Description
Optional[Any]

Message or None if not set

DatasetNameNotFoundError

Bases: DynamicIOError

Error raised when a model does not name its dataset.

TableNameNotFoundError

Bases: DynamicIOError

Error raised when a model used with a Postgres resource does not name its table.

SchemaNotFoundError

Bases: DynamicIOError

Error raised when schema is not specified in the provided source.

SchemaValidationError

Bases: DynamicIOError

Error raised when schema validation fails.

MissingSchemaDefinition

Bases: DynamicIOError

Error raised when schema is not specified in the provided source.

DataSourceError

Bases: DynamicIOError

Error raised when the data source fails to load.

ColumnsDataTypeError

Bases: DynamicIOError

Error raised when the validated data does not have the expected data types.

NonUniqueIdColumnError

Bases: DynamicIOError

Error raised when the data source fails to load.

NullValueInColumnError

Bases: DynamicIOError

Error raised when the data source fails to load.

NotExpectedCategoricalValue

Bases: DynamicIOError

Error raised when the data source fails to load.

InvalidDatasetTypeError

Bases: DynamicIOError

Error raised when dataset type is not one of [parquet, json, csv, h5].

NoObjectsFoundError

Bases: FileNotFoundError, ValueError

Error raised when an S3 path prefix, an S3 parquet file_path that ends in /, or a local_batch source has no objects.

It is a FileNotFoundError, so a caller can treat it as a missing source. It is also a ValueError: in earlier releases, a path prefix or a local_batch source with no objects raised the pandas ValueError: No objects to concatenate, and callers that catch ValueError must not change.

ObjectNotFoundError

Bases: FileNotFoundError, NoFilesFound

Error raised when the S3 key of a csv or json file_path does not exist.

It is a FileNotFoundError, so a caller can treat it as a missing source. It is also an awswrangler NoFilesFound: earlier releases raised NoFilesFound for a missing key, and callers that catch NoFilesFound must not change.

S3KeyNotFoundError

Bases: FileNotFoundError, ClientError

Error raised when the S3 key of a parquet or hdf file_path does not exist.

It is a FileNotFoundError, so a caller can treat it as a missing source. It is also the botocore ClientError that S3 returned: earlier releases raised that ClientError for a missing key, and callers that catch ClientError or read its response must not change.

is_missing_key classmethod
is_missing_key(error: ClientError) -> bool

Whether error is the S3 reply for a key that does not exist.

SchemaNotDeclaredError

Bases: DynamicIOError, TypeError, AssertionError

Error raised when an IO class does not set schema.

It is a TypeError, as the other errors of a wrong IO class declaration are. It is also an AssertionError: earlier releases checked this with assert, and callers that catch AssertionError must not change.

MissingMixinError

Bases: DynamicIOError, TypeError, AssertionError

Error raised when an IO class has no mixin that reads or writes the type of its resource.

It is a TypeError, as the other errors of a wrong IO class declaration are. It is also an AssertionError: earlier releases checked this with assert, and callers that catch AssertionError must not change.

InvalidS3UrlError

Bases: DynamicIOError, ValueError, AssertionError

Error raised when an S3 operation gets a URL that does not start with s3://.

It is a ValueError. It is also an AssertionError: earlier releases checked this with assert, and callers that catch AssertionError must not change.

Test harness

testharness

A harness for testing a pipeline task end to end against the resource definitions it already uses.

A task writes its outputs through dynamicio. In a LOCAL environment those land on disk, so a test can run the task and then compare every output with the reference file declared next to it, under the optional expected key of the resource definition::

FINAL_BAR:
  LOCAL:
    type: "local"
    local:
      file_path: "[[ TEST_RESOURCES ]]/data/processed/final_bar.parquet"
      file_type: "parquet"
    expected:
      path: "[[ TEST_RESOURCES ]]/data/processed/expected/final_bar.parquet"

The pieces, one class per module:

  • :class:EnvironmentSwitch: runs a test in an explicitly chosen environment of the resource definitions, instead of patching env_identifier.
  • :class:ExpectedOutputs: the session. Collects every mismatch, raises one error, cleans up, and can regenerate the expected files.
  • :class:Expectations: the curated rules of a comparison (order, dtypes, tolerances, ignored columns, custom assertions), strict or relaxed.
  • :class:FrameComparison: applies the rules to two frames.
  • :class:OutputFiles: finds, reads, removes and promotes the generated and expected files of a resource.
  • :class:OutputMismatchError: the error that reports all failed checks at once.
  • :class:OutputSandbox: redirects the local outputs of a task into a scratch directory, so tests that write the same files can run in parallel.
  • :class:SessionReport: the account of what a session verified, printed for passing runs too (by the pytest plugin of this package).
  • :class:ParquetSchemaComparison: compares the arrow schema and pandas index of two parquet files (Expectations(check_parquet_schema=True)).

Example::

with ExpectedOutputs(Expectations(ignore_columns=["run_id"])) as outputs:
    staging.main()
    outputs.check(raw_config, "STAGED_FOO")
    outputs.check(processed_config, "FINAL_BAR", list_columns_to_sort=["tags"], rtol=1e-3)

FrameAssertion module-attribute

FrameAssertion = Callable[[pd.DataFrame, pd.DataFrame], None]

A custom check (actual, expected) that raises AssertionError when the output is not acceptable.

EnvironmentSwitch

Context manager that makes configs select environment and restores their own on exit.

A task picks the resource definitions of its config's environment, so a test that needs a variant (LOCAL_REMOVED_UPDATED, LOCAL_E2E ...) must switch the configs the task uses. :class:ExpectedOutputs does this for every IOConfig that exists, so you rarely need this class; use it to switch only some configs, for example when you do not check outputs::

with EnvironmentSwitch("LOCAL_E2E", input_config, output_config):
    Task().run()

Parameters:

Name Type Description Default
environment Optional[str]

The environment name, as it appears in the resource definitions. None selects the default environment of each config (the first one its resource definitions list).

required
configs IOConfig

The IOConfig objects to switch.

()

Raises:

Type Description
ValueError

On entering, if a config defines no resource in environment (usually a typo).

of_all classmethod
of_all(environment: Optional[str]) -> EnvironmentSwitch

A switch of every IOConfig that is alive; those that do not define environment are left alone, unless none does.

available staticmethod
available(config: IOConfig) -> set

The environment names the resource definitions of config define.

Expectations dataclass

The curated rules of a comparison. Immutable: derive stricter or more relaxed variants with :meth:with_.

The defaults are what a pipeline test usually wants: the same data, whatever the order of rows and columns, dtypes included.

Attributes:

Name Type Description
check_row_order bool

Require rows in the same order (otherwise they are aligned by index label, or sorted by sort_by).

check_column_order bool

Require columns in the same order.

check_dtype bool

Require identical dtypes (int64 is not float64).

check_exact bool

Require exactly equal floats (otherwise rtol / atol apply).

rtol float

Relative tolerance for floats.

atol float

Absolute tolerance for floats.

sort_by Tuple[str, ...]

Columns to sort both frames by before comparing, for outputs without a meaningful index. Only used when check_row_order is False.

only_columns Optional[Tuple[str, ...]]

Compare just these columns.

ignore_columns Tuple[str, ...]

Leave these columns out of the comparison (timestamps, run ids ...).

list_columns_to_sort Tuple[str, ...]

List-valued columns whose element order does not matter.

assertions Tuple[FrameAssertion, ...]

Extra checks that run after the frames matched, e.g. lambda actual, expected: assert actual["price"].min() >= 0.

check_parquet_schema bool

For parquet outputs (other types are skipped), also require the same arrow schema and pandas index as the expected file (the loaded frames can hide a difference in nullability, timestamp units or a stored index). Applied by ExpectedOutputs.check.

strict classmethod
strict() -> Expectations

Rows and columns in order, dtypes and floats exact.

lenient classmethod
lenient() -> Expectations

Values only: dtypes may differ (int64 vs float64) and floats are compared with a looser tolerance.

with_
with_(**changes) -> Expectations

A copy with the given rules changed. Sequences are accepted for the tuple-valued rules.

Raises:

Type Description
TypeError

If a rule does not exist.

also_assert
also_assert(*assertions: FrameAssertion) -> Expectations

A copy that additionally runs assertions.

ExpectedOutputs

Lazily compares the outputs a task generated locally with the expected files declared in their resource definitions.

Use it as a context manager to raise the collected errors on exit, or call :meth:raise_if_failed yourself.

Parameters:

Name Type Description Default
expectations Optional[Expectations]

The default rules of every check (see :class:Expectations); a single check can refine them.

None
environment Optional[str]

The environment of the resource definitions, e.g. "LOCAL_E2E". Inside the with block every IOConfig that defines it selects it, so the task under test runs in it too, and the checks compare its outputs; the configs get their own environment back on exit. It defaults to the first environment each resource definition lists (conventionally the local one), whatever the process runs in. A single check can override it.

None
configs Optional[Sequence[IOConfig]]

Switch only these IOConfig objects instead of every one that exists.

None
keep_files bool

Keep the generated files even when they matched (by default they are removed, so the next run starts clean).

False
update_expected Optional[bool]

Overwrite the expected files with the generated outputs instead of comparing. Defaults to the DYNAMICIO_UPDATE_EXPECTED env var.

None

Attributes:

Name Type Description
errors List[Tuple[str, str]]

(name, message) of each failed check.

passed List[str]

Names of the checks that matched.

updated List[str]

Names of the checks whose expected file was rewritten.

skipped List[str]

Resources :meth:check_all left out because their paths have {placeholders} that no params fill; check those with check_each.

report property
report: SessionReport

What this session verified so far: the outputs that matched (with their size), failed, were updated or skipped.

read_generated staticmethod
read_generated(config: IOConfig, source_key: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> pd.DataFrame

The output the task generated for source_key.

read_expected staticmethod
read_expected(config: IOConfig, source_key: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> pd.DataFrame

The reference output declared for source_key.

clear staticmethod
clear(config: IOConfig, *source_keys: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> None

Remove leftovers of earlier runs (generated files of source_keys) so a stale file can not pass for a fresh output.

clear_all staticmethod
clear_all(config: IOConfig, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> None

:meth:clear every output of config that declares an expected path (those with unfilled {placeholders} are left alone).

combinations staticmethod
combinations(grid: Mapping[str, Iterable[Any]]) -> List[Mapping[str, Any]]

Every combination of the grid values, as params dicts.

clear_each classmethod
clear_each(config: IOConfig, source_key: str, grid: Mapping[str, Iterable[Any]], environment: Optional[str] = None) -> None

:meth:clear the generated file of source_key for every combination of the grid values.

compare
compare(actual: DataFrame, expected: DataFrame, name: str, expectations: Optional[Expectations] = None) -> bool

Compare two frames, recording the failure under name instead of raising. Returns whether they matched.

check
check(config: IOConfig, source_key: str, *, label: Optional[str] = None, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool

Compare the generated output of source_key with its expected file and record any mismatch.

On a match the generated file is removed (unless keep_files); on a mismatch it is left in place to be inspected. A missing generated or expected file is recorded as a failure too, so one broken output never hides the others. With update_expected the generated file replaces the expected one.

Parameters:

Name Type Description Default
config IOConfig

The IOConfig (in its LOCAL-style environment) the task wrote the output with.

required
source_key str

The resource name.

required
label Optional[str]

Name in the report (default source_key, with params appended when given).

None
params Optional[Mapping[str, Any]]

Values for {placeholders} in the generated and expected paths.

None
environment Optional[str]

The environment to check instead of the session's.

None
expectations Optional[Expectations]

Rules for this check instead of the session's.

None
**rules Any

Individual :class:Expectations fields to change for this check only, e.g. ignore_columns=["updated_at"].

{}

Returns:

Type Description
bool

Whether the output matched (or was updated).

check_all
check_all(config: IOConfig, *, skip: Sequence[str] = (), rules_for: Optional[Mapping[str, Mapping[str, Any]]] = None, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool

Run :meth:check for every resource of config that declares an expected path, so a new output is checked without touching the test.

Resources whose paths have {placeholders} that params does not fill are left out and listed in skipped; check those with check_each.

Example: outputs.check_all(stage_output_config, rules_for={"STAGED_CARGOES": {"list_columns_to_sort": ["product_id"]}}, skip=["STAGED_RAW"]).

Parameters:

Name Type Description Default
config IOConfig

The IOConfig the task wrote the outputs with.

required
skip Sequence[str]

Resources not to check.

()
rules_for Optional[Mapping[str, Mapping[str, Any]]]

For a resource, the :class:Expectations fields to change for its check only.

None
params Optional[Mapping[str, Any]]

Values for {placeholders} in the paths of every resource.

None
environment Optional[str]

The environment to check instead of the session's.

None
expectations Optional[Expectations]

Rules for these checks instead of the session's.

None
**rules Any

Individual :class:Expectations fields to change for every check.

{}

Returns:

Type Description
bool

Whether every check matched.

check_each
check_each(config: IOConfig, source_key: str, grid: Mapping[str, Iterable[Any]], *, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool

Run :meth:check for every combination of the grid values, for a resource whose path has several {placeholders}.

Example: outputs.check_each(config, "PART", {"table": ["events", "cargoes"], "shard": range(4)}) checks 8 files and reports each by its own params.

Parameters:

Name Type Description Default
config IOConfig

The IOConfig the task wrote the output with.

required
source_key str

The resource name.

required
grid Mapping[str, Iterable[Any]]

For each placeholder, the values it takes.

required
environment Optional[str]

The environment to check instead of the session's.

None
expectations Optional[Expectations]

Rules for these checks instead of the session's.

None
**rules Any

Individual :class:Expectations fields to change for these checks only.

{}

Returns:

Type Description
bool

Whether every combination matched.

raise_if_failed
raise_if_failed() -> None

Raise one :class:OutputMismatchError listing every recorded failure.

FrameComparison

Applies the rules of expectations to a generated and an expected frame and raises AssertionError on the first difference found.

prepare
prepare(df: DataFrame) -> pd.DataFrame

The frame as the comparison sees it: columns selected, list columns and rows ordered.

compare
compare(actual: DataFrame, expected: DataFrame) -> None

Raise AssertionError if actual does not satisfy the expectations against expected.

OutputFiles

The files of source_key in the environment config selects: what the task generated, and what it is expected to generate.

Parameters:

Name Type Description Default
config IOConfig

The IOConfig (in its LOCAL-style environment) the task wrote the output with.

required
source_key str

The resource name.

required
params Optional[Mapping[str, Any]]

Values for {placeholders} in the generated and expected paths, for outputs whose file name carries a runtime parameter such as a date.

None
environment Optional[str]

The environment to look the resource up in; by default the first one the resource definitions list.

None
resource property
resource

The definition of the resource in the environment under test.

placeholders property
placeholders: List[str]

The {placeholders} of the paths that params does not fill.

environment_name property
environment_name: str

The environment looked up: the selected one, else the first one the resource definitions list.

generated_path property
generated_path: str

The local path the task writes the resource to.

expected_path property
expected_path: str

The path of the reference file declared under expected.

is_parquet property
is_parquet: bool

Whether the resource is written as parquet.

read_options property
read_options: Mapping[str, Any]

Reader options for both files: the JSON orient and lines the resource is written with, overridden by expected.options.

declaring_expected staticmethod
declaring_expected(config: IOConfig, environment: Optional[str] = None) -> List[str]

The resources of config that declare an expected path in the environment: the outputs a test can check.

read_file staticmethod
read_file(path: str, file_type: str, **options: Any) -> pd.DataFrame

Read a file with plain pandas, so no schema or cast hides a difference. options go to the reader (orient and lines only to JSON).

read_generated
read_generated() -> pd.DataFrame

The output the task generated.

read_expected
read_expected() -> pd.DataFrame

The reference output.

generated_exists
generated_exists() -> bool

Whether the task wrote its output.

expected_exists
expected_exists() -> bool

Whether the reference file exists.

remove_generated
remove_generated() -> None

Delete the generated file if it is there.

promote_generated
promote_generated() -> None

Make the generated file the new reference (creating the expected directory if needed).

Raises:

Type Description
ValueError

If the reference is declared in another file type than the output, as copying would leave a reference the checks cannot read.

OutputMismatchError

Bases: AssertionError

Raised once, after all checks ran, listing every output that did not match its expected counterpart.

Attributes:

Name Type Description
errors List[Tuple[str, str]]

(name, message) for each failed check, in the order the checks ran.

passed List[str]

Names of the checks that matched, listed in the message so a failure also shows what is healthy.

OutputSandbox

Context manager that points every local output of configs below directory and restores the original paths on exit.

A resource at /data/out/foo.parquet is written to <directory>/data/out/foo.parquet (its directory is created); {placeholders} are kept. The expected files are not moved, and ExpectedOutputs follows the redirected path, so a test reads like one without a sandbox. This makes tests that write the same outputs independent: they can run in parallel (pytest-xdist) and leave nothing behind in the repository.

Pass the output configs of the task under test only: an input config would be redirected too, and the task would no longer find its inputs.

Parameters:

Name Type Description Default
directory Union[str, PathLike]

Where the outputs go, typically pytest's tmp_path.

required
configs IOConfig

The IOConfig objects (in their LOCAL-style environment) the task writes with.

()
redirected
redirected(file_path: str) -> str

The path file_path is redirected to.

ParquetSchemaComparison

The arrow schema (names, types, nullability, without the key-value metadata) and the pandas index columns of two parquet files.

differences staticmethod
differences(actual_path: str, expected_path: str) -> List[str]

Describe how the layout of the file at actual_path differs from the one at expected_path (empty when they agree).

compare classmethod
compare(actual_path: str, expected_path: str) -> None

Raise AssertionError listing the differences, if any.

SessionReport

Lists the outputs a session matched, failed, updated and left out, one line each.

Which outputs a test verifies can be decided at runtime (check_all looks them up in the resource definitions), so a green run says which ones they were. Sessions record their report in :attr:recorded, and the pytest plugin of this package prints them at the end of the run.

Parameters:

Name Type Description Default
matched Mapping[str, str]

Name of each output that matched, with a short description of it.

required
failed Sequence[Tuple[str, str]]

(name, message) of each output that did not match.

required
updated Sequence[str]

Names of the outputs whose expected file was rewritten.

required
skipped Sequence[str]

Names of the outputs check_all left out.

required
test str

The test that ran the session (PYTEST_CURRENT_TEST), when known.

''
is_empty property
is_empty: bool

Whether the session checked nothing.

title property
title: str

The headline: the counts, and the test.

lines
lines() -> List[Tuple[str, str]]

(status, text) per output, where status is matched, failed, updated or skipped, in the order of the checks.

text
text() -> str

The plain-text report.