API reference¶
Generated from the docstrings of the dynamicio package by mkdocstrings. The
reference documents the public classes and functions; names beginning with an underscore are left out.
Core¶
core ¶
Implements the DynamicDataIO class which provides functionality for data: loading; sinking, and; schema validation.
DynamicDataIO ¶
Given a src.utils.dynamicio.config.IOConfig object, it generates an object with access to a series of methods for cloud I/O operations and data validations.
Example
input_sources_config = IOConfig( "path_to/input.yaml", os.getenv("ENVIRONMENT",default="LOCAL") )
class IO(WithS3File, WithLocal, DynamicDataIO): schema = S
my_dataset_local_mapping = input_config.get(source_key="MY_DATASET") my_dataset_io = IO(my_dataset_local_mapping) my_dataset_df = my_dataset_io.read()
async_read
async
¶
Allows the use of asyncio to concurrently read files in memory.
Returns:
| Type | Description |
|---|---|
DataFrame | Iterable
|
A pandas dataframe or an iterable. |
read ¶
Reads data source and returns a schema validated dataframe (by means of _apply_schema).
Returns:
| Type | Description |
|---|---|
Optional[DataFrame]
|
A pandas dataframe or an iterable. None if the source does not exist and |
async_write
async
¶
Allows the use of asyncio to concurrently write files out.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data to be written |
required |
write ¶
Sink data to a given source based on the sources_config.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data to be written |
required |
validate_from_schema ¶
Validates a dataframe based on the validations present in its schema definition.
All validations are checked and if any of them fails, a SchemaValidationError is raised.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A dataframe to be validated. |
required |
Returns:
| Type | Description |
|---|---|
DynamicDataIO
|
self (to allow for method chaining). |
Raises:
| Type | Description |
|---|---|
SchemaValidationError
|
if any of the validations failed. The |
log_metrics_from_schema ¶
Calculates and logs metrics based on the metrics present in its schema definition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
A dataframe for which metrics are generated and logged |
required |
Returns:
| Type | Description |
|---|---|
DynamicDataIO
|
self (to allow for method chaining). |
Resource definitions¶
io_config ¶
Implements the IOConfig class, generating objects used as a configuration parameter for the instantiation ofsrc.utils.dynamicio.dataio.DynamicDataIO objects.
The IOConfig object, essentially parses a yaml file that contains a set of input sources that will be processed by a
task, converting filtering and converting them into dictionaries.
For example, suppose an input.yaml file, containing:
READ_FROM_S3_CSV:
LOCAL:
type: "local"
local:
file_path: "[[ TEST_RESOURCES ]]/data/input/some_csv_to_read.csv"
file_type: "csv"
CLOUD:
type: "s3"
s3:
bucket: "[[ MOCK_BUCKET ]]"
file_path: "[[ MOCK_KEY ]]"
file_type: "csv"
would be loaded with:
input_sources_config = IOConfig(
"path_to/input.yaml",
env_identifier="CLOUD",
dynamic_vars=config_module
)
and:
input_sources_config.config
would return:
{
"READ_FROM_S3_CSV": {
"LOCAL": {
"type": "local",
"local": {
"file_path": f"{test_global_vars.TEST_RESOURCES}/data/input/some_csv_to_read.csv",
"file_type": "csv",
},
},
"CLOUD": {
"type": "s3",
"s3": {
"bucket": "mock-bucket",
"file_path": "mock-key",
"file_type": "csv"
}
},
}
}
SafeDynamicResourceLoader ¶
Bases: SafeLoader
Implements a dynamic yaml loader that parses yaml files and replaces strings that map to [[ DYNAMIC_VAR ]].
Dynamic variables defined in a provided module object.
with_module
classmethod
¶
Creates a dynamic subclass of SafeDynamicLoader with the data_module attribute set to module.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module
|
ModuleType
|
A global vars module with all the dynamic values defined in it. |
required |
Returns:
| Type | Description |
|---|---|
type
|
The new loader subclass. |
dyn_str_constructor ¶
Responsible for the switching of one or more "[[ DYNAMIC_VAR ]]" strings with the respective attributes value in a given module.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
ScalarNode
|
Parsed item whose dynamic values that map to the "[[ DYNAMIC_VAR ]]" convention are replaced with the respective attributes in te provided module. |
required |
Returns:
| Type | Description |
|---|---|
str
|
Constructed |
IOConfig ¶
Generates an object that returns a sub-dictionary of the elements of that yaml file.
The file serves as a config for setting up DynamicDataIO objects. Requires a resources yaml file, an ENVIRONMENT value {CLOUD or LOCAL} and a vars module.
Example
input_sources_config = IOConfig( "path_to/input.yaml", env_identifier="CLOUD", dynamic_vars=config_module )
default_environment
property
¶
The first environment the resource definitions list (conventionally the local one), independent of the env_identifier of the process.
sources
property
¶
Class property for easy access to a list of sources.
Returns:
| Type | Description |
|---|---|
List[str]
|
All top level names of the available resources for the used resources yaml config. |
instances
classmethod
¶
Every IOConfig that is alive, so a test harness can select an environment for all of them at once.
get ¶
A getter.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_key
|
str
|
The name of the resource for which we want to create a config. |
required |
Returns:
| Type | Description |
|---|---|
IOEnvironment
|
A dictionary with the necessary fields for loading the data from a source. |
Example:
Given:
VOYAGE_DATA:
LOCAL:
type: "local"
local:
file_path: "[[ TEST_RESOURCES ]]/data/processed/voyage_data.parquet"
file_type: "parquet"
CLOUD:
type: "kafka"
KAFKA:
KAFKA_SERVER: "[[ KAFKA_SERVER ]]"
KAFKA_TOPIC: "[[ KAFKA_TOPIC ]]"
If you do:
input_sources_config = IOConfig(
"path_to/input.yaml",
env_identifier="CLOUD",
dynamic_vars=globals
)
voyage_data_cloud_mapping = input_config.get(source_key="VOYAGE_DATA")
then `voyage_data_cloud_mapping` is:
"KAFKA": {
"KAFKA_SERVER": "mock-kafka-server",
"KAFKA_TOPIC": "mock-kafka-topic"
}
Backends¶
S3¶
with_s3 ¶
This module provides mixins that are providing S3 I/O support.
InMemStore ¶
HdfIO ¶
Provides in-memory stream support for reading and writing HDF5 tables.
Uses PyTables to create in-memory file handles, enabling read/write operations on HDF content without persisting to disk.
create_file ¶
create_file(label: str, mode: str, data: Optional[bytes] = None) -> Generator[tables.File, None, None]
Create an in-memory HDF5 file using PyTables with optional preloaded data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
label
|
str
|
A label used for naming the temporary in-memory file. |
required |
mode
|
str
|
File access mode ('r' for read, 'w' for write). |
required |
data
|
Optional[bytes]
|
Raw file data to preload when opening for reading. |
None
|
Yields:
| Type | Description |
|---|---|
File
|
A PyTables file object representing the HDF5 structure. |
load ¶
load(fobj: IO[bytes], label: str = 'unknown_file.h5', options: Optional[Dict] = None) -> Union[DataFrame, Series]
Load a DataFrame or Series from an in-memory HDF5 file-like object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fobj
|
IO[bytes]
|
A file-like object containing the HDF5 data. |
required |
label
|
str
|
A logical name for the file (used in metadata). |
'unknown_file.h5'
|
options
|
Optional[dict]
|
Optional keyword arguments to pass to |
None
|
Returns:
| Type | Description |
|---|---|
Union[DataFrame, Series]
|
The object read from the HDF file. |
save ¶
save(df: DataFrame, fobj: IO[bytes], label: str = 'unknown_file.h5', options: Optional[Dict] = None) -> None
Save a DataFrame to a file-like object as an HDF5 structure.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The DataFrame to store. |
required |
fobj
|
IO[bytes]
|
The target file-like object. |
required |
label
|
str
|
A logical name used for the in-memory file. |
'unknown_file.h5'
|
options
|
Optional[dict]
|
Optional keyword arguments to pass to |
None
|
Notes
Data is first written to an in-memory PyTables structure, then streamed into the provided file-like object.
S3Keys ¶
List and delete the objects under an S3 prefix, for the overwrite and overwrite_partitions modes of a path prefix write.
S3TransferHandle
dataclass
¶
A dataclass used to track an ongoing data download from the s3.
WithS3PathPrefix ¶
Bases: WithLocal
Handles I/O operations for AWS S3 path prefixes (reads, and partitioned parquet writes), using boto3 transfers.
This mixin assumes that the directories it reads from will only contain a single file-type.
Parquet prefixes are read one object at a time, in memory, on the download threads, and concatenated in listing
order. Prefixes accumulate objects over years, so an object may predate a column that was added to the schema
later: such an object is read with the columns it has, the missing column is null-filled on concatenation, and a
warning counts the objects that lacked it. A column that no object has still raises a ValueError. Other file
types are unchanged.
WithS3File ¶
Handles I/O operations for AWS S3 using in-memory streaming for CSV, JSON, Parquet, and HDF files.
For CSV, JSON, and Parquet, AWS Data Wrangler is used for efficient direct reads from S3. For HDF files, content is streamed into memory using boto3 and then loaded via PyTables.
s3_sync_down ¶
s3_sync_down(source_url: str, dest_dir: str, include_pattern: Optional[str] = None, max_concurrency: int = _SYNC_MAX_CONCURRENCY)
Download every object under an S3 prefix into dest_dir, preserving the relative key layout.
One GetObject per file (no HeadObject), issued from a thread pool while the listing is still paginating.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_url
|
str
|
|
required |
dest_dir
|
str
|
Local directory to download into. |
required |
include_pattern
|
Optional[str]
|
Optional |
None
|
max_concurrency
|
int
|
Number of concurrent download threads. |
_SYNC_MAX_CONCURRENCY
|
Raises:
| Type | Description |
|---|---|
BotoCoreError
|
Whatever boto3 raises if listing or any download fails (a |
s3_read_down ¶
s3_read_down(source_url: str, reader: Callable[[BytesIO], Any], include_pattern: Optional[str] = None, max_concurrency: int = _SYNC_MAX_CONCURRENCY) -> List[Any]
Fetch every object under an S3 prefix into memory and parse it with reader, overlapping network and parsing.
Unlike s3_sync_down there is no temp-dir round trip: each worker thread does a GetObject and parses the bytes
straight away, so parsing of early files overlaps with the download of later ones.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_url
|
str
|
|
required |
reader
|
Callable[[BytesIO], Any]
|
Called with an in-memory file object for each object. |
required |
include_pattern
|
Optional[str]
|
Optional |
None
|
max_concurrency
|
int
|
Number of concurrent worker threads. |
_SYNC_MAX_CONCURRENCY
|
Returns:
| Type | Description |
|---|---|
List[Any]
|
The |
s3_sync_up ¶
s3_sync_up(source_dir: str, dest_url: str, acl: str = 'bucket-owner-full-control', max_concurrency: int = _SYNC_MAX_CONCURRENCY) -> List[str]
Upload every file under source_dir to an S3 prefix, preserving the relative layout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_dir
|
str
|
Local directory to upload. |
required |
dest_url
|
str
|
|
required |
acl
|
str
|
Canned ACL applied to every uploaded object. |
'bucket-owner-full-control'
|
max_concurrency
|
int
|
Number of concurrent upload threads. |
_SYNC_MAX_CONCURRENCY
|
Returns:
| Type | Description |
|---|---|
List[str]
|
The keys of the uploaded objects. |
Postgres¶
with_postgres ¶
This module provides mixins that are providing Postgres I/O support.
WithPostgres ¶
Handles I/O operations for Postgres.
Write options (passed as keyword arguments to write, or set under options in the resource definition):
truncate_and_append: bool: If True, truncates the table with CASCADE and appends new rows (faster but destructive). If False (default), safely deletes all existing rows and then appends new rows. Use truncate only if you are certain views and FK constraints can be dropped or tolerated.create_table: bool: If True, creates the table from the schema before writing when it does not exist. The table has no primary key, because the schema does not say which columns form it.
split_table_name
staticmethod
¶
The Postgres schema (None for the default one) and the table of a __table_name__: sales.orders is table orders of schema sales.
session_for ¶
Connect to a database using connection_string and returns an active session to that connection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection_string
|
str
|
|
required |
Yields:
| Type | Description |
|---|---|
Session
|
Active session |
Athena¶
with_athena ¶
This module provides mixins that support AWS Athena I/O.
WithAthena ¶
Handles I/O operations for AWS Athena using AWS Wrangler.
Note
The __abstractmethods__ = frozenset() is used to silence false positives from pylint,
which might wrongly assume this class has abstract methods due to NotImplementedError.
This class is not an abstract base class and does not use abc.ABC.
Kafka¶
with_kafka ¶
This module provides mixins that are providing Kafka I/O support.
WithKafka ¶
Handles I/O operations for Kafka.
Options (keyword arguments of write, or options in the resource definition):
- Standard: keyword arguments passed to the producer constructor (see
KafkaProducer.DEFAULT_CONFIG.keys()). key_generator: Callable[[Any, Mapping], T]: defines the keying policy to be used for sending keyed-messages to Kafka. It is aCallablethat takes atuple(idx, row)and returns a string that will serve as the message's key, invoked prior to serialising the key. It defaults to the dataframe's index (which may not be composed of unique values or string type keys). It goes hand in hand with the defaultkey-serialiser, which assumes that the keys are strings and encodes them as such.key_serializer: Callable[T, bytes]: Custom key serialiser; if not provided, a default key-serializer will be used, applied on a string-key (unless key is None). A custom key-generator that generates a non-string key is best provided alongside a custom key-serializer suited to the custom key type.document_transformer: Callable[[Mapping[Any, Any]], Mapping[Any, Any]]: Manipulates the messages/rows sent to Kafka as values. It is aCallabletaking aMappingas its only argument and returning aMapping, invoked prior to serializing each document. This can be used, for example, to add metadata to each document that will be written to the target Kafka topic.value_serializer: Callable[Mapping, bytes]: Custom value serialiser; if not provided, a default value-serializer will be used, applied on a Mapping.flush_timeout: float: The most seconds that a write waits for Kafka with no progress. A write raisesTimeoutErrorif a full local queue has no room for a message in that time, or if the finalflush()delivers no message in that time. Progress is a smaller local queue: an error event of the producer, such as a refused connection, is not progress. If not provided, a write waits until every message is delivered. It is not a deadline for the whole write: a write that keeps delivering messages can take longer.
Example
Given¶
keyed_test_df = pd.DataFrame.from_records( [ ["key-01", "cm_1", "id_1", 1000, "ABC"], ["key-02", "cm_2", "id_2", 1000, "ABC"], ["key-03", "cm_3", "id_3", 1000, "ABC"], ], columns=["key", "id", "foo", "bar", "baz"], ).set_index("key")
kafka_cloud_config = IOConfig( path_to_source_yaml=(os.path.join(constants.TEST_RESOURCES, "processed.yaml")), env_identifier="CLOUD", dynamic_vars=constants, ).get(source_key="WRITE_TO_KAFKA_JSON")
write_kafka_io = WriteKafkaIO(kafka_cloud_config, key_generator=lambda key, _: key, document_transformer=lambda doc: doc["new_field"]="new_value")
When¶
with patch.object(mixins, "KafkaProducer") as mock__kafka_producer: mock__kafka_producer.DEFAULT_CONFIG = KafkaProducer.DEFAULT_CONFIG mock_producer = MockKafkaProducer() mock__kafka_producer.return_value = mock_producer write_kafka_io.write(keyed_test_df)
Then¶
assert mock_producer.my_stream == [ {"key": "key-01", "value": {"bar": 1000, "baz": "ABC", "foo": "id_1", "id": "cm_1", "new_field": "new_value"}}, {"key": "key-02", "value": {"bar": 1000, "baz": "ABC", "foo": "id_2", "id": "cm_2", "new_field": "new_value"}}, {"key": "key-03", "value": {"bar": 1000, "baz": "ABC", "foo": "id_3", "id": "cm_3", "new_field": "new_value"}}, ]
Note
The __abstractmethods__ = frozenset() is used to silence false positives from pylint,
which might wrongly assume this class has abstract methods due to NotImplementedError.
This class is not an abstract base class and does not use abc.ABC.
populate_cls_attributes ¶
Set the dynamicio options (key_generator, document_transformer, key_serializer, value_serializer) from the options, or their defaults.
The options stay in self.options: _write_to_kafka keeps them out of the producer configuration.
Local filesystem¶
with_local ¶
This module provides mixins that are providing Local FS I/O support.
WithLocal ¶
Handles local I/O operations.
WithLocalBatch ¶
Bases: WithLocal
Responsible for batch reading local files.
Examples:
>>> READ_FROM_BATCH_LOCAL_PARQUET:
>>> LOCAL:
>>> type: "local_batch"
>>> local:
>>> path_prefix: "[[ TEST_RESOURCES ]]/data/input/batch/parquet/"
>>> file_type: "parquet"
>>>
>>> READ_FROM_BATCH_LOCAL_TEMPLATED_PARQUET:
>>> LOCAL:
>>> type: "local_batch"
>>> local:
>>> path_prefix: "[[ TEST_RESOURCES ]]/data/input/{templated}/parquet/"
>>> file_type: "parquet"
>>>
>>> READ_DYNAMIC_FROM_BATCH_LOCAL_PARQUET:
>>> LOCAL:
>>> type: "local_batch"
>>> local:
>>> dynamic_file_path: "[[ TEST_RESOURCES ]]/data/input/batch/parquet/dynamic/**/part_{runner_id}.parquet"
>>> file_type: "parquet"
Utilities¶
utils ¶
Mixin utility functions.
allow_options ¶
Decorator to filter **kwargs passed to a function, allowing only valid ones.
Options are filtered per reader or writer: a key that is not in options is dropped with a warning only.
A new dynamicio option (one that the target library does not accept, e.g. truncate_and_append) must be
added to the list of each reader or writer that reads it, as in [*args_of(pd.HDFStore.put), "pickle_protocol"].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
options
|
Union[Iterable[str], Callable]
|
A list of valid options or a callable that returns a list of valid options. |
required |
Returns:
| Type | Description |
|---|---|
Callable
|
A decorator that filters **kwargs passed to the function. |
args_of ¶
Retrieve a set of accepted keyword arguments from one or more functions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*funcs
|
Callable
|
The functions to inspect. |
()
|
Returns:
| Type | Description |
|---|---|
set[str]
|
A set of accepted keyword arguments. |
get_string_template_field_names ¶
Given a string s, it parses the string to identify any template fields and returns the names of those fields.
If s is not a string template, the returned Collection is empty.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
s
|
str
|
A string which is either a template, e.g. /path/to/file/{replace_me}.h5 or just a path /path/to/file/dont_replace_me.h5 |
required |
Returns:
| Type | Description |
|---|---|
Collection[str]
|
Collection[str] |
Example
get_string_template_field_names("abc{def}{efg}") ["def", "efg"] get_string_template_field_names("{0}-{1}") ["0", "1"] get_string_template_field_names("hello world") []
resolve_template ¶
Given a string path, it attempts to replace all templates fields with values provided in options.
If path is not a string template, path is returned.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
A string which is either a template, e.g. /path/to/file/{replace_me}.h5 or just a path /path/to/file/dont_replace_me.h5 |
required |
options
|
MutableMapping[str, Any]
|
A dynamic name for the "replace_me" field in the templated string. e.g. {"replace_me": "name_of_file"} |
required |
Returns:
| Type | Description |
|---|---|
str
|
Returns a static path replaced with the value in the options mapping. |
Raises:
| Type | Description |
|---|---|
ValueError
|
if any template fields in s are not named using valid Python identifiers or if a given template field cannot be resolved in |
pickle_protocol ¶
Downgrade to the provided pickle protocol within the context manager.
The lock serializes the callers of this context manager only. pickle.HIGHEST_PROTOCOL is process-wide, so another thread that pickles while the
context is open also gets the downgraded protocol. Lock order: take this lock before with_local.hdf_lock, never the other way, or two writers deadlock.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
protocol
|
Optional[int]
|
The number of the protocol HIGHEST_PROTOCOL to downgrade to. Defaults to 4, which covers python 3.4 and higher. |
required |
get_file_type_value ¶
Get the value of the file type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_type
|
Union[str, Enum]
|
The file type, which can be a string or an Enum. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The value of the file type. |
Errors¶
errors ¶
Hosts exception implementations for different errors.
DynamicIOError ¶
Bases: Exception
Base class for DynamicIO errors.
message
property
¶
Easy access for optional message argument.
Returns:
| Type | Description |
|---|---|
Optional[Any]
|
Message or |
DatasetNameNotFoundError ¶
TableNameNotFoundError ¶
Bases: DynamicIOError
Error raised when a model used with a Postgres resource does not name its table.
SchemaNotFoundError ¶
SchemaValidationError ¶
MissingSchemaDefinition ¶
DataSourceError ¶
ColumnsDataTypeError ¶
NonUniqueIdColumnError ¶
NullValueInColumnError ¶
NotExpectedCategoricalValue ¶
InvalidDatasetTypeError ¶
NoObjectsFoundError ¶
Bases: FileNotFoundError, ValueError
Error raised when an S3 path prefix, an S3 parquet file_path that ends in /, or a local_batch source has no objects.
It is a FileNotFoundError, so a caller can treat it as a missing source. It is also a ValueError: in earlier releases, a path prefix
or a local_batch source with no objects raised the pandas ValueError: No objects to concatenate, and callers that catch ValueError must not change.
ObjectNotFoundError ¶
Bases: FileNotFoundError, NoFilesFound
Error raised when the S3 key of a csv or json file_path does not exist.
It is a FileNotFoundError, so a caller can treat it as a missing source. It is also an awswrangler NoFilesFound: earlier releases raised
NoFilesFound for a missing key, and callers that catch NoFilesFound must not change.
S3KeyNotFoundError ¶
Bases: FileNotFoundError, ClientError
Error raised when the S3 key of a parquet or hdf file_path does not exist.
It is a FileNotFoundError, so a caller can treat it as a missing source. It is also the botocore ClientError that S3 returned: earlier
releases raised that ClientError for a missing key, and callers that catch ClientError or read its response must not change.
is_missing_key
classmethod
¶
Whether error is the S3 reply for a key that does not exist.
SchemaNotDeclaredError ¶
Bases: DynamicIOError, TypeError, AssertionError
Error raised when an IO class does not set schema.
It is a TypeError, as the other errors of a wrong IO class declaration are. It is also an AssertionError: earlier releases checked this with
assert, and callers that catch AssertionError must not change.
MissingMixinError ¶
Bases: DynamicIOError, TypeError, AssertionError
Error raised when an IO class has no mixin that reads or writes the type of its resource.
It is a TypeError, as the other errors of a wrong IO class declaration are. It is also an AssertionError: earlier releases checked this with
assert, and callers that catch AssertionError must not change.
InvalidS3UrlError ¶
Bases: DynamicIOError, ValueError, AssertionError
Error raised when an S3 operation gets a URL that does not start with s3://.
It is a ValueError. It is also an AssertionError: earlier releases checked this with assert, and callers that catch AssertionError must not change.
Test harness¶
testharness ¶
A harness for testing a pipeline task end to end against the resource definitions it already uses.
A task writes its outputs through dynamicio. In a LOCAL environment those land on disk, so a test can run the task and then compare every output
with the reference file declared next to it, under the optional expected key of the resource definition::
FINAL_BAR:
LOCAL:
type: "local"
local:
file_path: "[[ TEST_RESOURCES ]]/data/processed/final_bar.parquet"
file_type: "parquet"
expected:
path: "[[ TEST_RESOURCES ]]/data/processed/expected/final_bar.parquet"
The pieces, one class per module:
- :class:
EnvironmentSwitch: runs a test in an explicitly chosen environment of the resource definitions, instead of patchingenv_identifier. - :class:
ExpectedOutputs: the session. Collects every mismatch, raises one error, cleans up, and can regenerate the expected files. - :class:
Expectations: the curated rules of a comparison (order, dtypes, tolerances, ignored columns, custom assertions), strict or relaxed. - :class:
FrameComparison: applies the rules to two frames. - :class:
OutputFiles: finds, reads, removes and promotes the generated and expected files of a resource. - :class:
OutputMismatchError: the error that reports all failed checks at once. - :class:
OutputSandbox: redirects the local outputs of a task into a scratch directory, so tests that write the same files can run in parallel. - :class:
SessionReport: the account of what a session verified, printed for passing runs too (by the pytest plugin of this package). - :class:
ParquetSchemaComparison: compares the arrow schema and pandas index of two parquet files (Expectations(check_parquet_schema=True)).
Example::
with ExpectedOutputs(Expectations(ignore_columns=["run_id"])) as outputs:
staging.main()
outputs.check(raw_config, "STAGED_FOO")
outputs.check(processed_config, "FINAL_BAR", list_columns_to_sort=["tags"], rtol=1e-3)
FrameAssertion
module-attribute
¶
A custom check (actual, expected) that raises AssertionError when the output is not acceptable.
EnvironmentSwitch ¶
Context manager that makes configs select environment and restores their own on exit.
A task picks the resource definitions of its config's environment, so a test that needs a variant (LOCAL_REMOVED_UPDATED, LOCAL_E2E ...) must
switch the configs the task uses. :class:ExpectedOutputs does this for every IOConfig that exists, so you rarely need this class; use it to switch
only some configs, for example when you do not check outputs::
with EnvironmentSwitch("LOCAL_E2E", input_config, output_config):
Task().run()
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
Optional[str]
|
The environment name, as it appears in the resource definitions. |
required |
configs
|
IOConfig
|
The |
()
|
Raises:
| Type | Description |
|---|---|
ValueError
|
On entering, if a config defines no resource in |
Expectations
dataclass
¶
The curated rules of a comparison. Immutable: derive stricter or more relaxed variants with :meth:with_.
The defaults are what a pipeline test usually wants: the same data, whatever the order of rows and columns, dtypes included.
Attributes:
| Name | Type | Description |
|---|---|---|
check_row_order |
bool
|
Require rows in the same order (otherwise they are aligned by index label, or sorted by |
check_column_order |
bool
|
Require columns in the same order. |
check_dtype |
bool
|
Require identical dtypes ( |
check_exact |
bool
|
Require exactly equal floats (otherwise |
rtol |
float
|
Relative tolerance for floats. |
atol |
float
|
Absolute tolerance for floats. |
sort_by |
Tuple[str, ...]
|
Columns to sort both frames by before comparing, for outputs without a meaningful index. Only used when |
only_columns |
Optional[Tuple[str, ...]]
|
Compare just these columns. |
ignore_columns |
Tuple[str, ...]
|
Leave these columns out of the comparison (timestamps, run ids ...). |
list_columns_to_sort |
Tuple[str, ...]
|
List-valued columns whose element order does not matter. |
assertions |
Tuple[FrameAssertion, ...]
|
Extra checks that run after the frames matched, e.g. |
check_parquet_schema |
bool
|
For parquet outputs (other types are skipped), also require the same arrow schema and pandas index as the expected file (the loaded frames can
hide a difference in nullability, timestamp units or a stored index). Applied by |
lenient
classmethod
¶
Values only: dtypes may differ (int64 vs float64) and floats are compared with a looser tolerance.
with_ ¶
A copy with the given rules changed. Sequences are accepted for the tuple-valued rules.
Raises:
| Type | Description |
|---|---|
TypeError
|
If a rule does not exist. |
also_assert ¶
A copy that additionally runs assertions.
ExpectedOutputs ¶
Lazily compares the outputs a task generated locally with the expected files declared in their resource definitions.
Use it as a context manager to raise the collected errors on exit, or call :meth:raise_if_failed yourself.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
expectations
|
Optional[Expectations]
|
The default rules of every check (see :class: |
None
|
environment
|
Optional[str]
|
The environment of the resource definitions, e.g. |
None
|
configs
|
Optional[Sequence[IOConfig]]
|
Switch only these |
None
|
keep_files
|
bool
|
Keep the generated files even when they matched (by default they are removed, so the next run starts clean). |
False
|
update_expected
|
Optional[bool]
|
Overwrite the expected files with the generated outputs instead of comparing. Defaults to the |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
errors |
List[Tuple[str, str]]
|
|
passed |
List[str]
|
Names of the checks that matched. |
updated |
List[str]
|
Names of the checks whose expected file was rewritten. |
skipped |
List[str]
|
Resources :meth: |
report
property
¶
What this session verified so far: the outputs that matched (with their size), failed, were updated or skipped.
read_generated
staticmethod
¶
read_generated(config: IOConfig, source_key: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> pd.DataFrame
The output the task generated for source_key.
read_expected
staticmethod
¶
read_expected(config: IOConfig, source_key: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> pd.DataFrame
The reference output declared for source_key.
clear
staticmethod
¶
clear(config: IOConfig, *source_keys: str, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> None
Remove leftovers of earlier runs (generated files of source_keys) so a stale file can not pass for a fresh output.
clear_all
staticmethod
¶
clear_all(config: IOConfig, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None) -> None
:meth:clear every output of config that declares an expected path (those with unfilled {placeholders} are left alone).
combinations
staticmethod
¶
Every combination of the grid values, as params dicts.
clear_each
classmethod
¶
clear_each(config: IOConfig, source_key: str, grid: Mapping[str, Iterable[Any]], environment: Optional[str] = None) -> None
:meth:clear the generated file of source_key for every combination of the grid values.
compare ¶
compare(actual: DataFrame, expected: DataFrame, name: str, expectations: Optional[Expectations] = None) -> bool
Compare two frames, recording the failure under name instead of raising. Returns whether they matched.
check ¶
check(config: IOConfig, source_key: str, *, label: Optional[str] = None, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool
Compare the generated output of source_key with its expected file and record any mismatch.
On a match the generated file is removed (unless keep_files); on a mismatch it is left in place to be inspected. A missing generated or expected
file is recorded as a failure too, so one broken output never hides the others. With update_expected the generated file replaces the expected one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
IOConfig
|
The |
required |
source_key
|
str
|
The resource name. |
required |
label
|
Optional[str]
|
Name in the report (default |
None
|
params
|
Optional[Mapping[str, Any]]
|
Values for |
None
|
environment
|
Optional[str]
|
The environment to check instead of the session's. |
None
|
expectations
|
Optional[Expectations]
|
Rules for this check instead of the session's. |
None
|
**rules
|
Any
|
Individual :class: |
{}
|
Returns:
| Type | Description |
|---|---|
bool
|
Whether the output matched (or was updated). |
check_all ¶
check_all(config: IOConfig, *, skip: Sequence[str] = (), rules_for: Optional[Mapping[str, Mapping[str, Any]]] = None, params: Optional[Mapping[str, Any]] = None, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool
Run :meth:check for every resource of config that declares an expected path, so a new output is checked without touching the test.
Resources whose paths have {placeholders} that params does not fill are left out and listed in skipped; check those with check_each.
Example: outputs.check_all(stage_output_config, rules_for={"STAGED_CARGOES": {"list_columns_to_sort": ["product_id"]}}, skip=["STAGED_RAW"]).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
IOConfig
|
The |
required |
skip
|
Sequence[str]
|
Resources not to check. |
()
|
rules_for
|
Optional[Mapping[str, Mapping[str, Any]]]
|
For a resource, the :class: |
None
|
params
|
Optional[Mapping[str, Any]]
|
Values for |
None
|
environment
|
Optional[str]
|
The environment to check instead of the session's. |
None
|
expectations
|
Optional[Expectations]
|
Rules for these checks instead of the session's. |
None
|
**rules
|
Any
|
Individual :class: |
{}
|
Returns:
| Type | Description |
|---|---|
bool
|
Whether every check matched. |
check_each ¶
check_each(config: IOConfig, source_key: str, grid: Mapping[str, Iterable[Any]], *, environment: Optional[str] = None, expectations: Optional[Expectations] = None, **rules: Any) -> bool
Run :meth:check for every combination of the grid values, for a resource whose path has several {placeholders}.
Example: outputs.check_each(config, "PART", {"table": ["events", "cargoes"], "shard": range(4)}) checks 8 files and reports each by its own params.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
IOConfig
|
The |
required |
source_key
|
str
|
The resource name. |
required |
grid
|
Mapping[str, Iterable[Any]]
|
For each placeholder, the values it takes. |
required |
environment
|
Optional[str]
|
The environment to check instead of the session's. |
None
|
expectations
|
Optional[Expectations]
|
Rules for these checks instead of the session's. |
None
|
**rules
|
Any
|
Individual :class: |
{}
|
Returns:
| Type | Description |
|---|---|
bool
|
Whether every combination matched. |
raise_if_failed ¶
Raise one :class:OutputMismatchError listing every recorded failure.
FrameComparison ¶
Applies the rules of expectations to a generated and an expected frame and raises AssertionError on the first difference found.
OutputFiles ¶
The files of source_key in the environment config selects: what the task generated, and what it is expected to generate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
IOConfig
|
The |
required |
source_key
|
str
|
The resource name. |
required |
params
|
Optional[Mapping[str, Any]]
|
Values for |
None
|
environment
|
Optional[str]
|
The environment to look the resource up in; by default the first one the resource definitions list. |
None
|
placeholders
property
¶
The {placeholders} of the paths that params does not fill.
environment_name
property
¶
The environment looked up: the selected one, else the first one the resource definitions list.
read_options
property
¶
Reader options for both files: the JSON orient and lines the resource is written with, overridden by expected.options.
declaring_expected
staticmethod
¶
The resources of config that declare an expected path in the environment: the outputs a test can check.
read_file
staticmethod
¶
Read a file with plain pandas, so no schema or cast hides a difference. options go to the reader (orient and lines only to JSON).
promote_generated ¶
Make the generated file the new reference (creating the expected directory if needed).
Raises:
| Type | Description |
|---|---|
ValueError
|
If the reference is declared in another file type than the output, as copying would leave a reference the checks cannot read. |
OutputMismatchError ¶
Bases: AssertionError
Raised once, after all checks ran, listing every output that did not match its expected counterpart.
Attributes:
| Name | Type | Description |
|---|---|---|
errors |
List[Tuple[str, str]]
|
|
passed |
List[str]
|
Names of the checks that matched, listed in the message so a failure also shows what is healthy. |
OutputSandbox ¶
Context manager that points every local output of configs below directory and restores the original paths on exit.
A resource at /data/out/foo.parquet is written to <directory>/data/out/foo.parquet (its directory is created); {placeholders} are kept. The expected files are not
moved, and ExpectedOutputs follows the redirected path, so a test reads like one without a sandbox. This makes tests that write the same outputs
independent: they can run in parallel (pytest-xdist) and leave nothing behind in the repository.
Pass the output configs of the task under test only: an input config would be redirected too, and the task would no longer find its inputs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
Union[str, PathLike]
|
Where the outputs go, typically pytest's |
required |
configs
|
IOConfig
|
The |
()
|
ParquetSchemaComparison ¶
The arrow schema (names, types, nullability, without the key-value metadata) and the pandas index columns of two parquet files.
SessionReport ¶
Lists the outputs a session matched, failed, updated and left out, one line each.
Which outputs a test verifies can be decided at runtime (check_all looks them up in the resource definitions), so a green run says which ones
they were. Sessions record their report in :attr:recorded, and the pytest plugin of this package prints them at the end of the run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
matched
|
Mapping[str, str]
|
Name of each output that matched, with a short description of it. |
required |
failed
|
Sequence[Tuple[str, str]]
|
|
required |
updated
|
Sequence[str]
|
Names of the outputs whose expected file was rewritten. |
required |
skipped
|
Sequence[str]
|
Names of the outputs |
required |
test
|
str
|
The test that ran the session ( |
''
|
lines ¶
(status, text) per output, where status is matched, failed, updated or skipped, in the order of the checks.