Writing data¶
Step 5: Writing out¶
Sinking data is done in a very similar way. You need to:
-
Define your output resource definitions, in our case in
staging_output.yaml(shown in Step 1), with theLOCAL,DEVandPRODenvironments and, inLOCAL, theexpectedpath a test compares the output with. -
You need to define the respective dynamic values found in your resource definitions in your
src/config/environment.py - You need to create an instance of the
IOConfigclass for thestaging_output.yamlin the__init__.pyfile (we already did this). - Define the additional
DynamicDataIOsubclasses in thesrc/resources/io.pymodule, dictating through the schema the list of columns, and their types (also used for schema validation). - Finally, instantiate instances of those subclasses and call the
.write()method, passing in the dataframe you want to write out, e.g.
demo/src/runners/staging.py:
...
StagedFoo(source_config=staging_output_config.get(source_key="STAGED_FOO"), **constants.TO_PARQUET_KWARGS).write(foo_df)
StagedBar(source_config=staging_output_config.get(source_key="STAGED_BAR")).write(bar_df)
Notice that you can pass all pandas options to write out, when for instance you are writing out parquet.
demo/src/config/constants.py:
# Parquet
TO_PARQUET_KWARGS = {
"use_deprecated_int96_timestamps": False,
"coerce_timestamps": "ms",
"allow_truncated_timestamps": True,
}
Of course this is not a problem as parquet is the format used by both resources in either environment. This is not always the case however. See in
demo/src/resources/definitions/transform_output.yaml (here with its LOCAL, DEV and PROD environments):
---
...
FINAL_BAR:
LOCAL:
type: "local"
local:
file_path: "[[ TEST_RESOURCES ]]/data/processed/final_bar.parquet"
file_type: "parquet"
options: <---- Options for Local writing as parquet
use_deprecated_int96_timestamps: true
coerce_timestamps: "ms"
allow_truncated_timestamps: false
row_group_size: 1000000
expected:
path: "[[ TEST_RESOURCES ]]/data/processed/expected/final_bar.parquet"
DEV:
type: "kafka"
kafka: &bar_kafka
kafka_server: "[[ DEV_KAFKA_BROKER ]]"
kafka_topic: "[[ KAFKA_TOPIC ]]"
options: &bar_kafka_options <---- Options for writing to a Kafka Topic
compression_type: "snappy"
max_in_flight_requests_per_connection: 10
batch_size: 262144
request_timeout_ms: 60000 # 60s
buffer_memory: 134217728 # 128MB
PROD:
type: "kafka"
kafka:
<<: *bar_kafka
kafka_server: "[[ PROD_KAFKA_BROKER ]]"
options: *bar_kafka_options
Here, we have a case where different options need to be used for each environment as it deals with a different source. This is gracefully managed through resource
definitions passing these arguments in the options key per environment.
Step 6: Full Code¶
The full code for the loading module in our example would live under:
and looks like:
"""Add module docstring...."""
import logging
from demo.src import staging_input_config, staging_output_config
from demo.src.config import constants
from demo.src.resources.io import InputIO, StagedBar, StagedFoo
logger = logging.getLogger(__name__)
def main() -> None:
"""The entry point for the Airflow Staging task.
Returns:
Void function.
"""
# LOAD DATA
logger.info("Loading data from live sources...")
bar_df = InputIO(source_config=staging_input_config.get(source_key="BAR"), schema=Bar).read()
foo_df = InputIO(source_config=staging_input_config.get(source_key="FOO"), schema=Foo).read()
logger.info("Data successfully loaded from live sources...")
# TRANSFORM DATA
logger.info("Apply transformations...")
# TODO: Apply your transformations
logger.info("Transformations applied successfully...")
# SINK DATA
logger.info("Begin sinking data to staging area:")
StagedFoo(source_config=staging_output_config.get(source_key="STAGED_FOO"), **constants.TO_PARQUET_KWARGS).write(foo_df)
StagedBar(source_config=staging_output_config.get(source_key="STAGED_BAR")).write(bar_df)
logger.info("Data staging is complete...")