CorePipelineConfig

Immutable configuration snapshot built via PipelineParameterManager.

Path pattern¶

{container}/{layer}/{domain}/{product}/{table_name}/{version}/output/{processing_method}

Fields¶

Group	Fields	Defaults
Required	`env`	string; defaults to "dev" when not provided
Structure	`domain`, `product`, `table_name`	empty strings
Versions	`bronze_version`, `silver_version`, `gold_version`	`v1`
Processing	`bronze_processing_method`, `silver_processing_method`, `gold_processing_method`	`incremental`, `incremental`, `delta`
Infrastructure	`env_vars` (dict)	empty dict

Infrastructure variables (specified in prepare_infrastructure()): datalake_name, datalake_container_name, Azure IDs, etc.

Note: the environment key used by the manager is env by default. Lookup follows the PipelineParameterManager case_fallback setting (exact name first, then uppercase/lowercase fallbacks if enabled). When env is not set in the environment, it defaults to "dev".

Accessing infrastructure variables:

tenant_id = cfg.env_vars["az_tenant_id"]
datalake = cfg.env_vars["datalake_name"]

Methods¶

get_lake_path(layer, processing_method_override=None, version_override=None)¶

Returns folder path. Overrides are optional.

cfg.get_lake_path("bronze")
# -> "data/bronze/sales/orders/order_lines/v1/output/incremental"

cfg.get_lake_path("gold", processing_method_override="full", version_override="v9")
# -> "data/gold/sales/orders/order_lines/v9/output/full"

get_work_path(layer, version_override=None)¶

Returns working path (ends with /work, no processing method).

cfg.get_work_path("bronze")
# -> "data/bronze/sales/orders/order_lines/v1/work"

validate_rules(layers=None)¶

Raises ValueError on rule failures. Called automatically by build_core_config().