Skip to content

architecture.md

Latest commit

 

History

History
261 lines (190 loc) · 5.78 KB

architecture.md

File metadata and controls

261 lines (190 loc) · 5.78 KB
Raw

Architecture

Purpose

The framework is designed as a banking-oriented ingestion kernel for Databricks-style platforms.

It aims to solve four things together:

  • governed ingestion into Bronze
  • curated state handling in Silver
  • semantic publication into Gold
  • persisted operational and control evidence

The design stays deliberately smaller than a full enterprise platform. It gives you the core execution and control model, while leaving approval workflows, enterprise access policy, and broader data-product governance outside the repo.

Core Concepts

SourceContract

SourceContract describes the inbound dataset.

It owns source-side concerns such as:

  • source identifiers
  • expected and mandatory columns
  • primary keys
  • schema mode
  • quality expectations
  • format-specific adapter options

Examples:

  • file ingestion source
  • Bronze table used as Silver promotion input
  • XLSX sheet-driven source

DatasetTarget

DatasetTarget describes where the dataset is published and how that target behaves.

It owns target-side concerns such as:

  • catalog.schema.table
  • medallion stage
  • state model
  • managed storage path

This split is intentional:

  • SourceContract says what the source promises
  • DatasetTarget says how the target should behave

SourceAdapter

Adapters convert one source type into an IngestionDataset.

Current adapters:

  • CsvSourceAdapter
  • XlsxSourceAdapter
  • SparkTableSourceAdapter
  • StaticSourceAdapter for tests

These adapters normalize source behavior into one pipeline contract, even though the actual source forms are different.

Pipelines

The pipeline layer orchestrates:

  • discovery
  • schema validation
  • quality evaluation
  • technical metadata enrichment
  • stage preparation
  • persistence
  • tracker state changes
  • audit emission

Concrete pipelines:

  • FileToBronzePipeline
  • FileToSilverPipeline

Shared orchestration:

  • BaseIngestionPipeline

Persistence and Ops

The framework persists several kinds of state:

  • target table writes via DeltaRepository
  • file-tracker state via TrackerStore
  • audit history via AuditLogger
  • quarantine outputs for failed validation
  • reconciliation evidence via DeltaReconciliationStore

Repository Map

src/databricks_ingestion/
  adapters/         source readers: csv, xlsx, table
  audit/            audit events and Delta persistence
  contracts/        source and target definitions
  pipelines/        orchestration and stage-specific logic
  policies/         schema, quality, bronze, scd2
  runtime/          Spark/runtime bootstrapping
  storage/          Delta repository and tracker state
  job_config.py     config parsing for bundle-style jobs
  job_runner.py     execution helpers for jobs
  jobs.py           public façade / CLI entrypoint
  reconciliation.py controls and persisted evidence
  semantics.py      Gold semantic publication

End-to-End Flow

Bronze Flow

landing files
  -> adapter discovery
  -> processed-file filtering
  -> source read
  -> schema policy
  -> quality policy
  -> Bronze metadata enrichment
  -> append-only Delta write
  -> tracker success
  -> audit success

Bronze is designed to answer:

What arrived, from where, and under which run context?

Silver Flow

Preferred Silver flow:

Bronze table/view
  -> SparkTableSourceAdapter
  -> schema policy
  -> quality policy
  -> stage preparation
  -> append/latest/scd2 write
  -> reconciliation controls

Silver is designed to answer:

What is the curated business record or business state?

Gold Flow

curated Silver table/view
  -> SemanticViewSpec
  -> optional joins
  -> dimensions / measures / filters
  -> Gold table or view
  -> semantic metadata properties
  -> reconciliation controls

Gold is designed to answer:

What is the trusted output for analytics, controls, or reporting?

Example Architecture Walkthrough

1. Bronze ingest from CSV

contract:
  source_id: customer_accounts
  source_system: bank_feed
  format: csv
  landing_path: /Volumes/main/raw/customer_accounts
  primary_keys: [account_id]
target:
  catalog: main
  schema: banking
  dataset_name: customer_accounts
  stage: bronze

Outcome:

  • rows land append-only in main.banking.customer_accounts_bronze
  • file-level lineage is stamped into the table
  • tracker and audit state are persisted separately

2. Silver promotion from Bronze

contract:
  format: table
  landing_path: main.banking.customer_accounts_bronze
  table_options:
    source_table: main.banking.customer_accounts_bronze
target:
  stage: silver
  state_model: scd2

Outcome:

  • Silver does not reread landing files
  • Silver state is promoted from governed Bronze
  • current and historical versions are handled in the curated layer

3. Gold publication

semantic_spec:
  name: customer_accounts_summary
  source_table: main.banking.customer_accounts_silver AS accounts
  shared_measures:
    - name: total_amount
      expression: sum(accounts.amount)
  measure_refs: [total_amount]
  materialization: table

Outcome:

  • a Gold table is published from curated Silver
  • semantic metadata is written to table properties
  • reconciliation controls can block publication

Why This Shape Works Well

This architecture is useful because it separates three concerns that often get mixed:

  • source truth
  • curated state
  • published semantics

That makes it much easier to reason about:

  • lineage
  • controls
  • ownership
  • replay
  • BCBS239-style evidence

What the Architecture Intentionally Does Not Try To Be

This repo is not trying to own:

  • enterprise workflow approval
  • enterprise access policy enforcement
  • enterprise semantic serving APIs
  • full product-registry or marketplace behavior

Those layers can sit above the framework, while the framework remains the ingestion and control kernel.