ETRM System Architecture

A single malformed price file from an ISO can silently overwrite a day of nodal positions, and by the time the variance surfaces on a T+30 statement the wrong number has already propagated through pricing, loss, and imbalance. The failure mode this component prevents is exactly that: uncontrolled data crossing the ingestion boundary and corrupting a settlement run that must later be reconciled to the cent. Within the Core Architecture & Market Taxonomy for Energy Settlements framework, the Energy Trading and Risk Management (ETRM) platform is the operational backbone that turns physical delivery schedules, financial derivatives, and grid-operator mandates into auditable, replayable records. Its architecture must be decoupled layer by layer so that a fault in one stage — a stale telemetry feed, a schema drift, a duplicated hour on a fall-back day — is isolated at its boundary rather than allowed to reach the ledger.

The diagram below maps the decoupled layers of a production ETRM pipeline, from schema-agnostic ingestion through temporal alignment and reconciliation to financial posting, including the fallback path that preserves continuity when feeds degrade.

Each layer owns exactly one responsibility and exposes exactly one contract to the next. That separation is what makes the pipeline replayable: any stage can be re-run from its input snapshot and must produce a bit-identical output.

Data Ingestion and Normalization Layer #

The ingestion pipeline is the primary control boundary for every downstream financial process. ISOs and RTOs distribute settlement artifacts through disparate delivery mechanisms — SFTP batch drops, RESTful market-data APIs, EDI 867/820 transactions, and real-time telemetry streams — and each jurisdiction enforces its own schema conventions, interval granularities, and versioning protocols. A production-grade ETRM architecture deploys a schema-agnostic ingestion engine that normalizes raw payloads into a canonical internal model before any downstream processing runs. That normalization step enforces strict validation against the ISO/RTO Data Format Standards, guaranteeing that locational marginal price (LMP) components, transmission loss multipliers, congestion rents, and ancillary-service credits all map to standardized dimensional keys.

The canonical model preserves the LMP decomposition the calculation layer depends on. Every nodal price resolves into a system energy component, a congestion component, and a marginal-loss component:

$$LMP_n = \lambda + \mu_n + \nu_n$$

where \(\lambda\) is the system marginal energy price, \(\mu_n\) is the congestion component at node \(n\), and \(\nu_n\) is the marginal-loss component. If ingestion does not carry all three components through as separate typed fields, the Pricing Logic Implementation downstream cannot prove that a nodal price reconstructs from its parts, and a congestion charge silently disappears into the energy line.

Python automation teams typically use polars for high-throughput parsing and pydantic for strict schema validation, rejecting malformed records at the boundary so that contaminated data never reaches the ledger. Because the field contract is the same one enforced by the Schema Validation Frameworks on the trade side, the ingestion model is shared rather than duplicated.

from decimal import Decimal
from datetime import datetime
from pydantic import BaseModel, Field, ValidationError
import polars as pl

class LmpRecord(BaseModel):
    node_id: str
    interval_start: datetime          # tz-aware, UTC-anchored at ingest
    energy_component: Decimal         # lambda
    congestion_component: Decimal     # mu_n
    loss_component: Decimal           # nu_n
    lmp_usd_mwh: Decimal              # published nodal price

    def decomposition_holds(self, tol: Decimal = Decimal("0.005")) -> bool:
        parts = self.energy_component + self.congestion_component + self.loss_component
        return abs(parts - self.lmp_usd_mwh) <= tol

def normalize_feed(raw_df: pl.DataFrame) -> list[LmpRecord]:
    """Reject malformed records at the ingestion boundary; never let them through."""
    records, rejects = [], []
    for row in raw_df.iter_rows(named=True):
        try:
            rec = LmpRecord(**row)
            if not rec.decomposition_holds():
                raise ValueError(f"LMP does not sum from components: {rec.node_id}")
            records.append(rec)
        except (ValidationError, ValueError) as exc:
            rejects.append({"node_id": row.get("node_id"), "reason": str(exc)})
    if rejects:
        route_to_dead_letter(rejects)   # isolate, do not halt the run
    return records

Specification & Standards Reference #

Ingestion is not a free design; it is constrained by the same tariff and reliability envelope that governs the rest of the settlement chain. Charge codes, statement cadence, and the components a participant is entitled to receive are defined per market:

• PJM Manual 28 and the PJM settlement XML schema specify charge-code semantics and the preliminary/final statement cadence.
• ERCOT Nodal Protocols, Section 9 governs settlement calculation and the fixed-width extract layouts ERCOT publishes.
• CAISO Business Practice Manual for Settlements & Billing defines the resource-level statement structure and dispute windows.
• NAESB WEQ standards define the wholesale electric quadrant messaging that EDI 867 (meter usage) and 820 (payment/remittance) transactions ride on.
• FERC approves the Open Access Transmission Tariff under which each RTO settles, and Order 2222 pulls DER aggregations into the resource taxonomy the model must represent.

Because a charge code can change meaning across a tariff revision, the canonical model must be versioned against a jurisdiction and an effective date — a pipeline that hard-codes today’s semantics will mis-settle tomorrow’s intervals.

Temporal Alignment and Cycle Management #

Energy settlements operate across overlapping temporal horizons: day-ahead (D-1), real-time (D), preliminary (D+1), final (D+30), and regulatory true-up (D+90+). Misalignment between trade-execution timestamps, SCADA metered intervals, and market-clearing windows is the leading cause of reconciliation breaks. The architecture routes every canonical record through a deterministic Settlement Cycle Mapping engine that synchronizes UTC, local market time, and daylight-saving transitions without introducing rounding drift. Interval aggregation rules respect 5-minute, 15-minute, or hourly boundaries while preserving fractional MWh precision, apply time-weighted averaging for partial intervals, and explicitly flag any gap exceeding the market’s allowable tolerance.

Step-by-Step Implementation #

Building the reconciliation core from the canonical records is a fixed sequence of production steps.

Step 1 — Anchor and align intervals. Convert every reported timestamp to UTC at ingestion and align it to the market operating-day grid so that a metered interval and a market interval share one key.

from zoneinfo import ZoneInfo
from datetime import datetime

def align_interval(local_ts: datetime, market_tz: str) -> datetime:
    """UTC-anchor a reported timestamp; DST tables live in the cycle-mapping engine."""
    return local_ts.replace(tzinfo=ZoneInfo(market_tz)).astimezone(ZoneInfo("UTC"))

Step 2 — Model the settlement line item with Decimal money math. Every monetary and volumetric field is a Decimal; binary floats accumulate drift that eventually flips a rounding boundary and breaks reconciliation.

from decimal import Decimal, ROUND_HALF_EVEN
from datetime import datetime
from pydantic import BaseModel, Field

CENT = Decimal("0.01")

class SettlementLine(BaseModel):
    trade_id: str
    node_id: str
    interval_start: datetime          # UTC-anchored
    iso_zone: str
    scheduled_mwh: Decimal = Field(ge=0)
    settled_mwh: Decimal = Field(ge=0)
    lmp_usd_mwh: Decimal
    loss_factor: Decimal = Decimal("1.0")
    variance_tolerance_pct: Decimal = Field(default=Decimal("0.5"), ge=0, le=Decimal("5.0"))

    def charge(self) -> Decimal:
        gross = self.settled_mwh * self.lmp_usd_mwh * self.loss_factor
        return gross.quantize(CENT, rounding=ROUND_HALF_EVEN)

Step 3 — Reconcile idempotently. Match scheduled against settled volume per (trade_id, node_id, interval_start); the key makes re-delivery of the same file a no-op instead of a double-count.

def is_reconciled(line: SettlementLine) -> bool:
    if line.scheduled_mwh == 0 and line.settled_mwh == 0:
        return True
    denom = max(line.scheduled_mwh, Decimal("1e-9"))
    variance = abs(line.scheduled_mwh - line.settled_mwh) / denom
    return variance <= (line.variance_tolerance_pct / Decimal("100"))

Step 4 — Route by outcome. Reconciled lines proceed to financial posting; breaks go to the exception queue for analyst review. The posting store is append-only and idempotent on the same key, so a replay proves identical.

def route(lines: list[SettlementLine]) -> None:
    for line in lines:
        if is_reconciled(line):
            post_to_ledger(line.trade_id, line.node_id, line.interval_start, line.charge())
        else:
            enqueue_exception(line)   # never blocks cleared transactions

This routing layer feeds the broader Settlement Calculation & Validation Engines, which price and validate the obligations these reconciled lines represent.

Edge Cases and Failure Modes #

Production feeds break in predictable, ugly ways. Each must be handled explicitly rather than caught generically.

• Negative LMPs. Congestion and oversupply routinely push nodal prices below zero; a Field(ge=0) on lmp_usd_mwh would reject valid market data. Prices are unbounded; only volumes are non-negative.
• DST boundaries. The spring-forward 23-hour day drops an hour-ending and the fall-back 25-hour day duplicates one. Anchoring to UTC at ingestion and carrying an explicit hour-ending index prevents the duplicate from overwriting or the gap from mis-aggregating.
• Zero-volume intervals. A curtailed resource reports scheduled_mwh == settled_mwh == 0; treat it as reconciled rather than dividing by zero.
• Stale telemetry. A feed that stops advancing looks healthy but repeats yesterday’s interval. Compare the max reported interval against wall-clock and flag staleness before it reaches reconciliation.
• Schema drift. A field that silently changes type or unit propagates a wrong number everywhere; the pydantic contract at the boundary rejects it to the dead-letter path.

from decimal import Decimal
from datetime import datetime, timedelta, timezone

def guard_interval(line: "SettlementLine", max_staleness: timedelta = timedelta(hours=2)) -> None:
    # Negative LMP is valid; do not reject it.
    if line.settled_mwh < 0 or line.scheduled_mwh < 0:
        raise ValueError(f"negative volume at {line.node_id} {line.interval_start}")
    if datetime.now(timezone.utc) - line.interval_start > max_staleness + timedelta(hours=1):
        flag_stale(line.node_id, line.interval_start)   # telemetry not advancing

Security & Access Boundaries #

Regulatory compliance mandates strict segregation between trading, scheduling, and financial-accounting functions, and the ingestion architecture inherits those Security & Access Boundaries directly. Role-based access control enforces least privilege, isolating position management from the settlement calculation engines. Data residency and encryption-at-rest requirements align with the NERC Critical Infrastructure Protection Standards — CIP-005 electronic security perimeters, CIP-007 system security management, and CIP-011 information protection — so that pricing curves, counterparty credit limits, and settlement statements remain cryptographically secured. Every data mutation carries immutable lineage: inputs, the tariff and price versions in force, a UTC timestamp, and a SHA-256 content hash, enabling forensic reconstruction during a regulatory inquiry or a dispute.

Fallback Routing and Threshold Configuration #

Market-data feeds are inherently volatile — network partitions, API rate limits, and delayed batch submissions all interrupt the primary channel. Production ETRM systems implement circuit-breaker patterns, exponential backoff, and dead-letter queues to isolate failed payloads without halting the broader pipeline. When a primary ingestion channel fails, fallback routing switches automatically to a secondary SFTP endpoint or a cached market snapshot, preserving continuity for critical D+1 and D+30 runs. Message brokers such as Apache Kafka or RabbitMQ provide at-least-once delivery; idempotent processing in every reconciliation microservice prevents a duplicated market update from inflating position ledgers or triggering a false break.

Alerting is tiered so that a transient blip does not page an analyst but a systemic outage does. The parameters below are configurable per market and feed the same escalation model as Threshold Tuning & Alerts.

from decimal import Decimal

class CircuitBreaker:
    def __init__(self, fail_threshold: int = 5, reset_after_probes: int = 3):
        self.fail_threshold = fail_threshold
        self.reset_after_probes = reset_after_probes
        self.failures = 0
        self.state = "CLOSED"

    def record_failure(self) -> None:
        self.failures += 1
        if self.failures >= self.fail_threshold:
            self.state = "OPEN"            # divert to fallback SFTP / cache

    def record_success(self) -> None:
        self.failures = 0
        self.state = "CLOSED"

def break_rate_exceeds(breaks: int, interval_count: int, tier: Decimal = Decimal("0.02")) -> bool:
    if interval_count == 0:
        return False
    return (Decimal(breaks) / Decimal(interval_count)) > tier

Multi-ISO Cross-Market Reconciliation #

Traders operating across multiple balancing authorities face compounding complexity when reconciling cross-jurisdictional positions. A cross-market reconciliation engine normalizes disparate pricing zones, currency denominations, and transmission loss factors into a unified settlement view, applying the correct multiplier per node using the same discipline as Loss Factor Mapping Strategies. The validator below performs pair-wise matching of executed trades against ISO-issued settlement statements with polars and pydantic, flagging interval mismatches under strict Decimal-exact type safety across heterogeneous feeds.

from decimal import Decimal
from datetime import timezone
from typing import List
import polars as pl

def validate_and_reconcile(lines: List["SettlementLine"]) -> pl.DataFrame:
    """Flag reconciliation breaks across markets; cleared lines proceed to posting."""
    results: List[dict] = []
    for line in lines:
        denom = max(line.scheduled_mwh, Decimal("1e-9"))
        variance_pct = (abs(line.scheduled_mwh - line.settled_mwh) / denom * Decimal("100"))
        results.append({
            "trade_id": line.trade_id,
            "iso_zone": line.iso_zone,
            "interval_start": line.interval_start.astimezone(timezone.utc),
            "is_reconciled": is_reconciled(line),
            "variance_pct": str(variance_pct.quantize(Decimal("0.0001"))),
            "requires_manual_review": not is_reconciled(line),
        })
    return pl.DataFrame(results).sort(["iso_zone", "interval_start"])

This layer integrates directly with automated settlement workflows, routing unreconciled records to exception queues for analyst review while allowing cleared transactions to proceed to financial posting. For comprehensive implementation guidance on schema-validation patterns, refer to the official Python pydantic documentation.

Testing and Reconciliation Verification #

The architecture is only trustworthy if it is provably reproducible. Two verification techniques apply on every deployment. First, a shadow calculation: replay a closed settlement run through the current code and assert the ledger output hashes identically to the archived run — any drift means a logic change altered a historical number. Second, edge-case unit tests that pin the exact failure modes above so a refactor cannot silently reintroduce them.

from decimal import Decimal
from datetime import datetime, timezone

def _line(**kw) -> "SettlementLine":
    base = dict(trade_id="T1", node_id="N1", iso_zone="PJM",
                interval_start=datetime(2026, 6, 1, tzinfo=timezone.utc),
                scheduled_mwh=Decimal("10"), settled_mwh=Decimal("10"),
                lmp_usd_mwh=Decimal("42.15"))
    base.update(kw)
    return SettlementLine(**base)

def test_negative_lmp_is_valid():
    line = _line(lmp_usd_mwh=Decimal("-18.40"))
    assert line.charge() == Decimal("-184.00")          # oversupply, not an error

def test_zero_volume_reconciles():
    line = _line(scheduled_mwh=Decimal("0"), settled_mwh=Decimal("0"))
    assert is_reconciled(line) is True                  # no divide-by-zero

def test_charge_is_cent_quantized():
    line = _line(settled_mwh=Decimal("3"), lmp_usd_mwh=Decimal("33.333"))
    assert line.charge() == Decimal("99.99")            # bit-exact, ROUND_HALF_EVEN

Frequently Asked Questions #

Why must an ETRM settlement pipeline use Python’s decimal module instead of float? #

Charges are quantized to the cent and reconciled against an ISO statement to the cent. Binary floating point cannot represent most decimal fractions exactly, so summing thousands of intervals accumulates drift that eventually flips a rounding boundary and breaks reconciliation. Quantizing every amount with Decimal keeps the ledger bit-exact and reproducible across preliminary and final runs.

Why should the ingestion layer never reject a negative LMP? #

Congestion and oversupply routinely drive nodal prices below zero, so a negative LMP is valid market data, not a malformed record. Only volumes are non-negative. Enforcing ge=0 on price fields silently discards real intervals and understates congestion cost; the constraint belongs on scheduled_mwh and settled_mwh, not on lmp_usd_mwh.

What makes an ETRM run replayable and auditable? #

Every posted line carries its inputs, the tariff and price versions in force, the code revision, a UTC timestamp, and a SHA-256 content hash, written to an append-only store keyed idempotently on (trade_id, node_id, interval_start). Because re-delivering the same file is a no-op, any historical run can be replayed and proven identical, satisfying FERC traceability and counterparty dispute requirements.

How does the architecture keep a degraded feed from halting settlement? #

A circuit breaker diverts a failing primary channel to a secondary SFTP endpoint or a cached snapshot, malformed records fall to a dead-letter queue, and reconciliation breaks go to an exception queue — none of which block cleared transactions. Idempotent processing means the at-least-once redelivery that follows a fallback cannot double-count, so the critical D+1 and D+30 runs complete even while one feed is down.

Related #

• ISO/RTO Data Format Standards
• Settlement Cycle Mapping
• Security & Access Boundaries
• Schema Validation Frameworks
• Settlement Calculation & Validation Engines