Loss Factor Mapping Strategies

A single mismapped loss factor quietly rescales an entire node’s delivered volume, and because the distortion rides through pricing and imbalance untouched, it does not surface until a T+30 statement fails to reconcile against the ISO’s shadow settlement. The failure mode this component prevents is exactly that: a stale, null, or anomalous loss multiplier crossing into the calculation core and rescaling megawatt-hours that must later tie out to the cent. Within the Settlement Calculation & Validation Engines framework, loss-factor mapping is the transformation that reconciles physical network topology — where transmission and distribution losses are real, node-specific, and time-varying — with the commercial settlement boundary, where those losses become a multiplicative adjustment on billed energy. Get the mapping wrong and every downstream charge inherits the error; get it deterministic and every interval settles on a provable, versioned factor.

The diagram below shows the loss-factor mapping and validation flow this page describes: published factors are joined to metered intervals, checked for nulls and outliers against historical bands, and applied multiplicatively to produce settlement-ready volumes.

Every stage after the join is a pure, replayable transformation over interval-indexed data: given the same published factors, the same metered volumes, and the same code revision, the mapping layer must emit the same delivered volumes whether it runs for the preliminary D+1 cycle or the D+90 true-up. The sections below define the factor taxonomy the layer operates over, the tariff and standards that bound it, the stage-by-stage implementation, and the validation and alerting controls that keep a bad factor from ever reaching the ledger.

Factor Taxonomy and the Delivery-Factor Formula #

Independent System Operators and Regional Transmission Organizations publish loss information in incompatible shapes, and the first job of the mapping layer is to canonicalize them into a single delivery factor per settlement node per interval. The canonical form is a multiplier applied to metered energy:

$$V^{\text{adj}}{n,t} = V^{\text{meter}}{n,t} \times \delta_{n,t}, \qquad \delta_{n,t} = 1 - L_{n,t}$$

where \(V^{\text{meter}}{n,t}\) is the metered volume at node \(n\) in interval \(t\), \(L{n,t}\) is the fractional loss, and \(\delta_{n,t}\) is the delivery factor. A marginal loss factor (MLF) is published as the incremental loss on the next MWh and must be resolved to \(\delta\) before it can scale a volume; an average loss factor (ALF) already approximates \(1 - L\) over a zone. Conflating the two — applying an MLF where the calculation expects a delivery factor — is one of the most common silent errors in settlement automation.

Because a factor’s meaning depends on the tariff revision in force, the canonical record must be versioned against a jurisdiction and an effective date. The detailed node-resolution rules — canonicalizing node identifiers, handling topology renames, and reconciling a published factor to the right settlement point — are covered in Mapping transmission loss factors to settlement nodes. The interval alignment those factors depend on is produced upstream by the Settlement Cycle Mapping engine, and the field-level shape of each published feed is enforced by the ISO/RTO Data Format Standards.

Specification & Standards Reference #

Loss-factor mapping is not a free design; the multiplier, its publication cadence, and its dispute window are fixed by tariff. A mapping layer that ignores the governing document will mis-settle even when its arithmetic is flawless:

• PJM Manual 27 / Manual 28 define the marginal loss surplus allocation and the loss-factor components that flow into PJM settlement statements and charge codes.
• MISO BPM-005 (Market Settlements) specifies the marginal loss factor calculation and the day-ahead versus real-time loss-charge line items.
• ERCOT Nodal Protocols, Section 13 governs transmission and distribution loss factors and the profiles used to allocate unaccounted-for energy.
• CAISO Business Practice Manual for Market Operations defines the marginal loss component of the locational marginal price and its interval publication.
• NAESB WEQ business practices define the wholesale electric quadrant messaging on which EDI 867 meter usage — the metered volumes the factor scales — is delivered.
• FERC approves the Open Access Transmission Tariff under which each RTO settles losses, and its Uniform System of Accounts requires every loss adjustment be traceable to a metered interval and a published factor version.

For authoritative loss-factor methodology, the PJM Interconnection Loss Factor Methodology is the reference implementation many pipelines validate against. Because a loss factor can change meaning across a tariff filing, the canonical record must carry the tariff version and effective date so a replay of a historical interval reconstructs the factor that was actually in force.

Step-by-Step Implementation #

Building a resilient mapping layer is a fixed sequence of production steps. Every monetary and volumetric field is a Decimal; binary floats accumulate drift that eventually flips a rounding boundary and breaks reconciliation.

Step 1 — Model the mapping record with Decimal money math. The record carries the metered volume, the resolved delivery factor, and enough lineage to prove which tariff version produced it. Validation is enforced at the boundary with pydantic, the same discipline applied by the Schema Validation Frameworks on the trade side.

from decimal import Decimal, ROUND_HALF_EVEN
from datetime import datetime
from typing import Optional
from pydantic import BaseModel, Field

MWH = Decimal("0.000001")   # settle volume to the microMWh

class LossFactorRecord(BaseModel):
    node_id: str
    settlement_interval: datetime          # tz-aware, UTC-anchored at ingest
    metered_mwh: Decimal = Field(ge=0)     # volumes are non-negative
    loss_factor: Optional[Decimal] = None  # resolved delivery factor delta = 1 - L
    tariff_version: str                    # e.g. "PJM-2026.02"
    factor_source: str = "published"       # published | fallback | historical_mean

    def adjusted_volume(self) -> Decimal:
        if self.loss_factor is None:
            raise ValueError(f"unresolved loss factor at {self.node_id} {self.settlement_interval}")
        gross = self.metered_mwh * self.loss_factor
        return gross.quantize(MWH, rounding=ROUND_HALF_EVEN)

Step 2 — Join published factors to metered intervals. The join key is (node_id, settlement_interval); a left join from metered volumes keeps every settlement point even when a factor is missing, so the null path can handle it explicitly rather than dropping the interval. pandas gives vectorized alignment over the interval grid.

import pandas as pd

def join_factors(metered_df: pd.DataFrame, published_df: pd.DataFrame) -> pd.DataFrame:
    """Left-join so every metered interval survives; missing factors become nulls to route."""
    merged = metered_df.merge(
        published_df[["node_id", "settlement_interval", "loss_factor", "tariff_version"]],
        on=["node_id", "settlement_interval"],
        how="left",
    )
    merged["factor_source"] = merged["loss_factor"].notna().map(
        {True: "published", False: "missing"}
    )
    return merged

Step 3 — Route nulls through a deterministic fallback chain. Telemetry dropouts and feed delays leave gaps; a production layer fills them from a versioned history in a fixed order — never a random or non-reproducible guess — so a replay yields the identical factor.

from decimal import Decimal

def apply_fallback(merged: pd.DataFrame, historical_df: pd.DataFrame,
                   method: str = "rolling_median") -> pd.DataFrame:
    """Deterministic per-node fallback for missing factors; records the substitution in lineage."""
    null_mask = merged["loss_factor"].isna()
    if not null_mask.any():
        return merged

    grouped = historical_df.groupby("node_id")["loss_factor"]
    if method == "rolling_median":
        fill = grouped.median()
    elif method == "prior_interval":
        fill = grouped.last()
    else:
        raise ValueError(f"unsupported fallback method: {method}")

    merged.loc[null_mask, "loss_factor"] = merged.loc[null_mask, "node_id"].map(fill)
    merged.loc[null_mask, "factor_source"] = "fallback"
    # Any node with no history at all stays null and must not settle.
    still_null = merged["loss_factor"].isna()
    if still_null.any():
        merged.loc[still_null, "factor_source"] = "unresolved"
    return merged

Step 4 — Screen published and filled factors against a historical band. A factor that survives the join can still be anomalous — a decimal-point error, an MLF where an ALF was expected, a topology-change artifact. Screen each factor against its node’s historical distribution and revert outliers to the historical mean before they distort settlement.

from decimal import Decimal

def screen_outliers(merged: pd.DataFrame, historical_df: pd.DataFrame,
                    sigma_multiplier: Decimal = Decimal("3.0")) -> pd.DataFrame:
    """Revert factors more than N standard deviations from the node's historical mean."""
    stats = historical_df.groupby("node_id")["loss_factor"].agg(["mean", "std"])
    mean = merged["node_id"].map(stats["mean"])
    std = merged["node_id"].map(stats["std"])

    deviation = (merged["loss_factor"] - mean).abs()
    threshold = std * float(sigma_multiplier)
    outlier_mask = (deviation > threshold) & std.notna() & (std > 0)

    merged.loc[outlier_mask, "loss_factor"] = mean[outlier_mask]
    merged.loc[outlier_mask, "factor_source"] = "historical_mean"
    return merged

Step 5 — Apply the delivery factor once, and only once. The loss factor is a multiplicative delivery factor, so settled volume is the metered volume scaled by \(\delta\) — applied a single time, never double-counted across the pre- and post-settlement paths. The result flows to the Pricing Logic Implementation layer, which multiplies it by the locational marginal price.

from decimal import Decimal, ROUND_HALF_EVEN

MWH = Decimal("0.000001")

def apply_delivery_factor(merged: pd.DataFrame) -> pd.DataFrame:
    """Scale metered volume by the resolved delivery factor; refuse to settle unresolved rows."""
    if (merged["factor_source"] == "unresolved").any():
        raise ValueError("unresolved loss factors present; run cannot settle")

    def _scale(row) -> Decimal:
        gross = Decimal(str(row["metered_mwh"])) * Decimal(str(row["loss_factor"]))
        return gross.quantize(MWH, rounding=ROUND_HALF_EVEN)

    merged["adjusted_mwh"] = merged.apply(_scale, axis=1)
    return merged[["node_id", "settlement_interval", "adjusted_mwh",
                   "loss_factor", "factor_source", "tariff_version"]]

Whether the delivery factor is applied here (pre-settlement, scaling metered volume before price multiplication) or as a post-settlement financial adjustment is an architectural choice: pre-settlement mapping demands high-frequency validation against SCADA and AMI telemetry, while post-settlement mapping requires an audit trail rigorous enough to satisfy FERC accounting. Either way, the two-stage protocol above — resolve every factor, then bound every factor — must run before the multiplier touches a volume.

Edge Cases and Failure Modes #

Production loss feeds break in predictable, ugly ways. Each must be handled explicitly rather than caught generically.

• Stale telemetry. A feed that stops advancing repeats yesterday’s factor while looking healthy. Compare the max published interval against wall-clock and route to the fallback chain before the repeated factor reaches settlement.
• MLF/ALF confusion. An incremental marginal loss factor (e.g. 0.03) applied directly as a delivery factor scales a node’s volume to near zero. Canonicalize to \(\delta = 1 - L\) at ingestion and range-check that \(\delta\) sits in a plausible band (roughly 0.85–1.15).
• DST boundaries. The spring-forward 23-hour day drops an hour-ending and the fall-back 25-hour day duplicates one; a factor keyed on local hour will either miss or double-map. Anchor to UTC and carry an explicit hour-ending index.
• Zero-volume intervals. A curtailed node reports metered_mwh == 0; the multiplication is valid and yields zero, but the factor must still be resolved so the lineage is complete and a later true-up can restate the volume.
• Topology reconfiguration. A node split or rename between the historical window and the current run leaves no history for the outlier screen; the record must fall to unresolved rather than silently pass an unscreened factor.
• Negative or absent standard deviation. A node with a single historical observation has std == 0 or NaN; the outlier screen must skip it (as coded in Step 4) rather than revert every factor to the mean.

from decimal import Decimal

def guard_factor(delta: Decimal, node_id: str,
                 lo: Decimal = Decimal("0.85"), hi: Decimal = Decimal("1.15")) -> None:
    """Reject a delivery factor outside the plausible physical band — catches MLF/ALF confusion."""
    if delta is None:
        raise ValueError(f"unresolved delivery factor at {node_id}")
    if not (lo <= delta <= hi):
        raise ValueError(f"implausible delivery factor {delta} at {node_id}; check MLF vs ALF")

Threshold & Alerting Configuration #

Not every deviation warrants the same response. The outlier sigma_multiplier, the fallback method, and the plausible-band bounds are configurable per market and feed the same escalation model as Threshold Tuning & Alerts. Tiered routing keeps a transient dropout from paging an analyst while a systemic feed failure does.

from decimal import Decimal

def classify_alert(fallback_rate: Decimal, outliers: int, unresolved: int) -> str:
    """Map mapping-run metrics onto the escalation tier."""
    if unresolved > 0:
        return "halt"
    if outliers > 0:
        return "critical"
    if fallback_rate > Decimal("0.01"):
        return "warning"
    return "info"

Every fallback substitution and every outlier reversion is written to an append-only audit log with a UTC timestamp, the node, the original and substituted factor, the tariff version, and a SHA-256 hash of the mapping snapshot — so a dispute or a FERC inquiry can reconstruct exactly which factor settled which interval. When mapped factors misstate real grid conditions, the artificial volume swings surface downstream as false imbalances; robust Imbalance Allocation Algorithms depend on the mapping layer having already isolated true physical deviation from mapping artifacts.

Testing & Reconciliation Verification #

The mapping layer is only trustworthy if it is provably reproducible. Two verification techniques apply on every deployment. First, a shadow calculation: replay a closed settlement run through the current mapping code and assert the adjusted-volume output hashes identically to the archived run — any drift means a logic change altered a historical number. Second, edge-case unit tests that pin the failure modes above so a refactor cannot silently reintroduce them.

from decimal import Decimal
from datetime import datetime, timezone

def _rec(**kw) -> LossFactorRecord:
    base = dict(node_id="AECO", settlement_interval=datetime(2026, 6, 1, tzinfo=timezone.utc),
                metered_mwh=Decimal("100"), loss_factor=Decimal("0.97"),
                tariff_version="PJM-2026.02")
    base.update(kw)
    return LossFactorRecord(**base)

def test_delivery_factor_applied_once():
    rec = _rec(metered_mwh=Decimal("100"), loss_factor=Decimal("0.97"))
    assert rec.adjusted_volume() == Decimal("97.000000")   # scaled once, cent/microMWh exact

def test_zero_volume_stays_zero():
    rec = _rec(metered_mwh=Decimal("0"), loss_factor=Decimal("0.97"))
    assert rec.adjusted_volume() == Decimal("0.000000")     # valid, not an error

def test_unresolved_factor_refuses_to_settle():
    rec = _rec(loss_factor=None)
    try:
        rec.adjusted_volume()
        assert False, "unresolved factor must raise"
    except ValueError:
        assert True                                          # never settles on a null

def test_mlf_alf_confusion_caught():
    try:
        guard_factor(Decimal("0.03"), "AECO")               # raw MLF, not a delivery factor
        assert False
    except ValueError:
        assert True

Shadow reconciliation runs as a diff: historical_run.adjusted_mwh minus shadow_run.adjusted_mwh must be zero to the last quantized digit for every (node_id, settlement_interval). A non-zero diff is a regression, not a rounding artifact, because every amount is Decimal-quantized. For reproducible data-transformation patterns, the official pandas documentation is the reference for the vectorized joins and group-by aggregations the mapping layer depends on.

Frequently Asked Questions #

What is the difference between a marginal loss factor and a delivery factor in settlement? #

A marginal loss factor (MLF) is published as the incremental loss on the next MWh delivered to a node, while a delivery factor \(\delta = 1 - L\) is the multiplier that scales metered volume to delivered volume. The settlement calculation expects a delivery factor, so an MLF must be resolved to \(\delta\) before it is applied. Multiplying a volume by a raw MLF such as 0.03 instead of a delivery factor of 0.97 collapses the node’s settled energy to near zero — one of the most common silent mapping errors.

Why must loss-factor arithmetic use Python’s decimal module instead of float? #

Delivered volumes feed directly into charges that reconcile against an ISO statement to the cent. Binary floating point cannot represent most decimal fractions exactly, so scaling thousands of intervals by a loss factor accumulates drift that eventually flips a rounding boundary and breaks reconciliation. Representing the factor and the volume as Decimal and quantizing with ROUND_HALF_EVEN keeps the adjusted volume bit-exact and reproducible across preliminary and final runs.

How should a settlement pipeline handle a missing loss factor? #

Never drop the interval and never guess. A left join keeps every metered interval, and missing factors route through a deterministic fallback chain — a rolling per-node median or the prior valid interval — with the substitution recorded in lineage. A node with no usable history stays unresolved, which halts the run rather than settling on an unbounded factor, so no interval is ever billed on a fabricated multiplier.

Should loss factors be applied before or after price multiplication? #

Either is valid, but the choice must be explicit and consistent. Pre-settlement mapping scales metered volume by the delivery factor before the locational marginal price is applied and demands high-frequency validation against SCADA and AMI telemetry. Post-settlement mapping applies the loss adjustment as a separate financial line and requires a rigorous audit trail for FERC accounting. The critical rule is that the factor is applied exactly once — double-application across both paths is a recurring reconciliation break.

Related #

• Mapping transmission loss factors to settlement nodes
• Pricing Logic Implementation
• Imbalance Allocation Algorithms
• Threshold Tuning & Alerts
• Settlement Calculation & Validation Engines