Metering Data Integration

A meter export lands at T+2 with a spring-forward gap where an hour of consumption should be, three channels reporting in watt-hours while the rest report in kilowatt-hours, and a block of intervals silently estimated by the utility’s head-end system with no flag to say so — and if any of that reaches the calculation core unresolved, the settlement charges built on top will be internally consistent and wrong against the ISO. Metering Data Integration is the subsystem that stands between the Meter Data Management (MDM) platform and the money math, turning raw interval usage into a clean, gap-free, correctly-united, settlement-interval-aligned series that the rest of the stack can trust. Within the Core Architecture & Market Taxonomy for Energy Settlements framework it is the tier that feeds validated consumption into Settlement Calculation & Validation Engines; everything the pricing and imbalance layers compute rests on the integrity of what this component emits.

The core discipline here is VEE — validation, estimation, and editing — the industry-standard three-stage process every utility applies to raw meter reads before they are declared settlement-grade. This component re-implements or verifies that VEE state on ingestion, aligns the reads to the market’s settlement interval, normalizes units to a single canonical basis, and flags every interval that rests on an estimate rather than a measured value. The diagram below traces those stages from raw MDM export to the persisted, settlement-ready series.

Every interval that emerges from this component carries three pieces of provenance the money layer depends on: a canonical unit, a settlement-interval-aligned timestamp, and an estimate flag stating whether the value was measured or substituted. Strip any one of those and the downstream ledger loses the ability to prove, at true-up, how much of a settlement rested on measured actuals versus interpolated estimates. The sections below fix the standards this VEE process answers to, walk the ingestion-to-alignment steps in Python, catalogue the failure modes that break interval integrity, and set the thresholds and reconciliation checks that keep the series honest.

Specification & Standards #

Meter data is not free-form. The physical register format is fixed by ANSI C12.19 (the end-device data tables) and transported under ANSI C12.22 or, for consumer-facing exports, the ESPI / Green Button XML schema; the interval records this component parses are structured views over those standards, whose field-level differences across markets are catalogued in ISO/RTO Data Format Standards. What the meter reports and what settlement may use, however, are governed by two further layers.

The first is VEE itself. Most state jurisdictions mandate a documented VEE process — validation, estimation, and editing — as a precondition for using interval data in a billing or settlement determination. Validation applies deterministic checks (register-read continuity, sum checks against the cumulative dial, physical range limits, spike and flatline detection). Estimation fills validated-out or missing intervals using an approved method (like-day profiling, linear interpolation, or reference-channel proration). Editing is the auditable act of substituting an estimate for a measured value, and every edit must be logged with its method and the identity of the interval it replaced.

The second is the interchange and interval framework layered on top: NAESB Wholesale Electric Quadrant business practices define how metered quantities and interval boundaries are expressed for settlement, and the market’s own tariff fixes the settlement interval (typically 5-minute for real-time or hourly for day-ahead) that reads must resolve to. The NAESB business practice standards are the reference against which interval boundaries and quantity conventions are validated. Because these obligations are versioned and jurisdiction-specific, this component treats the applicable VEE ruleset and settlement interval as configuration keyed to the trading day, not hard-coded logic — the same pattern the calculation core uses to pin tariff revisions.

This table is the contract the rest of the component keys off: a source maps to a VEE status, a status maps to a substitution rule, and the substitution rule stamps an estimate flag that travels with the interval into the ledger.

Step-by-Step Implementation #

The component is a sequence of pure transformations over an interval-indexed frame. Each step is idempotent and keyed by (meter_id, interval_start), so a re-run after a late or corrected export replaces exactly the affected intervals. The concrete pandas mechanics of parsing, localizing, and resampling a single export are covered end-to-end in Normalizing interval meter data with pandas; the steps below frame the production process the component orchestrates around that transformation.

Step 1 — Parse and localize the raw export. Read the MDM file, resolve the reported channel and unit metadata, and localize every timestamp to the market clock as a timezone-aware index. Naive timestamps are rejected at the boundary — an ambiguous fall-back hour or a nonexistent spring-forward hour must be resolved explicitly, never collapsed silently.

import pandas as pd
from zoneinfo import ZoneInfo

def load_and_localize(path: str, market_tz: str) -> pd.DataFrame:
    """Parse a raw MDM interval export and localize to the market clock."""
    raw = pd.read_csv(path, parse_dates=["interval_start"])
    # Localize DST-aware: shift nonexistent (spring-forward) forward, flag
    # ambiguous (fall-back) so the duplicated hour is not silently dropped.
    idx = pd.DatetimeIndex(raw["interval_start"]).tz_localize(
        ZoneInfo(market_tz), nonexistent="shift_forward", ambiguous="infer"
    )
    raw = raw.set_index(idx).drop(columns=["interval_start"])
    return raw.sort_index()

Step 2 — Normalize units to a canonical basis. MDM channels report in Wh, kWh, or MWh depending on the register scale, and settlement math expects one basis. Convert every channel to kWh using its declared multiplier before any validation, because range checks and sum checks are only meaningful once the magnitudes are comparable.

from decimal import Decimal

# Canonical settlement basis is kilowatt-hours; map each source unit to it.
UNIT_TO_KWH = {"Wh": Decimal("0.001"), "kWh": Decimal("1"), "MWh": Decimal("1000")}

def normalize_units(df: pd.DataFrame) -> pd.DataFrame:
    """Convert every metered value to a canonical kWh basis."""
    out = df.copy()
    factors = out["unit"].map(lambda u: float(UNIT_TO_KWH[u]))
    out["usage_kwh"] = out["usage_raw"] * factors
    out["unit"] = "kWh"
    return out

Step 3 — Apply VEE validation. Run the deterministic validation gates: physical range limits, interval-sum reconciliation against the cumulative register, spike (velocity) limits, and flatline detection. Intervals that pass are marked MEASURED; those that fail carry a reason code into the estimation step.

Step 4 — Estimate and flag. For each failed or missing interval, apply the approved substitution rule from the specification table and stamp the corresponding estimate flag. Estimation never overwrites a measured value; it only fills where validation left a hole.

def apply_vee_status(df: pd.DataFrame, range_max_kwh: float) -> pd.DataFrame:
    """Assign a VEE status and estimate flag per interval; never overwrite a
    measured value. Downstream substitution reads these flags."""
    out = df.copy()
    valid = (out["usage_kwh"] >= 0) & (out["usage_kwh"] <= range_max_kwh)
    out["estimate_flag"] = "MEASURED"
    out.loc[out["usage_kwh"].isna(), "estimate_flag"] = "ESTIMATED_PROFILE"
    out.loc[valid & out["usage_kwh"].notna(), "estimate_flag"] = "MEASURED"
    out.loc[~valid & out["usage_kwh"].notna(), "estimate_flag"] = "ESTIMATED_INTERP"
    # Interpolate the range failures; profile-fill the missing reads elsewhere.
    interp_mask = out["estimate_flag"] == "ESTIMATED_INTERP"
    out.loc[interp_mask, "usage_kwh"] = (
        out["usage_kwh"].mask(interp_mask).interpolate(method="time")[interp_mask]
    )
    return out

Step 5 — Align to the settlement interval. Resample the validated, unit-normalized series to the market’s settlement grid, summing energy across sub-intervals or forward-distributing where a coarser read must be split. Alignment happens after VEE so estimates and measured values are bucketed consistently. The settlement interval this resamples to is fixed per market by Settlement Cycle Mapping.

Step 6 — Persist with provenance. Write the aligned series append-only, each interval carrying its unit, estimate flag, VEE reason code, and a content hash over its inputs. This is what lets a later true-up prove which intervals were measured and which were substituted, and recompute only the corrected ones.

Edge Cases & Failure Modes #

The steps above are the clean path; production meter data spends much of its time off it. Five failure modes account for most integration incidents, and each needs an explicit, logged handling rule rather than a silent default.

Daylight-saving boundaries. The spring-forward gap deletes an hour and the fall-back overlap duplicates one. A naive index collapses the duplicate and drops the gap, silently mis-summing the affected day. The component localizes with explicit nonexistent and ambiguous handling and validates that the day’s interval count matches the expected 23, 24, or 25 hours for the boundary. The mechanics of resolving both cases are detailed in DST gap and overlap in pandas timestamp normalization.

Zero-volume and negative intervals. A genuine zero (an unmetered outage) and a missing read both look like absence, and a negative value is legitimate on a bidirectional (net-metered) channel but a fault on a unidirectional one. The validation gate keys the sign rule to the channel’s flow direction, so a negative on a delivery-only meter routes to estimation while a negative on a net meter passes.

Stale telemetry. A head-end that stops publishing leaves the last good interval carried forward if the ingestion naively forward-fills. The component bounds any forward-fill to zero intervals for measured energy — a gap is a gap, filled only by the approved estimation method and flagged, never by a stale carry-forward.

Schema drift. A channel that changes its declared unit, adds a column, or renames a field between exports silently corrupts normalization. Inputs are validated against an explicit contract at the boundary — the same discipline enforced in ETRM System Architecture — and a drifted record routes to a dead-letter path rather than propagating.

Unmapped meter identifiers. A read whose meter_id does not resolve to a settlement point must raise, not emit a silent NaN, because an interval with no location cannot be priced. The mapping itself lives in Reference Data Management.

Thresholds & Alerting #

Which of these conditions is tolerable is configuration, not code, and the thresholds adapt by meter class and market. Three parameters govern the component’s alerting posture:

• Estimation rate ceiling. The share of a meter’s daily intervals allowed to be estimated before the read is treated as untrustworthy. A residential channel might tolerate 10%, a settlement-quality generator meter under 1%. Breaching the ceiling routes the meter to review rather than into the preliminary run.
• Gap-length limit. The longest contiguous run of estimated intervals before the estimation method is downgraded from interpolation to a like-day profile — a two-hour gap can be interpolated credibly, a two-day gap cannot.
• Sum-reconciliation band. The tolerance between the summed interval energy and the meter’s cumulative register read. A breach signals dropped or duplicated intervals and blocks alignment until resolved.

Alerts route by tier the same way the calculation core routes exceptions: informational (an estimate was applied and flagged), review-required (an estimation ceiling was breached), and block (a sum-reconciliation failure that would corrupt the run). Tuning these bands so they surface real problems without drowning analysts in noise is the subject of Threshold Tuning & Alerts.

import logging

logger = logging.getLogger("metering_integration")

def check_estimation_rate(df: pd.DataFrame, ceiling: float, meter_id: str) -> None:
    """Emit a tiered alert if the estimated share exceeds the configured ceiling."""
    estimated = df["estimate_flag"].str.startswith("ESTIMATED").mean()
    if estimated > ceiling:
        logger.warning(
            "estimation_rate_breach meter=%s rate=%.3f ceiling=%.3f",
            meter_id, estimated, ceiling,
        )

Testing & Reconciliation #

The component is trustworthy only if its output is reproducible and reconciles two ways: against the meter’s own cumulative register and against the ISO’s settlement-quality meter data at final. The shadow-calculation approach runs the integration twice — once on the preliminary head-end export, once on the validated MDM export at true-up — and diffs the aligned series interval by interval, attributing every difference to either a replaced estimate or a corrected measured value.

Unit tests pin the deterministic behavior of each stage. A DST test asserts a spring-forward day localizes to 23 intervals and a fall-back day to 25; a unit test feeds mixed Wh/kWh/MWh channels and asserts a single canonical basis on output; an estimation test injects a missing interval and asserts the correct flag and substitution rule fire.

import pandas as pd

def test_sum_reconciliation():
    """Aligned interval energy must reconcile to the cumulative register read."""
    aligned = pd.Series(
        [10.0, 12.0, 8.0, 11.0],
        index=pd.date_range("2026-07-17", periods=4, freq="h", tz="America/New_York"),
        name="usage_kwh",
    )
    register_delta_kwh = 41.0            # cumulative dial movement over the window
    interval_sum = aligned.sum()
    assert abs(interval_sum - register_delta_kwh) <= 0.5, (
        f"interval sum {interval_sum} off register {register_delta_kwh}"
    )

Because every interval carries a content hash over its inputs, a reconciliation diff is not just a number — it is traceable to the exact export, VEE ruleset, and estimation method that produced each value, satisfying the same audit expectation the calculation core answers to. Raw parsing and vectorized transformation patterns shared with the ingestion tier are documented under Pandas for Trade Data Processing.

Frequently Asked Questions #

What does VEE mean in meter data integration? #

VEE stands for validation, estimation, and editing — the three-stage process that turns raw meter reads into settlement-grade data. Validation applies deterministic checks (range limits, sum reconciliation against the cumulative register, spike and flatline detection). Estimation fills failed or missing intervals using an approved method such as like-day profiling or linear interpolation. Editing is the auditable substitution of an estimate for a measured value, logged with its method so a later true-up can prove exactly which intervals were estimated.

Why align meter data to the settlement interval after VEE rather than before? #

Alignment resamples the series to the market’s settlement grid — summing sub-intervals or splitting coarser reads. Doing it after VEE ensures measured and estimated values are bucketed on the same rules and that each aligned interval carries a consistent estimate flag. Aligning first would blur the boundary between a measured read and a substituted one, making it impossible to attribute a later reconciliation difference to a specific replaced estimate.

How should estimated intervals be flagged for settlement? #

Every interval carries an estimate flag stating whether it was measured or substituted and, if substituted, by which method — profile, interpolation, or baseline. The flag travels with the interval into the append-only ledger. This lets an auditor quantify precisely how much of a settlement rested on measured actuals versus estimates, and lets the final run replace only the estimated intervals when validated MDM data arrives, recomputing nothing else.

What happens when a meter reports in mixed units across channels? #

Unit normalization runs before validation: each channel is converted to the canonical kilowatt-hour basis using its declared multiplier, because range and sum checks are only meaningful once magnitudes are comparable. A channel that changes its declared unit between exports is treated as schema drift and routed to a dead-letter path rather than normalized against a stale multiplier, preventing a silent thousand-fold error from reaching the money math.

Related #

• Core Architecture & Market Taxonomy for Energy Settlements
• ISO/RTO Data Format Standards
• Settlement Calculation & Validation Engines
• Normalizing interval meter data with pandas