Regulatory Filing Preparation

The settlement ledger closed clean, every line item reconciled to the ISO statement to the cent — and then the FERC Electric Quarterly Report submission bounced because two transactions carried a counterparty name that did not match the registered legal entity, and the whole quarter’s filing missed its deadline window. Regulatory Filing Preparation is the compliance-output stage that stops that failure: it consumes settled ledgers from Settlement Calculation & Validation Engines and normalized trade records from Trade Ingestion & Matching Workflows, then assembles, validates, and hashes the exact reports the federal and state regulators expect on a fixed calendar. Where the Core Architecture & Market Taxonomy for Energy Settlements section defines the market vocabulary and the calculation section produces the numbers, this section is where those numbers leave the building — and once a filing is submitted to a regulator it is a legal representation, so every figure must trace back to a settled line item and every submission must be provably identical to what was reviewed and approved internally. The read and write paths that touch filing data are governed by the controls in Security & Access Boundaries, because a report that cannot prove who assembled it and that it was not altered in transit is not a defensible filing.

The diagram below traces how settled ledgers and trade records fan out into the three filing streams this section owns, each ending in a regulator-bound package with its own cadence and format.

Each stream reads from the same reconciled source of truth but obeys a different regulator, a different schema, and a different clock. FERC wants a quarterly XML or CSV extract of every wholesale transaction and contract. NERC wants continuous, tamper-evident evidence that the systems producing settlement-grade data enforced access control. State commissions want annual proof that renewable obligations were met and retail settlements were fair. The sections below define the mandates that govern each stream, the traceability discipline that binds a filed figure back to a settled interval, the assembly architecture, a working Python package-builder, and the audit controls that make a submission defensible years later.

Market & Regulatory Context #

A regulatory filing is not a report the company chooses to format however it likes — its content, cadence, and delivery mechanism are fixed by statute and by the tariffs and reliability standards layered on top. Three regulatory bodies dominate the filing surface for a US wholesale market participant, and each has a distinct enforcement posture.

FERC (Federal Energy Regulatory Commission). Under Section 220 of the Federal Power Act and the rules implementing it, market-based rate holders and most jurisdictional sellers must file the Electric Quarterly Report — a structured extract of every wholesale power sale and every transmission transaction for the quarter, plus the underlying contract and product data. The EQR is filed through FERC’s dedicated portal in the agency’s XML or CSV schema; a rejected or late filing is a compliance event that can draw penalties and, in the extreme, suspension of market-based rate authority. Because the EQR reprints the same transactions the settlement engine already priced, the report must agree with the ledger it was drawn from to the transaction.

NERC (North American Electric Reliability Corporation). The Critical Infrastructure Protection (CIP) standards family — CIP-002 through CIP-014 — governs the cyber and physical security of systems that support the bulk electric system. For a settlement operation, the relevant obligation is evidentiary: CIP-007 requires security event monitoring and logging, CIP-010 requires configuration-change and access tracking, and the standards collectively require that logs be retained and producible on demand for a compliance audit. NERC audits on a multi-year cycle and expects tamper-evident evidence, not a spreadsheet a technician could edit after the fact.

State PUCs (Public Utility Commissions). Retail-facing obligations — Renewable Portfolio Standard (RPS) compliance, Renewable Energy Credit (REC) retirement, and retail settlement true-ups — are enforced by each state’s commission on its own calendar and in its own format. A load-serving entity proves it retired enough RECs to cover its renewable obligation for the compliance year, and the retirement events are logged in a regional tracking system (WREGIS, PJM-GATS, NEPOOL-GIS, or M-RETS) whose serial numbers must appear in the state filing.

The unifying constraint across all three is traceability: every figure in every filing must resolve to a metered interval, a settled transaction, or a logged event that the company can produce on demand. That is why filing preparation is treated as a downstream stage of settlement rather than an independent reporting function — it inherits the reconciled ledger and the audit trail, and it must never introduce a number that the settlement record cannot substantiate.

Core Concepts & Filing Calendar #

Before any package is assembled, the vocabulary and the calendar have to be fixed, because a filing that is internally consistent but delivered in the wrong window or the wrong schema is still a compliance failure. A few terms recur across all three streams.

Filing period. The window of activity a report covers — a calendar quarter for EQR, a compliance year for RPS, a rolling retention window for CIP logs. Every figure in a filing must fall inside its declared period and nowhere else.

Submission deadline. The date by which the regulator must receive the package. Deadlines are keyed off the period close, not off the day the data became available, so a late meter correction does not extend the clock — it triggers an amended filing instead.

Filing manifest. The signed index of what a package contains: each file, its byte size, its SHA-256 digest, the source ledger run it was drawn from, and the responsible submitter. The manifest is what makes a submission provable after the fact.

Amended filing. A superseding submission that corrects a previously filed period, typically driven by a resettlement or a late true-up. Amended filings reference the original and carry a delta, never a silent overwrite — the same append-only discipline the settlement ledger uses.

The table below fixes the cadence, format, and traceability anchor for each stream. Exact dates vary by regulator, entity type, and state, so these are the standard windows a filing scheduler keys off; the per-market specifics are configuration, not hard-coded logic.

Two anchors in that table matter more than the dates themselves. First, every wholesale row traces to a settled transaction, which is why the Settlement Cycle Mapping that keys the ledger by run and interval is the same key the filing extract selects on. Second, the NERC rows trace to logged events rather than to financial line items — a different kind of evidence, produced continuously rather than assembled quarterly, and handled by NERC CIP Compliance Logging.

Traceability: From Settled Line Item to Filed Figure #

The single property that separates a defensible filing from a liability is that every filed figure carries a verifiable path back to the settled record it came from. A regulator questioning a reported megawatt-hour or dollar amount does not accept “our system computed it” — it expects the participant to produce the metered interval, the price version, the settlement run, and the transaction identifier that add up to the filed number.

Filing preparation enforces this by never re-deriving a figure. The EQR transaction extract, for instance, does not recompute a sale price; it selects the already-settled settlement_amount from the ledger and reprints it, carrying the ledger’s run_id and per-line hash into the filing manifest. The reported quarterly total is then simply the sum of the selected line items, and reconciliation is an equality check rather than a recalculation:

$$\text{Filed}{q} = \sum{i \in q} \text{settlement_amount}i \quad\text{and}\quad \text{Filed}{q} \stackrel{!}{=} \text{Ledger}_{q}$$

where the sum runs over every settled line item \(i\) whose interval falls in filing quarter \(q\). If the filed quarterly figure does not equal the ledger’s own quarterly total to the cent, the package is blocked before submission — a mismatch means either the selection filter is wrong or a line item changed after the filing was drawn, and either way the report cannot go out. Because the ledger is append-only and idempotent on \((run_id, node, interval)\), the selection is fully reproducible: re-running the extract against the same ledger snapshot yields a byte-identical package, which is exactly what an amended-versus-original comparison relies on. When a resettlement changes historical figures, Resettlement & True-Up Processing produces the corrected line items and the filing layer emits an amended package referencing the original rather than mutating it.

Architecture & Integration Patterns #

The plumbing that assembles a filing has to be as disciplined as the calculation engine that fed it, because a filing bug is discovered by a regulator, not by an internal reconciliation. Four properties keep the assembly layer trustworthy.

Read-only over an immutable snapshot. The filing layer never writes back to the settlement ledger — it reads a pinned snapshot identified by run_id and produces an output package. This guarantees that regenerating a filing months later reproduces the same numbers, and it keeps the filing process from ever perturbing the settled record it is reporting on.

Schema enforcement at the regulator boundary. Just as inputs are validated on the way into the calculation engine, the assembled package is validated against the regulator’s published schema before it leaves — field presence, enumerations (transaction class, product type, rate units), and referential integrity (every transaction’s counterparty resolves to a registered entity). A record that fails is routed to an exception queue, not shipped in a package that the portal will reject.

Idempotent, versioned package identity. Each package is keyed by (filing_stream, period, run_id) and carries a content hash. Re-assembling the same period from the same snapshot yields the same package identity; a genuine change (a resettlement) produces a new version that references the prior one. This is what lets the system distinguish “regenerated identically” from “amended with a delta.”

Least-privilege assembly and delivery. The credentials that read the ledger, sign the manifest, and deliver to the regulator portal are separated and scoped, following the boundary model in Security & Access Boundaries. The assembly job holds read-only ledger access; a distinct, tightly held credential performs the signed submission, so a compromise of the assembly stage cannot itself file to a regulator.

High-volume periods — a quarter with millions of wholesale transactions across several markets — fan the formatter out concurrently over independent partitions, joining only at the manifest stage, reusing the batch discipline documented in Async Batch Processing Pipelines. Because each partition is a pure selection over the immutable snapshot, partitioning never changes the result: the concatenated, sorted package is identical to a single-threaded assembly.

Filing Package Assembly in Python #

The core of this stage is a package builder that reads a settled ledger, aggregates it into the regulator’s reporting shape, quantizes every money figure with decimal.Decimal, and emits a signed manifest whose per-file and package-level SHA-256 digests make the submission tamper-evident. The pattern below assembles a FERC EQR-style transaction package; the same structure — select, aggregate with Decimal, serialize, hash — drives the CIP and PUC streams with a different formatter. Note that money never touches a binary float, and the manifest hash is computed over canonical bytes so a regenerated package hashes identically.

import hashlib
import json
from dataclasses import dataclass, field
from datetime import datetime, timezone
from decimal import Decimal, ROUND_HALF_UP
from pathlib import Path
from typing import Dict, List
import pandas as pd

CENTS = Decimal("0.01")


@dataclass
class FilingManifest:
    """Signed index of a regulatory filing package."""
    filing_stream: str          # "FERC_EQR" | "NERC_CIP" | "STATE_RPS"
    period: str                 # e.g. "2026Q2"
    run_id: str                 # settlement run the package was drawn from
    submitter: str
    files: List[Dict] = field(default_factory=list)
    assembled_at: str = ""
    package_sha256: str = ""


def _sha256_bytes(payload: bytes) -> str:
    """Deterministic content digest for a serialized file."""
    return hashlib.sha256(payload).hexdigest()


def build_eqr_package(
    ledger: pd.DataFrame,   # settled line items for the filing period
    filing_stream: str,
    period: str,
    run_id: str,
    submitter: str,
    out_dir: Path,
) -> FilingManifest:
    """
    Assemble a regulator-ready transaction package from a settled ledger.
    Every reported amount is reprinted from the ledger (never recomputed) and
    quantized with Decimal; the manifest carries per-file and package hashes so
    a re-assembly of the same snapshot is byte-identical.
    """
    out_dir.mkdir(parents=True, exist_ok=True)
    manifest = FilingManifest(
        filing_stream=filing_stream, period=period,
        run_id=run_id, submitter=submitter,
        assembled_at=datetime.now(timezone.utc).isoformat(),
    )

    # Aggregate settled amounts per counterparty and product for the EQR
    # transaction summary. Sum in Decimal to keep the filed total bit-exact
    # against the ledger's own period total.
    transactions: List[Dict] = []
    grouped = ledger.groupby(["counterparty_id", "product_type"], sort=True)
    for (counterparty_id, product_type), rows in grouped:
        total_amount = sum(
            (Decimal(str(v)) for v in rows["settlement_amount"]),
            start=Decimal("0"),
        ).quantize(CENTS, rounding=ROUND_HALF_UP)
        total_mwh = sum(
            (Decimal(str(v)) for v in rows["delivered_mwh"]),
            start=Decimal("0"),
        ).quantize(Decimal("0.001"), rounding=ROUND_HALF_UP)
        transactions.append({
            "counterparty_id": counterparty_id,
            "product_type": product_type,
            "quantity_mwh": str(total_mwh),
            "settlement_amount_usd": str(total_amount),
            "source_run_id": run_id,
        })

    # Serialize deterministically: sorted keys, no trailing whitespace, so the
    # digest is stable across machines and re-runs.
    payload = json.dumps(
        {"period": period, "transactions": transactions},
        sort_keys=True, separators=(",", ":"),
    ).encode("utf-8")
    file_path = out_dir / f"{filing_stream}_{period}_transactions.json"
    file_path.write_bytes(payload)

    manifest.files.append({
        "filename": file_path.name,
        "byte_size": len(payload),
        "sha256": _sha256_bytes(payload),
        "record_count": len(transactions),
    })

    # Package hash binds the manifest's file digests together: any change to any
    # file changes the package hash, making post-assembly tampering detectable.
    manifest_core = json.dumps(
        {"stream": filing_stream, "period": period, "run_id": run_id,
         "files": manifest.files},
        sort_keys=True, separators=(",", ":"),
    ).encode("utf-8")
    manifest.package_sha256 = _sha256_bytes(manifest_core)

    (out_dir / "manifest.json").write_bytes(
        json.dumps(manifest.__dict__, sort_keys=True, indent=2).encode("utf-8")
    )
    return manifest


def reconcile_filed_total(ledger: pd.DataFrame, manifest_file: Path) -> bool:
    """
    Gate: the package's filed total must equal the ledger's period total to the
    cent, or the filing is blocked. This is an equality check, not a recompute.
    """
    filed = json.loads(
        (manifest_file.parent /
         f"{json.loads(manifest_file.read_text())['filing_stream']}_"
         f"{json.loads(manifest_file.read_text())['period']}_transactions.json").read_text()
    )
    filed_total = sum(
        (Decimal(t["settlement_amount_usd"]) for t in filed["transactions"]),
        start=Decimal("0"),
    ).quantize(CENTS, rounding=ROUND_HALF_UP)
    ledger_total = sum(
        (Decimal(str(v)) for v in ledger["settlement_amount"]),
        start=Decimal("0"),
    ).quantize(CENTS, rounding=ROUND_HALF_UP)
    return filed_total == ledger_total

The builder keeps the aggregation vectorizable while forcing every persisted money value through Decimal, and the reconcile_filed_total gate refuses to let a package proceed unless its total equals the source ledger’s total exactly — the equality that guarantees the filing agrees with the settled record. In production the transaction serializer is swapped for the regulator’s exact schema (FERC’s XML for EQR, a tracking-system CSV for RPS), but the select-aggregate-hash spine is identical, and the detailed EQR extract is built in Generating FERC EQR transaction reports in Python.

Validation & Compliance Requirements #

Everything above exists to make one claim defensible: the filing the regulator received is provably the filing the company assembled from its settled records, and it can be reproduced on demand. Five concrete controls carry that claim.

• Total-to-ledger reconciliation. Before submission, the package’s aggregate figures are checked for equality against the source ledger’s own period totals. A mismatch blocks the filing rather than shipping a number the settlement record cannot substantiate.
• Per-file and package hashing. Every file in a package carries a SHA-256 digest, and the manifest carries a package-level digest over those file hashes. Any post-assembly change is detectable, and a regenerated package can be proven byte-identical to the original.
• Schema conformance at the boundary. The package is validated against the regulator’s published schema — required fields, enumerations, and referential integrity to registered entities — so a rejectable submission never leaves the building.
• Append-only filing register. Every submission is recorded with its stream, period, run_id, package hash, submitter, and timestamp. Amended filings reference the original; nothing is overwritten, so the full filing history is reconstructable.
• Tamper-evident evidence logs. The access and configuration events proving that the systems producing filing-grade data were controlled are logged immutably per the CIP obligations, with the specifics handled by Building tamper-evident audit logs for NERC CIP.

Together these satisfy the FERC expectation that every filed transaction traces to a settled interval, the NERC expectation that supporting systems produce tamper-evident evidence on demand, and the state expectation that renewable and retail obligations are provably met and reproducible under audit.

Working Across the Filing Streams #

This section is composed of three focused reporting streams, each targeting a different regulator, cadence, and schema. Use these as the entry points into the filing layer:

• FERC EQR Filing Automation — quarterly transaction and contract reporting to FERC, mapping settled wholesale line items into the agency’s XML/CSV schema and reconciling the filed totals to the ledger.
• NERC CIP Compliance Logging — tamper-evident audit and access logging for the critical settlement systems, producing the continuous, hash-chained evidence a CIP audit requires.
• State PUC Reporting Workflows — RPS and REC compliance plus retail settlement reporting to state commissions, tying tracking-system retirement serials and retail true-ups to each state’s format and deadline.

Frequently Asked Questions #

Why must a filed figure be reprinted from the ledger rather than recomputed? #

A regulatory filing is a legal representation, and a regulator questioning a number expects the participant to produce the exact settled record it came from. If the filing layer recomputed the figure, any drift between the settlement engine’s logic and the filing logic would produce a report that disagrees with the company’s own books — an indefensible position under audit. Reprinting the already-settled settlement_amount and carrying its run_id and hash into the manifest means the filed total is, by construction, the sum of the settled line items, and reconciliation is a simple equality check rather than a second calculation that can diverge.

How does a filing package stay tamper-evident after it is assembled? #

Each file in the package carries a SHA-256 digest computed over its canonical serialized bytes, and the manifest carries a package-level hash over the collection of file digests. Because serialization is deterministic — sorted keys, fixed separators — regenerating the package from the same ledger snapshot produces byte-identical files and therefore identical hashes. Any later modification to any file changes its digest and the package hash, so a mismatch between the recorded package hash and a recomputed one is proof of alteration. The hashes are stored in an append-only filing register at submission time.

What triggers an amended filing instead of a new one? #

An amended filing is driven by a change to already-filed data — most commonly a resettlement or a late true-up that revises historical line items after the original period was submitted. Because the settlement ledger is append-only and idempotent, the corrected figures arrive as new versioned records, and the filing layer detects that the period’s total no longer matches the originally filed package. It then assembles a superseding package that references the original submission and carries the delta, rather than silently overwriting the prior filing, preserving the full audit trail of what was filed and when.

Are NERC CIP filings the same kind of report as a FERC EQR? #

No — they are fundamentally different evidence. A FERC EQR is a periodic financial extract: a quarterly snapshot of settled transactions assembled from the ledger. NERC CIP obligations are continuous and evidentiary: they require tamper-evident logs proving that the systems producing settlement-grade data enforced access control and tracked configuration changes throughout the period. One is assembled on a quarterly clock and reconciled to money; the other is produced continuously and reconciled to logged events. Both flow through the same signed-package and append-only-register discipline, but their content, cadence, and traceability anchors differ.

Related #

• Settlement Calculation & Validation Engines
• Trade Ingestion & Matching Workflows
• Core Architecture & Market Taxonomy for Energy Settlements
• Security & Access Boundaries
• Resettlement & True-Up Processing