Parsing CAISO OASIS XML in Python

A settlement job pulls a CAISO OASIS LMP report, feeds the XML straight into a DataFrame, and every price lands as a float — so the 34.56000000000001 that comes back from float("34.56") propagates into a T+3 charge that disagrees with the market operator by a fraction of a cent per interval, per node, across thousands of rows. The fix is not rounding at the end; it is never letting the number become a float in the first place. Within the ISO/RTO Data Format Standards component of the Core Architecture & Market Taxonomy for Energy Settlements framework, this guide walks a CAISO OASIS SingleZip payload from compressed bytes to a tidy, interval-indexed frame whose price columns are decimal.Decimal, whose timestamps are UTC-anchored, and whose row count you can reconcile against the report header before anything settles.

OASIS does not hand you a flat CSV. A SingleZip request returns a ZIP archive that contains one XML document, and that document is namespaced, deeply nested, and encodes every price component as a repeated REPORT_DATA element tagged by a DATA_ITEM code rather than a column. Parsing it correctly means three things done in order: unzip without touching disk, walk the tree with the namespace bound, and pivot the long DATA_ITEM records into the wide component layout that settlement expects.

The flow below traces that transformation end to end.

Prerequisites #

• Python 3.11+ for zoneinfo in the standard library and modern typing.
• pandas 2.x for the pivot and interval index. No third-party XML library is required — zipfile and xml.etree.ElementTree from the standard library are enough, and defusedxml is recommended if the payload provenance is not fully trusted.
• decimal from the standard library for exact price arithmetic; the goal is that no price is ever a float.
• Data dependency: a CAISO OASIS SingleZip response for a price report such as PRC_LMP (Day-Ahead) or PRC_INTVL_LMP (Real-Time). Public LMP reports need no API key, but the request must carry the queryname, market_run_id, startdatetime, and enddatetime parameters OASIS expects.
• Namespace: OASIS documents bind the default namespace http://www.caiso.com/soa/OASISReport_v1.xsd; every element you address must be qualified with it. The raw retrieval mechanics — polling, retries, and the zip envelope itself — belong to SFTP vs REST API for ISO data retrieval, the sibling guide this parser consumes the output of.

Implementation #

The parser is deliberately staged: unzip, extract the report header for reconciliation, stream the REPORT_DATA records into rows, then pivot and cast. Keeping the value as a string all the way to the Decimal constructor is the single most important rule — Decimal(str_value) is exact, Decimal(float_value) inherits the binary-float error the string was protecting you from.

import io
import zipfile
import xml.etree.ElementTree as ET
from decimal import Decimal, InvalidOperation
from typing import Iterator

import pandas as pd

# OASIS binds this default namespace on every element.
OASIS_NS = {"o": "http://www.caiso.com/soa/OASISReport_v1.xsd"}


def unzip_oasis_payload(zip_bytes: bytes) -> bytes:
    """
    Extract the single XML document from a CAISO OASIS SingleZip response
    without writing to disk. OASIS archives normally hold exactly one .xml
    member; a payload with none is an OASIS error envelope, not data.
    """
    with zipfile.ZipFile(io.BytesIO(zip_bytes)) as archive:
        xml_members = [n for n in archive.namelist() if n.lower().endswith(".xml")]
        if not xml_members:
            raise ValueError("OASIS payload contained no XML member (likely an error report)")
        return archive.read(xml_members[0])


def _text(element: ET.Element, tag: str) -> str | None:
    """Return the stripped text of a namespaced child, or None if absent."""
    child = element.find(f"o:{tag}", OASIS_NS)
    return child.text.strip() if child is not None and child.text else None


def iter_report_rows(xml_bytes: bytes) -> Iterator[dict]:
    """
    Walk every REPORT_DATA node and yield one long-format record per node.
    Each record carries the DATA_ITEM code (LMP_PRC, LMP_ENE_PRC, ...), the
    resource name, the GMT interval boundaries, and the raw value STRING.
    The value is never coerced to float here; it stays a string until Decimal.
    """
    root = ET.fromstring(xml_bytes)
    for data in root.iterfind(".//o:REPORT_DATA", OASIS_NS):
        yield {
            "data_item": _text(data, "DATA_ITEM"),
            "node_id": _text(data, "RESOURCE_NAME"),
            "interval_start_gmt": _text(data, "INTERVAL_START_GMT"),
            "interval_end_gmt": _text(data, "INTERVAL_END_GMT"),
            "interval_num": _text(data, "INTERVAL_NUM"),
            "value_str": _text(data, "VALUE"),
        }


def to_decimal(value_str: str | None) -> Decimal | None:
    """Exact string-to-Decimal cast; suppressed or malformed values become None."""
    if value_str in (None, "", "NULL", "-999", "-999.0"):
        return None
    try:
        return Decimal(value_str)
    except (InvalidOperation, TypeError):
        return None


def parse_oasis_lmp(zip_bytes: bytes) -> pd.DataFrame:
    """
    Parse a CAISO OASIS LMP SingleZip payload into a tidy, interval-indexed
    frame. Rows are keyed by (node_id, interval_start), and each DATA_ITEM
    code becomes its own Decimal-valued price column.
    """
    xml_bytes = unzip_oasis_payload(zip_bytes)
    long_df = pd.DataFrame(iter_report_rows(xml_bytes))
    if long_df.empty:
        raise ValueError("No REPORT_DATA rows parsed; check queryname and date window")

    # UTC-anchor the interval boundary. OASIS GMT strings already carry a zone,
    # so parse as UTC and keep it explicit rather than dropping to naive local.
    long_df["interval_start"] = pd.to_datetime(
        long_df["interval_start_gmt"], utc=True
    )

    # Keep the value column as Python object holding Decimal; do NOT let pandas
    # infer a float64 dtype, which would defeat the exact cast.
    long_df["value"] = long_df["value_str"].map(to_decimal)

    # Pivot the DATA_ITEM codes into columns so each node/interval is one row
    # with MEC/MCC/MCL/LMP side by side, ready for the additive integrity check.
    wide = long_df.pivot_table(
        index=["node_id", "interval_start"],
        columns="data_item",
        values="value",
        aggfunc="first",
    ).rename(
        columns={
            "LMP_PRC": "lmp",
            "LMP_ENE_PRC": "energy",
            "LMP_CONG_PRC": "congestion",
            "LMP_LOSS_PRC": "loss",
        }
    )
    return wide.sort_index()

The pivot leaves you with one row per (node_id, interval_start) and up to four Decimal columns. Because CAISO reports the nodal price and its three marginal components as separate DATA_ITEM codes, the additive identity that governs every LMP holds on the parsed frame:

$$LMP_n = \text{MEC}_n + \text{MCC}_n + \text{MCL}_n$$

where \( \text{MEC} \), \( \text{MCC} \), and \( \text{MCL} \) are the marginal energy, congestion, and loss components CAISO labels LMP_ENE_PRC, LMP_CONG_PRC, and LMP_LOSS_PRC. Checking that identity on the Decimal columns — not on rounded floats — is the difference between catching a truncated payload and settling on one. The component decomposition itself is validated downstream by Calculating locational marginal pricing in Python, which reconstructs each price from the market clearing duals.

The DATA_ITEM codes you will encounter in the common LMP reports map as follows.

A defensive parser drops non-price DATA_ITEM codes rather than letting them create phantom columns, and it treats CAISO’s -999 sentinel and empty VALUE elements as None so a suppressed interval never silently reads as a real price. This is the same normalization discipline that separates a namespaced XML feed from a flat CSV one, contrasted in Parsing CSV vs XML trade feeds with pandas.

Verification #

Confirm the parse before the frame reaches settlement:

1. Interval count. A single trading day of Real-Time PRC_INTVL_LMP for one node must yield exactly 288 five-minute rows (assert len(frame.loc[node_id]) == 288); Day-Ahead PRC_LMP yields 24. A short count means the date window clipped an interval or the pivot collapsed duplicates.
2. Decimal dtype. Assert no price column is a float: assert all(isinstance(v, Decimal) for v in frame["lmp"].dropna()). A single float in the column means a value slipped through float() somewhere upstream.
3. Additive integrity. On rows where all four components are present, assert ((frame["energy"] + frame["congestion"] + frame["loss"]) - frame["lmp"]).abs().max() <= Decimal("0.00001") — the identity must hold in Decimal, not merely approximately in float.
4. Header reconciliation. Extract the report header’s stated row count and compare it to len(long_df) before the pivot; a mismatch means the XML was truncated in transit and must be re-pulled, not parsed.
5. UTC anchoring. assert frame.index.get_level_values("interval_start").tz is not None so no interval is stored as naive local time, which would corrupt the Settlement Cycle Mapping grid that keys on this index.

Compliance Note #

FERC requires that a settled figure be reproducible from the source data the market operator published, which is why the raw zip should be content-hashed and retained in a landing zone before parsing, and why every price is carried as Decimal rather than float — a binary-float rounding drift of a hundredth of a cent, compounded across nodes and intervals, is an audit discrepancy, not a display artifact. CAISO’s data dictionary and the Business Practice Manual for Settlements and Statements define the DATA_ITEM codes and interval conventions used here, and because CAISO versions its OASIS schema against a tariff effective date, the DATA_ITEM-to-column mapping must be pinned to the schema version in force for the settlement date being parsed. Any row whose additive integrity check fails must route to manual review rather than settling on an unverified decomposition. The market-by-market schema differences that motivate this discipline are catalogued in ISO-NE vs CAISO Reporting Schema Differences.

Frequently Asked Questions #

Why cast OASIS prices to Decimal instead of parsing them as floats? #

Because settlement is exact arithmetic and binary floats are not. Decimal("34.56") stores exactly 34.56; float("34.56") stores 34.560000000000002, and that residual compounds when you multiply by MWh across thousands of intervals and nodes. The rule is to keep the VALUE element as a string from XML all the way into the Decimal() constructor — never route it through float(), and never build a Decimal from a float, which would inherit the very error you were avoiding.

How do I handle the XML namespace when finding elements? #

Every OASIS element is qualified with the default namespace http://www.caiso.com/soa/OASISReport_v1.xsd, so a bare root.find("REPORT_DATA") returns nothing. Bind the namespace to a prefix in a dict — {"o": "http://www.caiso.com/soa/OASISReport_v1.xsd"} — and address every element with that prefix, for example root.iterfind(".//o:REPORT_DATA", ns). Using iterfind with a namespaced path is both correct and memory-friendly for large Real-Time reports.

What does an empty parse usually mean? #

An OASIS SingleZip that unzips to XML but yields zero REPORT_DATA rows is almost always an error envelope rather than data — a bad queryname, a date window outside the published horizon, or a rate-limit rejection returned as a valid ZIP. Check for the XML member first, then assert a non-empty row set before pivoting, so a rejected request fails loudly at parse time instead of producing an empty settlement frame downstream.

Related #

• ISO/RTO Data Format Standards
• SFTP vs REST API for ISO data retrieval
• ISO-NE vs CAISO Reporting Schema Differences
• Calculating locational marginal pricing in Python