Building Secure API Gateways for ETRM Sync

The failure mode this page prevents is the double-posted settlement charge: a trading-desk client retries a timed-out sync, the second request lands, and the same MWh position posts twice to the upstream Energy Trading and Risk Management (ETRM) engine — surfacing days later as an unexplained reconciliation break that no variance report can attribute. Within the Security & Access Boundaries component of the Core Architecture & Market Taxonomy for Energy Settlements framework, an API gateway engineered for ETRM sync is not a router — it is a deterministic enforcement layer that validates scope, deduplicates on an idempotency key, checks the payload schema with penny-exact typing, and forwards to the upstream engine with bounded retries so a transient outage sheds load instead of cascading into the settlement run.

The sequence below traces a settlement sync request through the gateway’s enforcement stages: header and scope validation, idempotency check, schema validation, and resilient forwarding to the upstream ETRM engine.

The gateway sits in a screened subnet between the public trading desks and the trusted settlement engine, and each enforcement control maps to a stage in its ingress-to-egress pipeline. The topology below shows where every control runs and which HTTP status a caller receives when a control denies the request.

Prerequisites #

This gateway targets Python 3.11+ and an async ASGI stack. Before running the implementation, provision the following.

Data and access dependencies:

• Upstream ETRM endpoint — a reachable POST /settlements/sync URL on the internal ETRM engine, mutual-TLS pinned in production. Local integration work can point at the ETRM System Architecture reference stub.
• OAuth 2.0 client credential — a short-lived JWT carrying a settlements:sync scope, minted from your secrets manager with a TTL bound to the batch window (see Automating ETRM sync with Python requests for the token-acquisition pattern this gateway consumes).
• Idempotency backing store — an in-memory dict works for a single replica; use Redis or Memcached with a shared TTL once you run more than one gateway pod.
• A canonical settlement schema — payloads must already conform to your Schema Validation Frameworks contract before they reach the gateway; the gateway re-validates at the trust boundary rather than trusting the caller.

Install with:

pip install "fastapi>=0.110" "pydantic>=2.6" "httpx>=0.27" "tenacity>=8.2" "uvicorn>=0.29"

Implementation #

The gateway pairs strict schema validation with SHA-256 idempotency keys, bounded retry-and-backoff on transient upstream errors, and explicit failure isolation — returning a clean 503 rather than cascading a settlement-run failure during upstream degradation. Financial fields (position_mwh, settlement_price) are parsed as Decimal, never float, so a penny of imbalance never drifts in from binary rounding. The retry schedule follows a capped exponential curve, \(t_{retry} = \min(t_{max},\ t_{base} \cdot 2^{n})\), so a struggling ETRM upstream is never hammered by a thundering herd on market open.

import time
import json
import hashlib
import logging
from datetime import datetime
from decimal import Decimal
from fastapi import FastAPI, Request, Response, HTTPException
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field, ValidationError, field_validator
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

# Structured JSON logging so every access event is SIEM-ingestible and
# attributable to a settlement run, trader scope, and contract.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    handlers=[logging.StreamHandler()],
)
logger = logging.getLogger("etrm_gateway")

app = FastAPI(title="ETRM Sync Gateway", version="1.0.0")


class SettlementPayload(BaseModel):
    """Canonical sync payload. Money and volume are Decimal, not float,
    to preserve penny accuracy across the trust boundary."""
    settlement_run_id: str = Field(..., pattern=r"^[A-Z0-9]{16}$")
    node_id: str = Field(..., min_length=8, max_length=32)
    position_mwh: Decimal = Field(..., ge=Decimal("-100000"), le=Decimal("100000"))
    settlement_price: Decimal = Field(..., gt=Decimal("0"))
    interval_start_utc: str = Field(
        ..., pattern=r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$"
    )

    @field_validator("interval_start_utc")
    @classmethod
    def validate_utc_instant(cls, v: str) -> str:
        # The regex accepts syntactically valid but non-existent instants
        # (month 13, hour 25). Parse to reject them loudly.
        datetime.strptime(v, "%Y-%m-%dT%H:%M:%SZ")
        return v


class IdempotencyStore:
    """Single-replica idempotency cache. Swap for Redis/Memcached in production."""

    def __init__(self, ttl_seconds: int = 3600):
        self._cache: dict[str, float] = {}
        self._ttl = ttl_seconds

    def check_and_set(self, key: str) -> bool:
        now = time.time()
        # Purge expired keys so the cache does not grow unbounded.
        self._cache = {k: v for k, v in self._cache.items() if now - v < self._ttl}
        if key in self._cache:
            return False
        self._cache[key] = now
        return True


idempotency_store = IdempotencyStore()


@app.middleware("http")
async def enforce_etrm_boundaries(request: Request, call_next):
    # 1. Mandatory settlement headers — reject, do not default, if absent.
    settlement_run = request.headers.get("X-Settlement-Run-ID")
    trader_scope = request.headers.get("X-Trader-Scope")
    if not settlement_run or not trader_scope:
        logger.warning("Missing settlement headers from %s", request.client.host)
        return JSONResponse(
            status_code=400,
            content={"error": "Missing X-Settlement-Run-ID or X-Trader-Scope"},
        )

    # 2. Deny anything without an explicit settlements:sync scope.
    if "settlements:sync" not in trader_scope:
        logger.warning("Scope %r denied for run %s", trader_scope, settlement_run)
        return JSONResponse(status_code=403, content={"error": "scope denied"})

    # 3. Derive the idempotency key from the run id + the exact payload bytes.
    body_bytes = await request.body()
    idem_key = hashlib.sha256(
        f"{settlement_run}:{body_bytes!r}".encode()
    ).hexdigest()
    if not idempotency_store.check_and_set(idem_key):
        logger.info("Duplicate sync suppressed: %s", idem_key)
        return JSONResponse(status_code=409, content={"error": "idempotency conflict"})

    request.state.settlement_run_id = settlement_run
    request.state.trader_scope = trader_scope
    response = await call_next(request)
    response.headers["X-Gateway-Trace-ID"] = idem_key[:16]
    return response


@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type((httpx.ConnectError, httpx.TimeoutException)),
)
async def forward_to_etrm(payload: dict, token: str, timeout: float = 5.0) -> dict:
    async with httpx.AsyncClient(timeout=timeout, http2=True) as client:
        resp = await client.post(
            "https://etrm-upstream.internal/api/v1/settlements/sync",
            content=json.dumps(payload),  # payload already Decimal-serialized to str
            headers={
                "Authorization": f"Bearer {token}",
                "Content-Type": "application/json",
            },
        )
        resp.raise_for_status()
        return resp.json()


@app.post("/v1/settlements/sync", status_code=201)
async def sync_etrm_settlement(request: Request, response: Response):
    try:
        raw = await request.json()
        validated = SettlementPayload.model_validate(raw)
    except ValidationError as exc:
        logger.error("Schema validation failed: %s", exc.errors())
        raise HTTPException(status_code=422, detail=exc.errors())
    except Exception:
        raise HTTPException(status_code=400, detail="Malformed JSON payload")

    # Serialize Decimals as strings so no float ever touches the wire.
    outbound = json.loads(validated.model_dump_json())
    token = request.headers.get("Authorization", "").removeprefix("Bearer ")

    try:
        upstream = await forward_to_etrm(outbound, token)
        logger.info(
            "sync ok | run=%s scope=%s node=%s",
            request.state.settlement_run_id,
            request.state.trader_scope,
            validated.node_id,
        )
        return {"status": "accepted", "upstream_ref": upstream.get("transaction_id")}
    except httpx.HTTPStatusError as exc:
        logger.error("upstream rejected %d: %s", exc.response.status_code, exc.response.text)
        raise HTTPException(status_code=502, detail="upstream settlement engine rejected payload")
    except Exception as exc:
        # Retries exhausted / upstream unreachable: isolate the failure and
        # shed load rather than letting it cascade into the settlement run.
        logger.critical("upstream unavailable after retries: %s", exc)
        raise HTTPException(status_code=503, detail="ETRM sync temporarily unavailable")

Verification Steps #

Confirm each enforcement stage independently before promoting the gateway — a boundary that silently passes is worse than one that fails loudly.

1. Happy path returns 201 with an upstream reference. With a valid token and a well-formed payload, expect {"status": "accepted", "upstream_ref": "..."} and an X-Gateway-Trace-ID header on the response.

   curl -s -X POST http://localhost:8000/v1/settlements/sync \
     -H "X-Settlement-Run-ID: RUN2026070300001A" \
     -H "X-Trader-Scope: settlements:sync" \
     -H "Authorization: Bearer $SETTLEMENT_API_TOKEN" \
     -d '{"settlement_run_id":"RUN2026070300001A","node_id":"PJM-WEST-HUB",
          "position_mwh":"125.50","settlement_price":"42.17",
          "interval_start_utc":"2026-07-03T00:00:00Z"}'

2. The same request replayed returns 409. Resend the identical body and expect {"error": "idempotency conflict"}. This is the single check that proves the double-post failure mode is closed.

3. A missing or wrong scope returns 403; missing headers return 400. Drop X-Trader-Scope or send market-data:read and confirm the deny path fires before any upstream call is made.

4. Decimal fidelity holds. Assert that str(SettlementPayload.model_validate(...).settlement_price) round-trips exactly — no 42.169999999. In a shadow check, sum position_mwh * settlement_price over a batch and reconcile against the ETRM engine’s total; the diff must be exactly Decimal("0.00").

   from decimal import Decimal
   p = SettlementPayload.model_validate(
       {"settlement_run_id": "RUN2026070300001A", "node_id": "PJM-WEST-HUB",
        "position_mwh": "125.50", "settlement_price": "42.17",
        "interval_start_utc": "2026-07-03T00:00:00Z"})
   assert isinstance(p.settlement_price, Decimal)
   assert p.position_mwh * p.settlement_price == Decimal("5292.335")

5. Upstream outage degrades to 503, not a hang. Point the forwarder at an unreachable host and confirm the request returns 503 after the retry budget is spent, with a CRITICAL audit line — never a socket timeout leaking to the client.

Compliance Note #

This gateway realizes the ingress trust boundary that NERC CIP-005 (electronic security perimeter) and CIP-007 (systems security management) expect market participants to enforce in code: mTLS termination, scope enforcement, and rejection of malformed payloads all happen before data reaches the settlement engine. The immutable, structured audit log — one JSON line per accept, deny, and upstream call, keyed to settlement run and trader scope — is the evidence NERC CIP-011 (information protection) and SOX IT general controls require to attribute any position change to a specific principal. Idempotency and the append-only forwarding contract satisfy FERC data-integrity expectations that settlement records be verifiable and tamper-evident; corrections must be issued as versioned true-ups, never as an overwrite. Align the retry, timeout, and deny-count parameters with the same Threshold Tuning & Alerts engine used for settlement-variance monitoring, and validate the token TTL against the batch window before each production release. The NIST SP 800-53 Rev. 5 access-control (AC) baselines — AC-3 enforcement and AC-6 least privilege — map directly to the scope gate above; the async and dependency-injection patterns scale under concurrent trading-desk load per the FastAPI documentation.

Frequently Asked Questions #

Why hash the request body into the idempotency key instead of trusting a client-supplied key? #

Because a client that crashes mid-retry may not resend the same idempotency header, but it will resend the same settlement payload. Deriving the key from the settlement run id plus the exact body bytes makes the gateway deduplicate on intent, not on a header the caller might mangle. If you also accept a client key, treat it as an additional guard, not the sole one.

Should the gateway use Decimal or float for position and price fields? #

Always Decimal. A float cannot represent 42.17 exactly, so summing thousands of intervals accumulates binary-rounding drift that shows up as a sub-cent reconciliation break the ETRM engine cannot explain. Parsing money and volume as Decimal in Pydantic and serializing them back to strings on the wire keeps penny accuracy end to end.

What should happen when the upstream ETRM engine is unreachable? #

Fail closed with a 503 after a bounded retry budget, never hang or partially post. The retry uses capped exponential backoff so a degraded upstream is not overwhelmed on market open, and the failure is isolated to the single sync request rather than cascading into the whole settlement run. A secondary reconciliation queue can absorb the shed load for replay once the upstream recovers.

Where should schema validation live — at the gateway or inside the ETRM engine? #

At the gateway, in the DMZ, before the payload crosses the trust boundary. Validating with the same Schema Validation Frameworks contract used for trade ingestion means a drifted or malformed record is rejected with a 422 before it can corrupt an hourly settlement factor, rather than being coerced deep inside the calculation engine.

Related #

• Security & Access Boundaries
• Automating ETRM sync with Python requests
• Schema Validation Frameworks
• ETRM System Architecture
• Threshold Tuning & Alerts