SFTP vs REST API for ISO data retrieval

A settlement pipeline misses a T+1 deadline because it was polling a REST endpoint every minute for a report the ISO only drops to SFTP once the market run finalizes — the code was structurally correct but pointed at the wrong retrieval mode, and no amount of retry logic fixes a channel that will never carry that file. Choosing between SFTP and a REST API is not a matter of taste; it is a settlement-timing and idempotency decision that the two channels answer differently. Within the ISO/RTO Data Format Standards component of the Core Architecture & Market Taxonomy for Energy Settlements framework, this guide puts the two modes side by side, gives working Python for each, and states plainly when to reach for which.

The two channels differ in how the data arrives, not just how you fetch it. SFTP is a scheduled file drop — the ISO writes complete, revision-marked files to a directory you sweep on a cron. REST is a request-response query — you pull a bounded window on demand and get back a payload synthesized for exactly that request. That single distinction cascades into every operational property below.

The decision table #

The trade-offs are best read as a matrix. The columns below are the dimensions that actually change a settlement pipeline’s design; the last row is the summary rule of thumb.

Neither channel is universally better. SFTP wins where the ISO’s canonical settlement artifact is a file and the deadline is tied to the market run finalizing; REST wins where the data is queryable, changes continuously, and you only want the delta. Most production stacks run both — SFTP for the authoritative settlement drop and REST for intraday freshness — and converge them on a single content-hashed landing zone so the downstream is agnostic to how a byte arrived.

Retrieving over SFTP #

SFTP retrieval is a directory sweep: list, filter to the files you have not already landed, download whole, and hash. The revision markers (_REV1, _FINAL) matter — a later _FINAL re-drop supersedes an earlier provisional file for the same settlement date, so the sweep keys on filename and content hash, never on filename alone.

import hashlib
from pathlib import Path

import paramiko  # SSH/SFTP client


def sweep_iso_sftp(
    host: str,
    username: str,
    key_path: str,
    remote_dir: str,
    landing: Path,
    already_landed: set[str],
) -> list[dict]:
    """
    Sweep an ISO SFTP drop for settlement files not yet landed. Keys on the
    remote filename to skip re-downloads, then content-hashes each payload so a
    re-sent _FINAL with changed bytes lands as a new immutable version while an
    identical re-drop is a no-op. Returns one audit record per file examined.
    """
    ssh_key = paramiko.Ed25519Key.from_private_key_file(key_path)
    transport = paramiko.Transport((host, 22))
    transport.connect(username=username, pkey=ssh_key)
    records: list[dict] = []
    try:
        sftp = paramiko.SFTPClient.from_transport(transport)
        for entry in sftp.listdir(remote_dir):
            if entry in already_landed:
                continue  # cheap skip before paying for a download
            payload = sftp.open(f"{remote_dir}/{entry}", "rb").read()
            digest = hashlib.sha256(payload).hexdigest()
            target = landing / f"{digest}__{entry}"
            landed = not target.exists()
            if landed:
                target.write_bytes(payload)
            records.append(
                {"file": entry, "sha256": digest, "landed": landed}
            )
    finally:
        transport.close()
    return records

The sweep produces immutable, hashed artifacts; batching and parallelising many such downloads is the job of Async Batch Processing Pipelines, which fans a directory listing out across workers without re-downloading what is already landed.

Retrieving over REST #

REST retrieval is a windowed pull with explicit back-pressure. Unlike SFTP, the server can and will refuse you — a 429 means slow down, not fail — so the client owns retry, exponential back-off, and pagination bookkeeping. The example below pulls a bounded interval window and honours rate limits; because prices ride along in the payload, any monetary field is cast to Decimal on the way in.

import time
from decimal import Decimal

import requests


def pull_iso_rest_window(
    base_url: str,
    query_params: dict,
    api_token: str | None,
    max_retries: int = 5,
) -> list[dict]:
    """
    Pull one bounded time window from an ISO REST endpoint with rate-limit-aware
    back-off. A 429 triggers exponential wait honouring Retry-After; a page
    cursor is followed until exhausted. Any price field is parsed as Decimal
    from its string form so no settlement value is ever a float.
    """
    headers = {"Authorization": f"Bearer {api_token}"} if api_token else {}
    rows: list[dict] = []
    params = dict(query_params)
    for attempt in range(max_retries):
        resp = requests.get(base_url, params=params, headers=headers, timeout=30)
        if resp.status_code == 429:
            wait_s = int(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait_s)  # respect the ISO's throttle, then retry
            continue
        resp.raise_for_status()
        body = resp.json()
        for item in body.get("data", []):
            rows.append(
                {
                    "node_id": item["node"],
                    "interval_start": item["interval_start_gmt"],
                    # String -> Decimal: never route a price through float().
                    "lmp": Decimal(str(item["lmp"])),
                }
            )
        cursor = body.get("next_cursor")
        if not cursor:
            break  # last page reached
        params["cursor"] = cursor
    return rows

The retry-and-back-off discipline sketched here is treated in depth for high-volume ingestion in Handling rate limits in async trade ingestion. The provider-agnostic version of an authenticated pull loop lives in Automating ETRM sync with Python requests.

Verification #

Confirm retrieval is correct and idempotent before the data settles:

1. Idempotent landing. Re-run the SFTP sweep against the same directory and assert every record returns landed is False — proof the content-hash dedupe blocks a re-processed _FINAL drop from double-counting.
2. Window completeness. For a REST pull of a full trading day, assert the returned interval count matches the expected grid (288 five-minute intervals, or 24 hourly) so a dropped page is caught, not silently settled short.
3. Rate-limit compliance. Inject a 429 in a test double and assert the client waited Retry-After seconds and then succeeded, never abandoning the window on the first throttle.
4. Decimal preservation. Assert every price pulled over REST is a Decimal, so no monetary field was coerced to float between the payload string and the row.
5. Cross-channel reconciliation. Where the same report is available on both channels, diff the SFTP file and the REST window for one date; the canonical values must be identical to the settlement tolerance, confirming the channel choice is transparent downstream.

Compliance Note #

FERC and the ISO tariffs require that settlement figures be reproducible from the operator’s published source, which means the retrieval layer — whichever channel — must land an immutable, content-hashed copy of every payload before transformation, so an auditor can replay the exact bytes that produced a charge. SFTP satisfies this naturally because the file is the artifact; REST satisfies it only if the client persists the raw response keyed by its exact query parameters, since re-issuing a query later may return revised data. Revision markers (_REV, _FINAL) and provisional-to-final REST updates must be preserved as distinct versions, never overwritten, so the settlement date can be reconciled against the schema and data in force when it was published. The market-by-market delivery differences that drive channel choice are catalogued in ISO-NE vs CAISO Reporting Schema Differences.

Frequently Asked Questions #

When should I choose SFTP over a REST API for ISO data? #

Choose SFTP when the ISO’s authoritative settlement artifact is a scheduled file drop and your deadline is tied to the market run finalizing — the file is immutable, revision-marked, and trivially replayable, which is exactly what settlement audit wants. Choose REST when you need fresh, bounded windows on demand, when only a query API exposes the data, or when you want just the incremental change rather than a whole file. Many pipelines run both: SFTP for the canonical drop, REST for intraday freshness, converging on one landing zone.

How do idempotency guarantees differ between the two channels? #

SFTP idempotency keys on filename plus content hash: re-listing a file you have already landed is a cheap skip, and a re-sent _FINAL with identical bytes is a no-op while changed bytes land as a new version. REST idempotency must key on the request parameters plus the content hash of the response, because the same query window can be re-issued and may return revised data; without persisting the raw response, a REST pull is not replayable. Both converge on a content-hashed landing zone that deduplicates regardless of channel.

Do I need to worry about rate limits with SFTP? #

Effectively no — SFTP has no request quota, so the only throttle is your own sweep cadence and the server’s connection limits. REST is the opposite: quotas and 429 throttling are explicit, and the client must implement exponential back-off honouring Retry-After, follow pagination cursors, and treat a 429 as back-pressure to respect rather than a failure to abandon the window on.

Related #

• ISO/RTO Data Format Standards
• Parsing CAISO OASIS XML in Python
• Async Batch Processing Pipelines
• Automating ETRM sync with Python requests