Data

Companies restate their accounts, and the data provider overwrites the old figures with the new ones. Download the same quarter today and next year and the numbers can differ. A backtest is only reproducible against the exact bytes it originally read.

So the cache is treated as a permanent record. Each snapshot is a dated version that is never edited or deleted, and every result names the version it came from.

data/cache/
  2026-05-18-v3/              one frozen snapshot
    screener/                 the screen as it ran that day
    fred/BAA10Y.parquet       one file per economic series
    <endpoint>/<TICKER>.parquet
  paper-2026-07-14-v1/        a nightly snapshot for the paper book

a 90-day lag on results

Companies publish a quarter's figures weeks after it closes. A signal may only use a quarter once 90 days have passed, so no backtest acts on numbers before the public could have.

the index as it was

The universe on any past date is rebuilt from 767 companies that have been in the S&P 500 at some point, using a snapshot of the index at each quarter-end. A company counts from the first quarter-end after it joined until the last one before it left. Using today's members would quietly drop every company that failed.

a fresh snapshot every night

Each night the paper book's data is frozen into a new version named for the date. The next morning's screen reads that snapshot with downloads switched off, so it cannot reach for a number that arrived later.

versions are named in every run

Each backtest records the data version it read. Its settings are hashed, so an identical run is recognised and stored once.

The provider returns at most 5,000 trading days per request, about 20 years, and company fundamentals thin out before 2006. Results from the late 1990s rest on roughly seven companies in ten. The research page shows how much of the backtest's return comes from those years.

offsite archive

About 18,000 cached files are copied to Cloudflare R2 and verified by checksum. The copy only ever adds files. A mirror would delete remote files that are gone locally, and the point of the archive is that a local mistake can never reach it.

deduplication

Consecutive snapshots share most of their files. They are deduplicated with copy-on-write clones. Hard links look equivalent, but the Parquet writer rewrites files in place, so one edit would have corrupted every version sharing that file.

result files

A 30-year run records daily holdings for about 1,200 companies across 7,800 days. As CSV that was 38MB per run. Compressed Parquet stores the same table losslessly in 1.2MB, 31 times smaller.

most of the price history silently missing

An early price fetcher returned at most about 1,255 rows per company whatever dates were requested, which quietly cut history off around 2021. A backtest built on it looked complete. The fetcher now requests ten-year chunks and stitches them together.

a credit series shortened by licensing

FRED cut the ICE high-yield spread back to its most recent three years after a licensing change. The 30-year backtest needs 1996 onwards, so it uses Moody's BAA spread over the 10-year Treasury, available daily since then. The ICE series is kept for live monitoring, where three years is enough.

a backup check that could not finish

Verifying the archive before a laptop was wiped reported 8,077 errors. Every one was a file the archive held and the laptop did not, which is the archive working. The check was comparing in both directions, and because it then exited early, a second directory was never examined. It now checks one way and reports every directory before stopping.