How to backtest a stock scan, no code needed

Every scan is a claim. "RSI crossing above 30 in an uptrend" claims that such stocks tend to rise afterwards. Until you check the claim you are trading folklore, and checking it takes about ten minutes.

Here is the method I use. It works with the replay panel on this site and it works with a spreadsheet and some patience.

Step 1: state the claim precisely

"Finds good stocks" is not testable. "Stocks matching this scan rise over the next 10 sessions" is. Fix the exact conditions and the holding horizon before you look at a single result, and decide what counts as success while you are at it. Beating zero? Beating the index? A swing scan graded on six-month returns tells you nothing about swing trading.

Step 2: replay, don't remember

For each trading day in your window, run the scan as of that day, record what matched, then record the forward return of those matches over your horizon. Every scan page here does this automatically over the last 250 sessions. The panel on the golden cross is exactly that loop.

The part that matters is that each historical day's list was computed only from data that existed on that day. Which is where this goes wrong quietly.

Step 3: know the three result-fakers

Survivorship bias. Test only on stocks that exist today and every strategy looks better, because the delisted names are precisely the ones your scan would have matched on the way down. Bars for delisted names stay in this database, but the replay measures the universe as of today, so it still leans survivorship-positive by some amount I cannot put a number on. A hand-built spreadsheet test is usually worse off than that.

Lookahead bias. Any condition computed with information from after the signal date. A price series adjusted for a split that had not happened yet. A "top 500 by turnover" list drawn from today's rankings. The subtle versions inflate results by a few per cent, which is more than most real edges are worth.

Overfitting. Tune the thresholds until last year looks perfect and you have memorised last year. The tell is fragility. If rsi < 32 tests beautifully and rsi < 30 tests badly, there is no effect there, only noise you fitted to. Real effects tolerate a nudge.

Step 4: read hit rate and payoff as a pair

A 45% hit rate where the winners are twice the size of the losers is a good system. A 70% hit rate that hands it all back on the losers is a famous way to bleed slowly. Neither number means anything by itself.

So when you replay a volume breakout scan, the question is not how often it was green. It is what the average win paid against the average loss, and whether enough signals turn up per month for any of it to matter.

Step 5: respect the regime

One year of replay is one regime. A momentum scan tested across a trending year flatters itself, and the same scan in a sideways year pays the whipsaw tax. Read every result as "how this signal behaved in this regime", then check whether the losses bunch into particular months.

I also weight scans by whether I can explain them. RSI oversold turning up in an uptrend has a mechanism behind it: one-sidedness resolving on favourable ground. The mechanism is what you fall back on when the recent numbers wobble, and they do wobble.

Do this before you trade the scan

Build the scan in the query language or start from a preset, replay it, and keep it only if the claim survives. Then retest when the regime turns. There is no code at any step and the replay does the bookkeeping.

Most of my scans have not survived this. That is the point of doing it.