How we test

Every strategy the lab looks at, whether we found it, someone imported it or someone asked about it, goes through the same steps below. Each step is described in plain words first and as a formula second. Every number on this page is read from the lab's code when the page is drawn, so the page says what the lab runs today.

In short

We pick a strategy's settings on old data only and then trade them on later data they never saw. We charge realistic trading costs. A result has to beat its benchmark by a margin that luck rarely produces, after we count every other configuration we have ever tried. Most strategies fail, and we publish how many.

Every count, by source and by the check that failed · One run with every number open

Walk-forward, with a purge and an embargo

A strategy usually has a setting to choose, such as how many days of prices a trend looks back over. The lab chooses it on a training window of 756 sessions (about three years), picking the setting with the best training Sharpe, and then trades that setting on the next 126 sessions (about six months), which the choice never saw. Then it rolls forward and does it again. Only the trading windows count: the result is out of sample from end to end.

A trade can last several days, so the last training days could belong to a trade still open when the test starts. The lab leaves a gap (the purge) of 5 sessions between each training window and its test, and skips 5 sessions (the embargo) after each test before the next one starts.

for each fold k:
  train_k = sessions [s_k - 5 - 756, s_k - 5)
  test_k  = sessions [s_k, s_k + 126)
  setting_k = argmax over the grid of Sharpe(train_k)
  s_(k+1) = s_k + 126 + 5
out-of-sample returns = the test windows' returns, joined

The candidate bar

A run becomes a candidate only if every check below holds. Each run keeps each check's value and threshold, so anyone reading a result can see why it passed or failed.

Information ratio and its t-stat

The active return is the strategy's return less the benchmark's each day (the 50/50 SPY and QQQ blend, or a single asset's own buy and hold). The t-stat uses Newey-West standard errors, which allow for days whose returns move together, with the lag set by the Newey and West (1994) rule.

a_t = r_t - b_t
IR  = mean(a) / sd(a) x sqrt(252)
t   = mean(a) / se_NW(mean(a)),  lag L = floor(4 (n / 100)^(2/9))
pass: IR > 0.00 and t > 3.00

Deflated Sharpe, counting every trial

Try enough settings and one will look good by luck. The deflated Sharpe ratio (Bailey and Lopez de Prado, 2014) asks how likely it is that the true Sharpe ratio is above the best one N unskilled trials would show, given the length of the record, its skew and its fat tails. N is every configuration tried in the run's whole pool of families, this run's grid included, not just the ones that were kept.

SR  = mean(r) / sd(r)                (per bar)
V   = variance of the trials' Sharpe ratios
SR0 = sqrt(V) x ((1 - g) x Z(1 - 1/N) + g x Z(1 - 1/(N e))),  g = 0.5772
DSR = Phi( (SR - SR0) x sqrt(T - 1) / sqrt(1 - skew x SR + (kurt - 1)/4 x SR^2) )
pass: DSR >= 0.95

Probability of backtest overfitting

Combinatorially symmetric cross-validation (Bailey, Borwein, Lopez de Prado and Zhu, 2017): the test-window returns of every setting are cut into 8 blocks. For every way of choosing half the blocks, the setting with the best Sharpe on that half is ranked on the other half.

PBO = share of the 8-choose-4 splits where the in-sample best ranks at or below the median out of sample
pass: PBO <= 0.20

False discovery across every run

Each run's t-stat gives a one-sided p-value. The Benjamini and Hochberg q-value ranks it among the p-values of every run evaluated before it in the same pool, so a lucky run in a crowd of tries cannot pass.

sort the m p-values: p(1) <= ... <= p(m)
q(j) = min over k >= j of p(k) x m / k
pass: q < 0.10

The lookahead and LLM cutoff checks

The lookahead check recomputes the strategy's position on 25 evenly spaced days from data cut off at each day; any difference from the full-history position means the rule read the future, and the run fails. A series a language model produced can only be tested on days after that model's training cutoff; a model with no known cutoff is refused. The cutoffs the lab uses:

The fast screen

A study the lab's own idea engine queues on many names first gets a quick screen: each setting's rank correlation with the next returns on the first training window only. One with no setting at a t-stat of at least 1.00 stops there. It still counts every configuration of its grid in the trial ledger, so the screen hides no trial.

The stress gate

Four more checks a real edge should survive, all on the out-of-sample days:

• Costs doubled. The returns less a second charge of the run's cost per unit traded, never less than 5 bps (2x costs). The information ratio must stay above 0.
• The trailing year. The last 252 bars alone, against the benchmark over the same bars. The information ratio must be above 0.
• A plateau, not a spike. The settings next to the chosen one in the grid must keep at least 50% of its Sharpe ratio.
• Enough trades. At least 100 out-of-sample trades.

Every run reports all four as warnings today. From 2026-11-07, the costs-doubled check and the trade floor join the candidate bar.

Costs

Every trade pays for crossing the spread and for moving the price. The lab estimates each name's spread from its own daily high, low and close (Abdi and Ranaldo, 2017) over the last 21 bars, never less than 2 bps for half the spread. Market impact grows with the square root of the order's share of the name's daily dollar volume (its median over 20 bars), capped at 10%. No trade costs less than the run's flat floor, 5 bps per unit traded by default. A run is priced as a book of $250,000 unless its spec says otherwise.

bps = max(floor, max(half spread, 2) + 30 x sqrt(min(order / ADV, 0.1)))
floor = 5 bps (the run's flat cost)

Capacity is the book size at which market impact eats 50% of the out-of-sample edge, found by bisection on the run's own trades. Any backtest Sharpe used to size a position is cut by 50% first, because about half of a published effect disappears after publication (McLean and Pontiff, 2016).

Point-in-time data

A signal on a day may only use what was known that day. The lab's data is stored so that this holds by construction:

• Economic data comes from ALFRED with every vintage, so a study sees the number as first published, never a later revision.
• Company filings (fundamentals, insider trades, 13F holdings, congressional trades) are used from their filing date, never the period they describe; annual facts are the original 10-K's, so a restatement is never seen early.
• Delisted names stay in. The stock universe is redrawn every 21 sessions as the top 500 US stocks by median dollar volume over 60 sessions (price at least $5.00, at least 252 bars of history), from every listed stock, active or not, so companies that later failed were members while they traded.
• Futures positioning (CFTC) is seen from its Friday release, not the Tuesday it describes; end-of-day options data from the next day.

One known gap: the SEC's company-to-ticker map is today's, so a company that stopped trading before the data was loaded often has no annual facts. Studies that rank on those facts say so on their result.

How trials are counted

Every run is counted in the trial ledger with every configuration it tried, whether it passed or not, and deleting a study never lowers the count. Runs are grouped in families by where they came from. The house families share one pool: the deflated Sharpe and the false discovery test of a house run count every house run ever made. Other families are counted on their own, so a burst of questions or imports neither loosens nor crowds the house bar. The lab's own search may try at most 200 configurations a day; its catalogue holds 17 published edges.

Source: config/portfolio.py, src/research_lab/kit.py, src/backtest/costs.py and src/research_lab/llm_cutoffs.py, read when the page is drawn.

Our bar against a profit-factor bar

Some sites rank strategies by a looser bar: at least 100 trades, a profit factor of at least 1.2 and a win rate of at least 40%. We ran both bars on the same market of pure noise our own canary test uses: 22 random walks with no drift and no pattern, 1,500 sessions each, seed 20261012. On noise no strategy has an edge, so every pass is a false one.

The profit-factor bar reads each strategy's whole history at the setting with the best profit factor among its four, the way a backtest tuned on a chart shows it, before costs. Our bar picks the setting on training data only and trades it on later data, charges 5 bps a unit traded, and runs the stress gate, while counting only each strategy's own four settings as trials (the lab itself counts every trial it has ever run, which is stricter). Of the 8 noise strategies the profit-factor bar passed, ours failed them on information ratio at 2x costs (6), deflated Sharpe (8), false discovery q (7), information ratio (4), Newey-West t-stat (8), probability of overfitting (7), out-of-sample trades (5).

Source: src/research_lab/noise_compare.py on the seeded noise market of src/research_lab/noise.py (the lab canary's, tests/test_lab_canary.py); seed 20261012, reproduced bit for bit by the lab's tests.

Research, not advice. Past results do not promise future returns. StockGaines publishes its method to everyone alike and does not advise anyone on whether a strategy suits them.