A modular cointegration-based statistical arbitrage system — Kalman-filtered spreads, regime-adaptive position sizing, walk-forward validation — with a controlled ablation that quantifies each component's contribution to out-of-sample performance.
A v31 audit found six result-changing bugs — an inverted stop-loss, a truncated exit search that silently dropped non-reverting trades, a same-bar exit fill, a look-ahead pair universe, features read from the study’s final day, and an invalid cointegration p-value. With all of them fixed, the out-of-sample result is negative: −0.28%/qtr across 10 selection-clean windows, 1 of 10 positive, pooled Sharpe −1.23. It also loses to the textbook Gatev (2006) distance baseline (−0.31 vs +0.26 Sharpe) once that baseline runs the paper’s own rolling re-formation instead of a single stale 42-month formation. Zero pairs survive a Benjamini-Hochberg correction. The deliverable is a rigorously validated research framework — and the discipline to keep auditing until the number told the truth — not a deployable alpha.
Each window: 252 trading days of training, 63 days of out-of-sample testing, models retrained from scratch, and — as of v31 — trading only pairs already selected as of its train-end. One of ten is positive; three (W17–W19) generated no trades at all, so the honest read is one profitable window out of seven that traded.
| Window | Return | Trades |
|---|---|---|
| W10 | -0.4634% | 3 |
| W11 | -0.4484% | 7 |
| W12 | -0.0931% | 12 |
| W13 | +0.5675% | 4 |
| W14 | -0.7372% | 5 |
| W15 | -0.9953% | 13 |
| W16 | -0.6481% | 2 |
| W17 | +0.0000% | 0 |
| W18 | +0.0000% | 0 |
| W19 | +0.0000% | 0 |
A sequential pipeline from pair selection through signal generation, regime filtering, and portfolio accounting. Each stage is independently testable and documented across 19 versioned entries (v7 → v31).
v31 figures, under realistic t+1 execution on both entry and exit. Every walk-forward window is now selection-clean: each one may only trade pairs whose quarterly re-selection date is already known at its train-end, so the nine early windows that previously reported a Sharpe of ~5 no longer exist — they were trading a universe chosen with their own future.
| Period | Windows | Avg return / qtr | Sharpe | Win rate | Profitable |
|---|---|---|---|---|---|
| Out-of-sample (10 selection-clean windows) | 10 | −0.28% | −1.23 | 21.2% | 1/10 |
| Main backtest (Jul 2023–2025) | — | −0.31% | −0.31 | 28.6% | — |
The pipeline doesn't just report a Sharpe — it tests whether the Sharpe is distinguishable from zero (significance.py). It is not — and the point estimate is negative: no t-stat clears |2|, every bootstrap CI includes zero, and the Deflated Sharpe (which penalises for the number of trials) is 0.0% out-of-sample.
| Test | Main backtest | Out-of-sample (stitched daily) |
|---|---|---|
| Annualised Sharpe | −0.31 | −1.23 |
| Newey-West t-stat (mean ≠ 0) | −0.49 (p=0.63) | −1.86 (p=0.06) |
| Probabilistic Sharpe P(SR>0) | 32.5% | 2.2% |
| Bootstrap 95% CI on Sharpe | [−1.70, +0.80] | [−2.41, +0.06] |
| Deflated Sharpe (vs best-of-27 trials) | 0.9% | 0.0% |
Per-window OOS test: mean −0.282%/qtr, t-stat −1.94, p=0.085, 1/10 positive, 95% CI [−0.611%, +0.047%]. The result is negative and not statistically distinguishable from zero — the strategy has no edge to deploy, and the project proves that to itself rather than asserting otherwise.
Gatev (2006) distance method and a random-pair control, run on the same universe and out-of-sample period (benchmark.py). The baseline now follows the paper’s rolling 12-month formation / 6-month trading scheme; it previously formed once over ~42 months and then traded ~30 months without re-forming, which quietly handicapped it. On a fair comparison the textbook baseline beats this pipeline. The random-pair row is not a consolation either: this pipeline wins on return but loses on Sharpe (−0.31 vs −0.02), and the five draws span Sharpe −0.45 to +1.08.
| Strategy | Return | Sharpe |
|---|---|---|
| Cointegration + Kalman (this project) | −0.31% | −0.31 |
| Distance method (Gatev 2006), rolling | +2.78% | +0.26 |
| Random-pair control (avg of 5 draws) | −1.74% | −0.02 |
Multiple-testing check (Benjamini-Hochberg FDR): of 36,524 pairs with usable p-values, 3,725 are "cointegrated" at raw p<0.05 — but ~1,826 of those are expected false positives by chance, and zero survive BH-FDR at q<0.05 (zero at q<0.10 too). The p-values are now Bonferroni-corrected across the two Engle-Granger directions; the previous min(p₁,p₂) was not a valid p-value and inflated the test size.
A transformer-based signal-quality scorer is trained (leak-free, per window) on whether an entry reaches its exit band, and used for opportunity ranking. It trains, but it barely discriminates: across the 27 logged training epochs the BCE never falls below coin-flip entropy (minimum 0.7700 vs ln 2 = 0.6931) and only dips modestly under the class-weighted base-rate loss (≈0.88 at a 36.6% base rate with pos_weight 1.73). Yet the main backtest and all five cost profiles do not depend on it by construction: the score only reorders same-day opportunities, and the sparse main backtest never has enough competing pairs on one day for the ordering to matter. The definitive ML-contribution test is the four-seed robustness check below (run on the v26.1 codebase): one training seed showed an OOS gain (+0.74%/qtr), but across seeds {42,1,2,7} it dissolved into noise — the OOS spread (0.30–0.74%/qtr) is larger than the mean effect, with one seed below the classical baseline. The v31 audit changed the trade population substantially (14 trades, down from 41), so the ablation was re-run against the corrected code rather than assumed. The two arms come back bit-identical on every metric — return, Sharpe, trade count, win rate, walk-forward pooled Sharpe, and all five cost profiles. The measured ML contribution is exactly 0, not approximately 0.
| Metric | Classical only | + Transformer | Δ |
|---|---|---|---|
| Main backtest return / Sharpe (v31) | −0.31% / −0.31 | −0.31% / −0.31 | 0 |
| Main backtest trades / win rate (v31) | 14 / 28.57% | 14 / 28.57% | 0 |
| Walk-forward OOS pooled Sharpe (v31) | −1.228 | −1.228 | 0 |
| Walk-forward OOS, 4-seed mean / range (v26.1) | +0.36%/qtr | +0.53%/qtr (0.30–0.74) | noise |
| Institutional cost profiles (5) | independent of the scorer by construction | 0 | |
PAIRS_USE_TRANSFORMER=0 disables the scorer · seeds {42,1,2,7} · reproducible on any machine
The engineering log records every version, bug, and failed hypothesis from v7 to v31. Selected research notes below.
Negative results are reported in full.
Where the effort actually went: training the model was maybe 2% of this project. The other 98% was evaluation and data — the significance tests, the leakage audit, and a survivorship-bias hunt that mattered more than any architecture. No model lowers the noise floor set by the information in the data (the Bayes limit; in markets, the efficient-market wall), and the evaluation here showed the signal sits at that floor. The job is to measure the ceiling honestly, not to pretend an architecture can raise it.
min(p₁,p₂) across two Engle-Granger directions, which is not a p-value. With all six fixed the out-of-sample result is −0.28%/qtr over 10 selection-clean windows (1 of 10 positive, pooled Sharpe −1.23, per-window p=0.085), and the main backtest is −0.31% at Sharpe −0.31 on just 14 trades. Deflated Sharpe is 0.9% (main) and 0.0% (OOS). Of 36,524 pairs with usable p-values, ~1,826 of the 3,725 significant at raw p<0.05 are expected false positives, and zero survive BH-FDR at q<0.05. The comparison claim reversed too: the Gatev (2006) baseline had been formed once over ~42 months and then traded ~30 months without re-forming, while this strategy re-selected quarterly. Run properly — the paper’s rolling 12-month formation / 6-month trading — the textbook baseline returns +2.78% at Sharpe +0.26 and beats this pipeline (−0.31). Against a random-pair control it wins on cumulative return (−0.31% vs −1.74%) but loses on Sharpe (−0.31 vs −0.02), and the five draws are so dispersed (Sharpe −0.45 to +1.08) that the comparison settles nothing — the honest reading is that cointegration-based selection is not obviously worse than random, not that it beat it. Known limitations, all of which bias upward on an already-negative result: the universe comes from ~current Russell 3000 membership (survivorship bias — of 36 names that delisted/failed/were acquired in 2020–2025, only 1 is present), and the price cache predates the fix to the full-sample penny-stock filter, so names that fell below $2 were retroactively removed from earlier periods and cannot be recovered without a refetch. 14 trades is also far too little breadth for the Fundamental Law to give a meaningful IR at any plausible IC. The value here is the audit trail — a framework disciplined enough to keep finding its own errors until the number stopped flattering it — not a deployable alpha. Every claim is derived from logged, reproducible runs (logs/backtest_v31.log).
None of these is promised to produce alpha. Each is the specific, falsifiable experiment that would either confirm the null is real or locate an edge the free-tier study could not reach — and each maps one-to-one onto a limitation disclosed above. This is the difference between "give me resources and I'll figure it out" and knowing exactly what the next experiment is and why.