case study, two minute read
Building a crypto model that is allowed to lose in public.
The problem
Crypto "signal" sellers show winners and quietly delete losers, so their track records mean nothing. I wanted to build the opposite: a forecaster whose every call is recorded before the outcome exists, graded automatically, and protected against edits by database rules, so its real skill (or lack of it) is visible to anyone.
The approach
- Predict only moves big enough to pay trading fees: more than 1% over 12 hours, as a 3-class problem.
- Select models on net profit after 0.2% round-trip fees, not on accuracy.
- Walk-forward validation with thresholds chosen on inner folds; 39 configs logged, losers included.
- A six-month holdout locked away in code and evaluated exactly once.
- Tests proving no feature can see the future, and that training labels, the live grader, the database and the site all grade identically.
Architecture
Decisions that mattered
- Only live-reachable features. Binance futures data is geo-blocked from the US servers that run the jobs, so it was dropped even though it looked useful. Deribit funding answers from those servers, so it stayed, and removing it halved the walk-forward result.
- The database enforces honesty. CHECK constraints and triggers reject late inserts, signals that break the published rule, edits, deletes and re-grading, so a bug in the jobs cannot quietly rewrite one. Someone with owner access to the database could still change those rules. Since trust v2, receipts are hash-chained and the chain head is stamped hourly with OpenTimestamps; that only covers rows from the activation date shown on the home page.
- Late means skipped. If the hourly job starts more than 18 minutes after the candle closes, it logs a missed run instead of predicting a past it can half-see.
- Ship rules agreed before the holdout. Profitable on the holdout: ship as unproven. Not profitable: ship anyway, labelled as not beating fees. No moving the goalposts.
Honest results
The first model was right 52.8% of the time and still lost 92.6% in simulation, because its edge was 15 times smaller than the fees. Retargeting to fee-sized moves produced a walk-forward result of +91.0% (compounded, three equal sleeves, entry at reference closes, 0.2% fees, no slippage) over 988 trades (Sharpe 1.27, 95% CI 0.02 to 2.50). The edge was concentrated in late 2024 and early 2025 and faded. On the locked holdout the model found nothing worth trading and made 0 trades. Label: Does not beat trading fees in testing.
What I would do next
- Run an always-on collector for exchange funding and open interest so a real history exists to train on.
- Detect regimes from more than one market cycle before trusting any edge.
- Let the public track record run for months; a record made before outcomes exist cannot be fitted to them, though it is still a small sample.
See the live receipts or read how it works.