Methodology
- 1ObserveQuestionWhat you think you know, and why.
- 2ResearchIdeasPapers, footnotes, and who is on the other side.
- 3HypothesizeStress testA kill condition, written down first.
- 4ExperimentBacktestSandboxed code on a fixed snapshot of the past.
- 5VerdictVerdictRejected, unproven or promising, with the assumptions listed.
A moved goalpost is a new hypothesis, tested on new data.
Caliper Trading has a simple objective: let users understand their trading strategies via the scientific method. A caliper is a machining tool that measures materials to a high degree of precision, dating back to the 6th century BC. Our system is built to hold every output to the standard below.
Observe
An idea to test starts in the mind of the user, from observing the markets and the world. Brought to the chat, it is interrogated: is it grounded in a measurement, an anomaly, prior research, or first principles, that is, a mechanism for why it could be true? An idea without a grounding mechanism is likely noise, basically tantamount to astrology.
Research
Research fleshes the idea out. Looking beyond the headline and the abstract, combing through papers surfaces secondary findings, tables and footnotes that may hold a real edge. A known result is reproduced on overlapping conditions before its data count as evidence.
Every idea must also account for the counterparty, the other side of each trade. A credible hypothesis names the counterparty, explains why they make the trade, and why you are not facing adverse selection. Trading against better information and execution is normal in markets, and an idea needs a justification for going against these better operators. "Nobody else has noticed" is a delusion; every edge decays, and publication marks the top of the decay curve.
Hypothesize
From generating and researching ideas, the system arrives at a collection of hypotheses to test. Each hypothesis is falsifiable: if an experiment fails to meet a pre-registered criterion, it is rejected.
Experiment
The closest thing trading has to a scientific experiment is the backtest. Backtests face common setup failures that the design accounts for.
- No look-ahead. All mechanisms look at the past, data is served in the simulation's present, and prices are computed for that date.
- Costs are a grid. Fees and spread are priced under several sets of assumptions on both legs (buying and selling). Optimistic bounds are noted.
- Makers and takers operate differently with respect to a hypothesis and are thus tested separately.
- The counterparty price is built of their information: we investigate what that information is.
- Nothing is silently ignored. Exclusions come with a written justification, and filtered outliers are reported.
What runs is code, in an isolated sandbox whose only network access is to the historical data staged for it: a fixed snapshot, fully staged before computation begins. What comes out is statistical, so:
- Events are counted, not observations.
- If permutations of hypotheses are tested, significance is checked against permutation nulls.
- Concentration is reported: how much of returns come from a few flaky trades?
- A few winning trades (<20) warrants only a few units of trust.
- The following metrics are always reported: annualized return, a measure of risk adjusted return, maximum drawdown, a t-statistic, numbers of total and winning trades, and compound versus constant-size returns.
- Unproven is different from disproven; if a test produces a small sample size, the honest output is "unproven", alongside the sample size needed to decide it.
No analysis knows your capital, market access, fees or execution speed. Each conclusion therefore lists its operational assumptions and their sensitivity. An edge that needs institutional information and high-frequency computation is a different thing from an edge available to retail.
Verdict
The results are compared to the falsifiability conditions set out in advance, and the hypothesis is rejected or accepted. Adjusting the criteria after the fact is a new hypothesis, to be tested on different data. This discipline separates science from curve fitting.
After a verdict, resampling (shuffling or subsampling the backtest data) and simulation (drawing many hypothetical paths from a distribution fitted to it) stand in for the replication study the past only offers once.
The verdict states what did and did not work and why, with the metrics above. A rejected hypothesis is progress; negative results are part of the method. A positive backtest does not guarantee the future.
This is Not Financial Advice
A verdict is only about returns in the past, under stated assumptions. Resampling and simulation add some regime flexibility but are still derived from past data. We cannot predict the future, nor do we advocate any future action: no forecasts, no price targets, no recommendations.
