Intro to Prediction Markets, Part 2: Why Classical Metrics Fail

Previously, we made several comparisons to other financial markets, and one point that was touched upon was a question about how to evaluate strategies; in that context a big concern was the zero-sum nature of prediction markets talking about which hurdles needed to be cleared before using prediction markets as a source of capital investment. If we are to make inter-market comparisons to make those decisions, we need to make an apples to apples comparison. In this post, we look at many metrics that are used to evaluate trading strategies in classical markets, and how the discrete nature of a prediction contract puts a wrench in many of the metrics.

Naive Metrics

First it is worth mentioning metrics that are commonly reported but can be used to mislead unsuspecting readers to assume that a strategy is good.

Win rate

Win rate is a really simple metric to compute $$\frac{\# wins}{\# trades},$$ but says almost nothing about what the effectiveness of a strategy. As a refresher, if the market prices an event at $p$ that actually happens at rate $\theta,$ the expected P&L for the trade becomes $$\mathbb E[R] = \frac{\theta - p}{p}$$ If a trade were repeatedly made on that market, win rate at best approximates $\theta$, and says nothing about profitability. A trader who wins 95% of the time on a 96% market loses a small amount over several winning trades, but one who wins 3% of the time at a 2% price increases their money 50%.

Fundamentally, win rate only counts wins and losses, not the distribution therein. This holds even in traditional markets, and anyone advertising a win rate alone should be treated with suspicion.

Streaks

Streaks are likewise sensitive to the underlying distribution. 9 wins in a row is not good when you are priced at 95% (1 in 20 losing days), but is very good at 5% pricing (1 in 20 winning). Streaks are also subject to small-sample bias, and could be a indication that a backtested strategy with an abnormally long win streak does not have sufficient data.

Directional Hit Rate

Again, context is super important on what predictions are made: Does your system find a price $p^*$ which consistently beats the market? Picking winners regularly when the market is already at 90%, while producing a high hit rate, is probably not measuring a real edge.

Profit & Loss (P&L)

Advertising how much money was made using a strategy is another common tactic observed. Without any context, such as capital invested or the time it took to accumulate that amount, it is absurd to consider. A strategy making $1000 using$100000 capital and a year of time is much worse than one that makes $1000 on$10000 in a month.

In order for it to begin as a usable metric, enough information must be given to determine an approximate interest rate.

Return on Investment (ROI)

Even if one has a P&L number with a time-adjusted return on investment, fixing the problems of P&L, it does not do a great job at establishing success in terms of risk. Consider two strategies that return 5%, but one does so by primarily buying contracts priced around 90%, and the other does so buying primarily at 10%. The former will received their returns incrementally, where compounding returns play a large role. The latter will receive returns sporadically, so compounding does help, but not to the same degree.

Subtle Failures

The above are metrics commonly peddled by charlatans as a means to suggest success, both in prediction markets and in trading more broadly. The following metrics do have the strength of measuring something legitimate, but are not appropriate to measure a prediction market strategy in one way or the other.

Sharpe Ratio

The Sharpe Ratio measures excess returns in a period, where a strategy that returns $R_a$ is compared against the risk-free return $R_f$ (often a government bond), is defined by $$S_a = \frac{\mathbb E[R_a - R_f]}{\sqrt{\text{var}[R_a - R_f]}}.$$

Any strategy must clear $R_f$ to justify using capital. However, subtracting it from prediction market returns adds noise, because all participants face a zero-sum pool. Setting $R_f = 0$ for prediction markets better captures intra-market skill.

More importantly, Sharpe measures arithmetic returns and ignores leverage. We'll find a similar ratio that correctly reflects the compounding geometric growth, as well as a related drawdown metric.

Brier Score and Log Score

Both of these measure forecasting ability. Brier score measures the mean square error of forecasting $$BS = \frac1N\sum_{t=1}^N (f_t-o_t)^2$$ and Log Score measures the cross entropy $$LS = \frac1N\sum_{t=1}^N o_t\log f_t + (1-o_t)\log(1-f_t),$$ where $f_t$ is the forecast of outcome $o_t$. These measure how often you are right relative to the probability distribution. It has a proper use in prediction markets; recently Kalshi ran their market data through the Brier score to see how accurate pricing was, to justify their hypothesis that it is a useful tool for predicting the future. This is an appropriate use case.

For evaluating strategies as a whole, these are close to worthless because it does not center about returns. That is not to say that it couldn't be used to evaluate a forecasting component of a strategy, so it is still worth knowing these, because forecasting is part of the game. But if we're going to make comparisons to other prediction market strategies, let alone inter-market strategies, we need to have something that measures returns directly.

And we will work our way towards that in the coming posts.