Backtesting 101: Curve Fitting & Overfitting

Curve fitting & Overfitting are two terms that are sometimes used interchangeably around the internet. This can be especially prevalent amongst beginners. We hear the terms and draw our own conclusions on their actual meaning. We are all guilty of this in different areas and I am no exception. Our brains like to take shortcuts and if we can rationalize an idea without having to dig too deep, we often do it without even realizing.

Curve fitting and overfitting do go hand in hand but they are not the same thing! Only one of them needs to be treated with care.

Curve Fitting

Curve fitting is a process used in machine learning, predictive modeling, and data mining to create a mathematical formula that is able to fit a series of historical data. In other words, we are able to use this formula to recreate a logical path through the data. Once we are able to do this, the idea is that the formula can then be applied to other data series to try and predict future prices.

Predominantly, curve fitting techniques rely on using complex mathematical models to generate predictive formulas. This means we are more likely to be working with linear regression and discrete choice models rather than using stochastics, Bollinger bands and other common indicators. However, their are parallels with systematic trading and it is for this reason that the terms are so prevalent outside of the machine learning community. Those of us who base our entry and exit criteria using a combination of indicators with “if this, then that” logic, are essentially creating a “formula” to fit the “curves” of the historical data. We also believe that this formula has some sort of predictive power. Else why would we bother?

Real Word Applications

Curve fitting has many real-world applications outside of trading. Examples of these include, but are not limited to:

  • Sales and Marketing: Understanding customer behavior to predict/identify the optimum marketing materials, products and services that a consumer is likely to purchase.
  • Safety & Protection: Predicting and Identifying high-risk individuals. Such as child protection.
  • Churn Prevention: Identifing patterns of dissatisfaction amongst customers/
  • Medical Diagnosis: Used to analyze clinical parameters (symptoms, or in our world…Indicators!) trained with historical patient data to predict conditions and medical outcomes.

and many more…

Overfitting

Overfitting, by contrast, is a term used when a formula has been tweaked too much on a specific set of data and ends up representing only that set of historical data. In these cases, it rarely has any predictive power on other historical or future data series.

Distortion

Another way to think about it is to draw a comparison with audio signals. For example, listening to an AM radio signal might be a bit noisy sometimes but there is also important information we want to hear behind that noise. Assuming the noise stays within a reasonable level, we are easily able to ignore it and decipher the message. We don’t try to find meaning in the noise. We don’t try to decode it.

This is often quite different to trading. Whilst noise is also present, we seem to find ourselves trying to decipher the meaning of it. We want to catch every move, rise and dip. There is a real temptation to trade the noise. After all, this results in the best paper profits on a backtest. However, as soon as the noise changes its profile, the formula performs poorly even though the underlying message is the same.

Examples

To help visualize it, let’s take a look at some imaginary examples. In the first image, we have an example sine wave compared with a noisy sine wave. A sine wave is what a pure, clean audio wave looks like when viewed on an oscilloscope. In this example, the clean sine wave acts as the important information we want to hear. Then we add some noise to it. When we do this, the frequency of the wave remains the same (even if I have drawn it a little off!) but the path of the wave is no longer a smooth line. You might also notice that the wave starts to look a little more like the movement of price on a chart.

Sine Wave vs Sine Wave with Noise

Next, we take the nosy sine wave and add some imaginary trades on it.  In this example, we will see the difference between a well fit formula and an overfitted formula. The “well fit” imaginary trades might not be perfectly aligned to the tops and bottoms but the formula is able to capture the essence of the sine wave. The overfitted example, by contrast, has more trades and more profit but the formula will only work if this specific noise remains unchanged.

Finally, we have an example of what happens when the noise changes its profile. Notice that the first formula still catches the essence of the wave and enters/exits in sensible locations. However, the overfitted formula appears to now have quite random timing. The paper profits will not look so good now.

Different noise is added to the signal

Of course, the examples above are a little exaggerated but I hope they help visualize what overfitting can look like and how it will affect you as soon as you there are small changes in the price action. Even if the underlying theme is the same.

Avoiding Overfitting

So now we know what overfitting is, how can we take some measures to avoid it? Here are some suggestions and tips to avoid overfitting (listed in no particular order):

  • Break your test data into two parts. Fit your strategy to the first part of the data set (known as training data). Then test the strategy on the second part (known as the test data). If your strategy is well fit, it should have similar performance on the test data.
  • Test your strategy against other similar assets. If your strategy has predictive power, it should not perform wildly different on assets that have similar movement, volatility, and driving factors.
  • Minimize your parameters as much as possible. The more parameters you have, the more likely you can fall into overfitting. This is because you then have that many more combinations of parameters that can change the outcome of the test.
  • Do not try to catch every price move. Doing so often results in adding more indicators and parameters which as noted above leads to overfitting. It is ok to miss some setups. It is also ok not to have perfect timing.

Finally, If your returns are too good to be true, they probably are!