Admittedly, we have a bit of a puzzling title and a topic for this post. After all, forward testing does sound like the polar opposite of backtesting. So why does it appear in a backtesting 101? Well, in my opinion, they go hand in hand and are two parts of the same puzzle that we must consider when developing our ideas.
Backtesting lets us quickly develop and test on massive amounts of data quickly to weed out the good ideas from the bad. On the other hand, Forward testing is the only way we can start to see the real-world performance of our ideas. The only downside is that it comes at a huge time cost and as such, the two should be balanced appropriately in-line with your goals and risk tolerance.
A Case For Forward Testing
When we optimize our backtests on historical data we are in danger of over-fitting our strategy parameters to the data. We tweak and tweak so much that the settings result in buying and selling at just the right time. However, what we might fail to notice is that those settings might not be predicting future price movements. (For more on this topic see here: Backtesting 101: Curve Fitting). By forward testing we get to really see whether our ideas have merit. Any over-fitting will become immediately obvious.
Another benefit of forward testing is that we move onto using live data. Depending on the platform you are using, this can help weed out some deployment issues that you would not have found had you stayed in a backtest environment.
Forward Testing Tips
As simple as forward testing sounds, there are still some general tips that will help you to get the most out of it.
- Don’t tweak anything: You need an apples to apples comparison. Comparing with a lemon might leave a bitter taste in your mouth.
- Remember correlation is King: Forward testing is slow, we can’t run through years of tests in 10 minutes like Backtesting. Therefore do not panic if you start losing a few trades. Instead, wait until you have completed enough trades and focus on whether the results are showing any correlation with the Backtest results. Your Backstests likely had some losing streaks too. They just don’t affect you psychologically when they are over in a fraction of a second.
- Be just as wary of stellar results: Who could complain about that? Well, this links into the point above. If your forward testing results are not showing any similarity to the expected performance from the backtest, you should take a deep look and investigate why.
How Much Is Enough?
There are a few things we should consider when trying to find the right balance between the amount of forward/backtesting we perform. The balance should be tweaked according to your available time, risk tolerance and confidence level.
As noted in the tips section, Forward testing can be a slow endeavour. Especially so if you are working on a daily time frame or above. The temptation will be to cut it as short as possible. After all, we are likely to be excited about our great backtest and want to start sweeping up money now! This temptation can grow even greater if the first few trades are winners. However, If we have time on our side, we should try to remain calm. Consider whether we really have a seen a statistically significant amount of trades to confirm that we do indeed have a good idea. At this point, it is also worth reminding ourselves that a few trades means nothing. Even terrible strategies can have a string of winners.
As always though, not everything is black and white. We should try to identify opportunity costs that might arise from waiting too long. In other words, if you have any fundamental view that about the general market regime and your strategy relies on that view, then it may add some time pressure to deploy. After all, nothing lasts forever. As an example, if you spent 3 months Forward testing a long-only Crypto strategy in the last quarter of 2017, you would have missed the boat.
Finally, you might just have a personality that has a high-risk tolerance and needs to be in the thick of the action to remain interested. If that is you, it is not a problem to jump right in. Just start small and build size if/when the results start to confirm the idea.
Assuming you have time/patience to perform some forward testing, let’s try to discuss some reasonable figures:
Rules of Thumb
For those who like to keep things simple, there are a few numbers banded around the internet as a rule of thumb for a minimum amount of trades required. These generally range from around 20-30 trades. At the end of the test you can simply take a look at the key statistics such as win-rate, net profit, average PnL, drawdown etc to see if they are within acceptable ranges and have some sort of correlation with the backtest.
If you want to take a more scientific approach to verifying the signature of your forward test results, you might want to look at Hypothesis testing, and in particular, Z-testing. Z-Testing attempts to identify statistical significance between the two sets of results to determine if the forward test results are unlikely to have happened by random chance.
Interestingly, a Z-test is said to require at least 30 samples to be worthwhile. I.e 30 complete trades. This lends some credence to the rules of thumb above.
Here are some interesting links to get you started:
- Standard Deviation: https://www.khanacademy.org/math/probability/data-distributions-a1/summarizing-spread-distributions/a/introduction-to-standard-deviation
- Hypothesis Testing: http://www.statisticshowto.com/probability-and-statistics/hypothesis-testing/
- Z-Testing: https://www.investopedia.com/terms/z/z-test.asp
In the coming weeks, we may cover this topic and other statistics ideas in more detail!