Last Updated on 20 May, 2023 by Samuelsson
You rarely manage to find trading strategies that perform better in live trading than in backtests. Why is it so? This is because of the disadvantages of backtesting. You need experience in backtesting to avoid the many pitfalls along the way.
In this article, we look at the disadvantages of backtesting. There are many reasons why backtesting doesn’t work, like curve fitting, market cycles, chance, luck, and randomness. But the good thing is that you can avoid or at least minimize many of the disadvantages of backtesting. We are proponents of quantifying trading strategies, but you need to understand how to avoid the many pitfalls in backtesting. We explain why:
(Before we go on we’d like to mention that we have a backtesting course that covers all aspects of how to backtest.)
Why backtest and quantify trading strategies:
It’s good advice to do backtesting in order to gain some insight into how a trading idea might work. If you don’t do that you are basically just guessing how a certain strategy will perform.
Backtesting is a big leap in the right direction. You’ll get a hint on how the strategy performs, at least in the past, although obviously there is no guarantee that it will make money in the future.
Live trading will never be as good as theoretical backtesting
However, some words of caution are needed. Real results will never be as good as theoretical backtesting. It’s 100% certain that the real-time results will be worse than the actual backtesting.
Why? below you’ll find some personal thoughts of why the results differ so much.
The backtest might be liable to favorable conditions
The backtest is performed in a certain period and the markets may have been favorable to that trading strategy in that period. Most traders neglect this aspect. Testing over a longer time frame might minimize this.
Take for example so-called trend following strategies. They perform quite badly in certain periods covering many years, and much better in others.
If you are testing a moving average breakout, this might yield mediocre results over a five-year period. This is a typical strategy that needs to be tested over a period of at least 10 years.
Obviously, you don’t want to change a strategy in response to one year just because something didn’t work. That’s when you’re almost guaranteed that it would have worked the next year had you kept it as it was.
The ever-changing market cycles make trading a difficult task to follow. You have to accept drawdowns to make money and every strategy has drawdowns.
Backtests work because you know the after the fact
You can only trade the strategy after the fact.
For example, you are using entry at the close. Problem is, you only know if it’s a trade after the close. In order to trade on the close, you might have to guess/estimate there wil be a trade at the close.
So when backtesting, it’s crucial you take this factor into consideration. One way to do this is to trade at the open the day after the signal.
Backtesting involves elements of curve fitting
Another reason is the curve fitting aspect. This certainly applies if you’re having a lot of parameters or variables.
It’s easy to come up with a system that has performed remarkably well. You just need to put in a lot of parameters. That will explain the past, but most likely not the future.
The more simple the system, the more likely it’s to stand the test of time.
Curtis Faith explains in The Way of The Turtle some trend-following strategies that are incredibly simple. Over several decades they have worked well in currencies and commodities (not on stocks).
However, over a period of 1-3 years, they sometimes experience quite huge drawdowns. Still, these systems are so simple that they are less prone to be curve fitted.
Survivorship bias
Another problem is survivorship bias.
Put simply, this relates to the use of stocks/tickers that have “survived” the testing period.
For example in 2008/2009 a lot of stocks went bust (Lehman being an example). This means that companies that have gone bust are excluded from the analysis at later dates.
In day trading this might be less of an issue, but not when testing over a much longer time frame. If you download quotes for REITs back to 2005, this will exclude several stocks that went bust during the financial crisis.
Chance, luck, and randomness should not be ignored
If you’re testing a lot of strategies, some will show good returns simply by chance.
Unless there might be some logical reason for a strategy, you are guaranteed to find many good strategies the more you test.
Hence, there must always be some reasoning behind the parameters.
Backtesting involves garbage in, garbage out
The quotes you buy or download are usually not correct. If using high and low in the test, you can be sure there are errors compares to live trading. There are a lot of wrong quotes on high and low!
In real trading, this will have a huge impact, probably most of all the factors mentioned in this article.
Transaction costs are unknown
Slippage and commissions are easy to underestimate. Commissions are easy to calculate, but slippage is not.
This is a huge unknown, especially if you base your strategies on chasing the stock. If you wait to get hit, this is of course less of an issue.
Markets change – no backtest can accommodate change
The market changes all the time.
Obviously, the future is unpredictable and you can bet there will be totally random and dramatic changes in the marketplace. No one expected terrorists to hijack planes and send them into a skyscraper.
Such totally unpredictable disasters will happen sooner or later. Correlations among different asset classes also increase during such happenings. You can never backtest such things.
All traders do behavioral mistakes
You need to understand the most obvious trading biases. The psychological aspect is just as important as the strategy. Can you handle drawdowns and continue trading? Can you actually follow the strategy?
Based on my personal experience, this is something you have to consider thoroughly before you implement a strategy. It’s a lot easier said than done to follow a strategy 100%.
Always prepare for the worst
How can you prepare for the worst?
As a rule of thumb, it might be wise to expect a maximum of 50% of the profits from backtesting. You can exaggerate slippage and commissions, and expect a much higher drawdown than in the backtest.
Hope for the best, but prepare for the worst.
Never be too optimistic when seeing a very nice equity curve, the downfall will be bigger. There is only real trading that matters.
Does backtesting really work?
It depends. First, you need to understand and test valid and logical ideas. Second, you need to have an understanding of the disadvantages mentioned in this article.
I have probably tested close to 2000 ideas. So far no strategy has performed better in real life than in backtesting. So consider this as a fact of life.
When executing a backtested strategy live you discover a lot of factors you didn’t think of when backtesting. Hence, always start with the minimum amount of money and always papertrade a strategy before you go live.
A piece of advice on backtesting
I’ll finish the article with a piece of very good advice (at least it works for me):
It might be very boring and tedious to test strategies manually by hand but believe me, you can learn a lot more. It’s easy to use a program and to scan thousands of stocks.
However, by doing testing by hand you can extract info that otherwise would get lost. It might be a little boring doing it the old-fashioned way, but consider it as an investment.
For example, by testing one stock at a time by pasting quotes into Excel, instead of just scanning 1000 stocks in one minute using a fancy program, you can one by one look at each different stock. Look at its equity curve, look at max drawdown, etc.
If you use scanning, spend some more time doing it manually stock by stock.
Disadvantages of backtesting – conclusion
To get good results from backtesting, you need to understand the potentially many pitfalls of backtesting. A backtest can’t capture that markets change, are random, that you tend to curve fit, and you make behavioral mistakes. However, if you are street smart, you can even offset some of the disadvantages of backtesting.
