Walk forward analysis or optimization is becoming ever more popular with some traders to determine the robustness of a strategy. So, what is walk forward analysis?
Walk forward analysis is in sample and out of sample testing taken to the next level. In short, it works by dividing the data into several training and validation sets, and then walks through these sets by optimizing for the best values on the training sets, and applying them to the validations set. The validation sets are then stitched together to form an, in theory, out of sample equity curve.
In this article, we’re going to take a closer look at walk-forward testing. You’re going to learn why it’s becoming a more and more popular method among traders, and the disadvantages that not so many are aware of.
Why Even Bother With Walk Forward?
Any trader who has backtested for a while and seen the performance of their strategies on live data, will understand that historical performance is not indicative of future results. You can have a great backtest, but that won’t keep the strategy from failing as soon as subjected to new market data.
This issue is called curve fitting, and in essence, means that you have fit your strategy to random market noise rather than true market behavior. And since your strategy was based on randomness, so will be your future results.
Being one of the biggest and most expensive hurdles for traders, there are many methodologies that aim to keep you from curve fitting and help produce strategies that have a chance of working in live trading.
And walk forward testing is one of the more advanced of all the robustness testing methods out there.
How Does Walk Forward Analysis and Optimization Work?
As said, walk forward aims to help you produce strategies that don’t fall apart in live trading. In other words, it aims to separate the randomness of the market from the real edge.
It does so by using a concept that’s called out of sample testing.
Out of sample testing refers to the process of splitting your data into two halves or more, and saving one half to use as validation once the strategy is ready. As such the testing is not carried out on all the data, but only on a portion.
The portion of the data that’s used for initial testing, is called in sample.
The other that’s used for validation is called Out of Sample.
The rationale behind out of sample testing is that random patterns won’t persist through both data sets. As such, any strategy that was curve fit should fall apart once you test on the new data that is very unlikely to contain the same random patterns.
Walk Forward Takes it one Level Further!
Being a popular robustness testing form, the concept of out of sample testing inevitable has been tweaked into many variations. And the one that probably takes it the furthest, is walk forward testing.
Now, when you do a walk forward test, you split the data into several in sample and out of sample windows. Then you perform an out of sample test for each window, meaning that you optimize the strategy on the in-sample, and apply the settings to the out of sample.
Then you take these out of sample results and stitch them together which in theory forms an equity curve that’s completely out of sample!
This also creates a more realistic view of the strategy’s performance, since you’re deciding what settings to use BEFORE you apply those very settings. With hindsight, it’s easy to just pick the absolute best settings, but in real trading, that’s impossible, since your trades haven’t happened yet!
Let’s break down the process to make it easier to understand!
Doing Walk Forward: Step by Step
Let’s say that we have a trading strategy that uses the RSI indicator as a filter. We could just optimize the strategy on ALL the data, but instead, we decide to use walk forward to get a more realistic view of the strategy’s performance.
We decide to use 10 windows for our test.
Step 1: Optimize the Strategy on the First Window
First, we optimize the strategy on the first year of data. We find that 30 worked best.
Step 2: Apply 30 to the Second Year.
We then take the setting we got from optimizing on the first year of data, which was 30, and apply it to the second year. The results we get are out of sample, since the parameters were based on the results we got from the previous step.
Step 3: Optimize on the Second Year
We now optimize the strategy on the second year of data. We find that the best value was 40.
Step 4: Apply 40 to the Third Year
We apply 40 to the following window, which is the third year. In total, we now have two years of out of sample.
And so it continues…
We continue this way until there are no windows left. The out of sample results are then combined together and form an equity curve that’s completely out of sample.
Anchored Vs Rolling Walk Forward
As we now have covered the basics of how walk forward works, we wanted to mention the two types of walk forward optimization. Those are:
- Rolling Walk Forward
- Anchored Walk Forward
Both work fine, and some people like one more than the other. In short, the difference lies in how they manage the length of the in-sample window.
Let’s have a look at the differences in more detail!
Differences Between Rolling and Anchored Walk Forward
The rolling walk-forward method is the default one, which we showed you above.
However, the anchored walk forward is a little different. Instead of having the in-sample periods “roll”, which means that only the latest window is used as in sample, it uses all the previous windows as in sample.
As such, the size if the in sample grows bigger as the anchored walk forward progresses through the data windows.
For the first window, the in-sample is only 1 year.
For the second, it becomes 1+1 = 2 years.
The third time it becomes 1+1+1 = 3 years.
Is Anchored Better than Rolling?
Which one you choose depends on personal preference. There is nothing that says that one is better than the other.
While the rolling version responds quicker to market changes, the anchored version has its benefits too. For example, it could be that the anchored walk forward method works better since it uses more data to find the optimal settings.
It all comes down to how much of the historical data you think is relevant to the strategy!
Does Walk Forward Testing Work?
Many traders praise the walk forward for its ability to filter out curve fit strategies.
However, we’re not very convinced.
While walk forward is a great method that could add value to your strategy creation process, we see one big problem with it.
It adds a bias to your trading.
The main assumption that people who rely on walk forward make, is that a strategy must cope with having its parameters shifted regularly in order to be robust.
While this might seem sound, we’ve found that it leads you to dismiss great trading strategies, that don’t work well in walk forward.
We ourselves don’t use walk forward to create trading strategies but have another approach, which we teach in our algorithmic trading course. In fact, many of the strategies that we trade successfully fail miserably in walk forward tests!
Having said all this, walk forward could add a lot of value to your testing process. Just don’t trust it blindly. In trading, there is no foolproof way of doing things!
Walk Forward Test Settings
Now, apart from choosing between rolling and anchored walk forward, you have a few more options that will significantly impact the results. Those are the fitness function and the size of the in and out of sample windows.
1. Fitness Function
The fitness function is how you define what you’re looking for in a trading strategy. For example, if you optimize a strategy and use net profit as the fitness function, you will get the settings that yielded the highest net profit.
The fitness function you choose determines what strategy settings will be chosen by the walk forward optimizer. And this will have a significant impact on the results!
For example, optimizing for the highest net profit is very different from optimizing for the highest average trade. The former will have a tendency to include a lot more trades than the latter.
There are many fitness functions to choose from. Just be sure to select one that reflects what you think is important in a strategy!
2. Window Sizes
The size of the in sample and out of sample windows will have a significant impact on how the optimization is carried out.
For example, having only 5 windows instead of 10 means that the strategy will be reoptimized half the number of times, which could alter the strategy’s performance significantly. And following that the windows are fewer, they will be longer as well.
Depending on the strategy itself, it might benefit from having shorter or longer windows. However, a 1 year out of sample length with a 2-year in-sample period is a commonly used setting.
Curve Fitting with Walk Forward
While walk forward on its own is used to prevent curve fitting, you could curve fit a strategy even if it passes the tests!
Here are a couple of reasons for this:
1. The rules Were Curve Fit
In a walk-forward test, you bring in a trading strategy that you already have designed and set the rules for. The only part that you let the walk-forward optimizer change, is the parameter values.
For example, I might decide to include the ADX indicator in my strategy, and let the walk forward process play around with the exact indicator settings.
While most people think of curve fitting like selecting curve fit values, it could very well be that the condition itself, regardless of its setting, leads to curve fitting. And that’s not accounted for in a walk-forward test!
2. The Parameter Ranges Were Curve Fit
One thing to watch out for, is curve fitting the parameter ranges.
For example, let’s say that we once again are using the ADX. If we know that the strategy works well with ADX values between 20-40, and use that information to limit the settings that the walk-forward optimizer has to choose from, we have curve fit the parameter range itself. As such, the walk forward test will produce good results, which could very easily fool us!
While the example above was obvious, many times mistakes of this kind could be more subtle, so be sure to watch out for this!
3. The Walk Forward Settings Worked by Random Chance
Now, this is a major issue. As we covered earlier, you can make many adjustments to the walk forward test, in terms of window settings and fitness functions.
In short, this means that we have many variations, and just as with curve fitting on price data, it could be that one particular variation worked well by random chance.
This is something we see quite often when we do walk forwards. We like to play around with different fitness functions and settings, and often notice how a strategy that passes with one type of setting, might fail miserably if you just change the window sizes a little.
In our view, this makes walk forward an unreliable tool if you make your assessment only on the results you get on the first try. But trying different settings just to find one that works isn’t good either, since we’re facing the risk of curve fitting!
However, there is one method that aims to at least partially solve these issues, namely cluster analysis.
Cluster Analysis: Walk Forward Taken one step Further
Cluster analysis, also referred to as walk forward matrix, is when you run many walk forward tests with different settings, to see how a strategy performs across many combinations. Doing so, you get a better picture of how the strategy fares with walk forward in general, and not just individual runs.
The more in and out of sample combinations that a strategy works with, the higher the odds that it’s going to work going forward.
Walk forward testing is a quite advanced method to ascertain the robustness of a trading strategy. Many people believe it’s the solution to curve fitting, and while it could be to some extent, it’s not a foolproof solution by any means.
We recommend that you not make walk forward your principal method of strategy testing, since you will be throwing away quite many functional trading strategies that way.
If you’re interested to know how we test trading strategies ourselves, this is something we teach in our algorithmic trading course.
However, we also have a lot of free resources that will help you get started quickly. Here are a few of them!