Backtesting Strategies, Does It Work?

If you are a quant investor or a wanna be one, the first thing you do is to define an investment strategy, backtest it and eventually launch it live.

How many times have happened that the strategy has not performed on a real or demo account? If the answer is nearly always, let me explain you why it has happened and let me suggest you what I think is the best way to perform a backtest prior launching your new strategy live.

There might be four main reasons why your backtest did not work in real life:

1. You made a mistake in setting up the equations of your model
2. Your model is over-fitted
3. The market conditions are different from the ones you have used to train your model (e.g. new correlation among asset classes)
4. You have not given enough time to the model to perform

Assuming you have not done any mistake in setting up the equations and you are patient enough, most likely, your strategy did not work in real life because your model is over-fitted.

With the term “over-fit” I mean that the model knows quite well how to behave if the past would exactly represent itself while it would partially/total fail if future events are not exactly the same as past ones.

Let me illustrate this with an example. 

Imagine you have figured out the importance of rotating your investments among uncorrelated asset classes depending on the market conditions, now what you will try to do is to define some rules that will tell you in which asset class to be invested in. It is 2021, so you decide to use Machine Learning and more specifically you define a classification problem. Typically a classification problem tells you what to select depending on a given set of input variables. Input variables that in the case of investing could be: market volatility, return of a given asset, volatility of a given asset,… You then take your Python or Matlab software, build your model, train your data and… you are astonished by the outstanding performances of your strategy. I did the exercise and I have assumed that the strategy rotates among three asset classes: US T-bond, gold and US equities. I have then assumed that my holding period is between one week and one month. The result: 

You’ll then get super exited because when you look at the backtested data, you know that in no time you’ll be rich and you are confident enough that the strategy is robust enough to survive a market crash because your model was also trained with data including bad years like 2007, 2008 and 2020. 

What next? You go to your brokerage account, start implement the strategy, the first month does not go as expected, so you say that it was a one-off. Then eventually there is an equity market downturn and your drawdown is higher than 15%, you lose money and you: in the best case ask yourself what has happened and in the worst case you blame the rigged market.

What went wrong?

Long story short: your model is over fitted and this is not the way you run a back test.

The way I personally run a backtest is to mirror exactly what I would do in real life. If I have to take an investment decision in e.g. March 2016, the data I would have in real life to train my model are the ones available up to March 2016 and not September 2021. The way that a backtest should be structured is to iteratively train your model within the backtest. By doing so, your model is still over fitted (assuming we are still dealing with the previous example) but the outcome of using it is closer to what would happen in real life. The differences would be due to e.g. fees, orders slippage,…

​By running a backtest in this way, the performances of the strategy are now:

A huge difference is not it?

In my opinion, this is the most appropriate way to backtest a strategy and provide a very good idea on how your investment would perform in real life without losing money while implementing a possibly over fitted investment strategy.

I hope you find it useful and you have learned something new, to the next time!