Hi,
I am a bit confused with backtesting methods. Which one people usually use?
Can I apply moving average with train-test spilt?
If yes, what is the parameter from train data(generated position) I should be using with test data? A bit confused here. How can I employ test-train normally with any strategy.
Hi Vartika,
Let me first clarify the three approaches to backtesting that you have listed.
In-sample testing: In this method, the same dataset is used to develop and test the trading strategy. For example, suppose you consider a 10-year dataset for in-sample testing to test a moving average crossover strategy. In that case, the strategy parameters such as the window of the moving averages will be optimised to give the best performance in the in-sample period.
As you can guess, since the in-sample data is used to fit the parameters for best results, it will generate the same results while testing the strategy on the in-sample dataset of 10 years. There is an obvious risk of overfitting here.
Train-test method: In this method, instead of using the same data for strategy optimisation and testing, different datasets are used. For example, if you have 10 years of data, the first 70% (70% as an example, i.e., the first 7 years of data) can be used for strategy optimisation to find suitable parameters, and the strategy with optimised parameters would be tested on the remaining 30% of the data (the last 3 years of data).
This would decrease overfitting compared to the in-sample testing method. However, when this strategy is taken to live or for paper trading where it would be exposed to new data, it might not generate similar performance unless the train data represents all possible market conditions.
Since the strategy is tailored to train market conditions, in live trading or on new data, the strategy may still suffer from limitations if the train data is not representative of future market conditions.
Event-driven: Event-driven backtesting is a way to simulate trading conditions over historical data to simulate trades. In event-based backtesting, trades are executed based on specific events or conditions defined within the trading strategy. These events could include price movements, technical indicators crossing certain thresholds, economic releases, news events, etc.
Event-driven backtesting is not an alternative backtesting method to the in-sample testing method or train-test method; instead, it's an approach to simulating market conditions to generate trades.
However, event-based backtesting is an alternative to vectorised backtesting. Vectorised backtesting involves processing historical market data in batches or vectors rather than one event at a time. This approach leverages the efficiency of numerical computing libraries to perform calculations on entire arrays of data simultaneously.
Event-based backtesting is more flexible and realistic, allowing for the implementation of complex trading strategies based on specific market events. However, it can be computationally demanding. On the other hand, vectorised backtesting is more efficient and scalable, making it suitable for handling large datasets and performing complex calculations, but it may not always capture the nuances of real-world trading dynamics as accurately as event-based approaches.
As an example, if you want to test a moving average crossover strategy with simple entry and exit conditions without stop-loss or take-profit, a vectorized approach can be sufficient for backtesting. However, if you have complex trading rules, including stop-loss, and take-profit, event-based backtesting is more suitable.
Another way to backtest with reduced risk of overfitting is 'walk-forward testing.' You can read about it in QuantInsti's blog here
The choice of backtesting approach depends on various factors such as the complexity of your strategy, the availability of historical data, and computational resources, etc.
I hope this helps! If you have any doubts, feel free to share them with us.