Hi there. I have a background in Operations Research (Optimisation) and so spend a lot of time trying to find the best set of parameter values in backtesting trading ideas. However, a big stumbling block for me is overfitting. I try and optimise the ratio of mean return / standard deviation of these returns.
If one has 30 free parameters does that mean you need 30 validation sets? Or is it more a question of if you have 30 free parameters then you need to split as follows X% training | Y% validation | Z% testing?
Hello Phillips,
Overfitting is a significant concern when optimising trading strategies, and it's important to strike a balance between finding the best parameter values and ensuring the generalisation of your strategy to new, unseen data.
The number of validation sets you need is not necessarily tied to the number of free parameters, but rather it's about properly partitioning your data to evaluate the model's performance.
Here's a common approach to splitting the data:
- Training Set (X%): This is the largest portion of your data and is used to train your model. The model learns the patterns and relationships within this set.
- Validation Set (Y%): After training, you need a separate dataset to validate the model's performance. This set helps you fine-tune hyperparameters and check for overfitting. It provides an unbiased evaluation before moving to the test set.
- Test Set (Z%): This set is crucial for assessing the model's performance on completely new, unseen data.
Note that the percentages (X%, Y%, Z%) depend on the size of your dataset, and common splits are 70/15/15 or 80/10/10 for training/validation/testing. However, in financial markets where data can be limited, you might use more aggressive splits, like 60/20/20.
Now, regarding the number of parameters, having a large number of free parameters can increase the risk of overfitting. In such cases, regularization techniques (e.g., L1 or L2 regularization) and feature selection methods can be employed to prevent the model from fitting noise in the data.
It's also a good practice to perform multiple rounds of validation, tweaking parameters and assessing performance on the validation set before using the test set. This iterative process helps in refining your strategy without compromising its generalization to new data.
Remember, overfitting can still occur even if you have a validation set, especially if you tweak your strategy based on the performance of the validation set multiple times. So, it's essential to be mindful of this and prioritize strategies that demonstrate good performance across different market conditions and time periods.
I hope this helps!