What happens while you tune
A strategy has dozens of settings. Trying combinations, you inevitably find one that produces an excellent result on the chosen stretch. The trouble is that part of that result is coincidence: the particular candles of that particular period happened to suit those numbers.
Such a combination is called overfitting. It is recognised by one sign: on new data the result is sharply worse.
Why twenty runs are not twenty checks
Test twenty random strategies and one of them will almost certainly show a "statistically significant" result purely by chance. That is not a quirk of this service, it is a property of any search.
So the service counts the trials: how many runs you have already made on this instrument over an overlapping period — including runs of your other strategies. Ten strategies once each is the same search as one strategy ten times. The correction for the number of trials makes the significance bar stricter with every new run.
What to do instead of tuning
- Test over several periods, not one.
- Keep part of the history untouched and check against it at the end —
that is the out-of-sample test.
- Change one setting at a time.
- Look at stability, not at the maximum: a strategy with a moderate
result on every period is more trustworthy than one that shines on a single stretch.
A good check looks boring: similar numbers on different stretches. A curve that looks like a dream usually is one.