Still testing it but looks great.

I'm getting crashes when I run the WFO. What do I provide you to be able to debug it?

Let's start with the following details:

1. Is this reproducible?
2. Downloadable or custom Strategy? If downloadable, name is enough; otherwise we need to see the code in its entirety.
3. Single symbol or multi-symbol mode? Position sizing? Data loading settings?
4. Symbol(s) and data provider (including the data for external symbols)?
5. Which optimizer method?


The more details the better.

Oh, hi! Where can I find the WFO?
Thanks!

WLD 6.6 will be released in the days to come. Please stay tuned for the news on the built-in Home Page tool and on our website.

Thanks for the addition!

Is there some place where there is a more detailed explanation about it than in the User Guide? Specifically, what is WFO Efficiency?

Finally, while it's not really part of WealthLabPro, it would be great if you could reference a discussion of how to choose the "correct" Walk-Forward Intervals and Out-of-Sample Data %.

QUOTE:
Specifically, what is WFO Efficiency?

WFO Efficiency % and its formula are explained in detail in the User Guide (Help menu) > Strategy Window > Optimization > Walk-Forward Optimization > Results.

QUOTE:
reference a discussion of how to choose the "correct" Walk-Forward Intervals and Out-of-Sample Data %.

Robert Pardo in his book The Evaluation and Optimization of Trading Strategies mentions that the process is empirical, yet suggests that...

CODE:
Please log in to see this code.

Thank you. Excellent documentation as always. I just failed to notice the + next to Walk-Forward Optimization.

For WFO, where are the Strategy Parameter values for start, stop, and step sourced? I haven't found the way to set these in WFO as you can in Full Optimization.

I'm getting crashes when I run the WFO only when I choose Genetic Optimization. For a other choice of optimization method is working properly.

QUOTE:
I'm getting crashes when I run the WFO only when I choose Genetic Optimization.

Agreed. For me, Exhaustive works, Genetic consistently crashes. I don't use Monte Carlo.

We've been already aware that Genetic Optimizer crashes in WFO. Sorry for the delay with uploading the updated GA extension with this and other stability improvements. Will be done shortly.

UPDATE: Genetic Optimizer 2013.12 has been released today, please update to enjoy GA in your WFO optimizations.

akuzn wrote:

Good day!
Many thanks for new optimization step pushing to new ideas.

1. Just tried WFO with Monte Carlo optimization. In backtest setup i ve choosen "use worst trades".
Cant understand why when i press on WFO optimization result - strategy loaded with optimized parameters doesnt give same with WFO result? (All parallel computing methods are switched off).

2. Unfortunately there is not enough informationin in quick ref about how parameters are choosen in WFO?
How parameters selected (after optimization ) in one time window affect parameters in another window sliding or expanding?

3. If parameters are optimizing from beginning in every new time window - why there are 2 settings - for WFO and for optimization method ( same in optimization tab)?

Is it possible to ask you for some modifications?
1. For example expanding window but when starting period doesnt change, ending period can slide.
To be honest would like to test such kind of optimization especially in combination with GA. I did it many times manually.
2. Is it possible to modify GA - for example every new generation will have bigger time window or for example length of half of in sample period can mutate ga style?

Seems it wont be too difficult. But in depends on the mood.

LenMoz:

QUOTE:
For WFO, where are the Strategy Parameter values for start, stop, and step sourced?

They are sourced in the Optimization tab. WFO picks up changes even w/o applying to code. Try editing selected parameter and re-running WFO.

akuzn:

QUOTE:
Cant understand why when i press on WFO optimization result - strategy loaded with optimized parameters doesnt give same with WFO result?

Please refer to the User Guide: Strategy Window > Optimization > Walk-Forward Optimization > WFO FAQs: "Why do I get different results each time I run the same WFO in Portfolio Simulation mode?" Note that Monte Carlo is random in its nature.

QUOTE:
How parameters selected (after optimization ) in one time window affect parameters in another window sliding or expanding?

In each in-sample period, WFO finds an optimum set of parameters which it applied to adjacent out-of-sample block. If you're talking about trades that span two WFO intervals, then the WFO has special logic to correctly simulate a trade that spans multiple intervals.

QUOTE:
If parameters are optimizing from beginning in every new time window - why there are 2 settings - for WFO and for optimization method ( same in optimization tab)?

Suppose you picked GA as the method. You can choose Net Profit as the fitness function, but let the WFO select the OOS parameters based on another metric (highest Recovery Factor, for example) obtained during in-sample runs.

I am sorry, Eugene but "worst trades" is switched on as i told above.
I am asking - why when i try to use optimized parameters from result tab by pressing on them - i dont get same result as i see in Net Profit row?
Certainly Monte Carlo is random.

When i optimize not in WFO, but as previuosly GA for example when i try to use parameters from result tab everything is going ok. As for wfo when i try to use optimized parameters - WFO loads additional strategy to new window - i see optimized parameters in strategy parameters tab, but i have another results.

QUOTE:
I am asking - why when i try to use optimized parameters from result tab by pressing on them - i dont get same result as i see in Net Profit row?

I bet it's due to advanced nature of your strategies, equipped with all those parallel tasks, reflection and the rest. Load a plain vanilla strategy, say "Moving Average Crossover", and I'm sure the WFO OOS number will match the double clicked strategy opened in a new window. Mine always does regardless of chosen optimization method.

CODE:
Please log in to see this code.

CODE:
Please log in to see this code.

Odds to call are not convenient to some combo and open ends straight draw).

No everything is in sequentiel mode ( i have a switch).
Certainly before input question here i have tried with 10 lines strategie.

I have repeated In-Sample results. But where i can find Out-of-sample results?
From User Guide

QUOTE:
The WFO In-sample Results view provides the Strategy Parameter combination and Scorecard metrics for the optimum backtest (based on the selected metric) for each In-sample (IS) interval as well as for the current IS range. As if it were a regular backtest, Out-of-sample (OOS) results and trades for the Walk-Forward Optimization are given in the other Performance Visualizers you have selected

Or u mean - i have to input parameters including Out-of-Sample interval and then i see them regulat style?

Another question posted earlier: In WFO tab there are 2 groups of metrics parameters. Wich groups must be used?

QUOTE:
But where i can find Out-of-sample results?

Good question. Answer: right in the Strategy window. You probably didn't even notice that, but when you have just finished a WFO, there's a [WFO Out-of-Sample] note appended to the Strategy window's title. Now until you click Go/Execute to run a regular backtest using the default parameters, your Strategy window displays OOS results i.e. the combined results of backtested OOS 'chunks'.

QUOTE:
Another question posted earlier: In WFO tab there are 2 groups of metrics parameters. Wich groups must be used?

Already replied earlier (see reply #15 above). To rephrase, they serve diferent purposes: the one below configures selected optimization method, and the one above is the 'master' metric for the WFO itself.

If i understand right now OOS results after WFO with [WFO Out-of-Sample] note are shown for best WFO solution. Right?

For example if i want to see other WFO results i have to input them manually to new strategy window.

To be honest it is complicated.

May be it will possible to vizualize WFO results in visualizers tabs? Something like aggregate strategy with possibility to choose any strategy to see results.

I understand that wfo realisation is not easy to insert it in forms etc, but if u add such vizualisation it will be super.
In one click it will be possible to see all graphs for results, slide through results...Seems it must be modified versions of vizualizers.

QUOTE:
[WFO Out-of-Sample] note are shown for best WFO solution. Right?

Right, per the selected metric.

QUOTE:
if i want to see other WFO results i have to input them manually to new strategy window.

What are "other" WFO results? In WFO, you select the optimum, and use it for the next interval. Rinse and repeat.

Edit: If you want to see Exhaustive results of each WFO interval, double click the row's interval. It opens a Strategy [full] Optimization in another window with the interval's date range pre-selected. Run an Exhaustive Optimization.

QUOTE:
To be honest it is complicated.

Which part?

QUOTE:
May be it will possible to vizualize WFO results in visualizers tabs?

It's more than possible. It already happens! When you finish a WFO, the title bar shows [WFO Out-of-Sample], as you noted. It indicates that you are seeing WFO Out-of-sample (optimum) results in all the visualizers.

I supposed that intermediate results shown in WFO result tab may be visualised.

Those are the In-sample results which serve only to determine the optimum parameters to apply to the Out-of-sample test. With WFO, we're not as concerned about visualizing optimum results as we are about showing realistic trading based on the results of an optimization. As noted in the User Guide, though, "the WFO process chooses parameters that result in maximizing (or minimizing) a specified performance metric, it can and will pick peaks on an optimization surface that you may not have actually selected as the best choice for a stable system."

In cased you missed my edit above... if you want to see Exhaustive results of each WFO interval, double click the row's interval. It opens a Strategy [full] Optimization in another window with the interval's date range pre-selected. Run an Exhaustive Optimization.

It is not comfortable to open new window every time and each strategy has own interval - difficult to compare.
Would be nice to see on one equity chart on full interval.

Just what i am talking about. Certainly if possible.

I don't think that I understand what you're asking for. Is there another TA application that does it? (Which one?)

1. What are you comparing and for what purpose are you comparing it?

2. What, specifically, should be put into "one one equity chart on full interval"? Why an equity chart?

3. What is the "full interval"? Is it the entire test range or each of the n WFO intervals?

To be honest i dont know any other TA as good as WealthLab.
If i remember right there was even HFT realisation.) Only due to backtest capablities. Trading was on 1 bar. Strategy run 20 ms.

What i was doing before - something like manual expanding window logic.
Step 1: GA with recovery factor.
Step 2: after i compare 5 - 10 results from the list on IS + OS (full time window) and +25% i ve reserved to see let's say how it works in "future".
Step 3: reduced number or parameters on same IS + OS through steps 1 and 2.
Step 4: if needed step 3 with new ideas.

On each step i see results and i can see ratios and Equity chart. Graphics are always good.
For example 1 month ago i ve realised one good idea. There are 3 weeks i have tuned parameters for intraday strategies.
And seems they dont need changes. But i was analyzing many equity charts. It s easy and quick.
Best parameters were choosen only with graphics (sorting results by best ratios not enough). I often use Closed equity chart: easy to see quality of trading signals and quality of managing position logic in realised/unrealised profits. On Equity charts it s easy to see contribution of long and short trades.
Just one look and everything is clear. Only after that performance tab.
It s impossible to describe everything in ratios.

Here WFO is a black box. After 4 hours of computing i get only one result without any intermediate result to compare.)
If they could be drawed it will be easier to compare and see what was going on.

I m always trying to draw any new signal - easy to catch errors in realisation and logic.

Concerning WFO in other TA.
For example in OMEGA Tradestation there s more information about WFO runs.

After have run WFO many times seems it needs some additional filters.

May be it is possible to add not only highest/lowest metric but additional conditions wich could be taken from scorecards.
2 - 3 additinal filters: winrate more than, average profit / average loss more than. In WFO and in GA.

I ve found with "manual expanding windows" for all my 8 trading symbols winrate > 55%, for 2 hyper liquid futures on stocks >80%, Average profit / average loss for all strategies min 1.8, on average 2.2. But WFO has not found such parameters yet.

kbellare asked:

WFO is an excellent addition - i've been waiting for this for a while, didn't realize it was available in WL-6.6

Since the WFO window does not have a Strategy Parameters window like the Optimization tab does, if i need to reduce the strategy parameters to walk-forward-optimize, can i remove some Strategy Parameters from the Optimization tab and expect those to be removed in the WFO run? It's tedious to edit the code every time to remove or change strategy parameters for each WFO run.

thanks
Kiran

Yes you can do it. See my reply #14 in this thread.

On a closely related note, a request was submitted to be able to quickly omit Strategy Parameters from optimizations (probably using checkboxes).

There still isn't an elegant way to remove a parameter for optimizations, but you can do it by making the start, and stop values the same as the default value (step will automatically be changed to -1) for parameters you don't wish to optimize.

@Cone, there is a Remove Parameter button in the the Optimization tab that i've been using - does it not work for WFO?

There are many ways to skin a cat. One of them is like Cone suggested, but remove parameter will also do the trick.

ronc asked:

Questions on WFO:

1. Are the results each Out-Of-Sample test incorporated into the next In-Sample run, and if so, how? Or does each In-Sample run start anew, optimizing the parameters with no knowledge of the previous IS and OOS runs? If there is knowledge feed-forward, exactly how does WL combine the IS optimal parameters with the OOS run going into the next IS run?

2. WL Help says "Out-of-sample results for the walk-forward backtest are shown in the regular Performance Visualizers like any other backtest." I am looking for the the OOS results for each backtest in the series but cannot find them in the Perf Visualizers. When I click the Perf Visualizer tabs after a WFO run:

Performance Tab: shows some result, but since the Data Range in the Strategy Window still shows the range for the overall test, what do these results apply to? Is is the sum of all OOS runs, or something else? How do I see the results for each OOS backtest in the series?

Trades Tab: From the dates of the trades, I am guessing that these are all of the trades executed in all of the OOS backtests, combined together? If so is there any way to see just the trades from a specific OOS backtest, most importantly the last one? I assume that the IS trades are not included in this list?

3. WL Help describes Current Interval: "Wealth-Lab Walk-Forward Optimizations include a Current interval optimization. To trade a Strategy per the WFO methodology, one would assign the optimized parameter value(s) from the Current run for use as the Strategy's default or Preferred Values for live (or paper) trading." But what exactly is the "Current Interval"? From output, the Current Interval seems to cover the latter portion of the Strategy Window data range, but I am not clear what meaning that has. Is it the sum of of all of the OOS periods? What is is the source of the parameter values in it? Are they averages of the values for each In-Sample Run?

Thanks,
Ron

1 - No, each In Sample run starts anew, finding a new set of optimal parameters each time.

2 - After the Strategy has been executed on the set number of WF intervals, there comes the OOS backtest that is built from the results of the full-range runs. Simply put, the OOS results are the results of all backtested OOS "chunks", combined. This is what you observe in the Strategy window after performing a WFO - as indicated by [WFO OOS] in the window title. The OOS will be in effect until you re-run the strategy.

The WFO's Results tab display In-Sample results, each can be reviewed by double clicking on its row. Your question about visualizing the individual results of OOS runs was already asked (see posts #18, 22 above) and our position regarding it was expressed by Cone (see his reply #23). To reiterate, Wealth-Lab is not concerned about visualizing the results of individual OOS runs.

3 - Current Interval is an in-sample run that gives the ability to apply the parameters for the current in-sample period i.e. to live trading.

You can see what exactly Current Interval is on the graph in the User Guide (Strategy Window > Optimization > Walk-Forward Optimization > WFO Control). It illustrates the Current Interval for Expanding window where it's equal to a full optimization.

Here's a picture to give you a better understanding:

Let's say the StrategyTime Scale is 1 minute, the Strategy Data Range is 3 days, the in-sample range is 120 bars (3 hours), and OutOfSample = 33% (1 hour = 60 bars). When I have tried this sort of thing, the Results tab shows the Ranges with only date precision. Are the ranges actually at sub-day precision and the sub-day values are not shown, or are they rounded to the nearest day (in which case sub-day WFO won't work)?

I believe that the ranges are at sub-day precision. But don't take my word for it: set up a test optimization where only 1 parameter combination is possible and make sure that different In-Sample runs with identic parameters deliver different In-Sample results. Should the ranges were rounded, the In-Sample results would be identic (and they're not).

[You can't prove it by opening a window for the in-sample results because it won't work out on sub-day intervals, but that's a different story. The reason you already know: setting a Date Range to less than 2 days is not made possible. Since sub-day optimization doesn't make great sense anyway, I'm absolutely not worried about it.]

The tool divides the ranges into bars without regard to start-of-day or end-of-day. That design is probably not optimal (an unfortunate pun when referring to an optimizer), but in the grand scheme of things, it should not materially affect the results of an optimization with intervals that span several weeks and that have a statistically significant number of trades.

imho, if you're trying to optimize on a hourly basis, a) you're bound to be overfitting, and, b) how do you plan to apply it in live trading anyway? How many trades does your strategy take each hour?

I have been testing WFO and generally it works great but I'm having hard time to evaluate the results among multiple strategies.

Therefore I'm proposing the following new features to WFO.

1. In addition to WF intervals # and OOS data % in WFO settings, I would like to specify the intervals in date range (days/weeks/months/years). For example, interval is 4 weeks and OOS is 1 week, thus IS is 3 weeks.

2. WFO optimizer. This is so that you can optimize the intervals and OOS data range.

3. Support more wider interval and data% range. For example, WF Interval from 2~100 to 2~500, OOS Data % from 10~50 to 1~50. Current limitation doesn't comply very well with intra-day strategy for years time scale.

Is the WFO in WL incomplete?

The starting point is Robert Pardo’s book „THE EVALUATION AND OPTIMIZATION OF TRADING STRATEGIES “ which describes an easy to understand work flow to go through a WFO process. The procedure is consistent and is based on reliable statistical methods. So, if your trading strategy passes the WF test, it is highly significant that it makes money in realtime trading.

Comparing the Wealth-Lab WFO calculation to Pardo’s approach, there are some differences to mention – especially regarding the computation of two indicators: WFE and overall WFE. As these indicators play a crucial role in evaluating the robustness of a trading system, I would like to better understand why WL uses another calculation approach and how the deviations may influence the assessment of robustness of trading systems. To be more precise let me ask some questions:

WL uses net profits by the total number of bars for in-sample and out-of-sample results. In contrast Pardo takes annualized absolute profits. Why does WL deviate? In the help menu it is said that the special WL calculation allows it to apply to both portfolio mode and RawProfit Mode. Okay. But there is no indication whether this divergent computation from Pardo delivers different results or not. Maybe the WL approach and Pardo’s are one and the same. But it would be helpful and also necessary to explain in more detail when differences in results occur and when they do not occur and how reliable the WL calculation is compared to Pardo’s approach.
Does WL show out-of-sample results? If yes, I cannot find them. However, they are crucial for the WF test. To enhance transparency in the procedure it would help to illustrate the out-of-sample numbers in a clearly arranged visualizer-sheet, also to calculate WFE manually.
Pardo calculates the overall WFE for the complete analysis as follows:




with



and



where
APWF is the average profit of all WF profits
PWFi is the profit of walk-forward period i
Ywf is the number of years summed by all walk-forward periods
APOP is the average profit of the optimization period
POPj is the profit of the optimization period j
Yop is the number of years summed by all optimization periods
(For those who have the book at home got to page 255 ff). This is completely different to the overall WFE calculation in WL. From my point of view, the analysis in WL lacks the last but most important indicator for the robustness of a trading system. Therefore, the WFO in WL seems to be incomplete.

To put it in a nutshell: Pardo delivers a consistent workflow for walk-forward-analysis. It based on common statistics and, therefore, it is reliable. In the help menu WLs refers explicitly to Pardo’s process. So I cannot understand why WL differs from that procedure and why it works with other maybe also similar calculation methods. It makes conclusions about the long-term reliability of a strategy much more complicated.

Commenting only this one:

QUOTE:
Does WL show out-of-sample results? If yes, I cannot find them.

This question has been answered in this very thread. See my reply #19 above.

Hello Eugene,

I experience the same issue as 07nioe.

I think there is a misunderstanding here. If not, then the WL User-Guide does not tell the truth. Let me explain in more detail, step by step.

There are two issue:
1) The calculation of the Walk-Forward-Efficiency (WFE)
2) Where to find Out-Of-Sample Results

As WL already shows the Out-Of-Sample results, although I would like to see a little bit more of that, let’s focus on the first issue. We can discuss this later.

1 Calculation of the Walk-Forward-Efficiency (WFE)
1.1 Calculation of a single WFE for one In-Sample and Out-Of-Sample period
I am writing about the value highlighted with a red box in the picture below.



The Wealth-Lab User Guide says that instead of using annualized profit, as does Pardo, WL uses the net profit and the total number of bars in the In-Sample (IS) and Out-Of-Sample (OOS) periods.
I am convinced that the ratio still remains similar as using annualized profit. You can see the evidence in the screenshot further below.

@Eugene:
Please correct me if I am wrong. Could you please also confirm that this part of the Wealth-Lab User Guide reflects the single WFE calculation in the WFO?
07nioe asked why you are deviating here from Pardos original formula.

1.2 Calculation of the overall WFE for all In-Sample and Out-Of-Sample periods
I am writing about the value highlighted with a red box in the picture below.



The Wealth-Lab User Guide says that the overall WFE is calculated by taking the average of the single WFEs. Let’s try and reproduce the value above taking the displayed single WFEs.

Overall WFE = (93.99% + 49.51% + (-61.15%) ) / 3 = 27.45%

That is exactly the value shown in the screenshot above. Hereby we confirm that WL calculates the overall WFE taking the average of all single WFEs.

The issue is that Pardo uses a different formula to calculate the overall WFE. It is definitely not the mean or average of single WFEs! As 07nioe wrote, Pardos formula is described in his book on the pages 255 ff.

Let’s calculate the overall WFE using Pardos formula
In the screenshot below I summarized the IS and OOS results as well as the duration in years of each period. You can see there the single WFEs calculated Pardos way and the ones taken from WLs WFO.
The most important information that you can see in this screenshot is that the average WFE and the overall WFE are two different measures and that they are quite different.



As already stated in the Wealth-Lab User Guide an overall WFE above 50% indicates a robust strategy. Pardo says exactly the same and he refers to the overall WFE not the average WFE.

Request
Do you agree that the calculation of the overall WFE must be changed? If yes, would it be possible for you to show in the WFO results the overall and the average WFE?

If you would like to reproduce my numbers, you can find the strategy and the Excel-Sheet attached.
Here are the steps to reproduce

CODE:
Please log in to see this code.

Many thanks,
Konstantin

Hello Konstantin,

Thank you for your message. I don't see it as an issue, but leave this up to Cone to decide.

Why in your opinion using annualized profit is preferred?

Hi Eugene,

I do not prefer the annualized profit. For me it does not matter whether you use annualized profit or, how WL does it, per bar. The ratio is still very similar, the deviation is insignificant.

The only thing is, that Average WFE is not Overall WFE. But WL shows it as such. And I really would like to have the Overall WFE calculated by WL as well. Instead of using annualized profit you could use the following formula.



Cheers
Konstantin

Hi Eugene,

I am a little bit disappointed about the missing progress of this discussion. We have unfolded - let me say - some irregularities in your WFO. The applied metric does not help to make reliable assertions about a strategy's robustness. But this is what I expect from the WFO. From a coding perspective your approch may be fine. From a statistical point of view, the calculation is definitely incomplete. As coding should follow statistics, corrections as kribel described above are absolutely nessessary. I hope we agree in opinion.

Best
Niels

Hi Niels,

While I do not necessarily agree with your conclusion re: the usage of our WFO metric negatively affects strategy robustness analysis, that may be beyond my level of competence. From my limited perspective, it's a matter of preference.

As we're currently busy with different projects, I hope that Cone gets to the bottom of it when he has time. This is not a business critical issue. Consequently, I wouldn't expect any rapid changes and developments here.

I had been traveling during the recent part of this discussion and missed it entirely until now. We'll do our part to support a change to the overall WFE calculation per kribel / Pardo. It makes sense.

If you're going to be messing with the code anyway, one thing that would make the WFO Optimizer more useful for me is more control over the time periods used. Specifically, if I run the WFO Optimizer on 10 years of data, I would like to be able to have each period begin on January 1 and end on December 31. For smaller amounts of data, I would like to be able to have each period begin on the first day of the month and end on the last day of the month or the first day of a quarter and the last day of a quarter. Since that is how I adjust my strategies in real time, I would like to be able to back test them the same way.

Just an IMHO: this single-user option could be a complication of the otherwise logical and finite instrument's GUI. Wealth-Lab Developer customers may be using it in China with New Year starting in February, or in the Middle East where workweek ends on Thursday. New features should be approached carefully.

Currently, the selection of the IS and OOS periods are rudimentary by bar count. Developing a [different] GUI is a big deal, and I just don't want you to get your hopes up about something like that. Changing a calculation is a much easier task, so let's see if we can even get that first.

I can see the point of Panache and at the beginning I would have been with you. However picking fixed dates isn't really helpful. It might be more beautiful but it really doesn't matter that much - WFO is WFO.
The way I see it is really that it really shows you the results in a more realistic way. It is in no way someone should "optimize" on.

Hi Cone,

I also would really appreciate an option described by Panache. However, having a GUI where the user can pick dates for IS and OOS periods is not the best to go. Maybe it would be possible to generalize some options like smallest possible unit, e.g. one year, one month etc. Then the user can define when the smallest unit for that user begins and ends. This way you could make users around the globe happy.

Cheers
Konstantin

QUOTE:
generalize some options like smallest possible unit, e.g. one year, one month etc.

I agree with Konstantin. That would be my preference. His solution would make a strategy's WFO statistics more easily comparable to market performance.

Let me start by admitting that I am far from well versed in WFO Optimization.

If I load exactly 10 years of data (January 1, 2001 to December 31, 2010 for example) and select 9 walk forward intervals and 50% out of sample data, I would expect the resulting intervals to be January 1, 2001 to December 31, 2002, January 1, 2002 to December 31, 2003, etc. Therefore, I would expect to be able to control the beginning and ending dates of my intervals by the Date Range of my data, the number of intervals and the percent of out of sample data.

Obviously, the WFO Optimizer doesn't produce exactly my expected intervals.

Since neither Cone nor Eugene are excited about tweaking the GUI, perhaps a little work could be done on the back end, so the optimizer calculated the intervals based on dates, rather than what it does now. If it did that, I don't think it is necessary to change the GUI.

QUOTE:
If I load exactly 10 years of data (January 1, 2001 to December 31, 2010 for example) and select 9 walk forward intervals and 50% out of sample data, I would expect the resulting intervals to be January 1, 2001 to December 31, 2002, January 1, 2002 to December 31, 2003, etc.

Why? You did not consider the fact of each interval having different number of trading days due to market holidays and different number of days in calendar month.

Hi Eugene,

QUOTE:

You did not consider the fact of each inteval having different number of trading days due to market holidays and different number of days in calendar month.

This is rather a technical/mathematical issue that you are writing about. But a year is a year. Also a financial year starts and ends always on the same dates regardless of holidays. Also a daily bar is a daily bar regardless of the operating hours of the stock exchange, e.g. full time or early closing. I hope you get the point here.

I will try to prepare a possible GUI solution based on the current GUI to show that the required GUI changes are small.

Cheers
Konstantin

kribel, please don't waste your time. We (MS123, a third party) tried to get the Fidelity designer to do it this way when WFO was in development. It's not just a matter of creating a GUI, but it's also incorporating it into the WFO engine, testing it, QA, documentation, etc., etc. Changes in the UI do not come easily, so guys, I'm telling you to not get your hopes up about this. It's a matter of priorities and money, and I'm quite sure it's going to make it through either one of those filters any time soon. I'm just being pragmatic about it so everyone doesn't keep wasting their energy on this.

Konstantin,

Here's my two cents: creating "pretty" intervals is an unnecessary perfectionism. The WFO does not consider "financial year": it operates with chunks (intervals) of data, including intraday data. The proposed "smallest possible unit" feature sounds cosmetic to justify its use (and the number of unwanted new bugs it may introduce).

After a little spreadsheet work, I derived a method to get at least approximately "nice" WFO intervals. It involves properly setting the Date Range. The start date requires the most work.

1. Establish the end date. This should, of course, align with the granularity you seek (end of a year, quarter, month, etc.)
2. Decide on the "Number Of Walk-Forward Intervals."
3. Decide on the "Out-of-sample Data %"
4. Determine the days for the desired granularity. This would be near 90 for quarterly results, near 180 for half-year. In my spreadsheet I used 365.25 for days in a year and divided appropriately.
5. Calculate the Total Intervals that will be needed = Walk-forward Intervals(2.) + 1/Data%(3.) - 1
6. Calculate the start date = End Date(1.) - [Total Intervals(5.) * Out-of sample Days(4.)]

Example:
1. End Date = 1/1/2015
2. Number Of Walk-Forward Intervals = 4
3. Out-of-sample Data % = 20
4. OOS Days = 182.625 (6 months)
5. Total Intervals = 4 + 1/.20 - 1 = 8
6. Start Date = 1/1/2015 - 8 * 182.625 = 1/1/2011

Yielded in the WFO run:
In Sample Range -> Out of Sample Range
1. 1/3/2011 to 12/24/2012 -> 12/26/2012 to 6/27/2013
2. 6/28/2011 to 6/27/2013 -> 6/28/2013 to 12/27/2013
3. 12/28/2011 to 12/27/2013 -> 12/30/2013 to 7/1/2014
4. 6/29/2012 to 7/1/2014 -> 7/2/2014 to 12/31/2014
Current 1/3/2013 to 12/31/2014

Len

KGo asked:

Walk Forward Optimization is not exiting open positions on secondary symbols during the next Out of Sample segment. Is this a known design feature or am I doing something incorrectly?

How can this be reproduced? Taking "Pairs Trading" strategy for example.

I've read all the threads and user guide but have not found reference to this problem. Thanks for your help.
Above Eugene suggested

QUOTE:
Let's start with the following details:

1. Is this reproducible? YES
2. Downloadable or custom Strategy? Custom.code below
3. Single symbol or multi-symbol mode? Single Symbol mode, Position sizing? Fixed dollar 5k Data loading settings? 12/3/2010 to 4/7/2015
4. Symbol(s) and data provider (including the data for external symbols)? SPY, SSO, SDS from Fidelity
5. Which optimizer method? Exaustive
Used 3 intervals on daily bars. 40% OOS data, Sliding, Basic Scorecard, Highest, NetProfit, Exhaustive. 8 runs takes only a few seconds, A buy on 1/16/2013 is never exited.
I've tried this various ways and have had no success accessing and exiting the open position.

CODE:
Please log in to see this code.

Thank you for pointing it out Kevin. This looks like a bug to me.

Thanks Eugene. I believe this one is important to fix because many traders use long only leveraged ETF pairs like SSO,SDS or QLD QID instead of trading SPY or QQQ long and short. It would be helpful to use WFO on them.

Eugene,

I think there is still a bug on WFO leaving trades open. I just noticed it happening here but didn't found a solution around, is there some simple workaround on this?

Thanks.

Jairo,

How do we reproduce what you think is happening? Step by step please, with sample code, symbols and settings. See post #61 for an example of a bug report.

Eugene:

1. Is this reproducible? YES (maybe not exactly due to optimization values coming from Particle Swarm, not exhaustive, but usually I will get that problem with 1 or 2 simulations on this data.
2. Downloadable or custom Strategy? Modified.code from MA envelope, below
3. Single symbol or multi-symbol mode? Single Symbol mode, Position sizing? Fixed dollar 100, 100 dollar account, margin factor 2 to 1. Data loading settings? All
4. Symbol(s) and data provider (including the data for external symbols)? INSBTC (crypto), might have to upload it here
5. Which optimizer method? Particle Swarm

Using slippage 0.15% and fees 0.075% per trade.
Used 5 intervals on 15m bars. 25% OOS data, Sliding, MS123 Scorecard, Highest, APR%, Particle Swarm. A buy on 02-Apr-19 on OOS is never exited.

https://www.mediafire.com/file/nto1cc1fajjg7el/INSBTC.csv/file

The code is below

CODE:
Please log in to see this code.

Jairo, thanks for your detailed problem report. I noticed that your strategy peeks into the future with this code:

CODE:
Please log in to see this code.

Basically what it does is using a derivative of the Close price of the day of entry (which is unknown until market close) to exit on a stop that same day. Before we look into this, please fix it:

CODE:
Please log in to see this code.

I changed the code as suggested, but the problem remains, below is the first WFO I tested.

OK thank you. Could you re-run it with one of supported optimizers i.e. Exhaustive or Monte Carlo? And to make the developer's life easier please use Basic or Extended scorecard:

QUOTE:
5. Which optimizer method? Particle Swarm

QUOTE:
Used 5 intervals on 15m bars. 25% OOS data, Sliding, MS123 Scorecard, Highest, APR%, Particle Swarm. A buy on 02-Apr-19 on OOS is never exited.

I did basic scorecard + monte carlo, after 4 runs I got one with a open trade.

Looked into it. Made one more fix to your code and WFO works as expected (in my test on Basic Scorecard + MC):

CODE:
Please log in to see this code.

QUOTE:
I think there is still a bug on WFO leaving trades open.

Another customer reported this a couple months ago. We're still trying to find a reproducible case to pass it on to the developer. Thanks for the report.

I tested the modified code again, and after some monte carlo runs, I still got the problem.

That 07-Jan-2020 trade was opened half a hour before the data range's end. Where are you seeing the problem on this screenshot?

Indeed Eugene, my bad, it seems I couldn't see any other open trades by now, thanks!

As an avid user and fan of WFO, and not knowing exactly if this is the right topic to do so, but assuming that, I would like to know if the following thing is planned:

When using WFO, we usually set, let`s say, Net Profit as the metric to choose, but when selecting manually the parameters, I would select eg. the highest Net Profit that meets Recovery Factor > 2 and Ulcer Performance Index > 3, in order to not select blindly some parameters that may get highest net profit, but would bring too high drawdown that would be unbearable in real trading.

Is it possible to implement filters on the WFO selection? (I know that maybe that criterias won't be found on some simulations, but if that happens I will gladly discard that asset or strategy).

Thanks.

Here's an idea for a workaround for you as per my post #15 above:

"Suppose you picked GA as the method. You can choose Net Profit as the fitness function, but let the WFO select the OOS parameters based on another metric (highest Recovery Factor, for example) obtained during in-sample runs."

That unfortunately doesn`t exactly solves the problem. Let's say I want the highest net profit that has at least 10 trades done, but I still want Ulcer Performance Index > 3. Usually, if I let the WFO select the OOS parameters based on Ulcer Performance Index I will have selected parameters that have a very high Ulcer Performance Index, but it will have only 3 trades, and that won`t fit my criteria. The ideal parameters would be somewhere in between the highest Net Profit and Ulcer Performance Index.

Maybe even a manual selection would solve this. Other workaround is doing the WFO manually, but that kills the WFO function.

As far as I know, the multiple filters for WFO aren't considered. Seems like even its feasibility would require careful evaluation.

Eugene, I just found once again, unfortunately, an open trade on OOS coming from WFO.

The problem probably happened because a trade was opened on 19:15 on 6th june 2019, which is an ending date for one of the OOS data. I have a programmed exit to close any trade that is open at the start of the day at 01:30, that's why that trade wasn't closed on the same day.

The developer has fixed the bug in the upcoming build.

Very good news, thanks.

Hi Eugene!

Great Software! I think I will renew!

I have a question about WFO:

- I optimized only one Parameter: Holding period
- WFO optimized differnt holding periods for different time slots
- I ran: "Run Out-of-Sample Back-Test using WFO Parameters"
- in the "Back Test results - Positions" the holding period is not reflecting the WFO Results for the "Hold Max" parameter?

Thank
kurt

Hi Kurt,

Thank you, we strive to make WL the best product!

Since this is the legacy product forum (Wealth-Lab 6), could you start a new topic on the new website where everything related to WL8 is discussed?

https://www.wealth-lab.com/Discussions