"While TBB isn’t about stunning charts, it does what is needed for understanding how a system handled a trade and what is going on within that system when system internal values are displayed on a chart. TBB also supports stand-alone indicators and show-me location dots on the screen, and both can easily change color dynamically and be shown on a chart screen. Still, nobody should select TBB if their primary goal is a great charting package.

With that said, I think TBB is the best back testing platform available to the general public that I know about. It is the one to use because it is capable of providing a much more robust back test using a portfolio of markets and, or a portfolio of systems, that get tested by stepping through each market for each date in the portfolio of markets selected. To be clear about what this date-sequence stepping means, I’m saying that the testing of a market doesn’t move to the next available date in the data file until all the markets in the portfolio have been tested up that point and are ready to be tested on that date. If you are now wondering how this is different from other platforms, the difference is the other platforms test each market from the beginning of a market’s data to end of that file before they test the next market. This means that the system is never in a position to influence the trade signals based upon what is happening at the macro level of the account.

In real trading, we all must understand what size we can afford based upon the risk profile our system testing tells us we can afford to risk on each signal, so knowing where we are in the reality of the period that test date in the data file represents allows us to know what size to use with each signal. Other platforms allow this on a market by market basis, and it can be approximated roughly by running each market through a back test and then manually date aligning the equity profile results on each date in an add-on file the system reads, but that approach isn’t close enough when the action the system should take is to disable the signal because the risk profile for the account would be exceeded, or if the size to volume ratio, or some other risk filtering rule would be exceeded. Once again that can be adjusted for in the other platforms a little more by entering more code rules into the system to exclude trades for that market on that date again, and then manually reassemble the equity file again, but then again what happens when a trade in play closes out leaving more opportunity that would enable a trade signal that is still viable that was disabled because that trade closed was active? The user would then need to adjust the system logic for that market again, and then run the process again, and again, and again.

With date-sequenced testing, one system can be used to test all the markets in the portfolio and by just adding risk analysis and money management modules (Blox) to the system, the testing and new signal generation process becomes dynamic instead of iterative with a lot of manual interventions. By keeping the back testing dynamic and automated, results happens fast and it also allows the system developer to test the system variables for robustness and effectiveness from a portfolio perspective that is reflective of how the system will be applied in reality. Once again some of this can be done by hand with a lot of system code rule changes and testing iterations, but that is making work and risking errors where they don’t need to happen, and we can never forget that time is money."

Hi Eugene-
Not sure if Cone is out on holiday....I'd be curious as to your take on the comments made re: another competing backtesting site vs. Wealth Lab's?
Thanks-Rob

Hi Rob,

Sorry, too much work to dive into this.

Understood -no worries. If you ever have the time to comment on only one aspect, say their comment about "date-sequenced testing" and it's implications, great!

But I certainly understand it's not a priority right now.
Rgds -Rob

Rob,

I read this opus a couple of times and don't understand the "problem". As far as I understand his/her point, he/she basically implies that "other platforms" do raw profit runs, but only their "unique" platform "allows to understand what size we can afford based on risk profile blah blah". I understand this "date-sequenced testing" as a true portfolio level simulation -- something that Wealth-Lab has since I don't know when.

Ya- kinda what I read into into too - "borderline hot air", but wasn't sure if there was any credence in it all. So WL is as cutting edge as anyone else when it comes to simulating backtesting -great thanks!
Rob

Hi Rob,

I can shed some light on the post, or at least, on what date-sequenced testing means.

The issue that is being referred to is very important.

Essentially in platforms like Wealth-Lab, a trading system is run on each stock one at a time, for each stock in your datasource. Then all the trades that were created are sorted in date order, then the money management settings are applied. This means that when your rules are deciding whether or not to buy a stock, you cannot access "portfolio" level information to help make your decision. This is because there is no portfolio existing at that point in time that you run your buy rules on a specific stock.

Date-sequenced testing: With portfolio level simulators, instead of running the rules over each stock from start to end, the rules and position sizing are run in "date order"... in other words, the rules and position sizing are applied to every stock on the starting date, then every stock on the next day, then every stock on the next day etc. This means that portfolio level information is available to use in your trading rules... and is how you actually trade in the real world. When you are doing real trading, you work accross all stocks on the current day, and can make decisions on that day based on the portfolio you actually have already got.

As an example, imagine you wanted to buy a stock when some condition was met, but only if it didn't increase the risk in your portfolio. Or perhaps you want to do position sizing based on the risk that adding the position would create in your portfolio (effectively the "correct" way in finance). Or to take a position in one stock, you needed to balance the risk in your existing portfolio by changing position sizing in another stock (also the "correct" way in finance).

To the best of my understanding, Wealth-lab does not offer true "date-sequenced" testing.

However, I am happy to be corrected, as I am still using WL4, and have not changed to WL5. From reading previous posts and requests though, it appears that WL5 still uses the same approach as WL4 to run rules and create portfolios. WL4 does certainly NOT use date-sequenced testing.

Cheers,
Bruce

QUOTE:
To the best of my understanding, Wealth-lab does not offer true "date-sequenced" testing.

But can do essentially the same thing with its engine.

Portfolio level information is available to use in your trading rules. Here's just two examples one of which includes "but only if it didn't increase the risk in your portfolio":

Interacting Dynamically with Portfolio Level Equity
Multi-System Trading II (ActiveTrader Mag TSL 03-2009)

Hi Eugene,

I am familar with these chartscript-based work arounds.

However, in WL4, this does not amount to the same thing as the thread was asking for. The fact that chartscript has to be used to do this is evidence enough that the simulator does not do portfolio based date-sequenced backtesting.

To use portfolio-level rules is not possible in simulations, due to the fact that there is no "portfolio" available when the rules are "simulated".

When you trade in real-time, you work with the all the stocks, one day at a time. You make decisions based on all the information (including portfolio information) which is available at that time. That is what date-sequenced backtesting means. The backtest is created under the same conditions you would trade under.

Note that I am not complaining about what Wealth-Lab does or does not do, and I do not know anything about the product the original poster was referring to. I am simply explaining what date-sequenced backtesting means.

Personally, I do not find any other method than date-sequenced backtesting to be acceptable in a trading product. It is regrettable that the simulator does not natively work this way, although I fully understand why it has been designed the way it has.

Regards,
Bruce

Thank you Bruce, you shed the light on the "date-sequenced" term. I don't know if there's a single product priced in the same ballpark as Wealth-Lab is that can do the same as Wealth-Lab plus absolutely every other thing what people used to request: multi-currency portfolios support, event-driven backtesting, "date-sequenced" approach, multiple systems, walk-forward and genetic optimization etc. Hence the workarounds and approximations (although I do believe the 2 examples above are sufficient approximations). Hopefully, we'll see some innovations in Wealth-Lab 5 next year.

Hi man/men,

one general issue here: I figure you WL people get a lot of user requests and tips and the like. Say that for instance BruceV (no offence, man) above really wants his data-sequenced thing implemented, and perhaps other people really would like this, too. But then there might be some other people that are going on and on about something totally different.

How do you sort out all the requests, and give priority to any of them, avoiding being "penny wise pound foolish" (i.e. "most important stuff first")? One typical issue tends to be, I think, US people that really mostly/only do US products, and "the rest of the world" that tend to do more multi-currency stuff (at least that's my personal opinion after having attended several such conferences).

I know that other software systems run like a user forum gathering once a year or something, this is an opportunity for people that use the systems to get together and give specific feedback, and also listen to what the system developers are doing/thinking. Any maybe user votes/feedback on possible new functionality or something. Is that something you have considered?

QUOTE:
How do you sort out all the requests, and give priority to any of them,

We (MS123) don't sort or answer requests like these for the main Wealth-Lab app; that is done by Fidelity based on business requirements.

In this particular case, you can be sure that the entire architecture of Wealth-Lab 5 won't be changed, so if those that require pure date-sequenced testing can't live with the reasonable approximations, then Wealth-Lab 5 isn't the product for them.

On the other hand, Version 5.6 will have Position Sizer (PosSizer a.k.a. SimuScripts) capability, and this is the "built-in" answer to dealing with equity, cash, and drawdown trading decisions in v5 simulations by providing a method to size or reject (by sizing to 0) raw profit-based trades. There are some drawbacks, and it's not the same as date-sequencing, but it's a quite powerful feature and provides very reasonable results.

Letting the cat out of the bag, a Monte Carlo Visualizer is coming very soon. Those who have never used the legacy Monte Carlo-Lab, will finally discover that it's just not practical to put too much faith into a single "perfect" date-sequenced simulation. Trading is about probabilities.

Hi,

Eugene, the reason that the workarounds are not sufficient approximations is that for a particular date, when the trades are run, you have an existing portfolio, and a set of trades. You can then select from that set to determine which trade(s) best lower the risk of the portfolio.

This is not possible using simulator or chartscript.

With simulator, it isn't possible as the simulation isn't date sequenced.

With chartscript, it isn't possible as when you are scrolling through your datasource and you are half-way through it, you cannot accept or reject a trade on the basis of risk, because you do not know what other trades may come up as the chartscript progresses further down the datasource. ie. If I have 26 entries in my datasource named A, B, C ... Z, then I can't make a decision about stock C increasing or decreasing my risk without knowing the alternatives (perhaps M or X etc).

The "correct" way in finance is to iterate out the possibilities, and then decide which stocks to add/remove from the portfolio based on both the current state of the portfolio, and the potential stocks trigerred by the rules. It is not a decision which can be made on a stock-by-stock basis. It is a decision which can only be made on a day-by-day basis - just like when you are trading in real life.

Again, note that I am not complaining about the way WL works. I understand the reason it is implemented the way it is. However, the method of implementation of WL prevents this style of backtesting.

Cone: "Those who have never used the legacy Monte Carlo-Lab, will finally discover that it's just not practical to put too much faith into a single "perfect" date-sequenced simulation. Trading is about probabilities. "

Agreed, it is all about probabilities. However, this comment applies to the current implementation of the simulator as much as a date-sequenced one. The current implementation also is a one-off snapshot that does not reflect the range of possibilities.

Also, I am not suggesting that WL be changed in some way.

However, I do think users should be aware of what a product does or does not do, and glossing over an important fundamental mechanism such as the way backtesting "works" is not helpful. Hence, my attempt to shine the light on what the original poster was referring to.

Regards,
Bruce

Fully agree with Robert in that the value of a single backtest is slightly exaggerated. At the very least I'm interested to see a 3D graph of its optimizable parameters.

With regard to iterating over the datasource in alphabetical order, in the "equity-interacting" scripts it's possible to dynamically change the way of looping to some other criteria e.g. random order. Considering that "Multi-System Trading II" is all about portfolio volatility/risk at any given date, this prioritization could be closer to what we have in real life.

QUOTE:
This is not possible using simulator or chartscript.

I'd have to agree that removing stocks at the Portfolio decision level isn't possible (or at least not easy in v5), but controlling additions is possible assuming that you enter all the trades up to your buying power limits using Position.Priority. Further, if you didn't want to use up all the buying power, you could limit the risk of new Positions further with a combination of priority and PosSizer (SimuScript) logic, which in effect, takes a look at all positions, candidates, cash, equity, and drawdown on pseudo bar-by-bar basis (i.e., when new trade candidates appear).

That said, I do recognize that there is no "taking back" a trade entry from a raw profit run, and this is discussed here and in the v5 User Guide: Strategy Window > Backtesting Strategies > Inside a Portfolio Simulation, Single-Position Strategies with multiple triggers.

As always, thanks for the helpful comments! :)

Cone/Eugene -Great tete-a-tete here guys(bruce et al), but I am a semi-beginner and will ask you to take it down a notch for 'we mere motals'. I am one that has been self-taught and am using MonteCarlo extensivly. I am not sure what you are saying(per below), but in laymen's terms, can you explain/tell me that UNTIL the next iteration of WL comes out, can I continue to use (and depend on) the existing WL Monte Carlo optimization process to help me narrow down my criteria results to accurately reflect backtesting?

Since I brought up date sequencing, I am curious as to the existing WL5 version -does it have this work around built in? Am I utilizing it to my benefit with it all running seamless in background, or do I need to effect it somehow w/making a deliberate selection?

Also, what is Monte Carlo "Visualizer" and in practical terms, how will it benefit WL users?

Cone -"this single "perfect" date-sequenced simulation" you mention below. This is not what we have now in WL5 correct, is it? I am putting alot of stock in the WL5 prior to auto-trading my system (al beit, on a small scale). If you feel like I should look at other backtesters, I can, but I feel comfortable now w/WL.

Thanks,
Rob

Thank guys

QUOTE:
Monte Carlo optimization

rb, I'd recommend that you stop using the MC Optimizer and start using the Exhaustive method, which gives you visualizations of the optimization metrics and relationships between the parameters.

QUOTE:
what is Monte Carlo "Visualizer"

Here's the description of the legacy product: Monte Carlo-Lab

QUOTE:
If you feel like I should look at other backtesters

That's up to you. I'm telling you that one perfect simulation gets you nowhere. It will never repeat in real life.

Wealth-Lab Portfolio Simulations are accurate except in the case linked in my previous post. They also have the limitation of easily creating trading rules that interact with Portfolio Equity, Cash, and Drawdown, but this can be accomplished using an esoteric approach.

That said, date-sequenced simulations can still be accomplished in Wealth-Lab 5 as well as interaction with results of other symbols' trading. However, it must be programmed manually within the script looping over the symbols using the DataSetSymbols method for each bar of the simulation. If your trading decisions depend on what other symbols in the DataSet are doing on a bar-by-bar basis, then it can be done. Depending on exactly what is required, PosSizers (5.6) may be the answer without more complicated DataSetSymbols programming. These programming questions have to be answered on a case-by-case basis.

ok- will switch to exhaustive -will just need to be more patient....

My script/strategy are not interdependent with other symbols- each are independent of each other, so I should be ok there.

Cone, when do you think 5.6 will be ready?

QUOTE:
ok- will switch to exhaustive -will just need to be more patient....

Patience will pay off. I think we'll have a big optimization-related (never-before-seen-in-Wealth-Lab) surprise coming in the first half of next year.

5.6 is a December-ish release. We can't provide Fidelity release dates.

Very cool....Great to hear -look forward to that.

I guess I am wrestling w/terminology. We agree that the exhaustive method is the way to go. Can you clarify for me then -when I use the word "simulation and optimization" as one in the same(incorrectly?)- I assume they mean two different things?

We agree that Monte Carlo is a random sampling, while exhaustive is a complete(and much more robust analysis? We also agree that in real life trading, anything can and WILL happen-the least being (and more likely), what has never happened in the last 1-3 yrs of backtesting(witness 10/08-3/09).

Cone -you continue to mention trading is all about "probabilities"...is the key then, if I am to taking calculated bets, if I can narrow those probabailities to a small handfull of favorable probabilities, all the better. So if that being the case, as a general rule, the exhastive method will ALWAYS give me a more robust rules -based methodology, correct? Why I don't know - you mention visulaizers, but I don't know what that means, but that's ok. I just thought they were close but Monte Carlo was taking a sampling while exhaustive was complete, but you mention parameters and visulizers... You also mention that "one perfect simulation gets you nowhere". I guess that's also where I get hung up -is doing Monte Carlo -just one simulation? I thougt it was a sampling of many probabilities/choices that allows me to chosice base on criteria I want(IE: APR, or profit), allowing me to pick one and optimize the strategy? If it is, isn't the exhaustive method also one big simulation too????

Boy oh boy -wish I would have listened better in my 2nd yr University stats class....

Simulation = 1 backtest
Optimization = many backtests with different parameters each time

exhaustive = always finds the optimum in the parameter range
monte carlo = may not even come close to finding the optimum

Re: exhaustive visualizations - did you see User Guide: Strategy Window > Optimization > Graphs? You can't get that with MC.

Re: 1 simulation. Did you look at that Monte Carlo-Lab link? Hopefully it will be clearer when you run it and read the Help doc in 5.6.

Sorry for being short, but it's been a long day and I'm going to bed!

p.s.,

'Brevity is the soul of wit' so they say- no problem. Sleep is good Cone!!!!

So I read this -it was helpful. Just completed my first exhaustive opt -took 2 days, but yielded much better results. Ran one on my PC and one on the laptop. I assume the laptop, being slower was a function of RAM? Can I assume if I had a supper fast UNIX, or similar machine, the exhaustive would be much quicker?

Is MC simulation, or optimization? If the later, what is WL's product for simulation? Why would anyone ever want to do simulation(1 backtest -unreliable) when doing exhaustive opt is the way to go?

If I had a data set of just two stocks exposed to same parameters, better to run an exhaustive on each individually, or together? If together, doesn't the exhaustive opt ultimately take the avg. for the set diluting the results?
Thanks,
Rob

QUOTE:
Is MC simulation, or optimization?

You're still hung up on the term "Monte Carlo". It's just a cool way to say "random".

Monte Carlo optimization method
- uses random combinations of parameter values to try to quickly find "an optimum" - but it may not be "the optimum".

Monte Carlo-Lab
- an application that takes the results of 1 simulation (1 set of parameters) and scrambles it in different ways, hundreds or thousands of times.

QUOTE:
better to run an exhaustive on each individually, or together?

There is no "better". You may end up shooting yourself in the foot by "over optimizing".

QUOTE:
doesn't the exhaustive opt ultimately take the avg. for the set diluting the results?

See Preferred Variables.

Got it(for now)- thank you. If any more questions- will thread this over to "optimization" discusion.

If you were going to suggest just one book(maybe two) on optimization strategy involving the markets (that isn't too esoteric -don't need a degree in quantham physics/applied math to follow) what would they be/by whom?

I think I am at the point I need to satisfy my own curiosity further by really getting into it.
Thanks Cone

I can recommend:
The Evaluation and Optimization of Trading Strategies (Wiley Trading)
By Robert Pardo : )

Thanks! I'll pick it up on Amazon today!