QUOTE:
the relationship between optimizing and the effect of the PosSizer.

What do you mean?

(The question was asked in the topic, "ActiveTrader 2013-06 | Trendline Bounces fails in MSB mode")

This is an observation and a caution to strategy developers, not a complaint of any kind. The point is that under certain conditions, exhaustive optimization yields different results, run to run. Not only is maximum gain different (sort of expected), but the "best" parameters can be different, too.

Preconditions:
1. The strategy being optimized does not set LastPosition.Priority
2. The "Percent Of Equity" PosSizer is selected (or any other where not all signals result in trades)
3. Trades are bypassed due to insufficient capital

I ran an optimizer experiment, using published strategy, "ActiveTrader 2013-06 | Trendline Bounces" and the Exhaustive optimization method. My portfolio had only 3 symbols (because Exhaustive needed 2,160 runs. I used the "Percent Of Equity" PosSizer with a position size of 50%, to force signals to be excluded due to insufficient capital. I ran six tests, changing nothing between optimization runs, to see the effect of different signals being excluded.

Notice that "best" value for the 2nd parameter is not consistent across the runs, with 10, 20, 40, and 50 coming up as "best".

Dollar Range {"Best" Parameters}
#1-($33951.66) to $350165.91 {10,10,5,30}
#2-($33951.66) to $360917.11 {10,50,5,30}
#3-($33951.66) to $317181.70 {10,20,5,30}
#4-($33951.66) to $316198.23 {10,10,5,30}
#5-($33951.66) to $372288.72 {10,40,5,30}
#6-($33951.66) to $320216.31 {10,20,5,30}

This is another name Position.Priority.

Portfolio Simulation backtests won't yield the same results when you allow them to choose randomly between the trades they take during periods of high exposure.

You can easily remove the random nature of backtest by assigning priority to the positions. For more discussion, see the User Guide:
Strategy Window > Backtesting Strategies > How Trades Are Chosen, and,
Strategy Window > Optimization > Full Optimization > Optimization Control > Begin Optimization (see Note)

Thanks for the info.

I found the note to which you referred , "Portfolio Simulation sizing yields a true portfolio simulation, including dropped trades due to insufficient capital. To achieve repeatable optimization results, assign Position.Priority in your script, or enable 'Use Worst Trades in Portfolio Simulation' in Preferences (F12) > Backtest Settings."

For me, this just highlights the importance of setting priority, when you can, to choose the best trades in times of high volatility. It's sometimes difficult to insulate a strategy from the effect of the overall market, where virtually all instruments break through bands, for instance, on the same day.

QUOTE:
It's sometimes difficult to insulate a strategy from the effect of the overall market, where virtually all instruments break through bands, for instance, on the same day.

We have an answer for limit/stop priority too. Please see: SetTimeOfDayPriority. The solution requires intraday data (5 or 10 minute bars are usually sufficient). And, another advantage of using this method is that debug output will show you where the "spikes" are in your Daily data, i.e., the impossible trades. You can then use the intraday range to correct the bar(s).

Thank you for the update. I'm not sure SetTimeOfDayPriority applies to my case, since I have a source for Priority. Let me explain.

My use of priority is to identify higher quality trades, like the example in the User Guide which uses inverse of RSI as priority. I use a neural network in nearly all of my strategies. The neural network score correlates to the predicted forward gain, higher is better. I set Priority to the neural network score and echo it in the Entry Name. For my live trades, when not trading all signals, I place orders in descending order of neural network score.

Reference: From the User Guide topic, "How To Backtest And Trade With Priority"

QUOTE:
To create repeatable backtests, even when simulated buying power is insufficient for all trades, assign Priority to each Position. For example, if there is too little cash chasing many buying opportunities, it's customary to choose the most oversold stocks. In the example, since a low RSI value indicates an oversold condition, a higher priority is assigned by negating its value. The trick to actually trading with priorities is to pass the priority value as the signalName parameter in the entry signal. When trading, sort the trade Alerts by the Signal Name column to choose the orders with the highest Priority.

Can you comment on this?

If you use market orders, then you can set priority by any method that will create a number..

For an EOD system that uses limit orders, you must use time-of-day priority for the realistic/correct result, assuming the goal is to simulate what would have occurred.

I think I'll stick with my NN score, even with limit orders.

There's a big design flaw in WL multi-symbol backtesting that keeps it from being "realistic", at least for strategies that use the very common "if (IsLastPositionActive)" statement.

That "IsLastPositionActive" statement makes the assumption that, because a trade could be made (on price/volume), it has been made. So, the strategy switches to "holding a position" mode. A "long side" strategy will bypass running its "buy" logic, looking to sell the position (that was not chosen by the PosSizer). Until that fictitious position is closed by the strategy, valid opportunities to buy are missed, were cash made available from trades in other instruments. This could have substantial effect for long term strategies.

This stems from that fact that MSB design runs all bars for a symbol, rather than all symbols for a bar. I suspect it would take a major redesign to change that.

A realistic MSB would require running all symbols for just one bar, then running the PosSizer for that bar, bar by bar through the backtest period. With that design, "IsLastPositionActive" would be after PosSizer has run.

QUOTE:
I think I'll stick with my NN score, even with limit orders.

It's up to you but SetTimeOfDayPriority perfectly applies to your EOD limit order case, without which the resulting simulation, driven by the timeline-agnostic Priority assignment, will not be as correct as you'd think it is.

Re: LastPosition logic
That's an astute observation, but I don't agree about its extent. We discussed this years ago (on the Wealth-Lab 2, 3, and 4 forums), and in fact, there's still an old KB article that discusses it here. (Therein, "ChartsScripts" are "Strategies".) A long story short, if this is a 'big' problem for your strategy, then you can rewrite the strategy to process the symbols / trades as you describe. That design pattern is essentially what is used in the "Symbol Rotation" scripts.

QUOTE:
I think I'll stick with my NN score, even with limit orders.

I'll agree with Eugene here. That is a big design flaw If you're going for realism. It makes no sense for limit orders to use any other priority technique than [inverse] the time-of-day.

QUOTE:
It's up to you but SetTimeOfDayPriority perfectly applies to your EOD limit order case

QUOTE:
It makes no sense for limit orders to use any other priority technique than [inverse] the time-of-day.

We may have to agree to disagree on this one.

Having read the referred articles, "[inverse] the time-of-day" is not at all how I trade, because it makes the assumption that all signals become orders (Buying power is limitless). My real money end of day process is to look at the day's signals and my available cash, then place orders top down, in priority order, to the extent I can afford them. I'm ok with the fact that occasionally I'll miss an opportunity because the higher priority signal couldn't be met, while the next in line would have executed. It just leaves me in cash. I definitely wouldn't place ten limit orders when I could only afford three, hoping I could cancel the other seven before they create a large margin. That... makes no sense.

For the way you trade, then sure it works, you're right.

But there's no reason to miss opportunities because you're afraid of hitting a cash/margin limit. You can use the threshold feature in Wealth-Lab to prevent sending orders in excess of a cash or buying power limit. Also, you can let your broker reject new orders due to buying power constraints.

It's possible however, for the threshold feature not to work if too many orders are sent precisely (within 1 second) at the same time, which could happen, but isn't likely when using the Quotes tool to trigger/place orders. And, imho, if it happens in a margin account, it's no big deal. If you didn't catch it during the day and got a margin call at night, you just answer the call by selling back positions the next day.

Thanks for the feedback.

1. "threshold feature" is kind of off-topic, doesn't apply to backtesting/PosSizers
2. Not all of my accounts (IRAs) allow margin
3. imho, MSB as designed is a problem, as noted in #7, which contributes to convoluted workarounds. Many years ago, I learned the saying, "Get the data design right, and the programs write themselves." The same applies to applications. And the corollary is, "Get it wrong and you'll struggle for the life of the system."

Consider this high-level design for a (radically different concept) "Bar-level PosSizer". (Some of what follows is guesswork about the internals of WL and the capabilities of C#)

1. User initiates MSB backtest - "Bar-level PosSizer" is launched
2. The PosSizer compiles the strategy with overloaded(?) versions of all trade routines, BuyAtMarket, BuyAtLimit, etc.
3. PosSizer launches a version (child task?) of the strategy for each symbol of the portfolio
4. Child task runs until it calls a (overloaded) trade routine, now intercepted by the PosSizer (Some child tasks will end without making this call - no trades for the symbol) Child task is stalled.
5. PosSizer waits until all child tasks are stalled at a trade routine or have ended
a. Resulting signals are sorted by bar timestamp followed by priority.
b. PosSizer considers signals in order
c. For signals sharing the lowest timestamp, calls the corresponding base class trade routine (price/volume check)
d. if ok, runs PosSizer rules, else marks Position "not filled" and returns false to the strategy (and the strategy resumes)
e. Based on PosSizer rules, updates the Position status and quantity and returns true/false to the strategy (and the strategy resumes)
f. On a change in timestamp, PosSizer must again wait until all strategy subtasks call a trade routine or end
6. Repeat 5. until all child tasks have ended
7. Backtest is complete - run Visualizers, etc.

Len

When you develop this new perfect backtesting engine, don't forget the minor details of generating Equity and Cash curves so the the sizing mechanism has something to work with. Synchronization is not impossible, but it won't be easy.

Ah, ever the cynic. I love you guys. I don't claim perfect, but definitely better.

It seems that Equity and Cash curves could be (would need to be) maintained within the main loop described above and agree they seem to be minor. I expect synchronization would be the major issue.

Maybe akuzn could write it. It's likely beyond my current C# skills and knowledge of WL internal data structures.

Strategies that have excessive amounts of trades not taken because of "insufficient simulated capital" are amusing. When SetTimeOfDayPriority is not used, the strategy results often show better entries and exits than real life (in my experience). The accumulated excess return may compound favorably, and strategies relying on the number or "quality" of the ActivePositions end up observing a past and present trade history that is unrealistic.

Consider using the rotation strategy.. Use your NN to reduce the amount of potential trade signals on a given bar/day after evaluating all symbols, and use SetTimeOfDay priority for the Position.Priority value. Aim for an insignificant number of 'insufficient simulated capital' trades, and you can be sure the strategy backtest is realistic.

@LenMoz
Let's be realistic. Wealth-Lab isn't going to be completely re-designed to solve a minor and very marginal backtesting issue at the expense and risk of opening an entire flood of new and real problems. For the record, let's list the conditions that have to occur for this "big" issue to manifest:

1. The Strategy uses has single-position logic, i.e., holds only one position at a time, AND,
2. a trade must be rejected by some cause external to the Strategy rules (i.e., PosSizer rejects a trade due to high system exposure), AND,
3. another new entry signal must occur prior to the condition that would have exited the trade in 2 above had it not been rejected., AND
4. [in most cases] the strategy will have had to exit a different Position to free buying power prior to the new entry signal in #3.

That's a lot of ANDs. But if all these conditions are true, a Portfolio Simulation will miss the trade in #3. It hasn't been a real problem for any of the Strategies I've written and traded in the last 14 years, so it's difficult for me to believe that it's a statistically significant situation for 99% of the Strategies out there.

No TA platform will do everything that everyone wants exactly the way they want to do it, but if it must be your way, then you need to write it yourself. Cynical or not, that was my point.

Points taken. Thanks Cone and Dartboard Trader.

I'll revisit rotation strategies. My concern is whether neural networks aren't too computationally intensive. Mine typically have 10 or more DataSeries inputs, with moving averages up to 200 bars. Other DataSeries besides the neural network are also used (peaks, troughs, bands). If these are rebuilt with every GetExternalSymbol at every bar, tests might take a while, which is why I never even tried it. My S&P 500 strategy backtests already run 2-3 minutes. (dual core i3-2310m at 2.10 gHz).

Len

Bars and indicators (Series method) are cached, so references to those statements won't slow down execution.

Just to be clear, I'm not suggesting that you change your strategy to "rotation", but rather to use the "DataSetSymbols" pattern in the trading loop.

However, this alone won't solve the issue unless the strategy knows how many Positions that it can hold on a given bar. In other words, you're somewhat limited to using % of Equity sizing (i.e., 10% = 10 Positions) for this technique to work.

QUOTE:
Just to be clear, I'm not suggesting that you change your strategy to "rotation", but rather to use the "DataSetSymbols" pattern in the trading loop.

I'm not sure I understand that. To get the effect I'm looking for, all symbols need to be proceeding along the timeline together.
Example using 50% allocation per trade for simplicity (NN score in brackets):
Bar 200: Symbols A[44], B[53], C[49] issue Buy signals - Results in execution for B and C, but not A
Bar 215: Symbol B issues Sell Signal - frees up funds
Bar 220: Symbol A[56] would have issued another Buy, but is running its sell logic for (failed) trade at Bar 200(missed opportunity)
Bar 240: Symbol A[51] would have issued another Buy, but is running its sell logic for (failed) trade at Bar 200(missed opportunity)
Bar 250: Symbol A issues Sell signal, is again in synch with the portfolio

QUOTE:
However, this alone won't solve the issue unless the strategy knows how many Positions that it can hold on a given bar. In other words, you're somewhat limited to using % of Equity sizing (i.e., 10% = 10 Positions) for this technique to work.

The admittedly complex design I outlined (#12) solves this by using the PosSizer as the clock, stalling each symbol until all other symbols have attained the bar which PosSizer is currently "trading". The "Did the trade execute?" question is answered with greater accuracy. In C#, is that design even feasible?

QUOTE:
[Dartboard Trader @#15]Strategies that have excessive amounts of trades not taken because of "insufficient simulated capital" are amusing.

I think this is important to any strategy because of clustering of signals due to overall market conditions. I'd have to think that all but the very best strategies experience the phenomenon whereby a significant market move causes a flurry of buys (and sells). Ranking the (excessive) Buys seems important in this case. Further, it's this volatile case where, in live trading, I wouldn't place all the orders without regard to margin impact. Finally, it's after such an event that many symbols will be running sell side logic erroneously.

Side Note: 26 DataSeries are in the Bars Cache after my strategy executes. I earlier said "10 or more."

QUOTE:
I'm not sure I understand that.

The Rotation script always keeps n symbols Active. You have to use a similar design pattern, but (for your example) only allow 2 symbols to be active on any given bar.

For my strategies, I can't think of a way to choose the active symbols. Highest NN score is not the only buy criteria.

If I attempted to write it as a PosSizer launched by WL after a "does-nothing" strategy, I'll ask again, re #12 - "In C#, is that design even feasible?"