Wealth-Data: Error loading Provider: Collection was modified

DartboardTrader

#1

9/19/2016 12:38 PM

When running two strategies using multi-symbol datasets in WealthLab Pro 6.9.15, I always see a dialog:

"ZZWealthDataStaticProvider / Error loading Provider: Collection was modified; enumeration operation may not execute. [OK]"

Neither strategy is using a WealthData dataset directly. This is a blocking dialog, since one of the strategies will be stuck in "Executing strategy" unitl the OK is hit.

Workaround: Remove the WealthData extension.

Eugene

#2

9/19/2016 12:48 PM

I cannot reproduce this error even running two MSBs simultaneously (which is a tricky thing to do unless it's a slowly executing strategy) on a Wealth-Data DataSet (and Yahoo too). I guess it's either strategy-specific or data specific e.g. has to do with lookup of external symbols. We'd like to see more evidence to be able to duplicate and troubleshoot the issue.

Cone

#3

9/19/2016 2:50 PM

Please give us more information about the environment..
1. On which Provider's DataSets are you running the MSBs?
2. Do the strategies access external symbols?
3. Does the strategy do anything esoteric, like create a DataSourceManager?
4. Is Preferences > Backtest Settings > Benchmark Buy & Hold enabled? What symbol?
5. Anything else?

DartboardTrader

#4

9/23/2016 1:02 PM

The dialog appears typically once (twice if you hit OK soon enough) during the "Collecting Data.. {SymbolName}" phase of the backtest on one of the strategies.
Typically the other strategy is in the "Executing Strategy" phase.

1. The two separate MultiSymbol Backtest Strategies/scripts use the same Fidelity Daily Data Set of 102 symbols.
2. In one strategy, it uses GetExternalSeries this way:
DataSeries benchClose = GetExternalSeries("QQQ",Close);
... a couple lines lower...
DataSeries LC = Log_Series(GetExternalSeries("QQQ",Close));
... trading loop ...
this.SetTimeOfDayPriority(1);
In the other strategy, in the reverse order.
DataSeries LC = Log_Series(GetExternalSeries("QQQ",Close));
... a couple lines lower...
DataSeries benchClose = GetExternalSeries("QQQ",Close);
... trading loop ...
this.SetTimeOfDayPriority(1);
In both cases, both GetExternalSeries calls are outside the bars trading loop.

3. Both strategies use "this.SetTimeOfDayPriority(1);" at the end of their bars trading loops, as shown above.
I do have Fidelity 1 minute data sets of the same symbols.
4. Benchmark Buy & Hold has been unchecked off for ages, because of that other Benchmark B&H issue. The benchmark symbol is "SSO".
5. Let both strategies run against a single symbol in the dataset. Let one run against AAPL, one against AAL. Then, run both strategies on the same DataSet, In either single-symbol or multi-symbol runs, both strategies run in Raw Profit / Fixed position size. Both are set to use "Scale:" Daily ; Daily "Data Range:" 3 years.
Both strategies use the tabs:
DataSet, Strategy Summary, Editor, Alerts, Performance, By Symbol, Trades, Equity Curve, Drawdown, Profit Distribution, Performance+, Trades+, and WealthSignals.

Size:

Color:

The dialog appears typically once (twice if you hit OK soon enough) during the "Collecting Data.. {SymbolName}" phase of the backtest on one of the strategies.
Typically the other strategy is in the "Executing Strategy" phase.

1. The two separate MultiSymbol Backtest Strategies/scripts use the same Fidelity Daily Data Set of 102 symbols.
2.  In one strategy, it uses GetExternalSeries this way:
			DataSeries benchClose = GetExternalSeries("QQQ",Close);
                        ... a couple lines lower...
			DataSeries LC = Log_Series(GetExternalSeries("QQQ",Close));
                        ... trading loop ...
                        this.SetTimeOfDayPriority(1);
In the other strategy, in the reverse order.
			DataSeries LC = Log_Series(GetExternalSeries("QQQ",Close));
                        ... a couple lines lower...
			DataSeries benchClose = GetExternalSeries("QQQ",Close);
                        ... trading loop ...
                        this.SetTimeOfDayPriority(1);
In both cases, both GetExternalSeries calls are outside the bars trading loop.

3. Both strategies use "this.SetTimeOfDayPriority(1);"  at the end of their bars trading loops, as shown above.
I do have Fidelity 1 minute data sets of the same symbols.
4. Benchmark Buy & Hold has been unchecked off for ages, because of that other Benchmark B&H issue. The benchmark symbol is "SSO".
5. Let both strategies run against a single symbol in the dataset. Let one run against AAPL, one against AAL. Then, run both strategies on the same DataSet,  In either single-symbol or multi-symbol runs, both strategies run in Raw Profit / Fixed position size. Both are set to use "Scale:" Daily ; Daily "Data Range:" 3 years.
Both strategies use the tabs:
 DataSet, Strategy Summary, Editor, Alerts, Performance, By Symbol, Trades, Equity Curve, Drawdown, Profit Distribution, Performance+, Trades+, and WealthSignals.

DartboardTrader

#5

9/23/2016 1:29 PM

Created two test scripts that reproduce this issue when run simultaneously. Created support ticket #27446.

Eugene

#6

9/23/2016 3:35 PM

Thanks Michael. I confirm that I'm able to reproduce this behavior by simply including this.SetTimeOfDayPriority(1); in any two scripts which are being run simultaneously. My guess is that it has to do with SetTimeOfDayPriority depending on aux methods that in their turn rely on reflecting (System.Reflection) some undocumented internal methods. Something may be not thread safe (for example) about the ZZWealthDataStaticProvider so when it's hit, exception is thrown.

Do you really need to run those two backtests simultaneously?

Cone

#7

9/23/2016 5:10 PM

SetTimeOfDayPriority accesses/reads the binary .WL files to identify the time that a trade occurred. The method doesn't do anything fancy to control simultaneous access to files, and generally speaking, Wealth-Lab data providers do not do that either. I can also duplicate the error by opening a Workspaces with several strategies that operate on the same DataSet.

There's room for improvement, but there's no simple work-around for the moment... stand by!

Eugene

#8

9/23/2016 5:29 PM

I nailed down the bug. It turned out to be a thread safety issue in the Wealth-Data Provider as I suspected. We're going to test my fix thoroughly before committing it to production as it may have adverse effects.