I'm using a script to process stock symbols from an ascii file in which I apply fundamental, price and volume critieria before determining whether the stock is a candidate. I'm using "FundamentalsRatio.GrowthRateSeries" to compute the fundamentals etc.

My question is, can I perform critieria check on the fly (while processing the asci file) with "On Demand Updates" turned on, without having to first collect all the symbols from the ascii file, create a dataset then request the dataset be updated ?

It seems to work without having to create a dataset and update all symbols, but wanted to confirm with you to be sure.

thx

It's not clear to me the process (or processes) that you're using to "create a DataSet and then request the DataSet be updated" (or even why you have to do that if you're using ASCII data), but, fundamental data is collected/updated only via updates using the Data Manager - never during on-demand updates.

Can I write a script that automatically updates dataset every 30 min, for example?

I am talking about WL.net script.

Frankly, I haven't tried scripting that but you can experiment with undocumented interface calls on your own.

However, why not just put any Strategy in the Strategy Monitor to be run on the DataSet (applies to 10 minute or higher scales), provided you have select the 30 minute scale? This way, requested static data is stored on the disk.

Update:

Yes, there is a way to update DataSet(s) programmatically. However, a Strategy can't schedule events since "WL.net script" executes in real time only once when a new bar arrives. I could envision an external tool cycling through available DataSets periodically, but what's the point of having this possibility in a Strategy?

Question about "Update Data on Demand" option.

The User Guide states:

QUOTE:
Automatically update data for symbols on-demand when they are charted or accessed.

QUOTE:
Streaming charts ignore the "on demand" selection, and, in some cases will cause static data to be updated with the historically-requested bars.

This option automatically updates data for symbols when they are charted but *only if the streaming disabled*?
What if GetExternalSymbol is called from the strategy running in streaming chart? Does it update the data on-demand or not?
How about the symbols set on the activated strategy in Strategy Monitor? Does it update the data on-demand or not?

QUOTE:
How about the symbols set on the activated strategy in Strategy Monitor? Does it update the data on-demand or not?

Data updates by SM are documented in the User Guide > Strategy Monitor > Behavior as a function of Scale.

QUOTE:
· If the symbol is not in the local data store, this will result in a long delay. These data are updated on demand and saved locally.

Does the "saved locally" mean the symbol data saved locally to WLP process memory not in the disk? On-deman option doesn't alter the behavior?

QUOTE:
· Thereafter, data from new interval updates are not saved (eliminates disk access delay for saving multi-megabyte files).

I think this confirms above that the symbol data isn't saved to the disk.

QUOTE:
Streaming charts ignore the "on demand" selection, and, in some cases will cause static data to be updated with the historically-requested bars.

OK. I think it means streaming charts may save the symbol data to the disk regardless of the on-deman option, thus streaming charts ignore the on-deman option.

So what does "Update Data on Demand" option actually do?

Does it control whether or not saving the symbols data to the disk when the symbols are download by "Non-streaming Charts" therefore it may degrade the backtesting performance?

If the on-demand option controls "Non-streaming Charts" only, it can explain my observation at #35.

But why GetExternalSymbol with dataSetName parameter doesn't download the symbol data?

QUOTE:
Does the "saved locally" mean the symbol data saved locally to WLP process memory not in the disk?

No, it means the data is saved locally to the BarDataStore (e.g. inside FidelityStaticProvider folders).

QUOTE:
· Thereafter, data from new interval updates are not saved (eliminates disk access delay for saving multi-megabyte files).

Then this statement conflicts, doesn't it? Because this one says the updates are not saved.

Why? If a symbol from your DataSet hasn't been cached yet then WL will backfill the historical data and save it locally i.e. to the local data store (what we call the BarDataStore). The updates to it (the .WL file) are not saved, right - that's what the UG states.

There are (or used to be) different behaviors for Fidelity data.

1. Streaming charts never update data files, only the chart instance.
2. Strategy Monitor:
a. For 10-Minute bars and lower, bar updates in the Strategy Monitor are not saved to disk.
b. For higher than 10-Minute bars, updates are saved (generally these files are much more manageable in size).

That said, I'll have to check it now. When the Strategy Monitor was first designed it did not use "Streaming Bars". Since it does that now, it's possible that no files are updated by the Strategy Monitor.

I tested some scenarios and here is the results.

Streaming Chart / 1 Minute Scale / symbols in DataSets are updated and saved to disk: NO
Streaming Chart / 1 Minute Scale / symbol by GetExternalSymbol() is updated and saved to disk: YES
Streaming Chart / 1 Minute Scale / symbol by GetExternalSymbol(dataSetName) is updated and saved to disk: NO

Strategy Monitor / 1 Minute Scale / symbols in DataSets are updated and saved to disk: NO
Strategy Monitor / 1 Minute Scale / symbol by GetExternalSymbol() is updated and saved to disk: YES
Strategy Monitor / 1 Minute Scale / symbol by GetExternalSymbol(dataSetName) is updated and saved to disk: NO

Strategy Monitor / Daily Scale / symbols in DataSets are updated and saved to disk: YES

I verified the "Update Data on Demand" option and it has nothing to do with the results in these scenarios.

QUOTE:
Streaming charts never update data files, only the chart instance.

Chart instance is a chart window with the streaming disabled on which you run the backtest? I guess that's where "Update Data on Demand" option makes the difference, hence it affects the backtest performance?

As suspected, when using a Streaming [Bars] provider like Fidelity's, the Strategy Monitor does not save history to disk for any scale.

Re: Streaming charts never update data files with streaming data, only the chart instance.
i.e., you only see updates in the chart data when streaming; nowhere else.