Would it is possible to create two intraday data sets (with same set of symbols) that has different start date for the data. One data set would has all the historical data available and the second one would only has the last 3 yrs of data? Thanks.

Data providers typically request all data when backfilling a symbol, and even truncating the beginning of a symbol's history with the Data Tool would propagate to both DataSets. So if the DataSets belong to different data providers, that might be possible.

Question is, what's your objective? You can already accomplish this by simply setting a data range (which can even be saved with a strategy copy: Preferences > Advanced Options).

Thanks Eugene,

I have a strategy that use GetExternalSymbol to read in intraday data from a dataset that is differed from the controlled 'daily' symbol dataset and the intraday data set size is over a thousand symbols and each time the program run, it takes around half an hour just to read the intraday data alone (the data folder is reside in drive C (SSD). This is acceptable when I actually use the full range of the intraday data like in Monte Carlo simulations for system modelling. However, when use it to just for to look at new trades coming at end of day (and every day), it seems rather wasteful of resources and time for it to load all that data when not really needed. SetContext would not help me in this case since my 'clicked' controlled symbol is in a separated daily data set from that of the intraday.

Is on demand data update disabled?

Yes, if on demand data update is on, it will run for hours and I have to kill it.

1. If you can utilize more than one intraday data provider fo that, the workaround in post #2 could be the easiest. For example, keep IQFeed's complete history and have the other provider's data trucated.

2. If you have just one intraday data feed then I'd consider creating two Windows usernames. In this case you could separate them for running the time-consuming backtest task in one and have the other for daily trading tasks. Each copy would have separate WLD data folders. Both Wealth-Labs could run simultaneously, switching between accounts is done by Win-L shortcut.

I'd backfill the data once in the 1st account, copy it into the 2nd account's folder under AppData/Roaming, switch to the 2nd account and then run the Data Tool there to truncate the beginning of the data until the last 3 years.

Thank you Eugene. It seems like a neat solution. I only has one intraday data provider and will give the 2nd solution as you suggested a try. Thanks.

Glad I could be of assistance. Here's the Data Tool's dropdown box which should be switched from "After" to "Before". Then click on the date in the past to activate the checkbox:

Thank you. When I attempt to update data from IQFeed, WL got hanged. I am contacting DTN to see if it is from their end. It might be that they might not allow 2nd user account with a single license (?).

I think this is the case but want to confirm: both accounts would be able to access the same folder C:\Program Files\MS123\Wealth-Lab Developer 6?

QUOTE:
I think this is the case but want to confirm: both accounts would be able to access the same folder C:\Program Files\MS123\Wealth-Lab Developer 6?

Of course, both Windows usernames will run WLD from that same folder. That's not the issue. It does not contain user-edited nor data files. See the User Guide > Data > Where data are stored?.

QUOTE:
When I attempt to update data from IQFeed, WL got hanged. I am contacting DTN to see if it is from their end. It might be that they might not allow 2nd user account with a single license (?).

Sounds like simultaneous IQFeed updates in multiple Windows accounts can be problematic.

Assuming the 1st account is for long-running backtests and the 2nd is for trading, you could run the IQFeed Client in the former to maintain the complete history, copy over the data to the 2nd account, and truncate it every time. Clumsy but might work.

I checked and that updating data on 1st account from IQFeed still okay just the 2nd account is not working. Truncating daily data done in the matter of couple to few minutes but truncating intraday data has already running for about an hour and it still not done. Not sure if it got hanged or still running.

You can tell if it's running or has stuck by looking at the total file size in the data folder in e.g. Explorer.

If you break one big intraday DataSet into several ones, that should improve the Data Tool's responsibility. Perhaps I should've added a progress bar to the operation.

Yes, available disk storage size is increasing. It is still running and taking a while. Thanks.

What intraday interval are you using? second? ticks?

In any case, I think I see a way to speed up the load from cache for specified data ranges. We'll get that in the next update for IQFeed.

Thanks Cone. I think that would be great anytime we can get thing run faster and more efficient :).

The truncation process for the intraday data is completed and I think it took about few hours. I am using 5 minutes interval. The total size of the intraday data is around 38GB. With regard to IQFeed, it seems that I can update data on the 2nd account, given that I have to close out WLD on account 1 first. Otherwise WLD will hanged and the only way out is to kill the process. IQFeed's error message indicates it doesn't allow multiple logins. Anyhow, this seems like a viable solution to save time and resources when I run EOD simulations. Thanks.

Glad to see it works for you.

In v2020.10 of the Data Tool I will add a progress bar for all operatons that can be time consuming (data truncation, report generation etc.)

@Eugene, sounds good!

@tomphm,

QUOTE:
IQFeed's error message indicates it doesn't allow multiple logins. Anyhow, this seems like a viable solution to save time and resources when I run EOD simulations.

Disable File > Update Data On Demand,
1. Your simulations will go 100x faster
2. Wealth-Lab will not bother to connect to the IQFeed client.

I'm still working on the the next IQFeed Provider build which will also help loading shorter histories, but I'm trying to get Streaming Bars working for the Strategy Monitor first.. almost done.

@Cone, that would help with a faster IQFeed provider - thanks. My Update Data on Demand is OFF, otherwise it will go on for a long time. The challenge I am having here is that GetExternalSymbol loading data quite slow. It takes around 1 second to load a single intraday symbol. Even after truncate the intraday data range to only include the most recent 1 year in length, the load time using GetExternalSymbol does not change much. GetAllDataForSymbol speed is about the same as that of GetExternalSymbol. It would be nice if can get GetExternalSymbol run faster.

Once you truncated the intraday data to 1 year and disabled on demand data update, GetExternalSymbol can be made faster by:

1. disabling the Market Manager for IQFeed (i.e. the intraday data provider for external data requests)
2. calling GetExternalSymbol with dataSetName as parameter (see QuickRef).

Thanks Eugene. I will give it a try.

Wow, @Eugene, I don't know what you or @Cone did recently, the loading speed of GetExternalSymbol has increased by almost an order of magnitude. Previously loading an intraday symbol at 5 minute resolution at full data range takes about 1 second. Now, it takes approximately 150 msec. Awesome. Big thanks.

With regard to post #20,

1. Disable/enable Market manager seems does not change the speed much,
2. This option is the one I have been using.

So this must be the effect of Robert's great update (v2020.09.11) of IQFeed provider. Impressive!

@tomphm

Just FYI. Data Tool v2020.10 is released with the improvement.

Thank you Eugene.