Hi

After having written intraday 1 minute data to files with SaveToFile(), how is it possible to create a DataSet to work with the *.WL files like any other DataSets?

Or is the only option to use LoadToFile()? Or is there an option in the Data Manager? Or any other way to add native WL6 files to WL6, e.g. by adding it to some XML file or so (as I assume SaveToFile() would create native WL6 format)?

Thank you

SaveToFile() creates native WL format. It's possible to create a DataSet comprised of these files. Here's a how-to:

1. Find a provider a) capable of reading 1 minute data and b) that relies on Wealth-Lab to maintain its data in the local storage of .WL files. Preferred choice(s): Google, IQFeed. In plain English, your files must "belong" to some provider satisfying these two criteria.

2 - Now that you picked a provider, let's say Google, create a 1-minute DataSet with all your synthetic (?) tickers.

3 - Find the provider's data directory under the user's AppData folder:

"c:\Users\hlh\AppData\Roaming\Fidelity Investments\WealthLabDev\1.0.0.0\Data\GoogleProvider\1 minute"

If the "1 minute" folder hasn't been created before, create the folder manually.

4 - Now, time for some tedious work. Wealth-Lab's local storage follows a set of rules:

* Wealth-Lab expects to find symbols starting with a non-alphabetic character in the root of this folder. If the symbol name contains reserved characters impossible to use in file names (e.g. "+TEST"), you will have to rename the file accordingly (e.g. "%2BTEST.WL").

* Put these synthetic 1-min data files in a new folder: "A" for all tickers starting with A (A, AA, AAPL...), "B" for all symbols starting with B (BA, BRCM...). You get the picture.

If your symbol names are synthetic (e.g. "AAAA-1", then updating the pretend DataSet manually won't create the subfolders; otherwise if the symbols are real and the provider can find and update them ("AA"), Wealth-Lab will save you from the need to create the A,B... folders.

If you've done everything correctly, it will work out.

Update 7/26/2013

An even easier alternative:

1. Export to the legacy WL4 file format: Data | Exporting data out of WL6 to ASCII and binary files
2. Import with the built-in WL4Files provider. Voila.

UPDATE 04/14/2020: the WL4Files provider has been removed from WL installation packages so it may not be available. Your mileage may vary.

P.S. One more thing. Creating this pretend DataSet programmatically is also possible but will not be a time saver.

Very interesting, thank you Eugene!

Some more questions coming up:

A) If I would use existing (Google) symbols and would want to update my "regular" provider's data (Google Finance ticked) wouldn't that conflict with my generated ones? E.g. using AAPL.WL for my generated data.

B) So is it better then to use non existing symbols, like make AAPL AAPL_01min. But wouldn't that mean that a Google update would try to get all the xy_01min symbols (imagine 500+ stockdata time two for 1min and 5 min would request 1000+ non exiting symbols), and/or isn't there something which eliminates non-exiting data from online updates (then it might stop requesting them after a couple of attempts?).

C) So using SaveToFile() I then would direct it (change the target directory name) to dir and subdir according to beginning letter. Good to know it requires that subdir structure (and that it does not work like e.g. accessing ASCII data in one particular folder)

D) Is there a real advantage for me to save my synth data in WL native and not in ASCII form (I guess the size would be smaller in WL, but how about in terms of speed)? ASCII would allow me one folder and use the common symbols. And no concerns with updata issues (if any of my concerns above are real). And no mixed artifical data with downloaded historical one.

E) One last thing: As I prefer bars with a time-stamp at the beginning, e.g. 9:30 being the first RTH bar of a 5 minute interval and not 9:35 which WL uses for the bar from 9:30 to before 9:35, I did in the past (WL4) always work with "my bar time settings" on ASCII data, but I guess I should use WLs "ending time stamp" to avoid conflicts, right? But the only problem which my time stamp would cause is for outside RTH data at midnight. For getting High/Low of on a daily scale on RTH data, WL would still use all 78 candles (on 5 min RTH data), even if one is too early (9:30) and one is missing (16:00 as this would be my 15:55) - in other words, WL just grabs every bar with one bar date during set scale daily and returns daily High/Low from all that data, right?

Thanks!

F) One addition to the things above: A "WL6 Native Provider" would be a great option, meaning one which than can easily have no online updates (if this would generate conflicts) and might store all data in one singel directory (like ASCII) or not, depending onwhat is preferred.

G) Another thought: Would it be possible to use Aronow's Watchlist Provider for such things? I can access ASCII data as well as WL (Yahoo, Google) with this provider. If it would be possible to link/refer to "just a directory/ies holding those synthetic *.WL files might be an alternative to the suggested solution?

A) Of course it would conflict. By design.

B) Yes. So for purely synthetic symbols, for that reason I'd rather pick a provider that isn't being updated on a regular basis - like Random provider or IQFeed (if you're not using it).

D) No, there is not. The ASCII provider has ASCII Cache feature, speeding things up considerably (equal to binary files). Therefore I'd stick with ASCII for greater simplicity and not having to worry about B)

E) End-of-bar time stamp, that's right. However, you could try setting up a new market in the Market Manager (ASCII DataSets supported by the MM), add that synthetic symbol(s) to the group, and set a new open/close times to filter out unnecessary bars.

G) Not possible I think. WatchList Provider is based on a different idea and doesn't work with the data directly. Overall, ASCII is preferred.

Rebelion asked in a duplicate thread:

QUOTE:
Hello. I have WL file I created through SaveToFile() method in my strategy code (I generated bars by myself, they all have different datetime in 1 minute lag). But how to add this file to data manager? I tried to use WL 4 data provider, but I have Dataset empty (0 bars). But file isn't empty. What's the reason?

When referring you to this thread, I really believed that the presented procedure is complete. WL4 used a different WL file format, so SaveToFile() - and WL6 in general - creates data in a format that is not backward compatible with the WL4Files provider. Therefore what you're doing will not work. Please simply follow the procedure and use the Google (or Yahoo, or other - but not WL4Files) provider, as instructed. Hope this helps.

There exists a possibility to export to the legacy WL4 file format for importing with the WL4Files provider. Check it out under "Exporting as WLD 3/4 native binaries (*.WL)":

Data | Exporting data out of WL6 to ASCII and binary files

Hello Eugene

In post #3 you state

QUOTE:

P.S. One more thing. Creating this pretend DataSet programmatically is also possible but will not be a time saver.

Is there another thread that shows how to do this? I have a large dataset that I comb and segregate. it would be helpful to create a dataset for each subset.

Thanks

Hi Steve,

1 - See traderray's post #10 dated 4/3/2010 7:40 AM in this thread, and below:

How to create/convert GICS database for symbols WL5

However, creating the DataSet manually and only adding/removing the symbols using script, is a bit more easier to code:

Adding/Removing symbols from Data Manager
WL5.5 - Is it possible to manipulate DataSet Contents and change symbols within the datasets programmatically from a strategy?

Note: Changes to DataSets made programmatically will not appear in WL6 until it's restarted. Also, usage of these APIs is unsupported. Should you have followup questions, try asking them in those threads (not here).

Thank you Eugene. This will be helpful