The Data Tool is a handy new utility that integrates itself into Wealth-Lab's Data Manager, making it effortless to perform the following DataSet operations:

* Truncate last N bars of data
* Wipe the entire DataSet data (to help reload the data for an entire DataSet from scratch)
* Remove inactive (dead) symbols
* Remove selected symbols
* Change symbol
* Data validity check
* Clean bad data

Check out the online guide in the Wealth-Lab Wiki:

Data Tool

To install into Wealth-Lab 6.9.23+:

Direct link

Feel free to leave your suggestions w.r.t. this new tool here.

Nice tool,

Indeed I have to say that yesterday you released quite nice things. Thanks.

One suggestion about it. Could it be possible to select only some of the symbols in a data set and remove their data (not the whole dataset)?
I was thinking that it would be nice to remove the symbols from the datasets which have not been updated in the last year (but have some data), so one option could be to delete their data and then remove symbols without data.

Just an idea ;)

Thanks.

It sounds possible (to remove selected symbols data only) but I think there's better, single-step solution: allow "Remove inactive" to exclude such symbols from DataSet - in addition to wiping out the symbols with 0 bars.

My only concern is how to define a dead/inactive symbol without introducing more GUI options. When exactly a stock is no longer "very illiquid" or in a "trading halt" but has become "delisted": month, quarter, year since last trade?

One option could be "remove inactive" adding a date, so that if no updates have been received since that date, the symbol would be deleted. It would be the user the one who decides to delete those, being aware of the consequence.

Obviously is not the most complete solution but it would be probably the easiest for you.

..."without introducing more GUI options".

What's the concern of adding more GUI?

A right click context menu would be handy for deleting and changing symbols.

1. You could sort by last date, right click, "Delete Selected items"
2. For a single selection, "Change Symbol".

Just throwing it out there (but I'd rather you started working on the "Top Secret" Visualizer)

The idea behind not adding more GUI is simplicity where possible.

Thank you both for your suggestions, I'll try incorporating these features when time comes.

QUOTE:
A right click context menu would be handy for deleting and changing symbols.

...and perhaps reloading their data.

OK, let there be more GUI. Following ecorderob's suggestion, the tool will provide a cutoff date input field to allow cleaning DataSets from symbols that stopped trading before the specified date.

Cone's ideas will also be added:

QUOTE:
1. You could sort by last date, right click, "Delete Selected items"
2. For a single selection, "Change Symbol".

Data Tool has been updated to version 2011.11.

Highlights:

* Added a context menu with two options:

1. Change symbol - renames a single highlighted symbol.
2. Remove selected symbols - removes highlighted symbols from a given DataSet without affecting their data.

* Improved usability of remove inactive symbols feature by adding a DateTime control for specifying a cutoff date. Here's how it affects the tool's behavior:

1. When Date filter is inactive (default/old behavior): clears selected DataSet of any existing symbols with 0 bars
2. When a cutoff date was specified and the checkbox is engaged: additionally, removes any symbols that stopped trading before that date.

More details in the Wiki.

Thanks for your suggestions guys!

I have a few cosmetic suggestions...

* When I click on a dataset that has a lot of symbols, it takes a while to respond -- approx 30 seconds for a daily list of S&P 500 symbols. I didn't get an hour glass to show it was busy...

1) Please show hourglass so I know it's doing something;
2) Status bar -- which is shown on Data Sets tab while loading Symbol Details;
3) Optional: cancel button if it can take too long. 2000 symbols (Russell 2000) with 5 min bars may take quite a while.

* Dataset highlight went away when I clicked on the grid. Assuming this is a ListView or TreeView, you may want to set HideSelection = false (why it defaults to true I don't know) so I know what dataset I'm working with.

Overall a very nice tool! These were just some suggestions that I thought would improve the user experience. Nice job!!

Thank you for your suggestions Tim. I'll add these items to my queue.

p.s. I only wonder why it might take so long for you. With a typical 7200rpm HDD, it takes a few seconds for me on large DataSets - just like "Symbol Details" would do because showing DataSet details uses pretty standard API calls and apparently, there's hardly something to optimize there.

In what kind of environment is WLP running?

Eugene,

I have Windows XP (32-bit) with Pentium 4 3.4Ghz with 3G RAM. It took 4 or 5 secs for S&P 100 or NASDAQ 100 daily lists to open the first time. Subsequent times it was fairly instantaneous. The S&P 500 daily was over 20 secs, but fairly quick on subsequent tries.

I think the HDD is 7200rpm.

I didn't expect you to be able to optimize loading the symbols, but showing the status would help a user be patient if it was taking awhile to load them.

Tim,

I will show hourglass (but skip status bar) and fix HideSelection.

Frankly, seeing PC specs like these is what I expected. While optimizing performance is a worthy task and we shall not waste resources, you'll be definitely amazed what a (groundbreaking) difference a modern CPU makes. I'd expect a ten bagger speed increase if you load 64-bit WL6 on an i7 with 6+ Gb RAM compared to that P4 CPU. Nonetheless, loading S&P 500 on something in the ballpark of a 3 year old Core 2 Duo is not an issue either.

Eugene,

I know my computer is a bit old. I got it in 2005, but I've upgraded RAM and HDD since then. For most of the things that I do, this has been fine, but I've been considering getting a new machine. If I get a laptop, I may even consider getting a SSD drive which should make loads from disk screaming fast. ;-)

Btw, Have you noticed a difference between i3, i5 and i7 processors? I hadn't decided on the processor for the new machine, but it would have at least 6G RAM.

My hunch tells you're not after high octane number crunching with P4, otherwise you've already upgraded long ago (of course, I stand corrected if you've outsourced those jobs to CUDA or some other GPGPU flavor). So if I'm right, an i7 might not be cost effective for your tasks. Something like i3-2100K (or maybe even i5-2xxx) will bring quite a bit of joy and will most likely satisfy for another half a decade.

Thanks Eugene, That's good to know.

I'm considering introducing a data check feature here in the long run. Something along the lines of EODDataCheck from Community Components.

On the Truncate button, does it truncate bars for only the selected symbols or all the symbols in the DataSet? You might want to put up a message box stating "# bars from # symbols truncated" as feedback to the user.

"Truncate" always truncates the last N bars from all symbols of a selected DataSet.

Selection was a later addition.

I think that a dialog box would be excessive because the table immediately reflects the updated state of the selected DataSet.

Minor bug: I created a new Yahoo DataSet in Data Manager then updated prices. When I click on Data Tool, I can not see the new DataSet. I have to close Data Manager and reopen it for the Data Tool to reflect the new DataSet.

Very minor issue as I would not expect to do maintenance on a new DataSet right away but I thought I'd mention it when I noticed it. In this case I was just checking the bar count on some of the DataSets.

Btw, I think this is a great addition!

QUOTE:
Minor bug:

It's not a bug. Please read documentation before posting: scroll to "Important!"

Data Tool has been updated to version 2012.01.

Maintenance release; no highlights.

* Fixed: hourglass cursor missing on lengthy operations (requested by thodder)
* Fixed: DataSet highlight goes away when clicking on the grid (thanks thodder)

selecting a symbol, right click, change symbol doesn't open the new symbol screen. 2012.01

Because there is no "new symbol screen", and has never been.

Please re-read the online documentation, "Right-click menu", that says:

QUOTE:
Simply type in the new symbol name and hit "Enter".

In other words, the context menu that pops up after a right-click has an input box that contains the highlighted symbol. Type in a new symbol name, hit Enter, voila.

Data Tool has been updated to version 2012.02.

Change summary:

* New: Truncate whole "days", not just "bars" (to help reload bad intraday data)
* New: Data validity check feature
* Change: regrouped options into several tabs (redesign)
* Change: by request, made possible to install in WL 6.1+

The "Data check" takes the best from Cone's "Bad History Check" and my/fundtimer's "EODDataCheck" strategies, adding a Spike Detection option. For the complete list of detectable data errors please see the online guide in the Wiki.

The idea is to make all data-related operations (that WL6 doesn't already have) available where one expects to find them naturally: in the Data Manager, and without having to resort to a Strategy to validate a DataSet or to use Explorer to delete a bad file, and so on.

I got a chance to test drive the Data Tool today after it became v6.1 compatible. Its a nice tool - but can be made better.

I have the following observations to make about its Truncate feature:
- If you have a few datasets with non-overlapping symbols - and all of them are currently active - then, and *only* then, it will do its job well as deleting the last bar of data is synonymous with removing the last day's data.
- If you have inactive symbols in any dataset, since the Truncate feature is *not* date sensitive, running it on that dataset will delete the last bar of those symbols also with no hope of recovery as when symbols stop trading their data is not available for download any more. Since some users like to keep inactive symbols for backtesting purposes, this is a problem.
- If you use the Truncate last bar (or 'n' bars) feature across multiple datasets and they have some overlapping symbols, then the bar deletion is incremental (additive) e.g. if you delete last bar data from Dow30 and then again from Nasdaq100 datasets, then you have deleted 1+1 = 2 bars of data from the shared symbols (currently CSCO, INTC, MSFT).

All of these limitations can be avoided by adding the ability to Truncate data AFTER a specified date, as opposed to specifying number of bars; that way, inactive symbols' data will be preserved and there will be no deletion of any more bars than is necessary for active-but-overlapping symbols also. Besides, there are other valid reasons for deleting data forward from a date.

I would concurrently note that there are valid reasons for removing data BEFORE a particular date also - but only for individual symbols as opposed to for an entire dataset. Currently, the Data Tool only offers the ability to remove inactive symbols, as opposed to removing data prior to a date for active symbols.

As "The idea is to make all data-related operations...available where one expects to find them naturally..." I would like to suggest that you please add the ability to Truncate data Before/After a user-specified date as well, the former (Truncate Before a date) probably just confined to individual symbols rather than to an entire dataset (for safety reasons) while the latter (Truncate After a date) can be applied to either a symbol or to an entire dataset. This should make the Data Tool more robust and a more comprehensive solution for all data-related needs.

Thank you for your observations. First things first, I'll put the 'Truncate After Date' option on my list for evaluation. (The 'Truncate Before' is too exotic and I won't be considering this.)

A couple of remarks:

QUOTE:
Since some users like to keep inactive symbols for backtesting purposes, this is a problem.

This is a problem only if inactive symbols are mixed with tradeable symbols in a single DataSet. Create a DataSet that completely excludes the symbols which stopped trading and their precious data won't be affected when one massages the actual symbols's data with the Data Tool.

QUOTE:
- If you use the Truncate last bar (or 'n' bars) feature across multiple datasets and they have some overlapping symbols, then the bar deletion is incremental (additive)

Of course. It's up to you to care about that. The Data Tool operates with the raw data in individual DataSets, and there's no way for it to think what's going on 'outside'. Analogy: you can have dozens of Windows shortcuts to a file (read: DataSets), but if the file (read: a .WL file on HDD) was deleted, they all have become obsolete.

To know when a symbol becomes inactive is not an easy task, especially when you are talking of 1000's of symbols. It is further compounded by the fact that even active symbols become temporarily inactive due to a variety of reasons - lack of interest, pending FDA approval (for drug companies), and so on. No one can, or should, be expected to stay on top of every symbol. At least, some of us have a life outside trading.

Data truncation using date is pretty much the accepted norm. You can offer truncation using bars as an additional method but not in lieu thereof.

'Truncate After Date' will be implemented as an option (next release).

Good to hear that!

Data Tool has been updated to version 2012.03.

Change summary:

* New: "Truncate after date" option (requested by Sammy_G)

Data Tool has been updated to version 2012.05. Update to Wealth-Lab 6.3+ to be able to install/update the extension.

It comes with a considerable usability enhancement: reopening the Data Manager or restarting Wealth-Lab is no longer required after making certain changes to DataSets.

I need to delete eroneous pricing data in yahoo for many tickers. I have been here and found the reverse of what I need (truncate, delete data forward):
http://www2.wealth-lab.com/WL5Wiki/DataTool.ashx?HL=data

How do I do the opposite of truncate, delete a couple of years of daily pricing from a date to the start date of that data set?

Also, a logic check for price reported with no volume should be in the requirements list for the next version of Data Tool.

My previous need is not applicable to a data set, but on a few tickers within a data set.

Thanks,

Hank

QUOTE:
How do I do the opposite of truncate, delete a couple of years of daily pricing from a date to the start date of that data set?

This exact option is not considered for the Data Tool, but the closest thing is the "Remove all data" button. It will wipe out the data for the instrument so you could reload it from scratch next time you do a regular DataSet update in the Data Manager.

For Yahoo provider, there's an alternative way: specify the Starting Date for the new DataSet (at the time of creation) so that it won't include the erroneous period.

QUOTE:
Also, a logic check for price reported with no volume should be in the requirements list for the next version of Data Tool.

Thank you for the suggestion. An existing rule in the data validity checker considers a bar erronenous when its Volume < 0. I may add "No volume i.e. Volume = 0" to the list of erroneous bar rules in a future version.

hankt, you may wish to look at this thread: Data Truncation

QUOTE:

How do I do the opposite of truncate, delete a couple of years of daily pricing from a date to the start date of that data set?

This exact option is not considered for the Data Tool, but the closest thing is the "Remove all data" button. It will wipe out the data for the instrument so you could reload it from scratch next time you do a regular DataSet update in the Data Manager.

For Yahoo provider, there's an alternative way: specify the Starting Date for the new DataSet (at the time of creation) so that it won't include the erroneous period.

That won't work as it is a ticker issue, not a data set issue.

QUOTE:

Also, a logic check for price reported with no volume should be in the requirements list for the next version of Data Tool.

Thank you for the suggestion. An existing rule in the data validity checker considers a bar erronenous when its Volume < 0. I may add "No volume i.e. Volume = 0" to the list of erroneous bar rules in a future version.

Yes, not picking up erroneous tick data as volume = 0 would help in most cases. I notice that when tickers/symbols are recycled for IPOs, the old data is nearly always at zero volume.

Eugene,

I just looked at some of the data that I previously purchased from EOD Data - volume is sometimes missing for entire year's worth of data, thus we may want to selectively remove rather than blanket exclude.

Hank,

Missing volume is a nuisance but the OHLC data is there. In a future build, zero volume will be an option of the data check procedure, but selective removing is not on the radar. Zero volume is possible, it's not a definite error in the data.

Eugene,

Logically we can have no new price if zero volume. I understand the need to work around failures in data providers however.

I'm looking at the chart code that Sammy G wrote which enables a truncate from a point in time backward to help me. I haven't gotten it to work on a ticker yet but It looks like it should work.

QUOTE:
Logically we can have no new price if zero volume.

Good point, but should it be a limit up/down day or purely a data glitch and you might be throwing out the child along with the water. There are some guilty feeds (like the upcoming Morningstar data provider has fake bars with zero volume) but with the upcoming ability to detect zero-volume bars in the Data Tool, you'll have the list of bars to delete manually.

What's new in the Data Tool version 2012.06:

* Change: added "No trading (Volume = 0)" rule to the erroneous bar rule list (suggested by hankt)

sedelstein asked:

QUOTE:
In the Data Tool there is a section on "Supported" static data and "Compatible" data in Green and "Incompatible" data in red

I am unclear on Supported vs Unsupported and Compatible vs Incompatible. In the example on the page, GOOG and AAPL are compatible but MSFT is not

I would think that the data for MSFT would be compatible

Compatible: BBFree, CBOE, Fidelity, Finam, Forexite, Google, IQFeed, Market Sentiment, MSN, QuoteMedia, PiTrading, Random, TradingBlox, Yahoo

since the data can easily be gotten from Fidelity. I'm confused as to why it is in red. am trying to understand the sources of the data for my datasets

You're confusing "compatibility" with "highlighting". ;)

Compatible or not can be data providers (most are compatible). Particular stocks do not have anything to do with Data Tool compatibility. Think of the Quotes pane as an easy screener of a DataSet's performance on the last bar. GOOG and AAPL are green on the Wiki screenshot simply because their latest close was up (or unchanged), while MSFT is red because it lost a few cents that day.

Got it

The reason for the confusion is the word "Incompatible" is in red and (so is Microsoft) and the word "Compatible" is in green

What's new in the Data Tool version 2014.05:

* New: added "Truncate before date" option (requested by customers)

What's new in the Data Tool version 2014.11:

* Change: option to use U.S. market holidays when making data check for missing bars

Is the Data Tool still available?

Cannot find it in data manager tab.

Version 6.8.10.0

Thank you.

Sure it's still available and will also get upgraded to v2015.06 shortly to fix a minor issue.

If you can not find it, you either haven't installed the extension (see link in the top post) or are subject to scenario explained in this FAQ: Extension installed in Wealth-Lab doesn't show up after restarting application.

What's new in the Data Tool version 2015.06:

Fix: "Remove inactive symbols" and "Change Symbol" was deleting all symbols from the DataSet if provider does not support DataSet modification

Recently started using Data Tool to remove symbols with too few bars causing an index issue, however when I remove the symbols from the dataset and then rerun the script they still cause an error until I restart WL. Is there a way to refresh the buffered data without restarting WL?

I realize this isn't unique to Data Tool, because I can replace symbols in a dataset by cut and paste under the "Data Sets" tab and have the same problem of scripts using the old buffered symbols.

This can happen if you did not close the Strategy window being backtested on that DataSet before applying changes. Everything works correctly if you close the Strategy window(s) before removing inactive symbols with the Data Tool.

Thanks, I thought I had deleted symbols before and remembered it working, so that would explain why it appeared inconsistent.

Hi Eugene, The "truncate before date" feature hangs on one minute data. I experienced this with a 1m Fidelity dataset containing one symbol (.SPX) - the date was 7/1/2013. Please note this was done on a fresh install of WLP 6.9 and a fresh install of the DataTool extension v2015.06

Please advise. Thanks.

Hi Marc,

There must be a million or more bars of data to proceed so it should be a very time consuming process (if it succeeds). Especially if you're not an SSD user and/or your PC isn't fast enough. I wouldn't use it on huge intraday data files. There's nothing to optimize in the code so if it doesn't work, try it with EOD instead.

Thanks for the reply. The Fidelity .SPX 1m file has about 867K bars - not more than a million. I let the Truncate operation run for well over an hour. The WL file was never modified (based on the modified date in Windows Explorer), WLP was using only 50% of the CPU, and disk activity was nil (except for the occasional Windows OS 8.1 services). Even the page file (which is on a separate partition), never grew - nor did memory. In short, there was no sign of progress.

My goal is a base 1m dataset for the S&P500 index without data gaps. Fidelity's 1m bars for .SPX have gaps (I have a complete list) - and unfortunately Fidelity has made no fixes (yes, I tried them first). Without good 1m data, all higher scales have errors (although some may be "inside" the range of higher scale bars so the data gaps are effectively hidden). The last 1m data gap in Fidelity .SPX is 8/7/2015 - less than a year ago. I intend to run strategy verification for intraday trading on at least one year of data (and waiting until this August is a bit too far away even assuming there will be no more gaps in Fidelity's data from today forward). I am hoping to use Google static 1m data to add missing bars to Fidelity 1m data should my strategy detect any more gaps after 8/7/2015.

Any ideas on what is happening with the apparent Truncate "hang"?

Thanks.

As a workaround, try creating a new Bars object in a .WL file using a WealthScript Strategy. See QuickRef (Bars object > SaveToFile & LoadFromFile) and this thread: How to create a DataSet from SaveToFile() *.WL data.

Got it. Thanks.

What's new in the Data Tool version 2015.11:

* Change: Spike detection (Data validity check) is disabled on Yahoo datasets
* Change: possible to install in WL 6.8+, requires .NET 4.5

Hi Eugene,

Is it possible to add a new feature to be able to check holidays for other markets than US ?
The best would be to have the possibility to select from a list box the market name available in the market manager tool.

Hi Alexandre,

First, some background.

The Data Tool is currently hardcoded to check for holiday dates against the Data\Markets.xml file in Wealth-Lab's main folder under Program Files. This file is Wealth-Lab's central location for market's open/closing times, the time zone it trades in, and its holidays and days that have special hours. Wealth-Lab hadn't been architected with multi-currency or multiple market support in mind so even the current level of internationalization was a later addition (5.4 or 5.6 if memory serves). Being a non-essential component which wasn't included in WLP until some 6.9.x build, the Market Manager operates with its own database of markets stored in the MarketManagerMarkets.xml file under AppData\Roaming.

Now with Market Manager's wider adoption your idea starts to sound interesting to me. If said file could be found and it has valid MarketInfo entries we could import them from the MM. Otherwise the Data Tool would fall back on its default source. Making no promises that it would be implemented but I'll mark your feature request for future investigation.

Thanks a lot Eugene.

Alexandre, I take my words back and have to reject your request from post #61. On second thought, adding this feature would only lead to poor user experience. (Same as with another related request as I explained - maybe not clear enough).

The dates of market holidays change from year to year. Through the years, MS123 (and Fidelity?) invests time and effort into supporting the history of holidays and special hours for U.S. (i.e. what you see in Markets.xml). No such record exists for any other market. Therefore choosing a "Use non-US holidays" option in the Data Tool would not work in principle as there's no history to back it up. Sorry.

This is pity. If you spend time like me to document Market Manager, this is important to "maximize" the usage of the data ;-).

For your information, you can find on this website market holidays for a lot of exchange with the previous data (https://www.investing.com/holiday-calendar/).

Thank you for the link. The data is extractable from 2010 up but it's a can of worms I'd prefer not to open:

1. Who's to support it going forward as new holidays occur every week? If I learnt something from my development of screen scraping bots, it's about commitment to the ongoing effort of fixing unplanned changes to websites and cancelation of service. Like Google Finance last Sunday that affects a number of seemingly unrelated other projects.

2. Suppose we managed to pack the extracted data into MarketManagerMarkets.xml. How to distribute it to the users? How to keep them up to date? Rebuild Wealth-Lab Developer periodically? Use the Wiki for hosting it like Community.Rules? Been there done that. It's not user-friendly.

3. How do you factor in new changes when a user has already configured his Market Manager for some other markets? A crash course in manual XML editing for him?

I can think of more pitfalls but the point here is that certain feature requests call for a better multiple currency/market support than there is to be implemented. Even then, there's not enough user demand to invest time and effort into it (i.e. bang for the buck).

So thanks, but no thanks. :-)

Eugene,

Just to be sure; have a look on my "screenshot". My original idea is just a basic list box to select where to find the holidays date entry by the user in the market manager. By default, you display US Market as you cannot delete it in the market manager.

It's up to you the decision ;-).

Alexandre, to design a feature means to foresee one and even two steps ahead. What you're proposing here is where we started in post #61. This would only work for the minority of users who took the trouble to create a record of holidays in the Market Manager on their own. Everyone else would still be limited to the default behavior. To me this creates an impression of a half-working feature with a chance to raise support issues.

On a closing note, let me kindly suggest you take Cone's Bad History Data Check strategy which is able to detect gaps in the data according to your Market Manager presets. Unlike the Data Tool which must work with the raw data it doesn't have such limitation:

CODE:
Please log in to see this code.

Thank you Eugene.

What's new in the Data Tool version 2019.11:

* New: "Truncate" button changes to "Truncate Selected" and acts accordingly if some symbol(s) are selected.
* Change: warning message on "Remove all data"

I'm running Data Tool 2019.11 in WealthLabPro 6.9.21.15. When I go to the Data Tool tab, I only see the first two of my Data Sets. What do I need to do to see the rest?

When you have a problem, please explain it clearly, with as many facts as you can think of, and possibly attach a screenshot to illustrate what you're seeing.

1. With regard to compatible data providers listed here: Data Tool > Compatibility, what DataSets did you expect to see which aren't displayed?
2. If it breaks on some particular DataSet of a compatible provider, what is its name, what symbols does it contain and what is the data provider?
3. Was it also the case before 6.9.21 or are you a new user of the addin?

The more details the better the chances to track the problem down and help you.

QUOTE:
1. With regard to compatible data providers listed here: Data Tool > Compatibility, what DataSets did you expect to see?

Many of the DataSets which are missing are Fidelity Data Sets, so that isn't the problem.

QUOTE:
3. Was it also the case before .21 or are you a new user of the addin?

I've been using Data Tool off and on for many years. This is the first time I've used it since upgrading to .21. When I initially tried to use it, it showed no DataSets. I then installed 2019.11 and it showed my first two DataSets.

QUOTE:
2. If it breaks on some particular DataSet, what is it name and what symbols does it contain?

The last DataSet it was showing was named Bar and the next DataSet I expected to see was named Fidelity. Both of those are Fidelity DataSets and contain a lot of symbols.

To isolate the problem, I renamed the Bar DataSet ZBar and the Fidelity DataSet ZFidelity, so they would appear at the end of the list. I then created new DataSets A Temp, B Temp and C Temp, all with only the symbol A in them. If I tried to use Data Tool right away, it showed no DataSets, so I closed and reopened WealthLabPro each time I did this, so the DataSets would be in alphabetical order in Data Manager. (The Fidelity lock out to prevent multiple successive log ins means this took a lot longer than I would have liked.) After doing this, I am still only seeing the first two DataSets, which are now A Temp and B Temp. Obviously, I would expect to see C Temp as the next DataSet in DataTool.

Is there a file containing the names of all the DataSets? Assuming the Data Tool works for you, my best guess is that file is corrupted.

Note the message at the bottom of Data Tool saying, Last Update Log saved 3/20/2019. File Explorer shows the file LastUpdateLog.txt was last modified today at 9:18 AM. That might be because I have relocated my Wealth-Lab Pro directory using symbolic links, and the symbolic link is time stamped 3/20/19.

Thanks for the detailed response. There hasn't been any need in such tooling as a file that caches the names of all the DataSets. However, relocation of the WLP AppData folder using symbolic links is a good clue. I have never tested it with the Data Tool nor can offer any support since it's not supported by application itself (WLP). If you wish to continue troubleshooting kindly undo the relocation using symbol links. Alternatively, you might want to reduce the number of your DataSets from 136 to a more manageable number to help debugging.

The natural first step would be to close WLP, move some of the DataSets' XML files out, start WLP, log in (I know it's a drag - sorry), test the Data Tool, repeat with the next portion of DataSets etc.

The answer is an old BBFree or Google DataSet will cause Data Tool to stop when it reaches either of those DataSets. Symbolic links to another directory are not a problem.

I first moved all of the DataSets except A Temp, B Temp and C Temp to another directory without breaking the symbolic link and everything worked fine. I then moved all the Fidelity DataSets back and that was also fine. Same for Nasdaq, QuoteMedia, Tiingo and Yahoo. As soon as I put either a BBFree or Google DataSets back, Data Tool only showed the DataSets up to either of those.

Thanks, an interesting observation. Added a note to Data Tool's Wiki page > Limitations section.

What's new in the Data Tool version 2019.12:

* New: added a new column "52-week High/Low" to indicate if symbol has hit a yearly high or low today

QUOTE:
There hasn't been any need in such tooling as a file that caches the names of all the DataSets.

Are they being cached in memory? A related bug, both with the prior version and 2019.12 -- If you change the symbols in a Data Set (and click Apply Symbol Changes made above to DataSet) and then go to Data Tool, none of the Data Sets will be displayed. Closing and restarting Wealth-Lab solves the problem and makes them all available again.

If I change the symbols in a DataSet from the Data Manager (DM) and then go to Data Tool, none of the Data Sets will be displayed, I get an unhandled exception. This is caused by the feature introduced in v2019.12. Will fix it in v2020.01.

Otherwise the changes I make in the DM or DataSets tree (using Y! for test purposes) have always been picked up by the Data Tool. Cannot reproduce.

What's new in the Data Tool version 2020.01:

* Fix: exception on changing DataSet symbols

What's new in the Data Tool version 2020.02:

* New: Data Cleaner tab with Remove weekend bars feature
* Fix: spike detection is also disabled for AlphaVantage/Tiingo DataSets (like Yahoo, they're kept in raw unadjusted form and thus are incompatible)
* Change: targets .NET 4.6.2 or higher

What's new in the Data Tool version 2020.03:

* New: options to cure data on Data Cleaner tab. Now you can:

1) Remove zero bars (O/H/L/C=0)
2) Fix "Open,Close < Low" and "Open,Close > High"

What's new in the Data Tool version 2020.04:

* New: option to detect and remove synthetic bars (thanks Panache)
* Fix: improved removal of zero/weekend bars

After migrating WLP files to WLD files, why is the Data Tool not showing any datasets? How do I fix that? See attachment.

How many DataSets do you have - only 7? If the actual number should be greater then I think it stumbles on a DataSet by a defunct provider (a documented case in the online user guide). Another cause may be some 'bad' symbol in the first DataSet's title (a speculation). For troubleshooting, close WLD and move all XML files away from the user's AppData\...WealthLabDev\..DataSets\ folder, then add a bunch and start WLD, then repeat. I believe Yahoo's DataSets should be fine but the failing one may be among the rest.

P.S. By chance you tried to move any DLLs (like data providers) from the WLP installation folder by hand? Don't try that.

QUOTE:
How many DataSets do you have - only 7?

The migration tool only moved over the one Yahoo dataset, but none of the Fidelity datasets.

Thinking that one Yahoo dataset might be corrupted, I deleted it. But that didn't help.

QUOTE:
P.S. By chance you tried to move any DLLs (like data providers) from the WLP installation folder by hand?

No, no. And I asked the migration tool not to move any DLLs as well. It's a clean install redownloading all the add-ons from the website.

QUOTE:
So the question remains, is the Data Tool itself corrupted, or is one of the files corrupted?

The Data Tool is not corrupted. I swapped the WealthLabDev directory for WealthLabDev-beforeMigrate, and the Data Tool works after the swap. So the Data Tool is not the problem. Maybe the migration tool corrupted the datasets that were copied from WealthLabPro?

But at least I know it's a data file problem now. Maybe a linefeed or return character was added (or dropped) to these data files after the migration.

I knew from the start it's a DataSet / data provider DLL problem. Would you create a support ticket and attach your problematic DataSets folder?

What's new in the Data Tool version 2020.04:

* Fix: DataSet by a provider not installed or discontinued

Using Change Symbol on AlphaVantage dataset as in Post #4 of another topic produces the Microsoft .NET unhandled exception detailed in the attachment.

Using WLD 6.9.23.1. Configuration problem?

Can't reproduce on a plain vanilla intraday AV DataSet. Can you be more detailed about what you're renaming and how?

Tested on daily and intraday datasets.

Followed your post #4 at: https://www.wealth-lab.com/Forum/Posts/How-To-Migrate-your-intraday-Fidelity-data-to-AlphaVantage-40221
And Data Tool Wiki:

QUOTE:
The right click menu serves two purposes:

Change symbol: renames a single highlighted symbol. Simply type in the new symbol name and hit "Enter". Both the symbol name and its data file will be renamed. Helpful when symbol name changes for whatever reason.

Exception occurs when I hit enter with MVV or other symbols renamed to MVV-BK or MVV_BK. These were Fidelity symbols now AlphaVantage but it occurs on any dataset with any symbol.

Still not reproducible. Renamed MVV to MVV-BK back and forth with data or not in a pure AV DataSet. As a last suggestion, try to reopen the Data Manager before/after renaming.

I've tried without success opening Data Manager with the only action to do a rename. Likewise restarting computer and WLD has no effect.

The detail message attached to post #89 was beyond me. Hoped you might see something off there.

Yes, the error detail gives a clue about accessing a property that isn't initialized (null) in the routine that changes a symbol name. I've added some protections. Please create a support ticket and let's try out a beta build in hope it fixes the error you're having. Zip and attach your DataSets folder for me to reproduce the condition.

Update:

The problem was rooted in DataSets by WL4Files provider supposedly pulled over from WLP by the Migrator tool. Although such DataSets by an inexistent provider no longer break the Data Tool since v2020.08, they continue poisoining it by e.g. not letting rename symbols :)

Removing them fixes the issue.

Using WLD 6.9.23 (WLPro -> WLDev migration 2 months ago), Windows 10.

Sometimes symbols change. WLPro/Data Tool had a simple and easy way to change symbols and this worked across ALL watchlists DataSets the symbol was located in (for all compatible data providers).

It seems to me the only way to change a symbol in WLD is to do it manually, and separately for each watchlist DataSet (assuming one even knows all the lists the symbol is in).

Suggestion: It would be nice to have a simple way to achieve this in the Data Tool....input old symbol in one box, new symbol in another box, click a button and... voila! job done.

Thanks for the suggestion.

QUOTE:
Sometimes symbols change. WLPro/Data Tool had a simple and easy way to change symbols and this worked across ALL watchlists DataSets the symbol was located in (for all compatible data providers).

No it hadn't. The Data Tool has always let you rename symbols on a per DataSet basis. Probably you're confusing the Data Tool with Fidelity static provider's feature. Wealth-Data and Norgate providers handle symbol changes and do not require manual rename. How frequently do you rename symbols across the multiple DataSets of other providers and how many time and effort it takes in the Data Tool?

Yeah, the option to rename symbols was located under Fidelity provider's tab BUT it worked across all compatible data providers such as Yahoo & Morningstar (the only two I used besides Fidelity back then). A similar functionality would be nice to have in WLD too as a symbol may be scattered across multiple datasets.

What's new in the Data Tool version 2020.10:

* New: added progress bar on potentially lengthy operations
* Fix: Error on symbol rename
* Change: possible to install in WLD 6.9.23+

QUOTE:
Data Tool has been updated to version 2011.11....
* Added a context menu with two options:

1. Change symbol - renames a single highlighted symbol....

Yes, but that feature assumes you know what dataset the symbol is in in the first place, which I don't.

All I know is that the IQFeed log says the symbol(s) is(are) "invalid" and a web lookup says its ticker name has been changed. So my question is how do I rename a symbol (without knowing the dataset it's in) to the new ticker name?

Basically, I want to do the same thing the Fidelity provider does, which is to rename symbols that had a ticker name change without knowing which dataset they are in. Perhaps this feature could be added the the "inactive symbol" Data Tool function since a ticker name change would automatically make the old name "inactive". (Perhaps one could "borrow" the code out of the Fidelity provider to do this.)

The Data Tool works as is, and it will not work some other way. If you wish to use the Data Tool's feature you should find the symbol in your DataSets.

QUOTE:
The Data Tool works as is, and it will not work some other way. If you wish to use the Data Tool's feature you should find the symbol in your DataSets.

Yes, but there are over 50 datasets to look through. Can we add a Data Tool feature to find which dataset a symbol is in?

I think your point is that the IQFeed provider should be upgraded to work like the Fidelity provider does to update ticker name changes, and I agree. My comment is, if this feature was added in the Data Tool instead of the provider, then one could use this ticker name-change feature regardless of which provider they are using.

For now it will not take a big effort to do a simple search for symbol name across the XML files of your DataSets. This should get the job done.

When I'll be porting the Data Tool for 7.1, it's something to consider as a potential enhancement.

Is there any tricks to run datatool to truncate nightly (just 1 bar) like a batch before nightly dataSet update kicks in ?

No.