I am planning to create dataset at a minutes level and keep it current programmatically.

The idea came from the following thread where it has been mentioned that yahoo XML file is kept current with 2 weeks lag. After asking a followup question, it was identified that if a user wants to keep that file in sync with local, the user needs to hit a "Refresh" button. Please see the attached screenshot for details.

%AppData%\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\YahooStaticProvider\YahooClassification.xml

What I am planning is to grab the list from yahoo classification list from XML file programmatically and then create a separate XML for fidelity provider. I am thinking of ways to automate it. Is there a command-line program or piece of code I can call to refresh the yahoo classification XML file?

Let's break up the task in two steps.

1. Firstly, you will have to click that "Refresh" button (on your screenshot) manually to make Yahoo! provider download and process the Glassification Groups into YahooClassification.xml. Now that you have it, some script can make a DataSet out of it.

2. I am a believer in "don't code what you can beg, borrow or steal" but since you're the first one to wonder about it there's no readymade piece of code. While parsing the resulting XML file is entirely on you (there's no shortage of examples on the internet), on our forum you can find some guidance and probably some sample code re: programmatic DataSet creation:

How to create Dataset dynamically?
How to create DataSet XML file?

At the risk of repeating, what they say about having to restart WL to make a created DataSet appear still is and will hold true.

P.S. The lag isn't necessarily 2 weeks, it can range from same-day to weeks depending on our schedule. Although the scraping of websites of world exchanges for the actual list of index constituents is done by a bot, Cone does some review and postprocessing.

Thank you Eugene. Yes, creation of XML file is not a problem. Only thing I wanted to avoid is to manually refresh the yahoo classification group XML file. Hopefully, in future you expose the API calls for us to consume.

Thanks
Kuntal

Sorry Kuntal, exposing an API and/or sharing the source data isn't planned.

If you're interested in keeping an index DataSet current then you might want to have a look at Norgate Data (paid subscription). They offer survivorship bias free historical EOD data for U.S. stocks. The provider can be downloaded here (click).

Wealth-Data, also with classification groups, is free and keeps the Dow 30, S&P 100, Nadaq 100, and S&P 500 up-to-date within 1 day (usually). See the other advantages to using Wealth-Data at https://www.wealth-data.com

Hi Eugene,

I would like to create datasets at the subsector level by parsing the Yahoo.xml file provided with WLP. I have code written already, but not sure how to insert SOAP envelop in DSSTRING element. Here is the sample DSSTRING from a Yahoo dataset.

<DSString>{SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:clr="http://schemas.microsoft.com/soap/encoding/clr/1.0" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"}
{SOAP-ENV:Body}
{a1:YahooStaticSettings id="ref-1" xmlns:a1="http://schemas.microsoft.com/clr/nsassem/WealthLab.DataProviders.Yahoo/WealthLab.DataProviders.Yahoo%2C%20Version%3D2019.11.0.0%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3Dnull"}
{Version}2{/Version}
{_symbols id="ref-3"}AAPL,AOILQ,XNRG{/_symbols}
{_updateGroups}false{/_updateGroups}
{_groups id="ref-4"}{/_groups}
{_startDate}1900-01-01T00:00:00.0000000-05:00{/_startDate}
{/a1:YahooStaticSettings}
{/SOAP-ENV:Body}
{/SOAP-ENV:Envelope}
</DSString>

How can I create this programmatically? Here is the XML function to write dataset for Fidelity. Kindly help to convert it for Yahoo dataset.

CODE:
Please log in to see this code.

I'm fine with whatever solution you come up with. Be it XmlWriter based or something along the lines of plain text manipulation. If you decide to use the Yahoo DataSet's DSString as the template, the key obviously is the _symbols field:

CODE:
Please log in to see this code.

Hi Eugene,

So that I understood you correctly, you want me to hardcode texts from SOAP envelope other than the symbol fields to get it done? What are the chances that those texts will change in the future causing an issue?
Is there any other elegant way to do it?

At this time (and for some time), the WLP Classification is far out of date. We're looking for a solution to update it, but it seems that provider connection has been lost or otherwise unsupported.

Thanks Cone.

I see another file called "Gics.xml". Does that file gets updated?

QUOTE:
So that I understood you correctly, you want me to hardcode texts from SOAP envelope other than the symbol fields to get it done?

I would say it's a straightforward and simple solution with minimal effort.

QUOTE:
What are the chances that those texts will change in the future causing an issue?

At the moment, the chance of a breaking change to the Yahoo DataSet configuration file is seen as unlikely.

Thanks Eugene.

One concern from Cone's comment above. He appeared to indicate that Yahoo classification file does not get updated with new Symbols. Is that true? If that is correct, does "Gics.xml" gets updated?

Thanks
Kuntal

QUOTE:
He appeared to indicate that Yahoo classification file does not get updated with new Symbols.

Not true. He was talking about the Fidelity WLP Classification (GICS) rather than the Yahoo Classification Groups (which is updated on a weekly basis).

Right, we're talking about GICS. You said that you "would like to create datasets at the subsector level", so that requires GICS data. The bad news is that the Yahoo! GICS data is based on the same out-of-date WLPro GICS source. The Y! index constituent data is updated nearly every Friday.

Thank you Eugene and Cone for clarifications. I am out of luck then for the time being. Any idea when this will be fixed in the future?

Thanks
Kuntal

QUOTE:
Any idea when this will be fixed in the future?

Sorry, no. For now, my understanding that a lasting fix isn't feasible.

It's far from a permanent solution,but the next update (6.9.23.x) will have the latest GICS classifications, including Real Estate as an industry.

I don't have an ETA for you. It seems that due to the market environment, Fidelity implemented a moratorium on new software releases. When that passes, there will be a very nice update for everyone!