I see that in the last version of the MS123 Extra Fundamental/News providers extension, you have added the possibility to download dividend from Morningstar.

Is it possible for the next release to add more fundamentals data from this source ?

Morningstar is a worldwide database. Moreover, you can download fundamentals data directly on a csv file (see export feature on the following link)

http://financials.morningstar.com/ratios/r.html?t=BNP&region=FRA&culture=en-US
http://financials.morningstar.com/income-statement/is.html?t=BNP&region=FRA&culture=en-us

If you have the possibility to add also Morningstar has an historical/streaming Datasource extension, it would be nice for an international trader because :

- Quality of data is very good
- There is more than 10 years online of historical data
- 5 years of fundamentals data (more than 20 differents items
- There is realtime streaming for some markets
- it's free of charge

For an international trader like me, there is no others source at this time in WLD where I can match streaming/Historical/Fundamentals data because international symbol code are all the time differents depending of the source as I have explained several time here.

I know that I can develop my own dataprovider but what I would like is to spend my time is on trading strategy development and not on technical modules

WLD is a nice tool with a great support
Give for international traders a minimum of facility to trade on their local markets ;-)

Thanks for you understanding and I hope you will be able to do something :-)

Good day,

Some background on why did I pick Morningstar's dividend feed. Mainly for four reasons:

1. relative simplicity of implementation (compared to other Morningstar data items).
2. its exclusive dividend data fields missing in Fidelity's data.
3. the historical data depth surpasses Fidelity's (in this particular case).
4. based on experience, it might take a lot of time to introduce a change to Fidelity data feed.

Unfortunately, the data items you're proposing is a different story. As you pointed out, symbol names are different across different data providers. Wealth-Lab wasn't designed with symbol aliases in mind, and this feature is not on our radar this year. I think you'd agree with me that this Morningstar fundamental data provider would hardly be useful for EU stocks without a companion static data provider. Not to mention that the same stock data can be obtained from Yahoo! (BNP.PA) with full precision: for unknown reason, Morningstar rounds volume in their data e.g. "32.07 Mil".

Besides duplicating our effort of development, these Key Ratios are a challenge to parse. At first glance, it seems that the exported data presentation may require a sizable change of the design pattern I've been successfully using for other fundamental providers. Lastly, it's annual data like the already existing MSN/Yahoo fundamental providers.

I haven't made a decision yet, but chances of implementing this feed look pretty remote as of today.

If we focus only on getting a subset of fixed items from "Key Ratios", then I think it's doable (and even implementing the companion static provider). Please create a support ticket and refer to this thread so it's not forgotten.

QUOTE:
- Quality of data is very good

I. Quality of their static data is rather poor. How about lots of fake bars like this one particular example?


Now compare this with Yahoo, no sign of the fake 1/16/2012 bar:



Notice the different open price. This ain't some pink sheets, that's AAPL Nasdaq.

Not to mention that their static data (like Google) does not contain the open price in the 1990s.

And the worst thing, you could not weed out the fake bars if their volume is "0.0 Mil" - just because lots of perfectly valid symbols do have this "fake zero volume" too even though they were trading. But since the folks at Morningstar decided to lose precision by virtue of rounding, a stock still has a "0.0 Mil" volume if say 50,000 shares changed hands.

II. Their fundamental data suffers from two connected issues:

* Loss of true announcement date. For instance, fiscal year ended at 23 September but the data are marked as September - w/o the date.
* No date by default means 1st of month which is dead wrong if the data became available around the end of month. Consequently, arising possibility of peeking into the future. This is a serious problem that will be handled exactly like we're used to do with YCharts.

Their ASCII Export produces the most malformed CSV data I've ever seen in years:

* Systematic violation of CSV standard RFC 4180 e.g. guideline #2.4
* Bad formation of interim headers (sometimes split in two lines, somewhere not)
* Multiple currency separators in the file...
* ...with subsequent need to wrap some values in double quotes
* No uniform section headers
* No uniformity in dates in the headers. Could start/end as "TTM" or "Latest Qtr"
* Essentially duplicate fields: "Revenue"
* As the result, an irregular structure preventing from creating a robust parser. It could be done by making several assumptions about the particular sections, so end result can break at any time if data vendor considers a change.

Example

Nevertheless, I'll build the static and fundamental providers.

Thanks a lot ! Are you going to build also the streaming ?

No, streaming is not considered.

Why ? As you know, the problem with international stocks is the code translation.
If you don't consider it, we will have no possibility to scan in realtime orders.
It's petty because you have it with BBFree, yahoo and google extension. So, why not with Morningstar ?
Do not forget that Morningstar provides some realtime quotes depending of the market.

Well, maybe I was looking in the wrong place but there doesn't seem to be any sign of free streaming data. Can you point me?

In the quote page, there is the following note :

QUOTE:
ll data supplied by Morningstar Real-Time Data. U.S. intraday real-time exchange quotes are sourced from BATS when available. End-of-day quotes for Nasdaq, NYSE, and Amex securities will appear 15 minutes after close. Graph times are Eastern Standard.

If you want me to take a look, you have to point me to the place where it's streaming, and what is required to get there.

In the quote page, http://quote.morningstar.com/stock/s.aspx?t=XPAR%3aBNP, you can see that the last price is updated automatically every second.

Yep, I see their streaming quotes now that European market is open. Wasn't streaming for me before because I must had been visiting the page outside of EU market hours. Thanks.

Wish my JavaScript reverse engineering skills were better as the ins and outs of quote processing seems to be obfuscated. Nonetheless, it's clear to me how it's made. Let's see if I can work it out...

Here's good news for you abegy: Morningstar Streaming provider will also be built.

Thanks a lot ! :-)

The Morningstar Static (EOD), Streaming and Fundamental data provider for U.S. and international securities has been released. Registered Wealth-Lab customers please download and install it from Extensions:

Morningstar data provider

This thread is the place to discuss the provider. Make sure you've read the online guide before asking questions or reporting errors:

Morningstar Static and Streaming provider
Morningstar Fundamental Provider

I try to download fundamentals data for XPAR:BNP. When I update data, I have no problem (see the log here).

CODE:
Please log in to see this code.

But when I try to put a Fundamental item on a chart (for example dividend), I see nothing in the specific chart pane.
Is anything wrong ?
Thanks for your help.

Reason: the fundamental items from "Morningstar Fundamental Data" haven't downloaded. Server did not return any valid data. (The number of fundamental data items updated is not indicative in this case. By design.)

With Morningstar being a free data vendor, this can and will occasionally happen. I faced it a couple of times during the development. But don't give up after first try. Retry and the server will eventually return valid data.

The fundamental data feed has slightly changed their output format, rendering a dozen of key fundamental items blank (e.g. "Earnings Per Share"). A fix is in the works.

Do you have an example of the change ? I ask you because I haven't seen any change

They started attaching currency (EUR) and/or volume qualifier (e.g. Mil) to exported CSV files. For example:

Was: Revenue
Is: Revenue USD Mil

Consequently, the provider is no longer able to match the string against its list of fundamental items.

The Morningstar provider has been upgraded to 2012.11.1

* Fix: (Fundamental) almost dozen of key Financial items stopped working (zero values) due to a format change at Morningstar
* Change: (Fundamental) "Shares Mil" item was renamed "Shares"

Eugene, I see in the fundamental items that there is 2 items for morningstar Dividend :
- one with the morningstar logo
- another one with "f" as the logo

The first item report differents date for dividend operations but 0 for the cash dividend
The second one report only one date (I don't know for which operation) and the correct cash dividend amount.

For example, try with XPAR:BNP to see that.
Is it normal ?

Yes, it is normal. Reporting 0 is not normal though (investigating). Will post in the proper forum thread.

There are two independent Morningstar fundamental providers, and each one has a dividend item:

* Morningstar extended dividend data ("[ms] dividend") - part of MS123 Extra Fundamental/News Providers
* Morningstar static/streaming/fundamental provider ("[ms] Dividends", aka "the one with f") - this provider

p.s. The "f" is is Wealth-Lab's default fundamental item icon that appears because the obfuscator has swallowed the correct icon.

The Morningstar provider has been upgraded to 2012.11.2

Another hotfix:

* Change: (Fundamental) chart popups of fundamental items will display their title (to be easily recognized on the chart among 82 items)
* Fix: (Fundamental) fundamental icons appear as "f" (missing glyph)

Hi Eugene,

Is it possible to add the "TTM" values for all fundamentals items as the last value of the current fundamentals items ?
For example, I would like to see for XPAR:AC, -2.40 for Earnings per share instead of 0.12 in the current version of the extension.

Thanks a lot.

Alexandre, TTM values will be displayed in upcoming release of the provider (no ETA right now).

The Morningstar provider has been upgraded to 2013.05

Added TTM values (trailing 12-month).

Hi Eugene,

When I try to update historical quotes from Morningstar extension to have the last up to date data, I see that not all symbols are updated the first time. I need to update 2/3 times more to have it all.

Is it normal ?

I have no idea. Many factors may play a role, like network connectivity, or free data feed limitations (did you know that some of them don't like parallel or sequential downloads so much that in the past one could even get 'banned' at Yahoo?). It's a free data feed, and when you update thousands of symbols, anything can happen.

I try to update only 40 symbols. See the log. As you will see

QUOTE:
Daily data is being updated...

Updating Provider Morningstar Static ...

Up-to-date symbols: 0, Update required for: 42, New symbols: 0
Symbol: XPAR:ALU, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:BNP, Bars: 0
Symbol: XPAR:RNO, Bars: 0
Symbol: XPAR:SPI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UFF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:AC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ACA, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:AI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ALO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:BN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CA, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CAP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CS, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:DG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EAD, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EDF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FTE, Bars: 0
Symbol: XPAR:GLE, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:GSZ, Bars: 1, Last date: 7/15/2013
Symbol: XPAR:LG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:LR, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:MC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ML, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:MT, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:OR, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:PP, Bars: 0
Symbol: XPAR:PUB, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:RI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SAF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SAN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SGO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:STM, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SU, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:TEC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UL, Bars: 0
Symbol: XPAR:VIE, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:VIV, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:VK, Bars: 2, Last date: 7/15/2013

Provider update complete for Morningstar Static
----------------

QUOTE:
Daily data is being updated...

Updating Provider Morningstar Static ...

Up-to-date symbols: 37, Update required for: 5, New symbols: 0
Symbols already up to date: XPAR:ALU,XPAR:SPI,XPAR:UFF,XPAR:AC,XPAR:ACA,XPAR:AI,XPAR:ALO,XPAR:BN,XPAR:CA,XPAR:CAP,XPAR:CS,XPAR:DG,XPAR:EAD,XPAR:EDF,XPAR:EI,XPAR:EN,XPAR:FP,XPAR:GLE,XPAR:GSZ,XPAR:LG,XPAR:LR,XPAR:MC,XPAR:ML,XPAR:MT,XPAR:OR,XPAR:PUB,XPAR:RI,XPAR:SAF,XPAR:SAN,XPAR:SGO,XPAR:STM,XPAR:SU,XPAR:TEC,XPAR:UG,XPAR:VIE,XPAR:VIV,XPAR:VK,
Symbol: XPAR:BNP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:RNO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FTE, Bars: 0
Symbol: XPAR:PP, Bars: 0
Symbol: XPAR:UL, Bars: 0

Provider update complete for Morningstar Static
----------------


Update completed (4.92 sec)
----------------

This is a nightmare to manage the data with this extension. For quote, it's simple to see if there is a problem of update. But this is impossible to see if the same problem is available with the fundamentals items.

Is it possible for you to check again this extension to try to find a fix ?

Hi Alexandre,

Thanks for the update log. I was able to nail down the bug in my code. Testing shows that this DataSet can now be reliably updated multiple times w/o issues. The fix is to follow in next update.

P.S. Occasionally you might still see an uncommon "0 bars" when Morningstar actually returns an empty document. You can not combat this - it's a free feed.

Thanks a lot !

Eugene, is it possible to have the Morningstar extension compatible with paper accounts trading ?

Yes, in upcoming update.

The Morningstar provider has been upgraded to 2013.09

New: Paper Trading support (requested by abegy)

Does this provider work for anyone? I cannot get any data for AA or any other stock. Below is the log from the Data Manager. Print screen with an empty chart pane attached.

Updating DataSet Test ...
Preparing requests ...
Requests are ready to go.
Updating Security Names for 1 symbols...
Security Names updated.
[0] AA 11010 bars 08.21.2013 0 bars added
Updating Provider Morningstar Fundamental Data ...
AA 83 fundamental items updated
Provider update complete for Morningstar Fundamental Data
----------------

Update completed (4.189 sec)
----------------

You need to add the prefix exchange code before the symbol. For AA, it's XNYS:AA

Actually, Morningstar Fundamental Data updates without any issues. The prefix is not required, as I seem to get the fundamental data using Yahoo notation (like you just did, because Morningstar provider does not do any "Updating Security Names"):



The issue might be with your system, if the FRED provider fails for you for no apparent reason when it works for everyone else.

Indeed, the provider has a problem with non-U.S. delimiters. The bug will be fixed in an upcoming release. Thank you for spotting it.

UPDATE

Fixed in v.2013.09.01.

Areality asked:

QUOTE:
Please add fundamental fields which were presented exclusively in MSN and can be found from Morningstar. Being selfish I need a substitute for "[msn] depreciation" field. It can be found from Morningstar. Here is IBM example http://financials.morningstar.com/cash-flow/cf.html?t=IBM&region=USA&culture=en-US . The third row from top. "Depreciation & amortiz...". Your work is greatly appreciated.

Thanks for your request. I'll take a look later.

There's twice less data (5 years) for the Cash Flow (Balance Sheet, Income Statement) items than the usual 10 for the "Key Ratio" items, though.

Areality,

QUOTE:
Being selfish I need a substitute for "[msn] depreciation" field.

I determined that the task of adding is too complex (requires significant effort), and the data is very limited. Considering you already are a YCharts Pro user who have requested to add a depreciation item to that provider, it would be way easier to add a range of depreciation items to the YCharts provider early next month.

Just wondering if anyone at WealthLab looked at Morningstar Premium as an alternative to YCharts Pro. Morningstar data quality seems to be better and the price tag is only $199.00/year vs $1000-$2000 for Ycharts. Maybe it is also a more friendly animal for Wealth Lab developers? Does anyone has experience using Morningstar Premium? How did you like it? I can arrange access to Morningstar Premium for Wealth Lab developer interested in the project.

The only difference between Free vs. Premium I've found is the data loading depth:

* Quarterly: 5 vs. 10 quarters
* Annual: 5 vs. 10 years

Is it worth it? Do 10 quarters (i.e. less than 3 years) make great difference when compared to the data loading depth at YCharts?

Hi Eugene,

On the Morningstar websute, there is a specific page where you can access a specific market indicator calculated by Moningstar. There are available on the following page (http://www.morningstar.com/market-valuation/market-fair-value-graph.aspx).

Is it possible for you to give us the possibility to download all of them inside the extension ?

Thanks a lot.

Hi Alexandre,

Sorry, this looks neither promising nor feasible.

Trying to use Morningstar to pull in revenues.


Used this code.

CODE:
Please log in to see this code.

Pulls in Fiscal Yearly Revenue from 2012.

What am I doing wrong.

Is provider no longer valid?

Thanks.

The provider is working correctly. For all the Dow 30 symbols I've just tested it on, the last change happened in 2014. Note that the data is annual.

Make sure you're running the latest version 2013.09.01 (see Extension Manager > Others). What were the symbols?

It was MSFT and it was pulling in


http://financials.morningstar.com/income-statement/is.html?t=MSFT&region=usa&culture=en-US

the 73723 revenue num for June 2012.

Can not reproduce. MSFT or Dow 30, still works perfectly. Note the 86,833 number for 2014-06. Follow my advice on getting the last version.

Will re install..

Thanks.

As an aside..

How did you get such a nice yearly chart above?

QUOTE:
How did you get such a nice yearly chart above?

By switching to Monthly scale. There are two ways as per the User Guide:

+ Charting > Chart Window > "8. Chart Scaling"
+ Charting > Chart Control > Data Panel

The Wiki for this provider states, "•The last quarter or trailing twelve months value is dropped.". This does not appear to be true.. When I run the script below for Alcoa, "AA", it shows the last fundamental data item to be the TTM value on the Morningstar web site. It's dated 1/1/2015. I get...

CODE:
Please log in to see this code.

On the Morningstar web site, the 18.4 value is titled "TTM" , and 16.3 is titled "2013-12".

CODE:
Please log in to see this code.

TTM are no longer being dropped based on customer's request (see messages #27 to #29 in this thread). The Wiki page has been corrected. Thanks for the heads-up.

Problems in Morningstar Provider:

Updating DataSet Morningstar Dow 30 ...
Up-to-date symbols: 0, Update required for: 30, New symbols: 0
Error setting min/max download threads
Error setting min/max download threads
Update completed (0.125 sec)

This bug occurs under very rare conditions. I'll upload an updated extension (ETA end of month). Thank you for the heads-up.

What's new in Morningstar provider's recent version 2016.05?

* Fix: "Error setting min/max download threads" message prevents static provider data update
* Fix: removed artificial limit (.NET default) in New DataSet Wizard preventing from creating DataSets which contain more than ~6500 symbols (32767 chars)
* New: due to holiday bars in EOD data, added support for Market Manager in static data provider
* Change: .NET 4.5 and Wealth-Lab 6.8+ are required to install/update extension
* Change: (internal) updated browser agent version
* Change: (internal) increased thread download timeout to prevent failed requests
* Change: (internal) Multithreaded download routine rewritten. Should improve work on modern multi-core CPUs (e.g. servers)

Morningstar extentionI I have noticed is using the latest annual financial reportings data versus trailing twelve month, probably I am using it wrong, let us know the way we can switch it to TTM or this is prebuild? Thank you very much in advance.

The provider returns the data exactly as delivered by the free feed. But don't hesitate to visit the Wiki and search the forums: there's a good chance you'll find a workaround. For example, here's how to get TTM values:

Weath-Lab Wiki FAQ > How can I get TTM, MRQ and FTM values of estimated earnings?

What's new in Morningstar provider's recent version 2016.11?

NOTE: This update is a must have. It's recommended that you reload all accumulated fundamentals by Morningstar.

* Fix: many values were recognized incorrectly (lower than they are e.g. 2,353 was presented as 2.353 rather than 2353)
* Fix: "Book value per share" is unrecognized due to item renaming in downloaded data
* New: added new items: Payout Ratio %, Interest Coverage, Free Cash Flow Per Share

What's new in version 2017.04:

* Fix: "Update all data" doesn't update any data during first-time update
* Change: Wealth-Lab 6.9+ required to install/update

What's new in version 2017.10:

* New: support for automatic bar data corrections

hi,

i dont get fundamental data for european stocks. i read here that there are some Problems with different european ticker Symbols but i dont know how to fix it.

thank you


here is a picture

Hi,

As for me, the 1st paragraph of the Wiki page you refer to tries to explain it pretty well that the Yahoo! static data for ADS.DE (like shown on your screenshot) cannot be used with the Morningstar fundamental data:

QUOTE:
Since most every data vendor has its own symbology, Morningstar is no different. This is not a big issue for U.S. symbols which are universal, but for overseas symbols it presents a problem. For example, the ticker for a European stock symbol will most likely differ from its counterpart at MSN, Yahoo or QuoteMedia. For example, the same BNP stock traded on XETRA is known as "BNP.F" at Yahoo but exists as "XETR:BNP" at Morningstar. Consequently, attempting to download the historical fundamental data for BNP.F from the vendor that knows the stock as XETR:BMP will fail because it's not recognized.

Solution is obvious. Please visit Morningstar's website and look up the correct ticker for Adidas in their notation (could be XETR:ADS, for example - I didn't check it). Create a DataSet with this symbol using the Morningstar static provider and it should be a good match to Morningstar's fundamental data.

hi,

thank you for your answer.now i get data for adidas .

is there a fast way to create databases for eurostoxx 50,dax30,etc.?or do i have to add every Symbol manually?

QUOTE:
or do i have to add every Symbol manually?

Yes.

P.S. You can only do that for some other providers, Wealth-Data, Nasdaq Nordic, Dukascopy (to some extent) and Yahoo to name a few. (Nasdaq Nordic doesn't support the named indices and Dukascopy only has a few stocks).

But as for DAX30, Wealth-Data has the cleanest static data possible (although its symbol name e.g. ADS.DE isn't compatible with Morningstar's fundamental data).

Hello,
I'm using the Morningstar version 2017.10 and want to use fundamental data for a strategy.
But if I want to use the fundamental data nothing is shown. It seems no fundamental data is existing - see attachment.
Do you know the problem?

Marco,

I already responded to this very question this month right here. Please re-read my answer in post #66 and the Wiki online guide it refers to. Pretty sure that this will clear your question and pay off in the long run.

Hello,

if I use the exact symbol of Morningstar I still don't get fundamental data - see attachments.

Hi Marco,

Everything works:

1. Check the proper "Morningstar..." fundamental provider (there are two)
2. Update fundamental data using "Update DataSet" (not "Price")
3. Reopen the chart

1. Check the proper "Morningstar..." fundamental provider (there are two)

--> I only find one - see attachment

Good for you, so you're picking the right one and that's not an issue of confusion:

Morningstar Fundamental Provider: extended Dividend data

P.S. Don't open a chart before downloading the fundamental data.

Ok, but it doesn't still work ...

Then you're doing something wrong. I'm pretty sure that it works (see my screenshot above) and the provider has been around for 5 1/2 years.

Hi, I'm having problems with updating fundamental data from Morningstar too.

Whenever I update dataset (not price) and reopen the charts, some companies' fundamentals are updated while others remain completely blank. And when I reupdated the dataset, some of the initially blanked companies data may become available, but some which were at first available now become blank.

I double checked with Morningstar website for these individual stocks and confirm the missing data are there. Do you know the problem and how to fix this? Cheers.

Both the static and fundamental providers work as expected for me. Most likely, your symbol does not have a fundamental data item in the output file. It doesn't matter what Morningstar website shows as the actual export file (parsed by the provider) may have differences. It's not uncommon to encounter empty rows or missing values in this free data source.

If you could be more precise about the symbol and fundamental item then I'd have an idea what went wrong (or maybe it's okay).

Sure Eugene. For example, the book value per share for XTSE:CVE was not there. When I reupdated dataset for Morningstar provider it's now available (and it might not be if I reupdate again). While for XTSE:BNS, the data was there. After my post above, I reupdated dataset and it's now missing. The same applies for all fundamental variables, and I scrolled through each company to see availability.

In short, whenever I updated dataset, fundamental data were successfully made available only for some companies. These keep on changing every time dataset is updated (some firms succeed, others don't).

I updated the two symbols several times and "[ms] Book Value Per Share" has always been present in both. What you describe sounds like a network issue or (more likely) a server-side issue. With a DataSet big enough it's not impossible that your requests are getting throttled or rejected by the M. server. After all this is a free datafeed and web scraping by bots may not be welcome. So you might retry from a different ISP to change your IP address and see if problem persists immediately or later.

P.S. Meanwhile I found a potential issue with the M. static provider which could lead to 0 bars downloaded on some occasions (if download took unexpectedly long). But the M. fundamental provider is altogether different and not affected by this. To be fixed in v2018.12.

Thanks, I'm afraid it still doesn't work. The DataSet consists of 60 stocks only. I'm wondering, because when we click on charts WL has this right click "Reload Chart History" function, is it possible to make something like "Reload Fundamental Data" for that particular stock? This way we can try "force" the data to reload on stocks with missing fundamentals.

Re your other info about 0 bars, I have the same problem for the historical price of all ASX stocks which took too long (well, because there are over 2,000 stocks). I would say over 75% constituents returned 0 bars downloaded.

Safwan, thanks for your detailed feedback and patience to describe the problems. It was very helpful.

The issue with "75% constituents returned 0 bars downloaded" is now fixed. It wasn't that much of an issue with Morningstar before but in my quick test if the download speed slackens the static provider could quit processing prematurely.

The problem with fundamental data didn't present itself on 2 symbols yesterday. When I looked into it closely I was able to reproduce it with a DataSet of your size where up to 50% fundamental data requests ended up empty. Just like I suspected, this behavior of Morningstar's server looks very much like protection against web scraping to me. Without going too deep into detail, even request(s) first in the queue may fail after a pause in download as server responds with empty data at random. :\

Starting with v2018.11 which you already can download (or update to), the fundamental provider (only) will retry up to 5 times if it has encountered no fundamental data. This has proven to be the solution against missing data. (If your DataSet contains symbols not recognized by Morningstar, you'd want to delete them to avoid having to waste a few cycles retrying them 5 times each unconditionally.)

What's new in version 2018.11 - a mandatory upgrade:

* Fix: frequent failure with the static provider (0 bars) if a download took unexpectedly long
* Fix: fundamental provider will retry a failed download on empty fundamental data returned by server at random

Thanks a lot Eugene and awesome work for the quick upgrade.

It works much better now. For historical prices of 200 companies the download works fine, while at around 1,000 firms or so I noticed just 1 or 2 missing prices (but simply reloading Chart History fixes this).

For fundamental data, I tested for several sized DataSets a few times: 20 - all fine; 40 - 2/3 missing (reupdate gives all data); 149 (NZ data) - around 40 missing; 200 (ASX data) - 17 missing. You said the fundamental provider will retry up to 5 times if it has encountered no fundamental data, so it will skip if the server doesn't return any after that. Is this cycle purely server-side or affected by my connection speed too? Also, is it possible (maybe in future versions) to update only data for missing symbols, or as per my post above having something like right click "Reload Fundamental Data" on individual stock chart so that we can manually update only stocks with missing fundamentals (without affecting those already available)?

QUOTE:
It works much better now. For historical prices of 200 companies the download works fine, while at around 1,000 firms or so I noticed just 1 or 2 missing prices (but simply reloading Chart History fixes this).

Was about to suggest this workaround as I noticed partial data returned for a couple symbols during my experiments as well. You can notice them by unusually small Bars count, and I see it by incomplete web pages in my traffic sniffer. But provider cannot make assumptions about how much Bars are 'enough' or 'not enough' so it will not attempt to retry.

QUOTE:
Is this cycle purely server-side or affected by my connection speed too?

Your connection quality or speed is probably not a factor here. Compared to the static data, the footprint of a fundamental data download is nothing. In my opinion, our data download attempts are intentionally counteracted at server side by skipping requests at random. You should get used to it with free data feeds: for example, StreetInsider's protection against bulk capture would now let you get the data for no more than 2-4 symbols before suspension.

QUOTE:
Also, is it possible (maybe in future versions) to update only data for missing symbols, or as per my post above having something like right click "Reload Fundamental Data" on individual stock chart so that we can manually update only stocks with missing fundamentals (without affecting those already available)?

The fundamental provider's design pattern is to reload all data for all participating symbols, and proposed change is undesirable. In my testing on DataSets like Dow 30 or Nasdaq 100, there always were 0 missing fundamental data as the web sniffer detected successful retry attempts. So as a workaround, try to break your bigger DataSets into several smaller ones and update them manually. Fundamental data does not require frequent updates (perhaps once a month).

If this looks inconvenient you might consider paid alternatives like YCharts or Quandl, but they may not deliver the fundamentals for international stocks.

Thanks for the detailed feedback.

Noted on the small bars count and possible needs to update as well as the free data feeds limitations. And good idea to break them into smaller DataSets; I can then add tickers with updated fundamentals to a pooled set if testing on a larger portfolio is required.

Indeed not all indices are available on YCharts or Quandl, so far I would say Morningstar is the best in term of wider coverage.

What's new in v2018.12:

* Fix: potential un-representable date/time error on static data update

What's new in v2019.09:

* Change: Improved fundamental data update speed by skipping requests for unsupported symbols
* Fix: Fundamental provider sometimes might incorrectly display how many data items were updated
* Fix: Extension Manager icon not shown

Morningstar OHLCV data missing for 7/7/2020. This is the log from 7/8/2020 7:30 a.m. EST...

QUOTE:
Updating DataSet Top3KTechxADRMStar ...
Up-to-date symbols: 173, Update required for: 28, New symbols: 0
Symbols already up to date: AAPL,ACIW,ACLS,ACN,ADI,ADTN,AEIS,AGYS,AKAM,AMAT,AMD,AMKR,APH,APPS,ARW,ASGN,ATVI,AUDC,AVGO,AVT,AZPN,BAH,BB,BDC,BHE,BLKB,BRKS,CACI,CALX,CCMP,CDNS,CHKP,CIEN,CLS,CMTL,COHR,COHU,CREE,CRM,CRUS,CSCO,CSGP,CSGS,CTSH,CTXS,CVLT,DDD,DIOD,DOX,DSPG,DXC,EA,EBAY,EBIX,EGHT,EGOV,ENV,EPAY,EXTR,FFIV,FICO,FLEX,FLIR,FN,FORM,GIB,GILT,GLUU,GTT,HLIT,HPQ,HSTM,IAC,IDCC,IIVI,INFN,INSG,INTC,INTU,IPHI,IRBT,IT,JBL,JKHY,JNPR,KLAC,KLIC,LDOS,LLNW,LOGI,LOGM,LPSN,MA,MANH,MANT,MAXR,MEET,MEI,MELI,MITK,MRVL,MSI,MXIM,MXL,NATI,NCR,NLOK,NSIT,NTAP,NTCT,NTGR,NUAN,NXGN,NXPI,ON,ONTO,OSPN,OSTK,OTEX,PBI,PEGA,PLAB,PLT,POWI,PRFT,PRGS,PRO,PTC,QCOM,RAMP,RBBN,RDWR,RMBS,SANM,SATS,SINA,SLAB,SMCI,SMTC,SNX,SPSC,SSNC,SSYS,ST,STMP,STX,SWIR,SWKS,SYKE,SYNA,TDC,TDY,TRMB,TSEM,TTGT,TTMI,TXN,TYL,UCTT,UIS,VECO,VIAV,VMW,VRNT,VRSK,VRTU,VSAT,VSH,WDC,XLNX,XPER,XRX,ZIXI,
Symbol: ADP, Bars: 1, Last date: 7/6/2020
Symbol: ADBE, Bars: 1, Last date: 7/6/2020
Symbol: ADSK, Bars: 1, Last date: 7/6/2020
Symbol: GLW, Bars: 1, Last date: 7/6/2020
Symbol: FTNT, Bars: 1, Last date: 7/6/2020
Symbol: ANSS, Bars: 1, Last date: 7/6/2020
Symbol: ENTG, Bars: 1, Last date: 7/6/2020
Symbol: GOOGL, Bars: 1, Last date: 7/6/2020
Symbol: IBM, Bars: 1, Last date: 7/6/2020
Symbol: MCHP, Bars: 1, Last date: 7/6/2020
Symbol: LSCC, Bars: 1, Last date: 7/6/2020
Symbol: MSFT, Bars: 1, Last date: 7/6/2020
Symbol: MU, Bars: 1, Last date: 7/6/2020
Symbol: LRCX, Bars: 1, Last date: 7/6/2020
Symbol: IPGP, Bars: 1, Last date: 7/6/2020
Symbol: MPWR, Bars: 1, Last date: 7/6/2020
Symbol: NFLX, Bars: 1, Last date: 7/6/2020
Symbol: NVDA, Bars: 1, Last date: 7/6/2020
Symbol: OLED, Bars: 1, Last date: 7/6/2020
Symbol: QRVO, Bars: 1, Last date: 7/6/2020
Symbol: ORCL, Bars: 1, Last date: 7/6/2020
Symbol: RP, Bars: 1, Last date: 7/6/2020
Symbol: SNPS, Bars: 1, Last date: 7/6/2020
Symbol: TER, Bars: 1, Last date: 7/6/2020
Symbol: V, Bars: 1, Last date: 7/6/2020
Symbol: VRSN, Bars: 1, Last date: 7/6/2020
Symbol: VHC, Bars: 1, Last date: 7/6/2020
Symbol: ZBRA, Bars: 1, Last date: 7/6/2020
Update completed (0.749 sec)
----------------

Note the "Last date: 7/6/2020". These should be 7/7/2020. I can't seem to bring these symbols up-to-date.

Yes, data delays periodically happen with Morningstar (especially) and other free data providers. A delay up to 3-4 market sessions is OK with Morningstar. Not a big deal. Check back tomorrow or later.

On a related note, a watchdog alerts us if any data scraper in the wild (all data providers, their Classification Groups, C.Components, public Strategies that downloads data on-the-fly, countless other utility scripts etc.) goes down or starts returning unexpected results. Such automation saves a lot of time and effort and lets me act proactively.

QUOTE:
For Morningstar symbology please visit their website.

https://www.morningstar.com/products/data?prd=data
Sounds like great advice, but none of the links for that page work.

Anyway, what's the ticker symbol for the S&P500 index on the Morningstar provider, and how did you determine that?


Also the link https://corporate.morningstar.com/us/documents/Quotes/QuoteSpeedQuickReference.pdf on the http://www2.wealth-lab.com/WL5WIKI/MorningstarProvider.ashx page is broken.

To get S&P (symbol SPI:SPX)...
https://www.morningstar.com/indexes/spi/spx/quote

...you could use their search page:
https://www.morningstar.com/search?query=S%26P

PDF link removed. Websites occasionally get redesigned.

On the Morningstar provider page:

QUOTE:
Streaming data ... To reduce server load and conserve bandwidth, partial bars are being updated once every 10 seconds ...

But I can't get streaming to work at all on WLD, and I do have Morningstar checked as the streaming provider. I also restarted WLD after checking it as directed. Does Morningstar still provide streaming data for free, or is that now a subscription service? I can't seem to figure that out from their website because none of the links there work.

QUOTE:
Issues and limitations ... Known issue (deferred): Company names are not being retrieved when requesting new symbols on demand.

Does that now also apply to static symbol data as well for Morningstar? I just created some new Morningstar datasets and updated them (static download update), but I don't see company names. Am I doing something wrong?

QUOTE:
To get S&P (symbol SPI:SPX) ... https://www.morningstar.com/indexes/spi/spx/quote

Thanks a bunch for that. I would have never guessed about the "SPI:" prefix.

It looks like the index lookup only shows the Close prices and not the OHLC candlesticks. Volume isn't showing either. I guess I'm expecting too much from a free service.

QUOTE:
But I can't get streaming to work at all on WLD, and I do have Morningstar checked as the streaming provider.
+
I just created some new Morningstar datasets and updated them (static download update), but I don't see company names.

Both features are broken. The endpoint at Morningstar appears to be decommissioned or redesigned. (Hence the missing PDF link in the Wiki which you discovered yesterday.) Let's see if it'd be possible to fix both sometime next month.

My watchdog (see post #89 above) somehow wasn't probing for Morningstar's streaming data which explains why I wasn't notified. Thanks for the heads-up.

QUOTE:
It looks like the index lookup only shows the Close prices and not the OHLC candlesticks.

Yes, this is what I'm also seeing in the data: no OHL and V. There are much better examples of free data feeds for indices like say Yahoo.

QUOTE:
Both features are broken. The endpoint at Morningstar appears to be decommissioned or redesigned.

Thanks for checking it out. (I kind of thought my install was failing, but I see that's not it.)

I inquired to Morningstar about their subscription data feed rates. I received this:

QUOTE:
Thank you for writing to Morningstar. To get information about data feed rates please each out to licensedproducts@morningstar.com or you can also call at 866-608-9567.

Part of the reason for asking about rates is that some of the other subscription providers are very expensive. But even if Morningstar's rates are more reasonable, someone would still have to write a WL provider plug-in for it.

Morningstar's broken Streaming provider is to be fixed in v2020.09 (ETA ~ late August).

What's new in v2020.09:

* Fix: Streaming data stopped working (a backend change)
* Change: rebuilt for .NET 4.6.2+

Just a heads-up. The Morningstar "Dividend" item is broken (endpoint decommissioned at morningstar.com). No fix is planned for the WL6 extension but we've already resolved it in upcoming Build 2 of "Fundamental" extension for WL7 -- to be released along with WL7 Build 8:

https://www.wealth-lab.com/extension/detail/Fundamental

The Morningstar static, streaming and fundamental providers for WL6 are broken beyond repair.

We will restore and improve the historical and streaming providers for WL8 as of Build 11.