I will soon transition from Fidelity Wealth Lab Pro to Wealth Lab Developer. I have the Yahoo! Finance extension installed on Wealth Lab Pro. I have been informed that I can use it in Wealth Lab Developer, so to test it I duplicated an existing Data Set and designated Yahoo Finance as the provider. When I update the data, Yahoo finds most symbols but it will not recognize others. When I enter the symbols in the Yahoo Finance website, it finds all of the symbols (that I have tried). Here is an excerpt from the update log from Wealth Lab Pro.

[4] FSDAX 9126 bars 7/21/2020 9126 bars added
[2] FSDCX 7573 bars 7/21/2020 7573 bars added
[9] FSHBX Error: The remote server returned an error: (404) Not Found.
[1] FSEAX 6865 bars 7/21/2020 6865 bars added
[3] FSDPX 8522 bars 7/21/2020 8522 bars added
[0] FSHCX Error: The remote server returned an error: (422) Unprocessable Entity.
[8] FSELX 8817 bars 7/21/2020 8817 bars added
[8] FSMVX Error: The remote server returned an error: (404) Not Found.
[1] FSLVX Error: The remote server returned an error: (404) Not Found.
[7] FSENX 9840 bars 7/21/2020 9840 bars added

The same thing occurred with some indexes:

[5] ^IXUT Error: The remote server returned an error: (422) Unprocessable Entity.
[1] ^IXBK Error: The remote server returned an error: (422) Unprocessable Entity.
[3] ^IXIS Error: The remote server returned an error: (422) Unprocessable Entity.
[4] ^IXK Error: The remote server returned an error: (422) Unprocessable Entity.
[5] ^SPX Error: The remote server returned an error: (422) Unprocessable Entity.

Can anyone tell me why this is happening and how I can fix it?

Thanks

Please search the forum before raising an issue. Thank you. Both have been discussed and explained:

Yahoo provider Error 404 on symbol
Yahoo provider Error 422 Unprocessable Entity

Summary:

1. It's not data provider's fault.
2. Code 422 is temporary, and some symbols may just not exist with the free data provider (404).
3. Retry later.

Apparently Yahoo! Data Provider doesn't support the symbols I trade.

QUOTE:
[1] GDXJ Error: The remote server returned an error: (404) Not Found.
[6] XLE Error: The remote server returned an error: (422) Unprocessable Entity.
[2] IBB Error: The remote server returned an error: (422) Unprocessable Entity.

Y! does support them, but mileage varies. These symbols work for me (hitting European servers). Try restarting to refresh the Yahoo! cookie (not sure if clearing IE browser cache too will help).

Note that Y! is generally fine for a free source to get started, but you shouldn't rely on it for trading. At least once or twice a year you can expect an outage for several days during which we have to respond to site changes or other obstacles for downloading data.

@kazuna

It looked like off topic in the topic where you posted this. I've moved our replies to a suitable thread like this one.

I'm running into similar errors with Yahoo. I was hoping to be able to use free sources of data with WLD (now that Fidelity is decommissioning WLP). Which providers are better?? I only need EOD so I thought the free sources would be fine.

Let's please not turn this very specific thread into yet another "looking for provider alternatives" topic. If you're running into errors 404 and 422, can we have a summary to work with:

1. what symbols (you may attach the data update log)
2. at what time of day
3. how many symbols in the DataSet
4. have you restarted WLD prior to the morning (i.e. is Yahoo's saved cookie from yesterday or is it fresh)
5. anything else you can think of

Thanks.

Mainly I'm posing the question as there are probably other Fidelity users who are migrating to WLD. That means we are experiencing moving from Fidelity data to another provider for the first time. I had some Yahoo datasets in WLP, but now my Fidelity datasets are Yahoo ones creating more symbols to update.

#1 I can create a support ticket for that if we need to go that route.
#2 I usually start my computer and WLD around 7am EST and update all data
#3 This morning the log entry stated "Updating Security Names for 1273 symbols..." for updating Yahoo data.
#4 Yes, computer and WLD restarted
#5 My datasets contain stocks, ETFs, mutual funds plus the indices dataset installed with WLD.

I assume it is volume related as Yahoo provider starts out fine, but after a while I start getting 404 and 422 errors more frequently. Some 404 might be valid as I need to clean out obsolete symbols. Fidelity had a Fidelity Data tab in Data Manager which helped rename and remove symbols that got errors. I'm not sure if there's a tool to help with that in WLD.

IMHO using like 1K symbols is pushing the limits of the free feed. Like suggested in other topics for the similar issue, a) break your DataSets into smaller ones; b) update them by one with a delay; c) reduce the number of symbols; d) retry later.

I definitely understand. I figured the number of symbols was part of the problem.

One reason I do "Update all data..." is to take advantage of the option "Delete data for Symbols that are not contained in Datasets" to clean up extra data that is no longer needed.

I'm following most of your advice except (b). I'm in the process of reducing datasets and checking datasets for obsolete symbols. I tend to update an individual dataset before running a strategy against it just to make sure it's up to date.

QUOTE:
2. Code 422 is temporary,...

That appears to be true. If I try to ...

1) reload the chart history and get "No Data"
2) then I push the "Update data for all selected providers" button on Data Manager.

I get the 422 error in the download logs.

3) Then if I wait an hour and try again, it works as expected.

So it's a throttling method on the Yahoo server side that's outside our control. I would avoid doing a "reload chart history" operation to avoid this error if you accessed the ticker symbol in question recently with Yahoo.

None of the suggestions work. Previous discussions and explanations no longer work.

There is some funny business going on over at Yahoo. There is a slight improvement when switching VPN server city location. But it is temporary and short lived fix.. This tells me they are tracking the IP address to the requests.

It looks to me like after about 100-120 symbols, they start jacking things up.

I uninstalled WLD... Deleted the data folder. Reinstalled WLD from scratch. Reset my Windows Firewall to default. Switch VPN to different city.
Extensions updated, relaunch. I hit update on the default data sets, first 120 no problem, just like with WLP and Fidelity, them come the 404 and 422. Every 3-6 symbols.

Click "Update all data..." again, now those first 100-120 that worked fine are now getting the 404 and 422 garbage.

I don't have time to tinker with it. My kids first day of school tomorrow and I was supposed to go read him a story before bedtime a half hour ago. I'll figure out what the combination is... number of symbols limit, time between requests... gotta be something.

I also notice that there are some symbols will update to today's closing but others are still on last session. Not sure why on that either.

Thanks for sharing your experience. Please try out more suggestions:

1. Firstly, update the data "tomorrow" morning rather than doing it after market close. That's more reliable with regard to "symbols will update to today's closing but others are still on last session".

2. Experiment with the number of Threads and Retry attempts in Yahoo's Data Manger tab.

QUOTE:
1. Firstly, update the data "tomorrow" morning rather than doing it after market close. That's more reliable with regard to "symbols will update to today's closing but others are still on last session".

Yes. Sometime between 11:30pm and 2:00am is when the bars get added for that previous day.

QUOTE:
2. Experiment with the number of Threads and Retry attempts in Yahoo's Data Manger tab.

The only difference this made was a drastic slowdown.

I see consistent results when I only do 100 symbols at a time and wait at least 1 minute until doing another 100. Yahoo seems to have implemented an API "rate limiting" routine.

QUOTE:
Yahoo seems to have implemented an API "rate limiting" routine.

This looks like a resonable conclusion to me. Thanks. Let's have it for the other Yahoo users out there:

1. Break your DataSets into batches of 100 symbols for the update to work consistently.

2. Schedule 'Automated Data Update' at or after 2am for Yahoo to work or simply update the next morning manually. However, with rate limiting in mind this may not work if you have more than 100 Yahoo symbols.

3. Use other free or paid data services which do not have these limitations.

Here is another Yahoo update method that extracts error symbols from the update log and then updates the errors only.

1. Do scheduled update in the morning for all providers including Yahoo.

2. Run the below strategy and copy the symbols from debug.

3. Select all symbols in a "Yahoo Error" dataset and paste the new ones.

4. Update the Yahoo Error dataset.

The process prints and manually updates symbols with 404 or 422 errors. So with 660 symbols a good day has only 1 manual update instead of 7. Since the update log is refreshed after each manual update, you can repeat the process if more errors occur.

Enter the full path to your LastUpdateLog.txt file if needed in code below.

CODE:
Please log in to see this code.

Nice workaround. For the record, how many Retry Attempts are set (Data Manager > Yahoo tab)?

QUOTE:
For the record, how many Retry Attempts are set (Data Manager > Yahoo tab)?

Set at the recommended 5 retry attempts.

Note only one error is produced per symbol. If multiple errors per symbol were produced only one symbol would survive after pasting into Error Symbols dataset.

QUOTE:
There is some funny business going on over at Yahoo. There is a slight improvement when switching VPN server city location. But it is temporary and short lived fix.. This tells me they are tracking the IP address to the requests.

Indeed this is true. I digged in Yahoo's responses today: for me, they were always 404 on valid symbols (no 422) with an incorrect error message attached (like if a problematic symbol was delisted). Tried adding some logic to the provider which caches the first trade date of each instrument (as returned by a Yahoo API) and then requests just the available history. This has no visible effect as the problem shows up after a few hundred requests. Then, updating even a DataSet like Dow 30 becomes impossible. After a break the shaper at Yahoo releases my IP.

Bottom line: it seems we cannot really affect this behavior. If only by throwing in some great timeout between requests, slowing down the provider to the crawl.

QUOTE:
After a break the shaper at Yahoo releases my IP.

It would be good to know 1) the minimum wait time before updating the error symbols and 2) if a too soon failed update resets the wait period.

Adding the local time to the update log would help track the required wait time.

For the last week Yahoo has been updating all symbols with no 404 or 422 errors. Maybe all have same experience. I added a confirmation message when no errors are found.

CODE:
Please log in to see this code.