I have been having repeated issues with Fidelity intraday data disappearing and not wanting to reload. I have read the various threads about how to resolve, but I never seem to be able to do it reliably and predictably.

My current process may seem like overkill, but it doesn't always resolve the problem on first try.
Here's what I do:

1. Delete offending minute folder from Fidelity Static Data
2. Delete *all* instances of offending symbol in *all* folders in Static
3. Delete offending symbol from *all* data sets
4. Restart WL
5. Create *new* dataset and recreate symbol within it, setting preferred minute data in the dialog when entering symbol manually.
6. Open clean chart window and click symbol (Result = No Data)
7. Reload data

Sometimes this will fix it, sometimes I have to do this over again. As you can imagine, this takes a lot of time and if since I am intraday trading on signals, it can be a problem.

I would like to know if there is a "failsafe" way to force the recreation of data for a given symbol, even if I have to delete all the data, datasets, folders, etc., and start over.

There seem to be two different data related issues with the Fidelity static provider. If one knew a surefire way to resolve an issue (which one?) it would be posted on the Wiki (or fixed altogether), saving you from having to create this (essentially duplicate) thread. I'm not sure if further fragmentation of the discussion with this new thread would help troubleshooting this live bug.

Depending on how exactly the problem presents itself in the Data Update Log you might want to determine the scenario appropriate in your case and react accordingly:

1. Following a (reverse) split:

WL will only update 1 minute Fidelity data
or Open Issues > Fidelity data can not be updated except for X minute, workaround #3 "Delete all directories..."

2. Corrupt data (more recent reports):

Fidelity 1 minute static data stopped updating
Fidelity data 0 bars when updating SPY 1 minute DataSet
Fidelity Static Provider doesn't update correctly

Restarting WL and new DataSet creation are unnecessary. It's the deletion of data directories that helps. How exactly? If #2 is your problem please refer to appropriate thread for the procedure.

Thanks, Eugene.

QUOTE:
1. Following a (reverse) split:

This is not my issue as it happens with major indices (which are never split)

QUOTE:
2. Corrupt data (more recent reports):

Fidelity Static Provider doesn't update correctly

...was the most relevant, and recent. My takeway is that there are multiple causes and some of this may be due to the data feed. It appears the cause(s) of this problem are not well understood, so it's unclear what to fix and who should fix it.


I have 2 computers with WL primarily to test these problems to see if it exists on only one machine. That would point toward local data corruption instead of data feed. I have these problems on both computers, but never with the same symbol, same interval.


It would be great if there were one proven procedure that, even if extreme, would resolve this issue, even if only temporarily.

Okay thanks, at least we're onto something: your problem looks like @kazuna's already resolved issue. At first we need to understand what exactly happens repeatedly when you run into the issue and what works for you to alleviate it and then submit a detailed bug report to Fidelity. We're far from a "failsafe way" since I'm missing a problem report from you. For now let's rename your thread to "Fidelity static provider doesn't update correctly #2" for lack of a better title.

Looking forward to the problem report which contains action sequence, relevant data update log lines, and what data is broken. Please find an example here. tia.

I have once again lost 1 min data on all symbols. I have this problem often with data shorter than daily. In the past I have always managed to get the data back, but could never find a repeatable pattern that works every time.

There are no subfolders or .WL files in the ...Data>1 minute folder. I can't seem to create either one. I have tried deleting & recreating 1 min dataset, deleting ...Data>1 minute folder and contents, tried Reload of chart (gives "Chart Data Reloaded" message but no data or folder or file created), and data update for Fidelity. Data update indicates "Updating 1 minute data" but there is no message following that or symbols, as there are with all the other updates.

I have restarted WL several times.

I have working with QQQ and SPY, but appears to be true for all symbols.

I am at a loss bas to what else I should try to get 1 minute data back.

Have you relocated the Data folder a different drive or directory?

---

Let's reset your WLP to factory default and see if it helps get the data back:

1. Find your User \Data folder (Enable hidden file visibility). Sounds like you already know how to.

C:\Users\Windows User Name\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\

2. Above the \Data folder, rename the WealthLabPro parent folder to WealthLabProX for example.

3. Start WL6.

+ Do not copy any files to the new WealthLabPro/Data folder (except the Strategies subfolder which contains your Strategy code).
+ Do not enable data relocation feature.

4. Re-create your DataSets from scratch, possibly omitting any symbols which had a reverse split lately, and retry data update. The less symbols the better for troubleshooting: I'd leave just QQQ and SPY.

Before I tried what you suggested above, I tried something different. I copied the 1 minute QQQ.WL file from my other computer into the subject computer 1 minute folder. I then did a data update for Fidelity data. This appeared to bootstrap an update for QQQ (data was about a year out of date, but also a complete update for a number of symbols that were not in the folder:

---

Updating Provider Fidelity Investments ...
Updating 1 Minute data ...
QQQ 244021 bars 6/28/2019 4:00 PM 96683 bars added 24 bars corrected in 7 chunks
.MID 1503531 bars 6/28/2019 4:00 PM 1503531 bars added in 96 chunks
.NDX 1512063 bars 6/28/2019 4:00 PM 1512063 bars added in 96 chunks
.SPX 1509705 bars 6/28/2019 4:00 PM 1509705 bars added in 96 chunks
SMDD 80511 bars 6/28/2019 4:00 PM 80511 bars added in 59 chunks
SPXU 944686 bars 6/28/2019 4:00 PM 944686 bars added in 62 chunks
SQQQ 837423 bars 6/28/2019 4:00 PM 837423 bars added in 59 chunks
SRTY 674744 bars 6/28/2019 4:00 PM 674744 bars added in 59 chunks
TQQQ 845068 bars 6/28/2019 4:00 PM 845068 bars added in 59 chunks
UMDD 95095 bars 6/28/2019 4:00 PM 95095 bars added in 59 chunks
UPRO 925357 bars 6/28/2019 4:00 PM 925357 bars added in 62 chunks
URTY 466571 bars 6/28/2019 4:00 PM 466571 bars added in 59 chunks

Updating 15 Minute data ...
Symbols already up to date: IJR
.RUT 100765 bars 6/28/2019 4:00 PM 100765 bars added in 33 chunks
QQQ 98994 bars 6/28/2019 4:00 PM 98994 bars added in 32 chunks

Updating 3 Minute data ...
Symbols already up to date: QQQ,RUT,SPY,.DJI,DIA,EEM,FXI,IWM,IYR,NDX,SPX

Updating 30 Minute data ...
Symbols already up to date: .MID,.NDX,.SPX,SPY

Updating 5 Minute data ...
Symbols already up to date: NDX,QQQ,SPX,SPY

Updating Daily data ...

.....[additional messages removed]

Provider update complete for Fidelity Investments
----------------

Update completed (17 min 48.579 sec)
----------------

All the folders and files within the 1 minute folder are now well-formed. Strangely, it did not update SPY or .RUT 1 min, even though they are in the data set.

There must be a properties file (XML?)somewhere outside of the FidelityStaticProvider folder that lists these symbols, and could be involved with this problem.

I noticed that my 10 minute folder was missing, so I loaded 3 minute QQQ into a chart and changed the Scale. After a brief delay, it displayed 10 min data, and in the process created the 10 minute>QQQ>QQQ.WL folders and files. So it seems that transplanting the QQQ 1 minute data file kickstarted a process that rebuilt a data file or files that were causing the problem.

I am going to keep a backup copy of FidelityStaticProvider and the next time I have this problem I will restore that folder with an older (known-working) backup, and see if that fixes that problem.

I suspect there is a more direct way to accomplish what I did, but I haven't puzzled it out yet....

Thanks for the detailed feedback. Cone should know whether the issue is fixed already for the upcoming WLP build or not.

QUOTE:
This appeared to bootstrap an update for QQQ...

I'd say that corrupt is either the QQQ.wl file or the file for symbol like SQQQ with recent reverse splits. Your observation seems to be in line with my post #2 above so far.

QUOTE:
This appeared to bootstrap an update for QQQ (data was about a year out of date, but also a complete update for a number of symbols that were not in the folder:

By virtue of "Also update symbols ... not contained in DataSets" option being selected.

QUOTE:
There must be a properties file (XML?)somewhere outside of the FidelityStaticProvider folder that lists these symbols, and could be involved with this problem.

No, I doubt it exists (unlike Wealth-Data which does rely on one, for example).

QUOTE:
I noticed that my 10 minute folder was missing, so I loaded 3 minute QQQ into a chart and changed the Scale. After a brief delay, it displayed 10 min data, and in the process created the 10 minute>QQQ>QQQ.WL folders and files.

By virtue of having On Demand Update enabled.

QUOTE:
So it seems that transplanting the QQQ 1 minute data file kickstarted a process that rebuilt a data file or files that were causing the problem.

The old QQQ file may be the primary suspect.

This occurred twice more this morning when doing extensive backtesting on 1 minute QQQ. I tried these times to be more surgical:

- Deleting QQQ and Q folder didn't fix.
- Deleting 1 minute folder didn't fix
- Replacing QQQ.WL with backup didn't fix
- Deleting all SQQQ symbols didn't fix

The only thing that worked (both times) was replacing entirecontents of FidelityStaticProvider folder with contents from my backup. Brute force / shotgun approach.

I wasn't able able to identify a single item or folder that would do the trick.

This failed again with QQQ 1 minute. I was only able to repair it [i]partially [/I ]by restoring.

New problem: QQQ 1 minute data only goes back to 12/16.

Updating only adds one bar.

Updating Provider Fidelity Investments ...
Updating 1 Minute data ...
QQQ 244320 bars 7/1/2019 2:29 PM 1 bars added
.MID 1503830 bars 7/1/2019 2:29 PM 5 bars added
.NDX 1512362 bars 7/1/2019 2:29 PM 5 bars added
.SPX 1510004 bars 7/1/2019 2:29 PM 5 bars added
SMDD 80516 bars 7/1/2019 1:16 PM 0 bars added
SPXU 944984 bars 7/1/2019 2:29 PM 5 bars added
SQQQ 837722 bars 7/1/2019 2:29 PM 5 bars added
SRTY 675015 bars 7/1/2019 2:29 PM 4 bars added
TQQQ 845367 bars 7/1/2019 2:29 PM 5 bars added
UMDD 95104 bars 7/1/2019 11:21 AM 0 bars added
UPRO 925655 bars 7/1/2019 2:29 PM 5 bars added

Reload Chart Data doesn't fix it either.

I went to my other computer and ran WL. QQQ 1 minute shows the same problem. So now I'm wondering if this new data shortage problem is a backend Fidelity feed issue that I possibly had not noticed before.

Are you able to see 10 years of 1 minute QQQ data if you add it as a new symbol?

While there is a slight chance that you may be hitting a "bad" server with QQQ data, the QQQ Bars.Count (which had been copied from your other PC) has confused me. There's too few bars like if you took the file in the wrong bar scale or something. Certainly it doesn't have 10 years of data on neither one of your PCs, more like 2. If you get more it's on demand update in action.

Anyway, for this discussion to stay productive please revisit my post #6 above and proceed as suggested. Keep a backup, take just the Strategies subfolder into the refreshed Data folder and keep the bar scales and symbols to the minimum possible now, like 1-2. Let this be the starting point for troubleshooting.

QUOTE:
While there is a slight chance that you may be hitting a "bad" server with QQQ data, the QQQ Bars.Count (which had been copied from your other PC) has confused me.

It confuses me too. I am thinking that I may have had this data shortage problem for 1 minute QQQ for a while but never noticed it before.

QUOTE:
Anyway, for this discussion to stay productive please revisit my post #6 above and proceed as suggested.

OK. Since the identical problem is happening on two different PC's, I wouldn't expect the problem to be located on the PC. But I'll try.

I decided to execute your instructions in post #6 on my second (backup) computer. On that computer I successfully created new QQQ 1 minute that goes back 10 years.

I then copied this QQQ.WL to main computer in 1 minute>QQQ folder. It worked.

For now I am up and running again. Thanks for sticking with me on this.

Glad to hear.

I continue to have problems. If I haven't been clear, the problem appears mainly to be that new symbols (or scales for existing Fidelity symbols) are not created on request. WL will create a new "letter" folder if it doesn't exist, but not the .WL file inside the folder. I immediately displays Data Not Available, instead of loading the symbol.

I decided to do your recommended process in #6 above on my main computer. I had the same problem after creation of a new WealthLabPro folder and before doing anything else. Then I noticed some other things I hadn't noticed before:

1. The same problem exists with Yahoo data (Daily) as it does with Fidelity data
2. It appears to be specific to certain symbols

For example, I was able to add the following symbols to both Yahoo (Daily) and Fidelity (3 min): data sets:

XLB, XLC, XLI, XLRE, XLY

I was not able to add the following symbols on either Yahoo or Fidelity:

XLR, XLT

In every failure case it does not create the .WL file in the "X" folder, and has nothing to update, but will create an "X" folder if none exists.

I tried this on my backup computer and (thankfully) it reproduced the same results as above.

Can you confirm that you can replicate this?

Not seeing any issue here. Neither XLR nor XLT are recognized as valid symbols by Yahoo Finance website. If once they had been and then stopped trading, one cannot get the data for delisted symbols from Yahoo. (Perhaps from Fidelity too, not sure).

Sorry, I failed to run one more test: charting or quoting the symbol to see it's still real. Its been challenging to sort out all the problems I've been having, apparently some self-inflicted. This one was a false alarm. Thanks for checking.

I am getting run-time errors backtesting index symbol .IXR (3 min) when Data Range is extended to 9 years or more.

CODE:
Please log in to see this code.

I found the source of the problem is corruption with 3 min bars that starts at 4 PM on 4/11/2011 after which it replicates data from 10 AM that day, and the next several days data are corrupted (based on non-strategy chart) until it recovers on 4/19/2011.

While I was preparing this topic and ran the symbol again to check my post accuracy, I got the notorious No Data Available message and the symbol disappeared (deleted by WL?) in the 3 min folder. As typical, I can't reinstall the symbol.

This is one of the symptoms I've been seeing (disappearing symbols) in the problem reports above.

Q1. To recover this symbol, do I have to delete the entire 3 min folder first (this symbol does not sit inside a letter folder)?

Q2. If I can recover the symbol, is there a way to repair the symbol data?

1. Is on demand update enabled?

2. When updating the data which becomes corrupt, what tool are you using:
[ ] "Update DataSet"
[ ] "Update DataSet (Price)"
[ ] on demand data update (using a chart)
[ ] "Update all data for selected providers now"

Check all that apply.

3. What Fidelity fundamental data providers are enabled?

4. Finally, please attach the corrupt .IXR.WL file from the 3 min folder. I'd like to peek inside it as I expect it to be corrupt in some known way. Other customers reported corrupt Fidelity data and your current situation suggests that this scenario (post #2 above) may be at play.

Thanks.

QUOTE:
1. Is on demand update enabled?

Sometimes yes, sometimes, no. Doesn't seem to matter.

QUOTE:
Check all that apply.

[ ] "Update DataSet"
[ ] "Update DataSet (Price)"
[X ] on demand data update (using a chart)
[ ] "Update all data for selected providers now"

QUOTE:
3. What Fidelity fundamental data providers are enabled?

None that I am aware of.

QUOTE:
4. Finally, please attach the corrupt .IXR.WL file from the 3 min folder.

Attached.

I got an error "bad file extension" when I attempted to upload the file. I see it didn't upload. I have changed the file extension to .TXT and trying again. Please change the file extension back to .WL.

---

Looks like that didn't work either.

I will attempt to zip it and send again.

Thanks. I would've thought you're using fundamental data but it's still interesting to take a look inside the file.

The list of allowed file types includes: *.jpg, *.gif, *.png, *.zip, *.rar, *.pdf, *.txt. However, the file size should be under 3 Megabytes. If still no go please zip and attach it when replying to our email:

CODE:
Please log in to see this code.

Sent by email.

Received, loaded the data. It's not corrupt. The symbol can be charted in WL. I cannot duplicate the issue.

You may disregard what I said in post #19 as my concern doesn't apply to your case.

P.S.

QUOTE:
If I haven't been clear, the problem appears mainly to be that new symbols (or scales for existing Fidelity symbols) are not created on request. WL will create a new "letter" folder if it doesn't exist, but not the .WL file inside the folder. I immediately displays Data Not Available, instead of loading the symbol.

Restart WL prior to updating data in a new bar scale.

You don't see problem with charting starting around 4/11/2011 with 3 min bars? I see it with chart on two different computers. What's going on?

I didn't say so. A data glitch is out of scope of this conversation and MS123's support mandate. Data quirks happen every time with every data source, Fidelity not being the exception.

There are at least 4 bars with a wrong Low value in April 2011 alone. And there's a bad bar (OHLC = 0) at 04/12/2011 09:33am, removing (or correcting) which should get rid of the "Basis price..." runtime error. If it's reproducible after Reload Chart History, you might want to report it to Fidelity.

But it's out of context for "Fidelity static provider doesn't update correctly #2" and can't make the data "disappear" as you described and I can't reproduce.

Understood. Can you point me to info on how I would go in and change / correct the data? Is there a WL tool for that?

It's explained in the Wealth-Lab User Guide > Charting > Chart Context Menu > Edit Bar Data...

That worked. Thanks.

If I have On Demand Data Updates checked, is it likely to overwrite the fixes I made with the original bad data, if I open this symbol and timeframe in a chart or strategy? Do you recommend leaving that option unchecked, in general?

It should not overwrite your edits with the data in the distant past.