IQFeed Provider v2020.09.11 is ready for some user testing.

Strategy Monitor will now use Streaming Bars from IQFeed!

Notes!
1. During our acceptance testing we noticed that Wealth-Lab works much better if the Monitor thread(s) start before the end of interval data arrives. It's important to at least synchronize with internet time, but it's even better to advance your local clock by 1 or 2 seconds to get the Monitor threads started early.

2. When you first open the Strategy Monitor, if your local clock is lagging IQFeed time, a message box will notify you of the condition:


Faster data loading:
o Don't always load "All Data" from the DataStore
o Disabled call to MarketManager when Regular Session enabled
o Simultaneous calls for the same symbol-scale-interval are blocked until the first returns at which time the cache is used for all other calls.

Heartbeat Restored
o Streaming charts will close at end of interval without requiring a tick in the next interval.

Potential Fix
o Marked IQFeedClient.DownloadHistory method with the MTAThread attribute in an attempt to prevent an intermittent error when using GetExternalSeries.

Thank you for the update.

I just tried it out but as you didn't mention it in your notes, the Form T trade issue remains.
However, apparently the backtesting runs much faster with this new version like it used to run with Fidelity data provider!

=== Form T trade issue ===

SOXX @ 2020/07/17 09/31

IQFeed - 1 Minute
O:280.56 <=== This should be 280.33.
H:281.00
L:280.31
C:280.64

IQFeed - Daily
O:280.33

Fidelity - 1 Minute
O:280.33
H:281.00
L:280.31
C:280.64

Fidelity - Daily
O:280.33

From the IQFeed Wiki:

In summary, for the Opening Price replacement feature to work, the following conditions must exist:

1. Regular Session option must be checked
2. Split Adjust Intraday option should be must be checked (or you're guaranteed to get some crazy intraday opening prices)
3. Daily bars must be updated first. This will occur automatically if you perform an IQFeed Provider update. You must have a Daily symbol that corresponds to each intraday symbol!
4. Corrections will be applied only to intraday data that comes in on new download requests.

Refresh you chart. The cache will not be adjusted!

>4. Corrections will be applied only to intraday data that comes in on new download requests.
Yes, I did delete IQFeedStaticProvider folder before starting WLD, then updated the data.
I see Daily bars are updated first before the Intraday bars.

BUT I still get 280.56 open price at 9:31am on 2020/07/17 for SOXX chart when using 1-minute scale data!

QUOTE:
o Simultaneous calls for the same symbol-scale-interval are blocked until the first returns at which time the cache is used for all other calls.

So you're saying if I call GetExternalSymbol("SPX.XO",true) several times (for different stocks), the provider will only download it once (until the next bar update)? Gee, that would be really nice. I'll check it out.

Impressive work Robert!

QUOTE:
if I call GetExternalSymbol("SPX.XO",true) several times... the provider will only download it once...

There will only be precisely one live request for a symbol-interval at any given time. If requests for the same symbol-interval come in before the first one comes back, all the rest wait and get the result from cache when the first returns.

Therefore, it's possible that if you're running 30 symbols with GetExternalSymbol("ABC"), maybe 10 will queue for the first request, and then 17 for a second request, and 3 for a third request. It's a matter of timing.

There could be room for more optimization for on-demand requests that have the partial bar option false. In this case, we could find the date of the last bar in the cache compare it to the current time and determine if additional bars might be available before sending a request for them. I can add that to the ToDo list.

Let me know how it goes with the current change.

@kazuna
It sounds like you're doing the right thing. I didn't re-look at it for this build, but the opening price for SOXX in my intraday data for that date is 280.33. Anyway, I'll give it another look.

===
I just performed on-demand request for SOXX in 5 and 30 minute intervals that I hadn't downloaded. In both cases, the open price for the first bar on 7/17 is 280.33

1. Please check the Extensions Manager and make sure that you have version 2020.09.11

2. If you do, check the Data Manager > IQFeed. Is the Regular Session Option checked?

3. If it is, pull up an intraday SOXX chart, right click and Reload Chart History. Did the result change?

4. If not, create a DataSet with a different interval, like 30 minute bars. Update the DataSet. Result?

Hmm. No matter how many time I wipe the data and reload the dataset, I get 280.56 open price from 1 minute scale data on any scale chart window (1, 3, 5, 15, 60-minutes and daily).

Of course if you use the same data for different scales, you'll get the same answer. #4: Create a new DataSet with a different scale.

What are the answers to my questions? I asked them for good reason.

OK. I just figured it out by toggling options and data scales.

>2. Split Adjust Intraday option should be checked (or you're guaranteed to get some crazy intraday opening prices)

I had to enbale this option, not because of the crazy intraday opening prices but for the Form T trade issue to get fixed.

Does it make sense?

I have imlemented my own split adjustor to calculate the entry price based on the intra-day opening price and the daily ATR, so I didn't need the split adjust option but apparently it is required for the Form T trade issue for some reasons. My understanding was that the split adjust option has nothing to do with the Form T trade.

By the way, I found an issue with your split adjustor.

XLF Applying split: factor 0.812347684809098, Ex-date: 2016-09-19
XLF Splitting result bars: factor 1231, Ex-date: 2010-10-12

XLF took 1231 for 1000 split on 2016-09-19 which looks good.
However, the 1 for 1231 split on 2010-10-12 is wrong.

Apparently it is coming from SplitsDetected.txt and IQFundamentals.csv as follows.

XLF,1231 10/12/2010,09/12/2020 162502

20200912 093702,XLF,7,,57698000,31.3800,17.4900,31.3800,17.4900,,,,,,,,,,FINANCIAL SELECT SECTOR SPDR,XLF,,1.20,,186563.0,,,,744245,1000.00 09/19/2016,1231.00 10/12/2010,14,4,,18.31,1,11,02/12/2020,03/23/2020,02/12/2020,03/23/2020,30.7800,,,,,0,,,,,,USD,,,,,BBG000BJ29X7,3,

Is it IQFeed side problem?

Now I have to enable the split adjust option for the Form T trade issue but so that it actually breaks the stock split.

Could you take a look it, please?

One more issue with IQFeed Provider.

While I was trying various option combinations, I noticed IQClientSettings.xml not updated very often regardless whether I open and close Data Manager window, restart WLD etc...

I had to manually change IQClientSettings.xml in order to preserve the option settings.

QUOTE:
Does it make sense?

It does. Documentation corrected from "should be selected" to "must be selected". Since Daily data are always split-adjusted, it's clear that we shouldn't use Daily data to correct unadjusted Intraday prices.

Re: XLF
IQFeed problem - yes. This one's bizarre, because I just added XLF to a 60-min DataSet, updated and got:

XLF Applying split: factor 0.812347684809098, Ex-date: 2016-09-19
XLF Splitting result bars: factor 1231, Ex-date: 2002-07-01
[1] XLF 23393 bars 09.11.20 16:00 23393 bars added

Indeed, the date in my files was 7/1/2002, not 10/12/2010! Since the IQFeed intraday series only starts in 2007, this wasn't a problem in my case, but that split doesn't actually exist and it seems like there's some random date coming in with it in their fundamental message. I'll mention to them.

Note!
The only reason the first split worked is because our embedded resource had that data. If not, we would have split XLF 1:1000 on 9/19/2016 per the IQFeed fundamental record. This is all clearly in error because in reality the split in 2016 was a special dividend adjustment for $4.44356 per share (according to Yahoo!) which calculates to a split factor of 1/1.23167675, which is approximately the 0.812 factor we're using.

Meanwhile, try refreshing the intraday chart and see if it changes for you.

QUOTE:
I had to manually change IQClientSettings.xml in order to preserve the option settings.

Thank you. I'm not sure what happened there, but I'll find out and fix it right away!

Edit:
It seems okay here. Changes are saved in a Settings class, but they're serialized and saved to the xml file only upon exiting Weath-Lab. It might not be a good idea to have that file open for editing when we try to write it ;)

IQFeed fixed the XLF Split data. It should be fine after refreshing now. To be sure the update comes in, close and restart Wealth-Lab to shut down the IQFeed client first.

The split issue with XLF has been fixed. Awesome!

XLF Applying split: factor 0.812347684809098, Ex-date: 2016-09-19

20200914 141303,XLF,7,,57698000,31.3800,17.4900,31.3800,17.4900,,,,,,,,,,FINANCIAL SELECT SECTOR SPDR,XLF,,1.20,,186563.0,,,,744245,0.81 09/19/2016,,14,4,,18.31,1,11,02/12/2020,03/23/2020,02/12/2020,03/23/2020,30.7800,,,,,0,,,,,,USD,,,,,BBG000BJ29X7,3,

However, the issue with IQClientSettings.xml remains. Apparently the setting isn't saved to the xml file more often when WLD is closed while there are chart windows remain opened. It may suggest that it is a timing issue when WLD is closing and saving the files.

It's a pretty simple "Save" in the Application_ApplicationExit event. Anyway, we'll save the settings to disk after every control selection event now (next update).

QUOTE:
we'll save the settings to disk after every control selection event now (next update)

That sounds promising. Thanks!

QUOTE:
Disabled call to MarketManager when Regular Session enabled

@Cone, is the above means that during trading hours, the checked box to IQFeed within Providers window of Market Manger should be unchecked? Please see attached. Or it is something that you already implemented internally and that we don't have to do any or is it refers to something else?

The question is referred to one of the statements under Faster data loading: on post #1.

Thanks.

The change was internal so as not to even call the Market Manager at all if the Regular Session Option is checked. Even though it doesn't change the result if the MM is disabled (like in your image), testing a boolean has to be a lot faster (1 msec?) than calling the Market Manager and having it do the check.

With regard to statement #1 of post #1, for both cases when clock is synchronized and when the clock is advanced by about 2 seconds, the difference in symbols updating speed in Strategy Monitor is not significantly noted. In either cases, the total time it takes to update 500 symbols at 5 minutes scale is about 2 minutes and (30 - 40) seconds. Can we increase the updating speed?

The update/execution cycle of Strategy Monitor has always been a disappointment to me. The problem isn't the data provider, it's the "internal workings" of the S. Monitor. Yes, I hope we can get to work on that in the near future.

Thank you Cone. Look forward to that.

Nothing [about Wealth-Lab] would please me more than to see the S. Monitor execute over a list of symbols as smoothly as it does in a simulation!

@Cone, I am relief that you see the SM needs improvement, since it the the component of WLD that where real live actions occur.

In some instances it takes almost 3 minutes (about 10 seconds shy) to update 500 symbols. I don't how the Strategy Monitor work but if the 3 minutes are just for updating and not included anything else (like program execution time, etc.,), then it implies the 500 symbols (maybe +/100 symbols) is the max limit of the dataset size that it can handle at 5 minute scale. It is observed that by the time, the message "time's up!" appears under the Action column within SM, there are about 10 seconds left before the next update cycle start. It was disappointed since my strategy is quite time sensitive at entry (than probably many others) and I was also looking to scale up the dataset size by a couple to few X in the future.

Perhaps, now with Fidelity is out of the picture, you and your team might have more flexibility to make WLD to become a better product. Thanks.