This release has many changes, bug fixes and new features. For my testing, it appears to be stable and works well, however, I'd expect to see some issues when more eyes are on it. Please reply to this thread if you find something!

For the options see Data Manager > IQFeed tab

1. Upgraded all sockets to Protocol 6.1 for large bandwidth savings.  

2. Regular session option now implements a dynamic Fieldset that only includes qualified trades

3. Implemented a trades-only watch so that changes in bid/ask do not create Quote events.  Bid/Ask are still included with each trade quote.

4. Split Processing
Turn on the option and refresh your IQFeed intraday histories! Fidelity intraday histories, should be compatible with the option selected.
- ImplementedOption for Intraday adjustment in settings (requires refresh)
- Added an embedded resource to extend split histories. Since only the 2 most-recent splits are available from IQFeed, the resource extends split lookup to apply to intraday histories.

Side note:
The Provider will now automatically split intraday and Daily histories only if IQFeed reports the split in a timely manner on the ex-date.

5. SecurityName fixed!
The correct company name will be updated with the data file during Data Manager updates

6. Partial Bar option to load Partial bar for Data Manager and On-demand historical requests

7. GetSessionOpen() is implemented.  
After the market open, calling this method will quickly cache the market opening price for all symbols in the DataSet. Returns 0 if the market has not yet opened or if symbol invalid. For weekend testing, GetSessionOpen() updates the DataSet and returns the opening price of the last bar, consequently, on a weekend this *slow* by design.

8. Wealth-Lab hourly bar convention.
Added option to timestamp 60-minute bars after the first hour of trading, e.g., 10:30, 11:30, ..., 16:00.  This option is required when using 60-minute bars in the Strategy Monitor.

9. Paper Broker support
Added GetQuote() and RequestHistoricalData() methods required by the Paper Broker Provider.  

10. TradeTicket Snapshot
You can now get a snapshot quote from the Trade Ticket

11. Fixed a potential invalid operation exception when canceling a Data Manager update

12. Fundamental lookup partially implemented. More on this in the next build.

Look for the upgrade in Extensions tonight, and let me know how it goes!

Impressive work Robert! Major upgrade in years.

Working on a bug to handle invalid or unauthorized symbols more gracefully.
Additionally, and I'm not yet sure if it's related, but some download threads are getting "lost".

=======
Edit: This was the problem.. Build 2020.08.26 is now available.

Re: Item 4: Split Processing

The split resource history contains split adjustments for special dividends for S&P 500 and Nasdaq 100 stocks. Since IQFeed doesn't adjust Daily data for special dividends, we'll remove those for intraday processing in the next build. That will require a refresh for some stocks - potentially affects 126 symbols.

This update appears to be very slow. I'm fetching 800 Daily bars from disk cache, and it can take the chart 3-5 seconds to load. It's like it's trying to fetch a lot more than the most recent 800 Daily bars from cache. Strange.

Other than that slowness, it seems to work fine. Thanks for fixing the security name.

Thanks so much for the terrific work! Now that you are free of the cumbersome Fidelity update approval process you are back to agile development. It will be better for all of us.

SecurityNames are working, Thank you. It is a little slower and looks like you are waiting for the SecurityName to come through before updating the screen. I have a Workplace with four different timeframe strategies on the same symbol and can see them come up as the security name is downloaded. You probably could catch the SecurityName the first time you get it and use that in this situation. But please push on IQFeed to improve their response. I responded to IQFeeds nice welcome emails offering support telling them of these problems. When I didn't get a reply, I asked again. I still didn't get a reply. You will probably have better results getting them to respond. However, the IQFeed data IS a definite improvement over Fidelity in quality and depth.

I REALLY APPRECIATE your terrific response on this and am delighted that we're working directly with you now.

I'll look into the reported slowness right away. The SecurityNames are saved with the data file during a DataSet or IQFeed Provider update in the Data Manager. That can't be slowing anything down during backtesting.

@superticker, just to be sure, you're not calling GetSessionOpen(), right?
It could take a few seconds for a DataSet the first time it's called, but all opening prices are saved at that point and it's lightning fast after that.

In general, I need more to go on because data is loading "instantaneously" from cache. On-demand requests, of course, have a round-trip and other processing involved - but even those generally finish within 1 second for me.

Cone,

I think the slowness is because I'm getting the Windows FEATURE update. It's taking all the resources. I didn't know that when I wrote. So stay tuned on the slowness, it may not be a problem.

The Windows "feature" update finally paused. It looks like the speed hasn't changed. IQFeed is a little slower than Fidelity, but not significantly. There was NO change from the IQFeed beta update. It was a coincidence that both updates happened at the same time. It was unfortunate timing, but happy ending. Sorry for the confusion, thanks again.

QUOTE:
@superticker, just to be sure, you're not calling GetSessionOpen(), right?

No, I'm not using GetSessionOpen(). And it has been been slow with a "plain" Chart load. But the behavior is also intermittent; I don't know why. I'll report back if I have more information.

My "guess" is that it's not loading from the disk cache when it should be.

UPDATE: Reproducing this slowness bug turns out to be complicated. It involves linked streaming charts, one chart in static cache (Daily) and the other not (non-Daily), and a GetExternalSymbol("CCP,s","SPX.XO",true) call which throws a threads abort error message. Let me develop screenshots of the problem and I'll post them. Until then, I wouldn't try to reproduce it since the steps to do so aren't clear.

Maybe File > Data On Demand is enabled?

You could leave On Demand enabled always for Fidelity and Daily bars would not be requested until the end of the next session. This actually had a bad effect for the reasons cited here:

Just know that IQFeed will always request a refresh of the last several bars if On Demand is enabled. You can remove all suspicions of not loading from cache if you turn off On Demand.

QUOTE:
ImplementedOption for Intraday adjustment in settings (requires refresh)

1. What do you mean by "refresh" in the above?
2. When I click on "Clear IQFeed Data", does it wipe out everything within the "IQFeedStaticProvider" folder?

All the tests so far with the previous release look great. Thank you.

Cone,

Would it is possible to have the "Split Adjust Intraday Data" option box CHECKED by default? I just install the latest release and forgot to check the box and have to do clear of all the data, copy over the Fidelity data and run the data update again. It's no big deal (and is not an issue when the release become more stable and this concern might not apply to everyone), what do you think? Thanks.

QUOTE:
1. What do you mean by "refresh" in the above?

Refresh is the same as "delete the data file cache and update everything" to start over. In the case that you're preserving Fidelity or other intraday history, copy the 1, 3, 5... Minute folders to the ..\Data\IQFeedStaticProvider\ folder.

QUOTE:
2. When I click on "Clear IQFeed Data", does it wipe out everything within the "IQFeedStaticProvider" folder?

Everything except the IQClientSettings.xml file (see below).

QUOTE:
All the tests so far with the previous release look great. Thank you.

Awesome!

QUOTE:
Would it is possible to have the "Split Adjust Intraday Data" option box CHECKED by default?

The reason it's not there by default is that there are already customers using it without that option. You only have to set it once, and then never again.

But keep in mind that this setting is stored here, so don't delete this file!
C:\Users\windows_login\AppData\Roaming\Fidelity Investments\WealthLabDev\1.0.0.0\Data\IQFeedStaticProvider\IQClientSettings.xml

This IQFeed slowness problem is allusive to reproduce. But it stems from the WL static-cached Daily chart using the Internet to download the chart (and its external symbol, SPX.XO) whenever the chart is in streaming mode. There's never a slowness problem when the chart is not streaming.

I think part of the reason it's hard to reproduce is because the IQFeed client caches the data "some of the time" masking the problem of WL failing to employ its own static cache to fetch the data. This makes troubleshooting the problem very difficult unless the IQFeed client cache can somehow be disabled.

That said, I wonder if it would be better to fix the thread abort problem first (see error message in screenshot), and put the slowness problem on hold until then. I don't know if they are related, but they might be.

To reproduce the slowness problem:
1) Setup two charts, one in static cache (Daily) and other not cached.
2) Make both charts streaming. The failing-to-use-the-WL-static-cache issue will not occur with non-streaming charts.
3) Employ GetExternalSymbol("CCP,s","SPX.XO",true) so an external symbol is also being fetched in both charts. Dismiss the error dialog box, which should only occur once. This error has always occurred with all previous versions of IQFeed--it's "normal".
4) Link the non-Daily, non-static-cached chart to the Daily one.


I was told GetExternalSymbol("CCP,s","SPX.XO",true) caches its result, but that may not be the case here.

Okay, thanks for the details. For sure, when you turn on Streaming, the first thing that's done is a request for recent history - 2 days before the last bar's date. You'll get a partial bar during the market session and streaming picks it up from there. GetExternalSymbol() will also require an on-demand request in a Streaming chart. So, the first request is to load the chart, then the code starts executing, and GetExternalSymbol() hits it with another on-demand request. That's 2 round trips, and totally expected when you start streaming. After that and during streaming, it's one round trip during code execution to update the external symbol.

So as not to required an on-demand request and update the "External Symbol" as quickly as possible, use this technique... Go here and use Workaround #3.

I'll see what I can do about the thread abort.

QUOTE:
GetExternalSymbol() will also require an on-demand request in a Streaming chart. So, the first request is to load the chart, then the code starts executing, and GetExternalSymbol() hits it with another on-demand request. That's 2 round trips, and totally expected when you start streaming.

Thanks for the explanation. I was told GetExternalSymbol() caches the symbol, but I don't think that caching is working in this case, which may be part of this slowness problem.

This might be a good time to introduce an unrelated on-demand error. I don't think it has anything to do with the slowness problem discussed in Post# 15, but I could be wrong. This error is common with all versions of the IQFeed provider, doesn't appear to affect the execution of the strategy, and rarely occurs if the On-demand Updates checkbox is unchecked.

I couldn't make this happen even once when trying yesterday... so, it's one of those that can't be fixed easily. I'll try as time permits. Meanwhile, you'll get MUCH better script performance using the workaround #3 mentioned above.

Aside...

Intraday and Split Adjustments
This requires more work. I just discovered that the IQFeed Split factor, which is favored for splitting intraday and historical data, is only precise to 2 decimals. That doesn't affect 2:1 (0.50), 4:1 splits (0.25), or 5:1 (0.20), but it affects common splits like 1.5:1 and 3:1. Example: a 3:1 split is given as 0.33 instead of 0.333333333, 7:1 is 0.14 instead of 0.142857143 (see AAPL 6/9/2014). I'm going to see if IQFeed can increase the precision.

QUOTE:
I couldn't make this happen even once when trying yesterday... so, it's one of those that can't be fixed easily.

Are you trying to say a race condition is involved because of a misplaced blocking Wait() that terminates a critical section for an async I/O call? If so, that's terrible news because that means I wouldn't be able to reproduce it under the debugger either since timing will be different. If that's the case, I wouldn't even try to the reproduce the problem. That's like trying to reproduce the Stuck Submitted problem on a different computer--it can't be done.

Can you look at the in-progress async I/O critical sections and see if the terminating Wait() [i.e. guard] looks like it's in the right place? This really needs to be fixed by inspection, and not with a debugger. You can't fix a race condition with a debugger.

Tell me more about this error message: WaitAll for multiple handles (I assume they mean Signals from multiple "ongoing" async I/Os) on a STA thread is not supported. Are you trying to use one Wait() to block multiple async I/Os as it implies? Perhaps DTN wants you manage all these Wait()'s separately?

Sounds like you need to call DTN and ask them for a code sample. Perhaps they expect you to queue up all the ongoing Signals (for each ongoing async I/O) and search the queue for identifying ones when you reach the blocking Waiting point. A code sample should confirm that. It does make sense to "pair" each Signal (i.e. async I/O) with it's own individual blocking Wait() call. I think that requirement is reasonable because each async I/O will complete at a different time and need a separate Wait() to proceed forward out of its critical section.

This is a little like the Stuck Submitted problem where critical sections weren't properly protected with carefully placed blocking Wait()'s, which lead to deadlock (actually livelock) between two async tasks (client & server handshaking).

UPDATE: I couldn't make this error message occur over the weekend, which is weird because it occurs all the time just before the market opens. There's another subtle condition requirement, but I don't know what that is. Let's put this problem temporarily on hold until I get back to you, but I would call DTN and ask them about these two error messages. I "think" caching of the external symbol might be involved, and certainly caching would change the "timing" for any race condition (assuming one is involved).

QUOTE:
you'll get MUCH better script performance using the workaround #3 mentioned above.

The problem there is DataSeries caching is failing on the external symbol. If caching were working, there would only be a slowdown on the first symbol rather than every symbol repeatably. Let me research this more, and post a new topic about DataSeries caching if there's a need.

For IQFeed v2020.09.02.0, when a DataSet name is included in the GetExternalSymbol("DataSet",Bars.Symbol,true) call, it returns bogus results for on-demand (intraday) data. It works as expected when the DataSet name is omitted or static (Daily) data is employed. See below.



Here's the code to reproduce this problem.

CODE:
Please log in to see this code.

Now you're probably wondering if this is related to the thread abort problem discussed above in Posts# 15 and 17. Honestly, I'm not sure. But the thread abort problem is definitely non-deterministic such that I can't reproduce it immediately after it occurs. That leads me to think it's a race condition created by an asynchronous I/O scheduling problem. That being said, the thread-abort error messages are an attempt to break the deadlock (really a livelock) between these two threads competing for the same resource. But I would call DTN to confirm this.

At any rate, fixing the GetExternalSymbol() call has higher priority for me over the thread abort problem.

I don't see a problem. If you don't specify a DataSet name, GetExternalSymbol gives you the data from the first DataSet, alphabetically, that it finds. Clearly, this is a DAILY DataSet in your example. If you want an external series that can scale to 30-minute bars, then make sure you specify a DataSet that has 1, 2, 3, 5, 10, 15, or 30 minute bars in it.

As for the thread abort, it has nothing to do with the IQFeed client. And even though I still can't make that error happen I've marked the Download history method with an MTAThread attribute for the next update (which will sometime tomorrow after I do some testing with the new Streaming Bars for the S. Monitor). Let's see if that helps.

QUOTE:
Clearly, this is a DAILY DataSet in your example. If you want an external series that can scale to 30-minute bars, then make sure you specify a DataSet that has 1, 2, 3, 5, 10, 15, or 30 minute bars in it.

The only datasets I store (on disk) are Daily datasets. For intraday data, I employ on-demand data. This is where the GetExternalSymbol() fails if the dataset name is included for any time scale that's not Daily or faster. Are you saying you can't reproduce this problem on your machine?

If the dataset name is excluded, GetExternalSymbol() works fine for all scales; 1,3,5,10,15,30-minute. So this is an on-demand problem only if the dataset name is included. I'm not sure why that is. Do you know?

The thread abort only occurs when GetExternalSymbol() is used. It sounds like the IQFeed client is throwing these abort errors. Are you saying WaitAll is part of the .NET framework instead? So this is a problem with the core WL code?

These strategies are four years old, and I never had the thread abort problem with Fidelity data, but I was calling ExternalSymbol.Series() instead of GetExternalSymbol(). The former is just too slow with IQFeed data, but worked fine with Fidelity data. I don't know why.