IQFeed provider runs backtest slower than Fidelity provider (Resolved)
Author: kazuna
Creation Date: 7/20/2020 11:22 PM
profile picture

kazuna

#1
8 Symbols / 1-Minute / 1 Year
IQFeed: 4517 ms
Fidelity: 1755 ms

8 Symbols / 1-Minute / 6 Years
IQFeed: 15670 ms
Fidelity: 10155 ms

26 Symbols / 1-Minute / 12 Years
IQFeed: 100078 ms
Fidelity: 59986 ms

Approximately IQFeed data runs 1.5x to 2.5x slower.

Does it make sense?
profile picture

Eugene

#2
This isn't data collection / update since you mention a date period. What does "runs" mean - is this a backtest or a Strategy Monitor run? How many runs did you make? One or two is not enough.

A. Backtest:

Have you disabled on demand data update? This greatly affects the backtest speed. Otherwise no, there will be no sensible difference as both static providers read their data the same way from .WL files using the same interface.

B. Strategy Monitor:

SM makes data update requests. Fidelity's server may be located 'closer' to you from a connectivity point of view.
profile picture

kazuna

#3
This is backtest. I run 10 times each to find the best time.
Yes, I disabled on demand data update all the tme.
I run the same strategy code and then just click the data set created with either Fidelity or IQFeed on the same WLP.
IQFeed data provider runs consistently slower than Fidelity.
profile picture

Eugene

#4
True. After thinking to myself "this may be external symbol calls" or "have you restarted WLD after disabling on demand update", I believe the answer is: the Market Manager. The Fidelity static provider does not support it therefore it appears faster. Running the data through the MM imposes a performance overhead. You can disable it for the IQFeed provider or for some group of symbols as suggested in the article. The speed will be on par with Fidelity's but you'll lose MM's features like filtering out premarket/aftermarket data etc.
profile picture

kazuna

#5
I have further tested some more scenarios and I concluded that the performance issue with IQFeed is due to the extra bars coming from the extended hours trades. Those symbols having more extended hours trades like SPY regressed the preformance significantly (order of 2-3 times) in certain time range backtesting.

Market Manager can get rid of the extended hours trades, however, unfortunately Market Manager introduces even more overhead than the performance gain by removing the extended hours trades from backtesting.

I think it is essential to consider providing a filter to remove the extended hours trades in the IQFeed data provider.
profile picture

Cone

#6
Closure for customers searching this topic...

We created an IQFeed option in the Data Manager to request only regular session data. Furthermore, when the option is checked, it completely bypasses calls to the Market Manager. Also, another optimization was implemented for v.2020.09.11 in which multiple simultaneous calls for the same data would all be served by a single request.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).