Hi,

80% of the time the Quotes window works as intended and at 9:30 the quotes start streaming and trades start triggering.

20% of the time it hits 9:30 AM and nothing happens and the quotes don't stream and then obviously trades don't trigger leading to missed opportunities.

I do the following before 9:30:

1) Log into Fidelity
2) Open the orders window and turn on live trading
3) Open my strategy, select all the alerts, and then press monitor in quotes window

If it fails, it's not kicking me out or asking me to log back in. I simply need to close the quotes window and reselect monitor quotes to open a new one which then starts streaming without a problem.

Any ideas on why sometimes the Quotes window fails? Sometimes the trades happen right at 9:30 and sometimes I can't be in front of a computer at that time. If I miss trades it's often painful because they are usually the profitable ones.

Thanks

Hi,

Thanks for the problem report. By chance do you remove some symbols from the Quotes tool? Just trying to see if there's any connection to a known issue:

QUOTE:
(163890) Quotes Stop After Removing Items. Streaming quotes in the Quotes tool may be halted inadvertantly after manually removing some items.

Or maybe there's any other pattern to those 20%?

I actually don't. I do the same thing except the only possible difference is the order of things. Like I may open the orders window after the quotes window. But I always log in to Fidelity first or I can't open a quotes window.

When did you start to experience this behavior?

It's not necessarily the problem, but make sure to do a fresh a restart each day before logging in.

Sorry for the delay. I was out of the country. It's happened from the first days I started using the quotes window. It happened again just today. Yes, I always do a fresh start.

It means I have to be at the computer at 930 and that isn't always practical and even if so I miss trades sometimes due to the really high volatility on the open.

Thanks for anything that can help with this.

Just a speculation but there may be a communication failure (or some other kind of issue at the backend) on a volatile market open. By chance if you noticed this on quiet days?

Today was very quiet.

I really really hope this gets fixed please. I just missed out on a lot of trades because it didn't start this morning while I was at work.

Could you please create a support ticket so this issue won't get overlooked accidentally?

QUOTE:
Or maybe there's any other pattern to those 20% [when the Quotes window fails to update]?

So next time and while this problem is occurring, go to http://www.speedtest.net/ and run an Internet speed test. We don't care about the bandwidth result, only the "ping time" is important here. There's a great deal of handshaking (interaction) between WL client and server for updating the Quotes window, so all we want to know is how long the latency is between the two. Bandwidth isn't that important.

I'm not suggesting long Internet latency is the problem. All I'm saying is long latency (and network congestion) is one factor triggering the problem. We just want to determine if long latency is an important factor here for tuning up the web (HTTP) server.

If this is a nondeterministic I/O problem, and I think you're suggesting it is, then there's a real-time scheduling problem (i.e. an asynchronous I/O problem) between client and server like we see with the "Stuck Submitted" problem. If that's actually true, then Internet latency will be a factor. These real-time asynchronous I/O problems are tricky to capture and debug.

---
On the WL client side, there should be a watchdog timer that aborts the I/O between client and server for the Quotes window when the handshake takes too long. And the abort should do a trace dump for the blocking I/O (or whatever operation is not responding) involved. This is necessary because the developer probably won't be able to reproduce the real-time problem on his own machine.

The developer can also have the exception handler attempt one retry to see if the Quotes window will come back to life. Sounds like there's an 80% chance it might.

I will check the next time it happens while I'm in front of the computer thank you.

How can I change the exception handler settings please?

The other thing that might help and that would actually be really useful is if when you open a quote window while the market is closed it still retrieved the quotes. Then if certain stocks are close to their triggers as a safety net you could manually trigger some trades before the market opened based on pre-market prices.

I created a support ticket.

Thanks

The next time it happens, check the status of your login. The only thing I can think of is that you're logging in more than 2 hours before the market open, and the login cookie is expiring.

If that's not it, my feeling is that there's probably not much you can do except to manage the issue when it pops up. fyi, an WLP update is in planning for next year to integrate a new streaming provider, among other upgrades. Maybe it will help, not sure!

It happened again today. Still logged into Fidelity but didn't start. Ping was 15ms

When did you log in to Fidelity today?

Sorry to hear this. I'm going to try to kick up some dust at Fidelity and see if they have any ideas.

Around 815 this AM. When I got back home for few min at around 1030 I was still logged in but the quotes window had not activated. Fortunately today it did not result in any lost trades.

Thanks

We're being told that a backend (server side) issue had existed (years ago) that was logging out customers after only 30 minutes. Fidelity created a work-around in WL Pro to keep the login alive behind the scenes. So, either that work-around isn't working for you, or, you don't have it.

Please check that you're using the latest WLP Version 6.9.19 by clicking Help > About Fidelity WL Pro ...
If you don't have 6.9.19, just click Help > Software Upgrade.

-------
p.s. Also we discovered that Fidelity logins (WL Pro, AT Pro, etc) are designed to keep apps alive for only 10 hours - sufficient for the market session. Indeed, I tested this and for sure if you log in at 3 am, for example, you will be logged out by 1:30 pm - in the middle of the market session (it was 10.5 hours for my test).

Hi,

I have the latest version. I wish it were that easy but as I mentioned sometimes this happens within 10 minutes of the market open and it isn't logging me out. I just close the quotes window and open a new one and it works right away without any logging in or out.

Any thing else I need to be checking?

Thanks

QUOTE:
Any thing else I need to be checking?

With reference to my post# 11, what is the network "ping time" at the very moment it fails to start? You have to catch it at that moment since ping times can vary.

Again, since this "appears" to be a nondeterministic behavior between two asynchronous machines (client and server), there's probably a race condition involved between the two machines, which is probably aggravated with long Internet latency.

Nobody is going to able to reproduce such a race condition unless they are on your Internet network. So the course of action is for the developer to write an error handler with a watchdog timer that does a trace dump to the error log when the watchdog timer times out. And that trace dump will reveal where the code in the client is deadlocking (hanging) and under what combination of circumstances (which won't be reproducible on anyone else's machine not on your network).

Now the actual problem (code fix) may be on the server side. We are just using the WL client error logger as the debugger to isolate the deadlock (technically it's a "livelock" because there's an impending operation, the Quotes window is updating) from the client's trace dump.

---
Perhaps you've never considered it, but nearly all activity on one's own system is "synchronous" (same process scheduler), so this nondeterministic behavior can't typically occur. But when you talk to an external system (like a server), which isn't synchronized to your machine, then you can have these nondeterministic problems if semaphores (scheduler directives preventing race conditions) aren't applied correctly to the "critical sections" of the code. (Google "critical section" if you want to learn more about how semaphores prevent race conditions. And this will be discussed in any graduate-level textbook about OSes under scheduling.)

QUOTE:
... fyi, an WLP update is in planning for next year to integrate a new streaming provider

In other words, any network race condition may disappear with the new server code and provider. So be patient. But it would still be nice to know if long Internet latency (i.e. long ping time) aggravates this problem with the current WL version so we know if we're on the right track.

You might also try reproducing this problem in a different city so you're on a different network with different latencies.

QUOTE:
I just close the quotes window and open a new one and it works right away without any logging in or out.

That's good to know.

For the time being, and until the integration of a new Fidelity Streaming Provider (probably before next summer), I can't think of anything that is going to help this intermittent condition (mainly because we don't know what's causing it) except to work-around it with a different streaming data/quotes provider - IQFeed. Click this link for information to sign up for a free trial, install the IQFeed client, and the Wealth-Lab Extension.

It's at least worth a try for the free trial, and if it works out, it should pay for itself for the next 6 months (just an estimate - could be longer) until a new Fidelity provider is ready.

QUOTE:
I just close the quotes window and open a new one and it works right away

Well, the race condition that caused the livelock in the first place may not occur again immediately--it's nondeterministic. That's why you need to measure the ping time immediately at the point that the market opens and the Quotes window fails to update. A ping measurement at any other time is useless.

For example, your enterprise may be scheduling network backups each morning when the market opens slowing your network down at that moment. You can try tethering your computer to your cell phone service for a different Internet latency. But my guess is the latency over 4G LTE would be worse.

If semaphores are applied right in both the client and server code to correctly synchronize their process schedulers, the network latency won't matter--the semaphore synchronizing forces processing to be more deterministic for all real-time (screaming) environments. My position is, even if the problem isn't in the WL client, the client should be fault tolerant enough to clear the livelock (e.g. Stuck Submitted) and put up a dialog box telling the user the streaming server quit responding. The watchdog trace dump to the error logger is optional and only for locating the problem code.

QUOTE:
work-around it with a different streaming data/quotes provider - IQFeed

Do let us know if switching data feed providers lessens or worsens the race condition causing the livelock. It's an interesting idea.

Thank you guys.

I will for now stay with this client until I can catch the latency when it happens. I travel a lot and it has happened in 4 different cities in 2 different states. My connection at home though is pretty rock solid but it happens there too.

Once we figure out if the latency is ok or not then I will try the new client to see. I'm reluctant to try a new client unless I can be in front of the computer.

Thanks and I'll keep you posted.

It would be nice if they fix it so that the latency glitch doesn't matter for the next update.

It happened again today while I was in front of the computer. The ping within 30s of the event was 16 ms.

QUOTE:
The ping within 30s of the event was 16 ms.

I think that latency (16ms) is reasonable and realistic. Moreover, if you have had this problem at other locations (home and office), then I don't think your Internet latency is a factor.

Since you're the only one reporting this problem, I don't think Wealth-Lab itself is a factor either; otherwise, many others would be having this exact problem,... and they are not.

What is particular to you is your Windows system itself. You have two options:

1) Try running Wealth-Lab on a different computer to see if you can reproduce the same problem. This may be the most revealing option.

2) Try running Wealth-Lab on your usual computer, but in Windows Safe Mode. You need to put it into Safe Mode just before it boots up Windows. Pick the Safe Mode version (just Google Windows Safe Mode for details) that still runs the Internet and disk, but does not load all the startup apps that weight the Windows scheduler down. See if you can reproduce the problem then. This option is harder to get working since Wealth-Lab requires many Windows services, so Option# 1 may be better. I'm not even sure you can get Wealth-Lab running in Windows Safe mode, but try and see.

I'm thinking there's a background process on your system that's loading its response time down. For example, every morning Windows Antimalware Service Executable, MsMpEng.exe, runs on my machine. (Check Task Manager and you'll see.) That really slows Wealth-Lab down big time, but not to the point that it livelocks any Wealth-Lab windows. Your system's problem is significantly worse, which makes me think there's a hardware problem affecting the device driver level. If this is really the case, then you won't be able to reproduce this problem on another computer with different processes and device drivers.

Also, check Task Manager (CTRL-ALT-DELETE) to see how much CPU time Wealth-Lab is using when the market opens. It should be minimal; otherwise, you may have some infinite looping operation in Wealth-Lab really slowing it down. You might also see if Task Manager can identify other apps running that are using a great deal of CPU time and running "Above Normal" priority. Wealth-Lab should be running at Normal priority so apps running Above Normal priority can starve it. (But I doubt a non-malicious app would be running Above Normal priority.)

And yes, a virus, malware, and especially a root kit can cause this problem. But these typically try to fly under the radar, and this problem isn't doing that.

If it were a software problem in the Windows application layer (For example, you should be running version 4.6.2 of the Microsoft .NET Framework to be compatible with the WL 6.9.19 build.), then the problem would be deterministic (consistently happening) rather than nondeterministic. So I don't think this is the case either.

I really appreciate all the time and thought that you are all putting into this and it makes sense it may be a problem with my machine although I have no idea why it would have a problem.

If I can I'll try the safe mode thing tomorrow.

Thank you and I'll keep you posted.

The Safe Mode test is to see if a non-Wealth-Lab process (including the device driver layer) is the cause. (And all your Windows applications would be affected to "some" degree if this were the problem.) But we still need to check the WL process itself with Task Manager.

In the attachment, I have a Process Explorer (you can use Windows Task Manager) screenshot of the Wealth-Lab CPU usage 5 minutes into the trading day. Notice its usage is only 1.1% with a couple strategy windows and a Quotes window streaming. Now if that's much over that, then something is running in Wealth-Lab that shouldn't be. And that's a problem that needs to be addressed.

Task Manger will show other processes besides Wealth-Lab. They aren't a factor unless they are running a higher priority than Wealth-Lab, which runs at Normal priority. And if they are all non-malicious apps, they should all be running at Normal priority so they aren't starving Wealth-Lab. You can verify that.

Ok, I think I have more information! So I just started a quote window since I'm home and it FAILED. I checked the event monitor per a suggestion by Cone after the suggestion here that it may be machine dependent in my ticket and an error I saw before popped up. Then I closed it and opened a quote window again and it worked without error. Cone posted some links for me to try but I won't have time for awhile. I'll keep everyone posted. Here are the errors with Perflib and Perfnet:

QUOTE:

(Perflib) The Open Procedure for service "BITS" in DLL "C:\Windows\System32\bitsperf.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

(Perflib) The Open Procedure for service "Lsa" in DLL "C:\Windows\System32\Secur32.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

(Perflib) The Open Procedure for service "MSDTC" in DLL "C:\Windows\system32\msdtcuiu.DLL" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

(Perfnet) Unable to open the Server service performance object. The first four bytes (DWORD) of the Data section contains the status code.

(Perflib) The Open Procedure for service "WmiApRpl" in DLL "C:\Windows\system32\wbem\wmiaprpl.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

Thanks again for all the help.

I would be very surprised if any of those errors had a slightest relation to WLP and your Quotes window problem. Nonetheless, this page has some more solutions:

The Open Procedure for service XXX in DLL "C:WindowsSystem32XXX.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code

QUOTE:
... would be very surprised if ... those errors had a ... relation to WLP and your Quotes window problem.

I tend to agree. "I think" this is a nondeterministic problem, which means it's a Windows process/threads scheduler issue. The error log discussed above would be referencing more deterministic (reproducible 100% of the time) problem.

I would concentrate on [1] how many CPU cycles the Windows scheduler is granting to the Wealth-Lab process (The Safe Mode test should maximize CPU access because fewer competing background processes [malware?] would be running.) and [2] how much CPU time is actually being consumed by the Wealth-Lab process itself at market open, which the Task Manager test should reveal.

What confuses me is the Quotes window appears to be livelocking (deadlocking with a pending update operation). That's a scheduling problem, but not because of starvation (typically) but because of resource allocation. Could you please tell us how and when you are populating the Quotes window in the first place? Please try a different way of populating the Quotes window to see if the problem goes away.

Why not open a second Quotes window and puts some pending trades in that. See if it livelocks too; I'm thinking it won't.

There may be other unrelated problems, but that's another issue.

Sure, I populate it as I described in the initial post. Log in to Fidelity, highlight all the alerts in my strategy and then press Monitor Quotes.

So open up a second Quotes window and populate it by manually entering the tickers and information. Let's see if that approach livelocks on your system. Perhaps the Windows.Forms.Quotes listener thread that updates the Quotes window is never scheduled to execute when he creates quote entries the first way.

And let us know about the Safe Mode and Task Manager test. It could still be a starvation problem by the Windows scheduler. If so, that may not be reproducible on another computer if it's a system-level problem.

Marc, how many quotes does your Quotes window have on average?

There were 217 symbols in the list supplied in the open ticket. Three of the symbols were dead, we removed those, but the problem still occurred. I've streamed 500 symbols before without a problem. However, for these market on open tests I've been using only 100 symbols.

Sorry I've been out of town. As Cone said it's 200+ symbols so manually entering won't work.

I'll try safe mode tomorrow, I forgot today.

What I did do today though is open 1 quotes window as normal, and turned off the auto-stage. Then I opened a second window with the normal auto-stage. My thinking is if it's a glitch opening the quotes window then maybe the first one will "catch" the glitch and the second will be ok. I just have to be really careful to turn off the auto-staging on the one or my position sizes will be too big.

The second window is also opening in a way different than the way I've been doing it in that I haven't been opening the quotes window first. Just having it open automatically from the strategy.

I'll keep everyone posted.

Thank you again