Ok, I've installed 5.1 and signed my Automated Trading agreement. Now I simply need to fine-tune the program I use to trade the daily bars to get going. At the moment I am trading the 1 minute bars manually which is supremely awkward.

Before I get too far down the road though a couple of questions to those of you who've been there already: what kind of latency can one expect with 5.1, in terms of signal-to-transact time? is there much additional delay for shorts?
(for liquid big-cap stocks and etf's)? is it necessary to pre-borrow securities? can you program in a price limit for the trade that keys off the closing price of the bar? for example, buy at close+.05 or better? Is it possible to limit the number of same-side sequential trades - say transact the first two sell signals in SPY and, if they are both still open, then ignore the next sell signal, if there is one; resume when one position comes off. Maybe a dollar limit per security (as opposed to a trade) could do that?

I know, a lot of questions

QUOTE:
what kind of latency can one expect with 5.1, in terms of signal-to-transact time? is there much additional delay for shorts?
(for liquid big-cap stocks and etf's)? is it necessary to pre-borrow securities?

It's a good idea to direct these order execution questions to Fidelity.

I have a routine that runs on 1 minute bars and at the start of the routine (when market opens), the loading of data from Fidelity is pretty slow, couple to few seconds per symbol. By the time all the symbols have loaded (more than 500), more than half hour have passed. The data range have kept to minimum to meet the program requirement. Cone have suggested earlier to contact Fidelity about Sever and data speed issue, which I did and they are currently looking into it. Frankly I don't think Fidelity will help much on this.

Any suggestions on how to improve the speed of 1 minute data loading at the begin of program execution?

Thanks.

We need more information about your strategy code and where it works. Where does your strategy operate, in Strategy Monitor or Strategy window? Is on demand data update turned off? Do you update the data before market open?

P.S. Some useful tips, see if they apply: What is optimal for my trading, Strategy Monitor or Strategy window?

Thanks for the reply. The code is operated in the Strategy Monitor. Data update on demand is turned off. I have all the data update about an hour before market start.

I will check out the link. Thanks.

Approximately how many bars (days = bars * 390) is required in your data range? How much memory does your machine have? How long does it take the strategy to execute one symbol? Is your computer dedicated to only this task?

On the surface, the kind of performance you described sounds like an over-tasked computer using virtual memory.

Thanks Cone. The longest historical data range required in the program is for 5 days - with 390 bars/day - that comes out about 2000 bars for the data range. Although, the number of bars being loaded for each symbol range from few bars to less a hundred bars (based on the info displayed in the Strategy Monitor). The PC has 8MB of RAM. Looking at the physical memory usage during the program execution, about 6MB is being used. I am not sure about the virtual memory usage and if it is a limitation and will check it out.

Once all the symbols data is being updated, based on looking at the Strategy Monitor, the rate of 'execution' of each symbol in the program go much faster (seems to be less than a second for each symbol) compare to when data is being loaded. Yes, the PC is dedicated to run the program only. Thanks.

The virtual memory usage during the start of the program where data is being updated is small in the hundreds of KB. So this seems the virtual memory is over extended. The page file is set to 10GB.

8MB of RAM - what is this, 1995? (Just kidding, I know you meant 8GB)

Thanks, I wanted to get an idea of parameters so that I can try it myself. There shouldn't be any memory problems as long as the strategy isn't doing something weird.

You said more than 500 symbols... have you checked to make sure all of the symbols are still valid and updating?

The S. Monitor will process 1-minute bars symbol-by-symbol as the data are received (so you'll get alerts right away), but symbols that don't return data will cause the thread to wait up to 50 seconds for the update before giving up. If some are NYSE stocks, it's possible that waiting for that first trade, which could happen after the first minute interval, would give the impression of slow data loading. I'll take a look at the S&P 500 today.

Although I had a lot of other realtime things going on (like another 100 symbols in Quotes), it looks to me that 500 symbols are just too many to put in a single thread for the Strategy Monitor. The timeline just gets overrun since the machine isn't fast enough to process all the symbol data serially.

I broke down the DataSet to 4 different ones (about 125 symbols per) and am looking at that. Then, I added the strategy 4 times, one for each sub-DataSet. With only 1 set running, operations are fast and completes in the first 15 seconds. With 2 sets running, things slow down, but complete within 35 seconds.

When I started the 3rd set, an error occurred in the thread and it hung up.

My machine isn't especially fast (a 3-year old Lenovo T520 with SSD), and, the strategy actually writes the data from the last bar to disk (a relatively slow operation). However, based on these results, I wouldn't try to run more than 250 symbols on my machine with 1 minute data. Processing power is constraint and it's going to vary by machine, strategy, etc. I'd recommend you whittle down your WatchList or find another machine to split up the task. Alternatively, you might think about trading your strategy on a higher timeframe, like 5 minute or 30-minute bars.

Thank you Cone for spent time on those experiments to gain more insights into this one. the info you gained is helpful and I think this is good for everyone who are using large data sets run on high resolution time line. I am started thinking about experimenting in that direction you have suggested after saw a post from someone doing pre-filtering the night before so to reduce size of the data set which will be used for next morning screening. Your data is reinforced this thinking. With large data set run at 1 minute step, I have experienced crashes and error message "The given key not present in the dictionary" and the crash possibly might related the fact that the thread was being overloaded with large amount of data being presented as you mentioned. the 500+ symbols data set is also too demanding for my CPU. During execution, the CPU usage is maxed out and the machine is pretty much frozen. So this is another reason I need to break the list down to smaller size. Again, thanks and I will take on your suggestions.

QUOTE:
"The given key not present in the dictionary"

Actually, this was the same error I got when I added the 3rd set of 125 symbols. I'll investigate that more to try and determine what key/dictionary is identified.

CODE:
Please log in to see this code.

... just to qualify this statement a bit - it's too much for a typical mid-range cpu for a 1-minute interval. Extend the interval by another minute (2-minute bars), it shouldn't be a problem, though Alerts could be delayed by up to 1-minute after the end of interval.

Re: "The given key not present in the dictionary"
In my case, this error was caused by the symbol JWM not having updated. It failed during a Data Manager update as well. Later, an on-demand chart request cleared the error; a corrupt entry in the binary file? It's still a bit of a mystery of what precisely caused this symbol to fail update because nothing was entered in the error log.

Anyway, when you see this error, perform a Data Manager update and look at which symbol the update terminates abruptly. The next symbol alphabetically is likely the problem - try an on-demand update in a Strategy Window, or refresh it.

Also, looking at this today made me aware that if you're only 1 minute behind and your DataSet has more than 25 symbols, the update process will [probably] never be able to catch up. The bad performance comes from saving those huge intraday files during the initial update after activating the Strategy. This is precisely the reason data are not saved while the Strategy is updating at the end of each bar during market hours. We need to eliminate saving initial updates too during market hours.

Thanks Cone. That's awesome. I am also in the process of understanding this error. When I got the "The given key not present in the dictionary" error, followed your suggestions, I checked the 'Update on Demand' in the Data Manager and ran a data update on the data set that I originally got the error. Afterward, review the data update log file, there is one symbol that stands out where its # of bars being updated is zero, while for all other symbols the # of bars updated is not zero. I deleted this symbol from the data set, re-run the program in the Strategy Window and stilled got the error. Not sure what happened here, possibly that there other symbol still there that cause the error originally. Like you mentioned, there is no error message in the log file that indicated there is something wrong with any of the symbol during update.

For the "update on demand" feature in the Data Manager to take effect, do you have to open a chart on the symbol that created the error and ran on it for it to be updated? If this is so, with the "update on demand" feature checked, would it is make sense to create a simple program and applied to the data set that has the error before run it in the Strategy Monitor.

Thank you.

Cone, do you suggest that to keep the data set to have max 25 symbols when work with 1 minute data because of the lagging in update?

Is there a fix to 'not saving initial update data during market hours" expected to be released in the near future? Thanks.

Not at all. For me, 125 symbols update very quickly in one thread in the S. Monitor.

The problem occurs if you miss activating a 1-minute Strategy before the opening bell. If you do, the lag in updating and saving the files takes too long (about 2 seconds per symbol) such that you'll never catch up if you have more that 25 symbols in the DataSet.

There are no magic bullets or quick fixes for the Wealth-Lab client (as opposed to our Extensions for Wealth-Lab). History suggests that Fidelity releases upgrades about 3 times per year, with one possibly coming next week - v6.8.

Thanks Cone. What do you mean by 'miss activating a 1-minute strategy' ? This might what I am experiencing now - under the 'Action' tab in the S.M. it indicates there is always constant update going on and even late into the day and there are several tens of symbols left to be updated.

If this error of lagging in update is indeed happening (for data set > 25 symbols), is this mean that the execution and receiving alerts also being delayed? and the delay is proportional to the size of the data set? What are other implications beside these?

I mean you need to get the Strategy Activated before the opening bell. But even if you do that keep in mind that if you have too many symbols running in 1-minute intervals, the timeline will get overrun and you'll be in the same boat as if you didn't Activate before the open. For me, 250 symbols updated within about 35 seconds, and I wouldn't attempt more than that for 1-minute intervals on this machine.

QUOTE:
several tens of symbols left to be updated.

Note that often you'll see "X Symbols left to update" just because there was no trading that occurred for that bar (so no update is available).

QUOTE:
receiving alerts also being delayed? and the delay is proportional to the size of the data set?

Yes and yes. Like I said, for me 125 symbols finishes within 15 seconds; 250 symbols consistently takes about 35 seconds to process... that means the symbols that finish processing last would generate Alerts 35 seconds after the end of interval.

It's disappointing that 250 symbols of data that are already in memory (except for the incoming bar) cannot be processed faster on an average machine, and in my mind it behooves us to investigate more to find the bottleneck (the answer may already be known to everyone but me).

Thank you Cone. That was very helpful info. I never thought much about the impact of when to active the strategy. That would be great to understand and improve on the size of data set to a number possibly 1.5 to 2X of the 250 symbols. For me with my machine probably not much different from yours in term of performance, (it something I need to understand better), the 250 symbols might be around my limit and this mean I might need to run in more than 1 machines or use 2 or 3 minutes scale.

This might stupid, but I want to throw this out, would it is be possible that the limit in update rate in this case for your machine at 250 symbols at 35sec be coming from Fidelity in their handling and providing of static data. By the way, I still have not heard back from Fidelity on my question if they can give me some info on any possible issues if any on their data servers that might slow thing down that they can see from about a week ago. Thanks.

Updates for Fidelity 1-min "streaming bars" are typically all received within the first 7 to 10 seconds. The bars are "pushed" to the S. Monitor with minimal delay shortly after they're built on the historical server. All the delay after those first 10 seconds is almost surely attributed to processing the data coming in, running the strategy, and showing the results (plus anything else the machine is doing).

In fact, you can see the messaging of the strategy processing symbols, which seems to go in spurts and then may stop for several seconds before "spurting" again. I'm not so sure displaying/updating that text is a good idea since it certainly has some contribution to the processing delay.

That makes sense. I remove all the visual indicators on mine and only setting enough text for send out with alerts. One question, I saw someone have used the command line "System.Threading.Thread.Sleep(1000)", my guess the reason for using this is for the purpose of data synchronization. Is this true? So if not needed, you would not want to use this delay command line since it only worsen the data updates?

Thanks.

QUOTE:
One question, I saw someone have used the command line "System.Threading.Thread.Sleep(1000)", my guess the reason for using this is for the purpose of data synchronization. Is this true?

When you have doubts, consider looking up unfamiliar .NET methods in the MSDN. What Sleep(1000) does is artificially pauses execution of the strategy by 1 second per each symbol.

Re: System.Threading.Thread.Sleep(1000)
It was probably used in a Strategy to help ensure that a secondary symbol had time to update before attempting to access it. That statement alone isn't a good way to do that, but if you use external symbol(s) in a intraday Strategy, you definitely need to do some extra work to ensure they're updated before processing their data.

Thank you both.

Cone:

Referring and continuing with the discussions we had from post #10, I have a setup in the SM running only 1 thread with 50 symbols in the data set at 1 minute scale. With the usual setup procedure in the SM; I have the strategy in the SM activated before market open and also log on into Fidelity before market open and also have the Auto-Trading set to one of the paper account. All data was being updated before market open and 'Auto-Update' feature in the data manager is turned off. The machine is not running any CPU and memory demanding tasks

As the market OPEN, in the SM, under the Action tab, I can see the display of 'Entering Streaming Filter Monitor Thread' and 'Leaving Streaming Filter Monitor Thread' messages switching back and forth. However, I don't see the usual spurting of actions of 'Update Completed...', Execute...' messages in the SM. The CPU meter also pretty much idle.

However, if I was to De-activate /remove the strategy from the SM, and Re-activate/Re-instate the strategy at a later time with the same data set and at 1 minute, the strategy would operate normally again, namely; I would see the spurting of actions; 'Update...', 'Execute...' and the spikes in the CPU meter. But (a big but), this Re-active/Re-Instate of same strategy NOT ALWAYS work. That is sometimes, Re-active or put back the same strategy in the SM does NOT work.

It seems like the strategy is stuck and I have to Re-Activate or re-instate it again in the SM after the market open to get it jump-start and running but I understand that this procedure is not correct.

From the limited number of observations I have, it seems the frequency of 'NOT work' from the actions of Re-Activate and Re-insert the strategy in the SM is higher (occurred more) right after the market open and less if the actions were done at more later time in the market session. Again, the sample of number of these observations are a few, so it might not be very liable.

Do you have any thoughts what might be causing the followings;

1. Why the strategy Activate before market open 'seems to get stuck' and until a forced De-Activate/Activate action or after it being Removed and Inserted to get it working

2. Why the De-Activate/Activate or the Remove/Re-Instate actions are not ALWAYS working?

Thank you for any insights you can offer.

1. or 2. - I haven't seen either one of these (but I don't trade intraday and only look at these things when required for testing), but I have seen Fidelity "Disconnect" at or prior to the open if I've been logged in for a while premarket. However, if you saw that, you'd almost surely have to restart WLP completely (like I do). To avoid this, I usually try to restart and log in after 0900 ET.

The only other thing I can think of is that your system clock is out of whack. Even though the S. Monitor uses a socket connection to Fidelity, the "Update" process is triggered by your system clock. (I really don't know why it's [still] designed this way after the socket bars were integrated.) If the clock is accurate, is your machine set to a weird time zone, like Mountain Time or something?

Thanks Cone. I think restart the machine is a good idea. I have not seen Fidelity 'Disconnect' and don't think this is the problem. As I observe the SM, at 9:31AM EST, the "Entering Streaming..." or "Leaving Streaming..." is display and it continue from there. So there is a smooth transition of the program entering and updating the thread as the market open.

I have checked and confirmed that the machine clock is set correctly and in synch with the clock at NIST.gov. The time zone is confirmed set to correct zone.

Hi. I'd just like to add that I've been experiencing the same sort of issues as tomphm with one minute data lately. I, too, will activate SM before the market open, but have found that it usually won't execute correctly unless I "jump start" it after the market opens. I, too, sync the clock to the .gov site. Thanks.