Hi. A friend of mine and I have been tinkering with the following strategy which is currently based on daily bars. It often generates more trade alerts than there is capital, and I would like to insert code that will make the strategy take the first 5 trades in any given trading day so that I could see a consistent backtest result. (I guess this will now involve intraday backtesting.)

Do you have a sugestion as to how that code will look like? Thanks.

CODE:
Please log in to see this code.

It's easy with the Position Options PosSizer. Click the "Max pos. per day" button and type "5" in the input field. Unlike the default "Max entries per day" PosSizer, this one will sum up positions taken during the day for intraday trades.

Hi Eugene. If I follow your instructions, would the PosSizer take the first[u][/u] 5 trade alerts that come along and not 5 trades chosen randomly? Say if there were 10 trade alerts in a day, I would like to take the first 5 trades and not 5 randomly selected trades amongst the 10.

Yes, the first 5 intraday alerts are taken.

Thanks.

This is a Limit order script for Daily bars. Eugene, unless I'm not seeing something that you are, to get the first 5 trades that triggered the limit order prices, joannakim needs to SetTimeOfDayPriority.

What do you think Eugene? And if Cone is right, how do I go about doing what Cone suggested: SetTimeOfDayPriority? Thanks.

Click the link Joanna.

I admit I didn't look closely, thinking that it's natural for a WLP user talking about intraday trades to use intraday data. Then the PosSizer would be the natural choice. Robert is right: with daily data you have to use the workaround.

When I clicked the link, under " Description," it says that it (SetTimeOfDayPriority) is valid for scripts that trade 1-symbol only. Does this mean trading one symbol only per trade? Becuase as time goes by, my strategy will trade many different symbols. Just wanted to make sure although this seemed kind of obvious. Thanks.

If the script uses SetContext() followed by a BuyAt or ShortAt, then it won't work. An example of a script that trades more than one symbol is a Pairs Trading script or a Rotation Strategy.

Cone, under the Usage Notes of SetTimeOfDayPriority,there is a discussion on "when a probable data spike is detected." What is that data spike referring to, and does it always happen?

Here's a made-up example.

Say yesterday's daily low in your MSFT data was 22.55 and your system theorectically "bought" MSFT at 23.98. Well, the day's actual low was only 26.76 (as you can see in the intraday data) so it's clear that the theoretical trade using the daily data could not have occurred.

The way the priority routine works is by finding the first intraday bar that matches your limit price. If that price doesn't exist in the intraday data, then it reports that a probably spike has been detected and you should investigate and manually correct your data with the daiy OHLC information provided.

That said, there are no guarantees that the "spike" doesn't also appear in the intraday data. But, experience shows that the routine does detect a fair number of spikes that could not have been realistically traded.

Cone, under the Usage Notes, there is also this: "Note that Position.Priority is set to -10000 for suspected spikes. Instead of correcting the data, you could use the Position Options sizer to "Reject a Position with Priority... Less than -1700" to filter these unrealistic trades." What are the -10000 and -1700 referring to?

Nothing. They're negative numbers, so their priority will be lower than all other trades, which means that they'll be the last trades processed/selected.

Hi Cone. I am a newbie in C# or programing for that matter, and looking at the code that I pasted above, I have a few probably very basic questions for you if don't mind. Don't laugh.

1. In the following segment of the code:
paramLimit = CreateParameter("LimitVar", 45, 35, 45, 1);
paramRatio = CreateParameter("Ratio", 2, 1, 2, 1);
paramROCHigh = CreateParameter("ROCHigh", 4, -4, 1, 100);

I tend to think that the first numbers: 45, 2, and 4 are the numbers that the respective parmeters should be, but I do not know why there are 3 more numbers on each line following the first number. Do you know what they are, and why they are there?

2. Some lines below the above mentioned item, there is " bar++." What is that?

3. A couple of lines below that, there is: Position p = LastPosition. Does this mean that p refers to last position? And what is "last position?"

Thanks a million. If only I could read code like you and Eugene do...... Alas.

1. Default, Min, Max, Step
Open an Optimizer, click on a parameter and you'll see.

2. Increment operator. Same as bar = bar + 1;

3. LastPosition returns the Position object for the most-recently created Position. The statement assigns the LastPosition object to the Position variable p. This is done really just a matter of convenience. If you need to refer to LastPosition more than once or twice in your code, it's just easier to type and read a shorter variable name, like p, instead of typing LastPosition many times.

joanna, 2 and 3 are basic C# statements, so I urge you to review the thousands of free resources for learning C#. If you don't want to program in C#, then you can use any .NET language for your strategies in your own dev environment, but then you can't use WL's convenient Editor and C# compiler.

Thanks for being patient with me Cone. Now that you reminded me of no.2, I remember reading that in a C# book. There is a lot to learn as a C# newbie; so even after learning stuff from books, etc., I often forget. Thanks for your help. I really appreciate it very much.

Hi again. I have been struggling to insert the code that Cone created for taking the first 5 trades of the day into the above stated strategy. Of course, being a true C# Newbie, I keep getting a bunch of error messages when I tried to compile. This kind of mishap probably is almost laughable to you, but do you think you could either help insert the code for taking the first 5 trades in any given day or at least give some guidance for me as to how to do it? Thanks again for your patience. It really is a little rough to be a Newbie.

The whole task boils down to inserting 1-3 new lines of code (depending on priority solution, Cone's or avishn's) -- well exemplified in the Wiki. Can we see how does your effort look like?

Hi Eugene. I think I basically tried the copy and paste method from what Cone created, and if I had to guess, I probably did not do that correctly, ha ha.

Here it is. I copied and pasted Cone's code at the bottom of the page.

CODE:
Please log in to see this code.

Missing this:

CODE:
Please log in to see this code.

Copy/paste act from the Wiki w/o reading the text and following the instructions:

CODE:
Please log in to see this code.

Where are the instructions that you are referring to? Thanks.

You must be kidding.

On the Wiki page in Cone's reply (8/2/2011 8:53 AM). That's where you made the copy/paste from.

I wish I could say that I was kidding...

In any case, Eugene, did I insert the missing line in the correct place, and what about the second item that you mentioned in your comment of 8/12/2011 4:49 PM? I don't even think I really understand what that line is saying if you don't mind explaining.

Thanks for your patience. I wish, some day, I could read and write code like you. Dream on, right?

QUOTE:
In any case, Eugene, did I insert the missing line in the correct place,

Yes, now you did.

QUOTE:
Thanks for you patience. I wish, some day, I could read and write code like you. Dream on, right?

The problem here is not in code reading skills, not nearly. Deciphering compiler error messages has nothing to do with it too. The answers could be derived from carefully reading the textual contents of the SetTimeOfDayPriority page, and this is what I'm trying to motivate you to do. The related text is just a dozen lines.

QUOTE:
I don't even think I really understand what that line is saying if you don't mind explaining.

I urge you to read the text, the intradayDataSet and (especially) the barInterval pieces. Then, review your copy/paste act (2nd line) and ask yourself:

* What does this line mean?
* Why I expected it work by unconditionally copying and pasting rather than immediately modifying it to suit my setup as per the online manual?
* Does it have parameters? If so, how does the online guide explains them? (I will not duplicate/explain it here since I couldn't do it better than Cone's text already does.)
* And if I have a Fidelity intraday data source, why did I past that line? What if I tried the alternate, simpler syntax mentioned in the "Syntax" block? (That's a straight hint.)

Eugene, in number 5 of Usage Notes under Description of Cone's SetTimeOfDayPriority, it says: " Note that Position.Priority is set to -10000 for suspected spikes. Instead of correcting the data, you could use the Position Options sizer to "Reject a Position with Priority... Less than -1700" to filter these unrealistic trades."

Where do you see Position.Priority having been set at -10000?

That's a different question. On how to set a Position's priority please refer to the QuickRef > Position object > Priority property.

Eugene, it says there in the Description: " Note that Position.Priority is[i][/i] set to -10000 for suspected spikes....." So I thought it has been set at some place, perhaps in the " [i] ActiveTrader Standard...." since I did not understand what the line: "{i} ActiveTrader Standard...." meant any way. What does that mean any way? Thanks.

The Position.Priority for spikes is set inside the SetTimeOfDayPriority method body and you need not to care about it. Concentrate on the very basic aspects of syntax first, and forget about "Description" for a moment - don't skip my instructions.

QUOTE:
What does that mean any way?

1. Carefully review the Syntax block: it's all explained there. Break up the line by components. There are two SetTimeOfDayPriority's up there. Which one does that line resemble? What the Positions thing can be? What type might "[i] ActiveTrader Standard Stock Portfolio (M5)" be in C#? Integer, IList or does it look like a string?

2. Having finishing with #1, read the Parameter Description block. Now that you, hopefully, understood what this string was all about, find the description of the parameter in question and read the text.

3. Finally, armed with this knowledge, carefully re-read the final paragraph in my reply from 8/13/2011 2:05 AM. Don't skip anything. Concentrate on the last sentence. Even though it went completely overlooked by you, it presented a very straight hint on finishing this quest in a minute.

My point was and still is: willingly copying and pasting w/o comparing your actions to the manual has led you in a wrong way. Should you just read the two-line Syntax block carefully, you've noticed that Cone described there a super easy way for Fidelity data users.

P.S. I enhanced the Wiki code example to illustrate the two ways of calling the method yet I hope you'll learn something from my instructions. Otherwise I was banging my head against the wall for no reason. ;)

Okay Eugene, I did not overlook your "straight hint." I was going to get to it, and change the code, but I got a little sidetracked by my curiosity about the other issue that I mentioned since I am a very eager student of C# at this point.

But nevertheless, thanks for changing the code in the Wiki. It is much easier for Newbies like my self: that is only if what I did in the code is correct, otherwise, I misunderstood what you did.

"bar interval" described in the Parameter Description is really ambiguous to me. But looking at the code that you wrote, it must mean the number of total trades in a day, right since it says 5?

QUOTE:
But looking at the code that you wrote, it must mean the number of total trades in a day, right since it says 5?

According to this logic it must as well mean the number of fingers on one hand. It is also 5, right?

barInterval:
The intraday interval of the DataSource you want to use to set priority by time of day.

Shiver me timbers if any single character of this text can make one believe it's about the number of total trades in a day!

QUOTE:
"bar interval" described in the Parameter Description is really ambiguous to me.

There exist two very different calls of the method: one specifically caters to Fidelity data users (exclusively) and does its magic after specifying the barInterval of an intraday DataSet, another (universally applicable) one has a different set of parameters.

Eugene, like I said in my previous post, I did revise my code which is located above: " last changed on 8/13/2011 8:07 AM." Is the code correct now?

And I guess the "(5)" means 5 minute bar intra-day data set from Fidelity, right?

Joanna, the placement of the method calls in charge for time-of-day priority is OK. The other code I did not look at.

QUOTE:
And I guess the "(5)" means 5 minute bar intra-day data set from Fidelity, right?

Bingo!

Oh good. Because taking the first 5 trades in any given day is the part of the code that I wanted you to help with. And according to what you just said, that portion of the code is correct?

And Eugene, if the code is correct, do I run the backtest on a 5 min Fidelity data set or on a daily Fidelity data set? And do I choose "5 min" in the "Scale:" box on the upper left corner of WLP or do I choose "Daily?"

1. Update your Fidelity Data before you start.
2. Run the script on a Daily DataSet. (The PositionHelper finds the priorities from your intraday data.)
3. Check the debug window for suspected spikes; OHLC data corrections using the intraday data are provided.
4. If spikes are detected, open a separate Daily chart for the symbol with the spike, double click the bar to open the Bar Data Editor, and enter the correction from 3. Repeat from Step 2. until no spikes are detected.

Hi. I ran the same strategy of mine twice on the same settings on the take-first-5-trades-in-a-day version of my strategy, and I got different results: different APR, etc. What do you think happened?

Also when I open the newly revised strategy of mine( with take first 5 trades of the day code included) with a Fidelity data set selected for it, the "Scale: box on the upper left corner automatically changes to' 5 minute." So the data set selected is a daily data set, but the "scale:" box says 5 min. Is that okay, and what is happening?

QUOTE:
What do you think happened?

Position priority is not assigned correctly. If you provide your script, then we can check it.

QUOTE:
So the data set selected is a daily data set, but the "scale:" box says 5 min.

It's okay if you want the 5-min scale, but if you want the Daily scale, just select Daily.

When I open my strategy by clicking on the link, the "Scale" box automatically goes to " 5 minute" although as you probably know, my code is written on daily with Cone's script added on to it. And when I ran it like that: with "5 minute" in the scale box, it produces very different results than if I had manually changed the "Scale" box to "Daily."

The number of trades taken on 5 min is 238 with trades not included due to insufficient simulated capital at 8. When I run the strategy with "Daily" in the "Scale" box, the number of trades included in the backtest results were: 1143 with Trades not included due to insufficient simulated capital at 11057. The APRs are very different too: the 5 min one produced much higher APR with much fewer trades.

Which is the correct way to run the strategy: on "daily" or on "5 min?" This is very very important an issue here, I believe. So I would really like to find out the correct answer to this. Thanks.

Also, on the backtest ran with "5 minute" scale, there were days when there were more than 5 trades which goes against the command: take first 5 trades in a day. What I thought that command meant was that it needed to stop taking any more trades after taking the first 5 trades in a day, do you agree on this? For example, on 3/17/2008, there were 9 trades, and on 3/20/2008 there were 6 trades.

Joanna, the strategy that you wrote with the intraday priority trick is supposed to run on Daily data. Not for intraday.

Should you design the strategy as intraday and apply the "Max trades per day" option in the Position Options PosSizer (only it and not the built-in PosSizer of the same name) as suggested in my first reply -- and it would save everybody a lot of time and nerves. :)

But Eugene, every time I open my strategy window, the "Scale" box automatically changes "Daily ( which is what I set it at)" to "5 Minutes," and I have to change it back to Daily. Why is that and I guess you are suggesting that I keep changing it to "Daily," right?

And of course, the data set that I selected was a daily data set from Fidelity. It's just that when I open my strategy window, the "Scale" box on the upper left corner of WLP always revert back to "5 minutes."

It's probably because you saved your Strategy that way and "Data Scale" is selected in Advanced Options. Just open your Strategy, change it to Daily, and save it again. Don't get hung up on this.

Cone, I got an error message in the Debug and Error Messages window saying this: ZIXI: (1) EOD data not in intraday date range for trade on 2/4/2004
1 -2500 1/30/2004
2 -2500 2/4/2004
3 -935 4/16/2004
4 -1000 5/3/2004
5 -935 5/5/2004
6 -935 5/7/2004
7 -935 5/17/2004
8 -1600 6/4/2004
9 -945 7/8/2004
1 -945 7/29/2004
ZRAN: (1) EOD data not in intraday date range for trade on 2/3/2004
1 -2500 2/3/2004
2 -1010 7/6/2004
3 -955 9/27/2004
4 -935 10/27/2004

I then opened the Bar Data Editor window , but do not know what to correct and how to correct them. The numbers such as -2500, -1000, etc does not mean anything to me.

-1000 = 10:00 (AM)
-935 = 09:35
-1400 = 14:00 (2:00 PM)
... these are not errors

Anything less than -1600, then, can't be in U.S. market hours. It means that the the trade price was not found in the intraday data for the date specified, so the priority is set lower than any "real" trade.

Since you're not getting the daily OHLC/V based on intraday data in the debug output, I must have not sent in my last update. If that's the case and you're using the latest version of Community.Components, you'll have to look through your intraday data to correct your daily bars.

How do I do that? I am not too uses to using WLP for the charts.

And are you going to send the last update, and to where?

QUOTE:
And are you going to send the last update, and to where?

Through the usual channels. To avoid duplicating the basics, everything related to extenion version checking is covered in the Extensions online guide as well as in the User Guide > Extension Manager.

But it seems that Cone is saying that since his latest version has not been sent, I should look through my intraday data to correct my daily bars. How do I do that? Sorry if this is too basic, but I haven't been using WLP for charts. Help...

I tried to download the community.components library, but I got a dialog saying that the extension is already up to date.

Oh, I think I got it. If I have problems, I will let you know.

I have a more important and broad question for you. It seems like there were several days for this particular stock that the intraday data is not agreeing the fill price if I am correct. And in this case there were 2 stocks each with several instances during the back testing period of 1 year. Is it worth it to correct these bars ( time consuming and tedious?) I guess if Cone send in the latest version, then maybe this process could be automized; is that correct? Otherwise would it make a noticeable difference in the APR for the year so that I would want to go through the time consuming process?

And Cone, would you send in your latest version right away?

You have the latest version. The debug messages are indicating that the the trades are outside the date range of the intraday data. Fidelity data only goes back to April 2004, so the trades in February 2004 cannot be evaluated.

If there's a spike, then the message will be "Probable data spike..." and the suggested OHLC/V correction will be provided. You still have to correct the data manually using the Bar Data Editor.

But number 3 onward: 4/16/2001 etc. are within Fidelity's data range, and I am able to see the 5 min data. So what are these items telling me? I guess that those need to be corrected in the daily bars, right?

And if I am correcting the data, am I supposed to always assume that the intraday data is correct, and hence I should correct the daily bar?

QUOTE:
So what are these items telling me?

I've already answered this about 4 hours ago. (see above)

QUOTE:
I guess that those need to be corrected in the daily bars, right?

Wrong. What I just said 20 minutes ago was that if you need to correct data, you'll get a "Probable data spike..." message.

QUOTE:
And if I am correcting the data, am I supposed to always assume that the intraday data is correct, and hence I should correct the daily bar?

You can do whatever you think is right. There are no guarantees.

Cone, did you mean to say: " anything LARGER than -1600...." rather than "anything less than...?" Because -1400. -935, etc. are all larger than -1600 and not less than -1600. This is from your post about 4 hours ago as you suggested.

Like I said, the items starting at number 3 were all correct data and within the US trading hours; so I don't understand what the problem is; why are these items there in the error window?

Could someone answer this for me please? It would be very helpful if I could make progress. Thanks.

I meant what I typed. If it's LESS THAN -1600 (e.g., -2500), then the corresponding "time" cannot be in U.S. market hours. Again, it's an invented number to make the priority lower than any of the other trades that actually occurred in U.S. market hours.

It's not an error window. It's a debug window. It's a convenient way to display information. But if you're confused about the information displayed, then you can ignore it.

What do you mean by a " debug" window? What is to debug?

And I ran another test, and I got a bunch of messages like this: MACE: Priorities could not be set.
C:\Users\owner\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\FidelityStaticProvider\5 minute\M\MACE.WL not found
1 0.884839483483154 4/21/2004
2 0.0266640451860912 6/3/2004
3 0.907593183642064 6/18/2004
4 0.472252475783346 7/7/2004
5 0.231115176450049 12/3/2004

What does this mean, and what do I do?

..FidelityStaticProvider\5 minute\M\MACE.WL not found means...
that the FidelityStaticProvider
5 minute data
in folder M
file MACE.WL could not be found.

If you don't have intraday data, then you cannot set the time-of-day priority, therefore you get the WLP default priority, a number between 0 and 1. Since this number is far GREATER THAN -930 (the highest time-of-day priority), it is a very high priority for the simulation.

Debug window - search User Guide.

Among the "trades" listed below, from the 3rd one down, the "trades" do not exist in the "Trades" tab in the back test window. It seems that with the exception of the first two which obviously have very low priorities (there was no intra day data for these, only daily,) these are non-existent trades. So why is the Debug and Error window bringing these trades up as problematic ones?

AAN: (1) EOD data not in intraday date range for trade on 7/26/2007
AAN: (1) EOD data not in intraday date range for trade on 9/20/2007
1 -2500 7/26/2007
2 -2500 9/20/2007
1 -1510 7/26/2007
2 -1005 8/9/2007
1 -1600 1/18/2007
2 -1025 8/16/2007
3 -1355 11/12/2007
1 -1505 8/1/2007
1 -1255 8/16/2007
2 -940 11/13/2007
1 -1330 10/30/2007
2 -1050 11/16/2007
3 -1520 12/12/2007

And as to the first two trades that did appear under the "Trades" tab which have daily bars but no intraday data, are they valid trades? Could the trades have been executed only with daily bars since my code was written on daily?

Could someone please try to answer the above question for me? I really need to make some progress. Thank you so much once again.

It seems like a lot of these items in the debug windows have daily bars but no intra day bars ( " No Data Available.") Is it possible for the system to have generated the trades without intra day data? My script is written on daily with limit price entry.

For the intraday priority to work as intended, it is required that you download the 5-min intraday data for each Daily symbol included in this backtest.

Then why are there trades in the "Trades" tab for those stocks that did not have intra day data?

Cone's good programming i.e. graceful error handling?

Are you serious about the last answer?

I did have the 5 min data set for the US stocks. I, however, highlighted the daily data set when running the back test since I think you told me to do that, right?

I re-downloaded the 5 min set after deleting the previous sets, and ran the back test again, and got very different APR than before: less than half of what I got before. One thing that I can think of is that previously I had 2 sets of 5 min data sets from Fidelity in my "Data Sets" list, but highlighted the Fidelity Daily Data Set when I ran the back test whereas this time, I had deleted both 5 min data sets that I had before, and re-installed one Fidelity data set, highlighted the daily data set, and ran the back test. Would that make a difference? I would like to hear what you think about this issue.

If you want my personal and final opinion on this overgrown thread, then the very first reply from 8/2/2011 7:12 AM said it all:

Consider tweaking the code to run intraday and use the "Position Options" to limit the number of intraday positions.

P.S. If you consider this, note that using fractional values for ATR series is meaningless and the trading loop is best started at bar #4 (minimum).

But Eugene, if you could be a little patient with me, I ran the same back test for the third time on the newly downloaded 5 min data set from Fidelity, and I got identical APRs 3 times in a row: since I started testing with the newly downloaded 5 min Fido Data set.

Previously I had 2 5 min data sets from Fido on my Data Sets list, and could it be that it was drawing data from both sets duplicating the result? In all times, I had the Daily Fidelity Data Set highlighted when I ran the back tests. And I also manually put "Daily" in the Scale box when it always showed 5 min as a default when I clicked, and opened the strategy.

It seems like, so far that deletion of 2 previous 5 min data set and installation of new 1 5 min data set has been the only change before I saw a drastic difference in APR.

I have invested so much time already in this method, and it would be really traumatic for me to go onto another method, and go through the learning curve again. The time consumption issue is becoming a major one here. What do you suggest? Thanks once again Eugene.

1. Your strategy runs on Daily data.

2. It creates trades without intraday data.

3. All Positions (trades) have a random priority, by default a number between 0 and 1.

4. However, you need intraday data to generate time-of-day priority (only). Again, intraday data is not used to create trades (see #1).

5. If you have intraday data for a symbol, Position Priority will be assigned a number between -930 (the highest intraday priority) and -1600 (the lowest intraday priority). You should understand what these numbers indicate, but if you don't, it's not important - you can ignore them because it's "working".

6. Since you don't have intraday data before April 2004, it's impossible to assign time-of-day priority to those positions. Therefore for trades before 4-2004 (or for any trade for which you don't have intraday data) a very low priority is assigned ... -2500. But look, even trades with low priority can be accepted in a backtest if the Portfolio has enough cash to buy it.

7. Finally, if a trade occurs within the date range of the intraday data, but the entry price is outside the intraday high or low for the entire day, almost surely the daily data has a non-tradable spike in it. You have to correct those daily bars.

All the answers are summarize here. This poor dead horse has beaten long enough.

QUOTE:
when it always showed 5 min as a default when I clicked, and opened the strategy.

You know why this occurs right? This should not be confusing in the least bit.

QUOTE:
The time consumption issue is becoming a major one here.

Exactly why I made a repeated suggestion (and it seems like everything is repeated on this thread a couple of times) of using the easiest method requiring only two inputs: the intraday data and the PosSizer's option. What could be easier than ticking a checkbox?

But Eugene, please read Cone's post on 8/2/2011 at 8:53 PM. My script is written on daily. What do you say? Would your idea still work?

My idea will work only if you tweak the script for application on intraday data. The choice is entirely up to you incl. what you're going to do about "ATR.Series( Bars, .01 )".

I would love to give it a try to code intraday if you could kindly give me some help. I know you have been so helpful for me, and I don't know how grateful I am to you especially sitting out here as a C# Newbie struggling. I would love to give it a try, and learn to code my strategy intra day, but I really need your guidance.

Would that be something that you might be able to do? thanks a million once again. I am a very motivated and excited student of C#.

Hi Eugene, any interest? I would really love to give it a go. That would be really exciting, challenging and rewarding for me!!! Thanks a million again.

Considering very high memory requirements of running a backtest on 7 years of 5-min intraday data on thousands of symbols, sticking with Daily would be the "lesser evil".

Eugene, I have a fairly powerful computers, and I can let them run over night. I think the more exciting thing for me is to try to code the intra day coding. Do you think maybe we could give it a go? I am running backtests on one year at a time any way and not 7 years. I have 8 gigs on both my PC and my laptop with Intel i-5.

1 day of 5-min data for a stock is ~20 Kb, or 5Mb a year. Thereby 1 yr of 5-min data for 3000 stocks takes ~15Gb RAM.

You may own 8Gb and at the same time don't have them: operating system's own needs, other processes in memory, memory fragmentation, lazy garbage collection in the heap of .NET 2.0. So, your nightly backtest has a good chance of dying with out of memory issues in the middle of the night, even if there are about 1500 stocks (5-min, 1 yr).

That was not a good idea of mine, I admit.

Do you think that I could do 6 months or 3 months at a time? The point for me is to learn to program intraday, and I will still run Cone's daily stuff for my main use, but the intra day stuff might come to use on fragments of time where I might want to compare results when I am a little unsure about the daily one's results because I do feel unsure about some of the daily tests that I have done so far. Then I can go back, and run that little segment of time only, and compare. But the true thrill to me is to try to program intra day!!! You do this kind of stuff probably in your sleep, but you would not believe how exciting it would be for me to try to do this. I would feel like a genius!!

Joanna, even though our constant participation on the forum may have created this illusion but the software didn't come with an on-demand consultant attached to it available at any time for virtually any kind of questions. If you want to learn, please do us a favor and learn. You're not the only customer to keep asking "what do you think Eugene?", and there's absolutely no subtitute for own experience and learning through trial and error. Good luck!

We'll do Eugene.

1. Considering that my script is written on Daily with Cone's Take First Five Trades of the Day addition included, is my strategy taking trades based on the daily bars ( I have limit entry) or based on intra day data?

2. I am checking on items one by one that appeared on the Debug and Error Messages window, and when there is a " Data Spike" message, and I go check on the chart, the chart is already free from the spike that the error message was reporting ( I checked the OHLC given by the message against the intra day bar data and daily data.)

So when there is a " Data Spike" message, have the bars on the charts already been corrected, and hence I don't need to go in, and change it my self? Thanks guys.

1. Cone have already answered it (see his summary above.)

Eugene, Cone seemed to have said that the Priority thing looks for the entry price in the intra day bars. That is what I thought until now. But while going through the items individually, I found that there were many trades made on stocks that did not have intra day data. That is why I am asking the question. Because if the trades are made based on the intra day bars, then if there is no intra day data, there should not have been a trade, but in many cases, I see trades.

What happened?

Joanna:

Specifically, see 8/21/2011 5:44 PM.

Thank you so much Eugene. I know you guys think that I am Duh... or something, but as a Newbie, it gets very difficult some times sitting out here, going through all those different things. Thanks once again for your patience.

Now, could you very kindly try to answer number 2 in my post above of 1:04 PM today?

When intra day prices and the daily bar prices differ, do I always correct the daily bar prices?

And if the answer is yes, why is that? Are intra day data infallible compared to the daily data?

QUOTE:
.. hence I don't need to go in, and change it my self?

Incorrect. There is no automatic correction. The message is "Suspected Data Spike" (or something to that effect). It means that the routine was unable to find the trade price in the intraday data. A possibility is that your trade occurred almost at the exact low, but due to tick rounding for stocks (0.01), the actual "double precision" low of the bar was slightly higher - just a guess, I'm not sure if it's even possible.

Give me an example of one of these occurrences. What was the 1) symbol, 2) trade price, 3) date, and 4) debug window message?

I will get to that a little later, but for now, I have been changing daily bars on LULU including in early part of 2010, e.g. 6/22/2010 continuously, and realized that I will be doing this forever because I think what happened was that Fidelity did not adjust the intra day prices for 2 for 1 stock split. And hence, ALL the trades regarding LULU which were many many appeared in the Error window. I can't sit here, and correct these one by one which will be forever. Is there a way to contact Fidelity, and enter a stock split into this stock so that all the bars will be corrected at once? thank you.

QUOTE:
Is there a way to contact Fidelity, and enter a stock split into this stock so that all the bars will be corrected at once?

From the chart's context menu, have you tried "Process a Stock Split"?

If I do that, then do I need to go back into the daily chart that I already corrected, and reverse the change that I have made?

I chose the stock split in the context menu on the chart, and got a dialog saying " stock split processed," but the prices still remain at the pre split level. I closed the chart window, and opened a new one, then got the stock up on the chart again, but the prices still remain unchanged.

what do I need to do to see the post split prices on the chart? Do I need to exit and reenter WLP? I would really prefer not to do that because I have many things that I got the results for in the current WLP window that took me a long time and I am still using. Thanks.

Did you pay attention to the split date in that dialog?

Why don't you just refresh the chart? Right click -> Reload Chart History

I did both: paid attention to the split date and reloaded the chart, and got dialogs after both actions saying that it did complete the tasks, but still pre-split prices.

I just don't see it. LULU closed yesterday at 50.94 or 50.93 on every chart scale I've looked at. That is the split-adjusted price.

Take a look at the daily chart between 7/27/2007 and 8/8/2007, and match that withe the 5 min chart for the same data range. The 5 min chart prices are in the 30s and the daily is in teens: half the prices of the 5 min chart.

Although I can see the LULU started trading on 7/27/2007 in the daily chart, the intraday charts (I checked 5-Min and 30-Min) start on 8/6/2007, and 8/6 thru 8/8 is in the correct range. I just refreshed these charts and get the same start data each time.

I agree with you about the 8/6/2007 being the first day for the 5 min chart. But my prices for 8/6/2007 through 8/8 is double the daily chart prices, and reloading the chart did not work for me. I described all the things that I did above, the the prices on the 5 min charts remain at the pre-split level: above $30.

It makes no sense that a split is applied on those dates, and it cannot be coming from the server since the only split in the LULU series occurred in July this year.

That said, I might know why your chart isn't getting refreshed. You've selected LULU in a Daily DataSet, changed the
scale to 5-Minutes to look at the chart, right clicked and selected "Reload Chart History". I know it's not intuitive, but since you selected LULU in a Daily DataSet, it's refreshing the Daily data.

Repeat the procedure, but click on LULU in a 5-minute DataSet first. It must work.

THANK YOU CONE. It worked. I highlighted the 5 min data set, instituted a stock split, then reloaded the chart, and it did it!

I added you to my "Heroes List." You just joined Eugene on that list. Thanks again guys. You are on the money!

Do I really need to go through all the items on the Debug and Error window, and compare the 5 min chart to the daily chart to see if they match, and correct the ones that do not? First of all, not all of them are incorrect. Many of them match already, and even the ones that do not, many are different by a cent or so on one of the items of OHLC.

The problem is that there are simply too many of these items in the Debug an Error window, and when we are talking about correcting data for multiple years, it becomes ridiculous. I will be doing this for the next 7 months.

There must be a better way. PLEASE make some intelligent suggestions, and save me from this despair.

Save us from guessing about what your talking about again by copying and pasting the results from your debug window here.

Hi. Here are the items copied from the debug widow for just some years 2008, 2009, etc.

There are so many items, I am simply overwhelmed. Please make suggestions as to how I could take care of these data issues so that the backtest results could be deemed trustworthy. Thanks a million.

Wow! Your HUGE log broke the thread.

Initially I thought to outsource it to Pastebin.org and Pastie.org but they are refusing to accept it as it's "slightly" above their limit of 64 Kb.

Since it's over 74000 lines let me kindly ask you to reduce the size to something manageable (like a few dozen lines). We certainly have tasks these days other than parsing a 2 Megabyte log! Thanks.

Sorry Eugene, but now, you must understand why I was so overwhelmed. Here is a smaller sample: from a part of 2008 log.

~4700 lines is somewhat better:

http://pastebin.com/eSjeKiaa

20.AAN: (1) EOD data not in intraday date range for trade on 1/22/2008
-> You don't have the intraday data on this date for AAN. You have to refresh this symbol's intraday data to try to get it.

170. ADI: Probable data spike on Thursday, 11/20/2008 at 16.71
171. O:17.47 H:18.40 L:16.73 C:16.75 V:5436852
-> The intraday low is 16.73, but the trade occurred at 16.71. This one's close, so it's up to you if you want to manually correct the daily bar or not.


272. - 281. AGCO: (1) EOD data not in intraday date range for trade on 1/9/2008
-> Same as 20.

292.AGL: Intraday data missing for 10/10/2008
-> Self-explanatory.

402.AIQ: Priorities could not be set.
403.C:\Users\owner\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\FidelityStaticProvider\5 minute\A\AIQ.WL not found
-> In this case, you have NO DATA for AIQ in the 5-minute interval - the data file does not exist.


592.AME: Probable data spike on Monday, 10/6/2008 at 23.51
593.O:36.29 H:36.29 L:33.13 C:34.77 V:1538409
594.AME: Probable data spike on Thursday, 10/16/2008 at 21.47
595.O:33.27 H:34.68 L:31.62 C:34.50 V:1467610
596.AME: Probable data spike on Friday, 10/24/2008 at 19.9267
597.O:29.89 H:31.32 L:29.10 C:30.57 V:1229079
598.AME: Probable data spike on Thursday, 11/20/2008 at 18.77
599.O:29.15 H:29.84 L:28.03 C:28.21 V:1586410
600.AME: Probable data spike on Friday, 12/12/2008 at 18.33
601.O:28.40 H:29.46 L:27.45 C:28.81 V:1170884

-> Clearly the daily data for AME does not match the intraday data. In this case, the intraday data was not properly split-adjusted for the 3:2 split on 12/22/2010. Refresh the intraday data.

etc. etc. That's the process you have to go through when you see these messages. There is enough information for you to compare intraday to daily data to determine what's wrong.

Data is everything, but too frequently WLP may miss applying a split, or there's a real spike in the data, or a data file becomes corrupt, or you forget to add a symbol to the intraday DataSet... It will may be an arduous process, and maybe it takes you a day or two. Once you get through it, you'll have data that you can trust - and you should back it up.

The alternative is to ignore the fact that your data has problems.

Thanks. I, however, have a couple questions.

1. You seemed to have picked only the longer lines with text in them. For example, what about the lines between 21 and 169, 172 and 271, 282 and 291 etc., etc? Aren't those shorter lines errors that need to be looked into?

2. How does one "refresh" a "symbol's intraday data" like you suggested for 20, and more?

QUOTE:
2. How does one "refresh" a "symbol's intraday data" like you suggested for 20, and more?

A. When there's a small number of affected symbol, the most convenient option is to right click the chart and choose the option to "Reload chart history".

B. Otherwise it might be possible to use the
Data Tool for that. Since you already have a list of tickers to refresh, I'd do it like this:

0. Install Data Tool and restart WLP.
1. Create a new 5-min DataSet with only the intraday symbols to refresh.
2. Open up the Data Manager. Switch to "Data Tool". (You might need to close/reopen the Data Manager for the new DataSet to appear in the Data Tool.)
3. Highlight the new DataSet in the Data Tool. When the table of quotes is shown, click "Remove all data". This will erase all stored data for the symbols of your new DataSet.
4. Go back to Data Manager's "DataSets" view. Highlight the new DataSet and click "Update (Price only)". This will download the 5-minute data from scratch.

Thanks Eugene. I will try those.

Could you and/or Cone try to answer the question number 1 as it is very important an issue it seems? Thanks again.

I answered number 1 about 3 weeks ago on 8/21/2011 5:44 PM; Point #5.

So Cone, the shorter lines without any text like 282 to 291, are you saying that they are not errors or bugs but that they are listed there to simply show the trades that occurred with the priority ( time of day)included? If that is the case, why are they showing up in the Debug and Error window?

And if that is the case, are you saying that it would be okay for me to review and correct just those lines with text included in the lines such as " Intradaay data missing" " EOD data not in intraday date range for trade on 11/12/2008," etc, etc? Thanks.

QUOTE:
If that is the case, why are they showing up in the Debug and Error window?

See the supporting Wiki help page > Usage notes > #4.

This is a little embarrassing, but I can't seem to find the Wiki help page. I went to Wiki, and clicked around, but can't seem to locate the page that Eugene was referring to.

Setting Priority for AtStop/AtLimit Orders (Same link was posted above on 8/2/2011 8:53 AM)

QUOTE:
are you saying that they are not errors or bugs but that they are listed there to simply show the trades that occurred with the priority ( time of day)included?

Of course I'm saying that.

QUOTE:
If that is the case, why are they showing up in the Debug and Error window?

Already covered on 8/19/2011 1:08 PM (also above).

Thanks Cone!

1. Cone, the link is not working: neither the one that you just gave me nor the one that you posted on 8/2/11.

2. Also, this will probably sound very redundant to you, but since the data is such an important issue, I think I would rather risk being redundant. So are you saying that it would be sufficient for me to just review, and correct the lines in the Debug and Error window that have text included in the lines such as "intraday data missing," " EOD data not in intraday date range for the trade on such and such date," " probable data spike," etc, etc, etc? Thanks again for your patience.

3. Also when I looked up the word "debug" in Wikipedia, it seems to say that the word is similar to fixing defects. So even if you say that it is a "Debug" window rather than an "Error" window like you said before, why would the trades appear in the "Debug" window if there are no defects? Thanks a million.

1. I'm sure the link is working, and I'm almost sure that you're not looking at the right tab in the browser.

2. The numbers without messages are showing you the priority that is being set, and myself being equally redundant, it means that it works! In retrospect, I wish I has not added the information and had only put messages that needed attention. But by now you should know which is which.

3. There is no way I will answer this question again. Instead, run this script:

CODE:
Please log in to see this code.

2. Eventually we'll remove the excessive debug output.

Yes, I agree with you Cone. I think it would have been much less confusing for people like me if you did not add the trade log into the Debug window although they are interesting to observe. Because I kept thinking: if there are no defects, these could not be showing up on a Debug window because there are no bugs in these lines to fix. I guess another way to solve this problem would be to change the name of the window, and add something that would suggest that the window is not only for debugging and for errors but also for other information such as the trade logs.

There is no problem to solve. Kindly accept the fact that a program, like the one I just demonstrated for you, can use the debug window however it wants to.

Cone, I am trying to follow your instructions in the 9/8/11 6:13 PM post.

QUOTE:
403.C:\Users\owner\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\FidelityStaticProvider\5 minute\A\AIQ.WL not found
-> In this case, you have NO DATA for AIQ in the 5-minute interval - the data file does not exist.

Why doesn't the file exist? And how does this differ from "intraday data missing" message?

QUOTE:
-> Clearly the daily data for AME does not match the intraday data. In this case, the intraday data was not properly split-adjusted for the 3:2 split on 12/22/2010.

How did you know that there was a split on 12/22/10?

The file doesn't exist because you haven't requested data for it - at least not for the 5-minute timeframe. It's possible you did request data and none were returned too.

Missing data means that the data is missing for the day that the trade occurred. If the trade is after April 2004, the history is incomplete and the data should be refreshed.

Re: Split
I looked at the Daily chart, making sure that I had selected to display Splits (F12 > Chart Annotations)

Cone, the items on the Debug window that have -2500 as priority, did these trades actually go through in the back test, that is: were these trades actually included in the back tests as actual trades?

ALL trades go through. We're just changing the priority. Isn't this clear?

(This was implied in the answer on 8/19/2011 1:08 PM.)

Your post of 8/18/2011, 2:09 PM said:

QUOTE:
Anything less than -1600, then, can't be in U.S. market hours.It means that the the trade price was not found in the intraday data for the date specified, so the priority is set lower than any "real" trade.

The Usage Notes also says: [QUOTE]"Note that Position.Priority is set to -10000 for suspected spikes. Instead of correcting the data, you could use the Position Options sizer to "Reject a Position with Priority... Less than -1700" to filter these unrealistic trades."

Shouldn't I be setting the Position Options sizer to reject a position with priority, say less than 1700, to filter unrealistic trades such as the priorities set at -2500, etc? Because if the price was not found in the intra day data, and therefore the priority was set at -2500, I thought I might want to eliminate these trades from back tests so that I will get more realistic and accurate back test results.

And if the price was not found in the intraday data for the specified date, why wouldn't it say the same thing as some other items: " probable data spike detected?"

Also I am still not getting intraday data for many of the symbols on certain dates even after doing some "refreshing," and I was wondering whether I could just move on from these items, and assume that the chances are that the daily bars selected the trades correctly. But since there are many of these, do you think that I would be risking a noticeable inaccuracy in back test results if I were to do this to the point where I should really try to find other solution(s) for finding the missing intraday data?

QUOTE:
I thought I might want to eliminate these trades from back tests so that I will get more realistic and accurate back test results.

Precisely correct.

QUOTE:
why wouldn't it say the same thing as some other items: " probable data spike detected?"

There are 3 different cases that we've been through:
1. If the day exists and the price is out of the intraday range, it's always a suspected spike. (correct the daily data)

2. If the intraday data doesn't exist for the day of the trade, nothing can be determined because you don't have the data for that day. (try refreshing)

3. Finally, if the intraday data file doesn't exist at all, then that's the message about the missing .WL data file. (request the data)

QUOTE:
I am still not getting intraday data for many of the symbols on certain dates

If the dates are before April 2004, then that's expected. Otherwise, give me a couple specific examples; symbols and dates that have no intraday data.

QUOTE:
give me a couple specific examples; symbols and dates that have no intraday data.

DICS 9/16/08 TO 11/21/08,
EEP 9/16/08,
ENTN 1/7/08,
EQU 3/17/08, 6/4/08, 6/25/08, 9/10/08, 9/30/08,
ES 1/18/08
ETFC 7/11/08

DICS is no longer a valid symbol, so even though the data that you have is missing data, it's not possible to refresh it.

EEP - the data for 9/16/08 will be there if you reload it.

All of the following are problems with the data history on the server, so the only way to attempt to recover it is by calling Fidelity and ask them to restore the missing data.

ENTN - the intraday history starts in Feb 2008 (after the trade)
EQU - the intraday history starts in June 2010 (after the trade)
ES - a mess of holes in intraday data
ETFC - another 1-day hole in intraday data

It's up to you to reject the trades based on a the priority assignment, but all of these examples could in fact be valid trades.

How did you know that DISC was no longer a valid symbol? And if it stopped being a valid symbol starting on a certain date, is there a way to know what that date was?

You said DICS before, DISC now. Neither return on-demand data, so that's usually enough to know that they're not valid.

After that you can use any symbol lookup resource available to you. Surely AT Pro has one, otherwise just use Y! Finance. Really you've never looked up a symbol before?

Unless you have Bloomberg, Reuters, Premium Data, or some other provider that has access to dead symbols, finding the last date of a previously valid symbol would take lots of detective work - hardly worth the trouble when you can get Premium Data delisted stock history back to 1950 for a very reasonable price. (DISC, Discus Corp, delisted in June 1992. I don't see a DICS.)

Of course, I looked up symbols gazillion times before. Thanks any way.

I recall hearing about Premium Data from friends too.

Aside from the explanations that you cited in your post of 9/16/11, 7:40 AM, can you think of other reasons for my not getting intraday data? I am already a little surprised by how many different reasons that you gave me in the above-mentioned post because those were from a very small list of the items that I sent you, and I was wondering whether I should anticipate a lot more of these different reasons (other than the ones that you already discussed) in addition.

Hi. I am sure that you guys are working on the issue that I raised above, but the erroneous data issue is such a serious one: " data is everything." Otherwise, why are we back testing?

So I hope to hear some helpful things about this matter in the near future. Thanks once again for your work.

QUOTE:
I am sure that you guys are working on the issue that I raised above

What issue is that?

Re: erroneous data
Here's deal - sometimes the data that you get from a Provider, like Fidelity, is erroneous or incomplete. Sometimes the bad data comes right from the exchange, so it's bad for everyone. No matter how many times you download a symbol's data, the same errors will still be there. I will not engage in a data discussion here, but the point is to know what data are bad and either fix those data points or work-around them programatically in your script.

Before you do a serious back test, use the Bad History Data Check [Rev. A] script in the Utilities folder to check (and help correct) the DataSets (Daily and Intraday) that you use. Instructions are on the Summary page. It's not that easy to use, but it can save lots of time when you need to refresh history of multiple symbols. Back up you data first, and use at your own risk.

Edit:
Regarding the Summary in the aforementioned script, the holidays in Markets.xml were extended back to 1990 (I think) with the 6.2 release.

What do you mean by the "Summary in the aforementioned script?"

aforementioned script: Bad History Data Check [Rev. A] script
Summary: The Strategy Summary - it's a tab on every Strategy Window you open.

If the stock is no longer trading, what should I do with that symbol and where?

Either you have the data to set the time-based priorities or you don't. If you don't have it, you buy it. If you don't want to buy data from delisted symbols, then you can't set the proper priority for the "near-perfect" simulation test. What else do you want me to tell you?

I got 3 trades in early 2009 for AGCO that said "EOD data not in intraday data range..." I checked both the daily and intraday charts, and both start on 5/4/2009: after the 3 aforementioned trades. How did this happen? How did these 3 trades occur, and what should I do about this problem now?

I did reload the charts for both, and nothing changed.

Same thing with AI. Both daily and intraday data starts on6/10/09. So the previous trades that are showing up on the Debug window, I don't know where they came from.

Again with ALR,both the daily and intraday data starts on 7/19/2010, and the Debug window shows that there were trades on 1/13/2009 and 2/27/09 with priority at -2500. But if there were no higher trade candidates with higher priorities on those dates, these trades would have gone through in the back test. How could this happen when these trade dates were much before when the data started to exist in both the daily and intraday. I am assuming that the stock did not start trading until 7/19/10, but do you agree with that?

QUOTE:
I checked both the daily and intraday charts, and both start on 5/4/2009: after the 3 aforementioned trades.

Impossible.

First, you can't create a trade if the data isn't there. Obviously the daily data exists before 5/5/2009 if a trade has been created there.

Second, since the intraday data does start on 5/4/2009, a trade create before that time is "not in intraday data range", just like the tool is telling you.

Cone, try it yourself with AGCO daily chart. I still get 5/4/2009 as the first day that the daily chart shows data. I reloaded it several times, but it is the same. And like I said before, there are 2 trades before this date: 1/15/09 and 2/10/09.

And obviously the same thing is happening with ALR which is the paragraph that I added via "Edit" in my 11:17 am post. I don't know if you have read it. You can try both these symbols your self, and let me know.

Again, same thing with ARC. The data only begins on 1/3/2011, but there were a whole slew of trades before that time: lots and lots.

To check your Daily charts, click on a Daily DataSet - like the one you're actually using to run the strategy.

I'm too lazy to find the thread, but we've through this before, recently. If you rescale your intraday DataSets to Daily, then you'll be looking at rescaled intraday data. So, that "Intraday-scale-to-Daily" chart can't show you anything more than the intraday range.

Question number 1: Cone, on ALR there were 2 dates that suggested:" EOD data not in intraday date range for trade on 1/13/2009" and on 2/27/2009.

However, as you can see on my paste below, there were many more trades that went through with priorities assigned.

When I looked into the intraday chart of ALR from 1/13/2009 to 12/27/2009 with the intraday data set highlighted, I get " No Data Available" message. I reloaded the chart twice, but nothing showed up. How can all these trades have occurred with priorities assigned when there were no intraday data? Thanks.

ALR: (1) EOD data not in intraday date range for trade on 1/13/2009
ALR: (1) EOD data not in intraday date range for trade on 2/27/2009
1 -2500 1/13/2009
2 -2500 2/27/2009
1 -935 12/30/2009
1 -1435 1/12/2009
1 -1110 2/20/2009
2 -1350 3/6/2009
3 -1450 10/28/2009
1 -1140 3/2/2009
1 -955 1/9/2009
2 -1245 1/14/2009
1 -1045 1/7/2009
2 -940 2/27/2009
3 -935 3/3/2009
1 -1425 10/28/2009
1 -1045 6/15/2009
1 -935 10/2/2009
2 -1400 10/27/2009
1 -935 4/21/2009
1 -1020 2/26/2009
2 -935 3/2/2009
3 -1335 3/6/2009
4 -935 4/15/2009
5 -1105 5/18/2009
6 -935 9/25/2009
1 -1345 1/13/2009
2 -1020 1/15/2009
3 -1140 2/20/2009
4 -1045 3/6/2009
5 -955 3/30/2009
6 -1230 7/8/2009
1 -1520 2/27/2009
1 -1135 7/29/2009
2 -1145 10/2/2009
1 -935 2/27/2009
2 -935 3/30/2009
3 -935 4/3/2009
1 -935 2/20/2009
2 -1030 4/21/2009
1 -1515 1/13/2009
2 -940 1/21/2009
3 -935 2/2/2009
4 -955 2/5/2009
5 -1035 5/8/2009
6 -1200 5/13/2009
7 -1010 10/22/2009
8 -1130 10/26/2009
1 -950 2/2/2009
2 -940 2/20/2009
3 -940 3/2/2009
4 -1215 5/15/2009
5 -1015 8/17/2009
6 -935 9/22/2009
1 -935 1/8/2009
2 -1245 3/6/2009
1 -940 1/8/2009
2 -1035 1/15/2009
1 -1540 10/28/2009
1 -935 2/20/2009
2 -950 3/20/2009
1 -940 2/11/2009
2 -935 3/6/2009
3 -940 5/21/2009
4 -955 6/17/2009
5 -1025 9/24/2009
6 -1145 10/1/2009
1 -935 1/8/2009
2 -1210 1/30/2009
3 -1015 2/18/2009
4 -1140 3/3/2009
5 -940 3/17/2009
6 -935 3/30/2009
7 -940 5/13/2009
8 -1020 6/8/2009
9 -955 6/17/2009
1 -935 6/2/2009
1 -935 2/17/2009
2 -935 2/27/2009
3 -1145 3/3/2009
4 -935 5/14/2009
1 -940 2/20/2009
2 -955 3/2/2009
3 -1235 7/8/2009
1 -1155 10/26/2009
2 -950 11/6/2009
3 -940 11/16/2009
1 -1355 2/11/2009
2 -1320 6/22/2009
1 -1005 5/14/2009
2 -935 7/28/2009
1 -935 3/6/2009
1 -935 5/12/2009
2 -1005 6/22/2009
1 -935 1/21/2009
1 -1305 4/13/2009
1 -935 1/13/2009
1 -1310 1/13/2009
2 -940 1/15/2009
3 -1020 2/18/2009
4 -935 6/22/2009
5 -1235 8/24/2009
1 -1040 3/30/2009
1 -935 6/12/2009
1 -1030 7/6/2009

Question Number 2: What do the numbers at the far left side of each line represent such as 1,2,1,2,etc?

Better late than never. Joanna, have you considered using either one of the two half-way methods: LimitPriorityLong and IntradayFillPriorityEstimate, found on the same page with Setting Priority for AtStop/AtLimit Orders?

They make no use of intraday data and produce no debug messages (like SetTimeOfDayPriority) and don't consume RAM (like applying "Position Options" on intraday data would). For that ease of use there's a trade-off: they're not as precise as using intraday data - they estimate a limit order's probability of being hit first using some logic revealed here.

I have invested so much time in Cone's method, and now I have to start from scratch again on a brand new method which I do not understand at all. I am not exactly a C# programmer.

Could you guys answer my 2 questions that I posted above at 8:42 AM any way please? Thank you.

Again it's time to remind you of my "All the answers are summarized here" post:8/21/2011 5:44 PM

See item 2.

QUOTE:
4. However, you need intraday data to generate time-of-day priority (only). Again, intraday data is not used to create trades (see #1).

5. If you have intraday data for a symbol, Position Priority will be assigned a number between -930 (the highest intraday priority) and -1600 (the lowest intraday priority). You should understand what these numbers indicate, but if you don't, it's not important - you can ignore them because it's "working".

I AM referencing what you said on 8/21/2011 or I think I am. My Question number 1 at 8:42 AM was: how could all those trades (many many as you can see on my paste) have received time-of-day priority when there was no intraday data on those days? These priorities are well within US market hours meaning that they could very well have been executed.

Could you please also kindly answer question number 2 above. Thanks

QUOTE:
I have invested so much time in Cone's method, and now I have to start from scratch again on a brand new method which I do not understand at all. I am not exactly a C# programmer.

Of course it's up to you. Just an illustration that a simple technique, if discovered by carefully reviewing the Wiki page, could've saved lots of time and effort.

I took a look, and got confused and scared. I will give it another look Eugene.

In the mean time, Cone, please try to answer my 2 questions in the 1:26 PM post. Thanks.

If the priorities were set between 0930 and 1600, the data are there.
If the priorities are set outside this range, the data are not there.
The first 2 trades tell you that the data are not there.
The data are there for the rest of the trades.

What is the problem with this?

Cone, please re-read my post of today at 8: 42 AM carefully. The data ara NOT there for all those trades below the first 2 trades, and they have priorities set within US market hours as you can see in my paste. Like I said in the post, I reloaded the chart multiple times with the intraday data set highlighted like you previously told me. I did it for almost the whole year of 2009. The data are not there. So how can there possibly be all these trades with their priorities set?

Also could you please answer my question number 2 of the 8:42 am post? Thanks.

Actually, I just realized that your script is outputting all that verbose information because you copied the example from the Wiki (now modified to comment out those lines). Consequently, the SetTimeOfDayPriority, does output only the information you need to see.

First, we're going answer these questions by modifying your script. In the /* Check priority */ block

CODE:
Please log in to see this code.

And run your simulation. You should be able to answer both questions yourself after that.

After that, I want you to delete the /* Check priority */ block of code altogether so that it never confuses you again.

Hi. I have been struggling to understand the above mentioned code in English so that I knew what was going on. I went through my C# book, and got bits and pieces understood or so I hope, but I still have some difficulty putting the whole thing together, and I was wondering whether you could be kind enough to lend a hand.

It seems like the first line is a comment.

As to the second line, you seem to be telling the computer to print something on the Debug window: the stuff that is within the ensuing parentheses. And what is within those parentheses seems to say to print: j+t ( the letter t rather than a tab) + p.priority, and this item is something that I am not clear about. Does p.priority refer to priority of positions? And then after this item, there is: + t ( the letter t) + p.Entrydate.ToShortDateString(). This last item, I do not understand.

3rd line is a comment again.

The 4th line seems to say to print in the Debug window what is in the parentheses,and that is: ...... I guess it might be easier for you to explain the 4th line to me rather than having to read what I say that I think what it might be.

I know that I could just copy and paste the code, but it would be so much better if I understood what was going on which, I believe, would help you later too because I might ask less of the really[i][/i] stupid questions later on. Thanks again for your patience.

Joanna, support doesn't include teaching you how to program and read the code. To support your decision to learn how to code, there's a large number of free web resources, books, community colleges, there's the QuickRef, WealthScript Programming Guide, the Wealth-Lab Wiki site. To stress my point: interpreting every line of code just because it's unfamiliar to you, is not covered by our support policy. Feel free to experiment and learn by example, trial and error.

You don't have to understand the code. Just do what I'm asking:

1. replace the first line of code with the second.
2. run the backtest
3. look at the results in the debug window to answer your own questions
4. Finally, after you finish steps 1 to 3, DELETE that block of code, save, and Compile. When you do that, only the "important" things will appear in the debug window and we'll never have to go through this again.

I will do that Cone. The, reason, however, why I asked about the content of the code was because my strategy is long only, and I did not understand why there was "To.ShortDateString()
" in the code. Thanks.

I have been correcting data: a very tedious and time consuming job, it turns out.

I am trying to put in a stock split on AMX which occurred on 7/21/2005 on the 5 minute chart. When I right click, and pick process a stock split, the chart to the left of 7/21/05 goes down 1/2 in price which is the right thing as the split was 2 for one, but the chart starting on7/21/05 remain at the previous level. I did this again, but the same result. Could you please look into this, and advise? Thanks.

If you're having a problem with the manual split, why don't you just right click and "Reload Chart History" instead?

I did that, and the chart reverted back to the pre-split prices.

Let me remind you of what is occurring. You have to be careful about clicking on a Daily DataSet and then swithing the chart scale, and then using Reload. You'll end up reloading for the DataSet scale, not the chart scale.

In other words:
1. Find a Fidelity 5-Minute DataSet
2. Click on AMX.
3. Right click and reload.

There will be no need to manually correct that split.

1. I get the following message in the Debug window:GGC: Probable data spike on Friday, 10/24/2008 at 54.71
O:55.00 H:58.00 L:55.00 C:55.50 V:14732

And so went into the 5 minute and the daily charts, and the $54.71 price does not exist in neither one of the charts. What happened?
And the same thing is happening over and over again: the probable data spike message with the spike price, and the spike price could not be found in either the daily or the 5 min chart. Does this mean that although the spike price was not there in either of the chart, that price generated and filled a trade?
E.g., there is this message: MDY: Probable data spike on Tuesday, 9/30/2008 at 120.19
O:129.33 H:131.87 L:127.15 C:131.83 V:8385796
But the spike price: 120.19 cannot be found in neither of the two charts, but was an order filled at that spike price?

2. I get this message: VRX: Probable data spike on Friday, 10/10/2008 at 7.17
O:15.41 H:15.88 L:14.58 C:15.00 V:2595408

There is no price even close to the spike price in either the daily or the 5 min charts. It seems awfully like there should have been a stock split since the spike price is close to 1/2 of the chart prices, but I couldn't find a stock split symbol on the charts. Do you see a stock split anywhere? Thanks

QUOTE:
GGC...and the $54.71 price does not exist in neither one of the charts

Sure it does. The Daily bar's Low AND Close is 54.25.

Look Joanna. You keep making the SAME MISTAKE OVER AND OVER (sorry for yelling, but I'm just not getting through any other way). When you want to see the Daily data, click the symbol in a Daily DataSet. When you want to see the 5-minute data, click the symbol in a 5-minute DataSet.

REPEAT: If you're looking at a 5-minute DataSet symbol, and rescale to Daily, then you're looking at the 5-minute data rescaled to Daily! Obviously if the price isn't in the 5-minute data, then it won't be in the Daily-scaled 5-minute data.

This was a thread that we worked on a while ago.

In the code that Cone suggested:
PrintDebug(Bars.Symbol + "\t" + j + "\t" + p.Priority + "\t" + p.EntryDate.ToShortDateString());
what was "\t" again please?

Joanna

It's the escape sequence for inserting a tabulation.

Escape Sequences

You could find the answer in under a minute if you used Google.

Voila! Never underestimate the power of Google.

Hi Eugene, and thanks Eugene.