As Strategy Monitor does not provide same functionality as manual runs and as I was referred to ask this question in the forum, here I am:

What is best practice to have one strategy run automatically once a day on a certain time going over different Datasets?

In the Strategy Monitor (if not other required functionality doesn't work their either):

As GetDataSetName does not work I guess I would have to name copies of the very same strategy like Script1, Script2, etc. and point them to individual Datasets?

Outside the Strategy Monitor, like clicking a couple of script once (in Multi Symbol Mode) and let them loop "forever" (in WL terms we know this might be not too long):

Update is supposed to run automatically at the scheduled time, but will the strategies automatically use the updated data from Yahoo and Google EOD data or stick to last bar as the bar which was there when execution was first initiated?

What is the best way to realize auto runs? Code a pause until the next 24hrs and then loop thru the code again? Can this be done in Multi Symbol Mode - as Single Symbol Mode crashes the PC after a few runs over S&P 500, not even intraday but EOD data? Can I use .NET timers in WL strategies?

As WL6 has a lot of problems which appear to be memory problems, who has practical experience keep WL open for weeks and months executing a couple of scripts?

Is there any difference in terms of stability and functionality when coding the strategy in Visual Studio and using dlls (as there is some obvious difference in writing to Excel files, i.e. code from VS can much better integrate Excel interop while doing the same from within WL has some restrictions)?

Any other advice to come to such a solution and keep it stable up and running is very much welcome. Thank you!

Remark about Strategy Monitor: It is not working! The very same script runs fine in editor mode and executes (=writes to file) over the S&P 500 symbols but does not do the same in SM. Always this Error(4): Index out of range. Must be...

Sometimes it lists the (for today) 7 symbols in the alert window, sometimes it doesnt. But never does it write the file (again, which it does in editor mode).

Generally: The chart loops also thru all 500 symbols at the start on the very first symbol only (all coded properly and within try/catch) and then loops thru them "natively" in SM (also write to file is within try/catch and using). So what to make it run like in manual all symbol mode?

Also changed the EditorSettings.xml to EditorSettings.bak as mentioned in the Wiki. Man, this WL6.1 is far from being a proper/stable software.

I don't want to argue about WL6.1 stability and x64 memory issues, as you know I ran repeated tests of your custom code in 64-bit environment, having noticed no memory leaks. It could even be some subtle issue in a RAM stick makes the whole difference, there's no way of telling it from here.

QUOTE:
Always this Error(4): Index out of range. Must be...

This is an exception happening in Strategy Monitor, anyplace during the strategy execution, and can be caused by data provider. Which data provider are you using in SM?

QUOTE:
But never does it write the file (again, which it does in editor mode).

This could mean that when that exception during execution happens (see above), it happens before the file write logic.

QUOTE:
As GetDataSetName does not work I guess I would have to name copies of the very same strategy like Script1, Script2, etc. and point them to individual Datasets?

Yes, that should work given the inability of GetDataSetName to function in SM.

Possible alternative that should work in Strategy Monitor: DataSet name. This approach will not work as expected if the symbol is not unique across DataSets.

QUOTE:
Is there any difference in terms of stability and functionality when coding the strategy in Visual Studio and using dlls

No difference. Built-in Editor is unavailable for compiled strategy libraries (natural limitation) and Strategy Parameters are missing when accessing a compiled strategy from the "Home Page" tool (bug). The ability to use VB, .NET 3.5, obfuscate code far outweighs these limitations.

QUOTE:
as Single Symbol Mode crashes the PC after a few runs over S&P 500, not even intraday but EOD data?

Just a note. When developing the Database provider, it was repeatedly tested on a database consisting of the whole NYSE/NASDAQ/AMEX universe of stocks. Hundred thousands trades, on a 32-bit OS, but no crashes.

Potential hardware issues aside, it's a misconception to think that our own code i.e. DLLs, strategies is not to blame. If "Bandwagon Trade" or some other simplistic preloaded code crashes the PC, it could be time to examine the hardware. If it works stable, consider profiling and optimizing your complex custom code.

Well, Strategy Monitor still does not write my file. It still writes the alerts in SM window, so as advised I took out anything before the file write logic. Didn't help.

A plain and simple strategy (written to check SM + filewrite) writes all 500 symbols to a file: works in editor and in SM. What I realized here - not my core problem yet but still annoying - is that in editor it loops "correctly" alphabetically thru the 500th S&Ps, but in SM it does the same in reverse order! So 'A' and not 'ZMH' is processed as the 500 symbol (line) [even when flushing the filewriter after every write to make sure it is written as being processed].

As there is still no solution to my problem I would like to ask again:

If I run a strategy (or more) permanently with an endless loop putting in a Thread.Sleep() to pause for a day will this work on updated bars or still only use the last bar from the very first/initial run?

Or if this would not work, can I run a script (even in SM if better for stability) which as such would loop and pause (or being triggered by a timer) and call/run/execute my other scripts so that they would run over meanwhile updated data (if so, is there an example somewhere how to run/start other scripts from within a script)?

I do not ask again if this would be stable as I know your answer (despite reality).

Thank you.

QUOTE:
Well, Strategy Monitor still does not write my file. It still writes the alerts in SM window, so as advised I took out anything before the file write logic. Didn't help.

In this case, the exception mentioned in my reply 4/12/2011 5:29 AM, might be caused by data provider. Have you tried replacing the data provider in SM to see if it positively affects the Actions column and, consequently, SM file writes?

QUOTE:
If I run a strategy (or more) permanently with an endless loop putting in a Thread.Sleep() to pause for a day will this work on updated bars or still only use the last bar from the very first/initial run?

I'm not afraid to tell that I don't know the answer to your question because I have never tried it.

QUOTE:
(if so, is there an example somewhere how to run/start other scripts from within a script)?

There are two pretty much different approaches I'm aware of:

1. Simple: WealthScript Techniques | MultiSystem, or Calling A Strategy From Another

2. Undocumented API calls (unsupported): WealthScript Techniques | Interacting Dynamically with Portfolio Level Equity (this is "running other scripts" literally)

Of course I'm not giving any guarantees it should work for your task. Again, I have not enough experience in this field of executing code by a timer.

Come on! Why the heck does the Strategy Monitor process all symbols in descending order! This is neither (well/at all) documented, nor did you guys locate/realize that issue in the trouble tickets/forum postings - despite having the code upon your request and following your advice to use

QUOTE:
if (Bars.Symbol == DataSetSymbols[0])

in a Strategy Monitor loop to operate only once on the first symbol. I have to enter lengthy discussions with your tech support. With WL6 I feel like a beta tester. I am pretty sure you will tell me that it is not a bug (maybe it is even a feature) that running a stratetgy in the editor over all symbols process them in ascending order, but then in Strategy Monitor it does the same the other way around! And the fun thing is, it still gives the alphabetically first symbol which it does process at the very end of the run the index[0] and starts to run the strategy with index[DataSetSymbols.Count-1]!

a) Is it at least save to assume that SM always turns the symbol sequence upside down, so that one can use different code for manually operated vs. Strategy Monitor run strategies?

b) And is there a method or a work around (as those many others) to request if a strategy got executed from the editor or the SM.

b) Sure there is. In the QuickRef, see ChartPane object > IsPricePane Property and Cosmetic Chart > ChartStyle Property for a code example.

a) If this is a feature, then I agree -- not properly disclosed/documented and not intuitive. We'll have to inquiry the developers about this one.

QUOTE:
b) Sure there is. In the QuickRef, see ChartPane object > IsPricePane Property and Cosmetic Chart > ChartStyle Property.

Using your sample code in a brand new strategy shows "Operating in Strategy window" when run from editor, both when a single symbol is clicked as well as Backtest on all Symbols.

But using it in my script the following way

CODE:
Please log in to see this code.

it does the following: It shows "Editor Mode" in debug window when a single symbol is clicked, but when Backtest on all Symbols is clicked (still from the editor, not the Strategy Monitor) I get "Strategy Monitor Mode"!?

The code above is very much on top of my strategy. Later on I check (now) with

CODE:
Please log in to see this code.

and then loop thru all the symbols with

CODE:
Please log in to see this code.

But this should not influence to get a run in "Strategy Monitor Mode" when run from the editor!

For me, in SSB the PricePane.IsPricePane code example works correctly and in MSB mode, states "Strategy windows" for the 1st symbol of the DataSet and "Strategy Monitor" for all the rest.

My records show, I updated these QuickRef entries before 5.4 so there's a chance of a change happening since then, behind the scenes.

We'll again have to find it out from the horse's mouth, too.

QUOTE:
1st symbol of the DataSet and "Strategy Monitor" for all the rest.

Actually, it indicates "Strategy Window" for the symbol whose chart you displayed last before clicking the DataSet. It makes sense since charts haven't been rendered for the other symbols, and it indicates that the ChartPane object exists for the last symbol displayed. I didn't even expect that in MSB mode, but it could be a G.C. issue.

So still no news on that bug? So fact is, that WL does run strategies upside down in Strategy Monitor which is neither documented nor confirmed that it will be that way to at least use different codes for one and the same strategy in cases like this one here? And it is also fact that the advised method to check whether or not a code is executed from the SM does not even work too?

So you do not deem it necessary to further investigate/comment on your softwares problems? You obviously prefer always pointing at some dubious developers at Fidelity, which does not interest and help us customers of WL Developer at all. We purchased licenses from You and we pay fees every year to You. You this is MS123 LLC (or whatever the name). So it is You having an obligation versus us, not Fidelity or any other firms or mysterious people.

Another bug/issue from Strategy Monitor: Executing strategies over portfolios like SP500, EOD - not even intraday or soemthing, is sloooow, sloooow, sloooow. Despite having ticked off 'Update Data on Demand', it almost looks like SM requests updates on that data (in this case Yahoo); or whatever the reason might be to have a script taking 15 minutes in SM when being exceuted in less than 10 seconds from editor. Any comment and working solution(!) on that?

Thank you very much!

QUOTE:
So still no news on that bug?

The bug was entered, and currently there's nothing to comment.

QUOTE:
whatever the reason might be to have a script taking 15 minutes in SM when being exceuted in less than 10 seconds from editor.

By design, EOD strategies run in SM only after all data are collected, the updated data are saved to the local store and then only updates are requested (once a day). Is your SP500 DataSet up to date?

Data Update is set to 1:00 AM ET, Strategy to be run from SM at 1:15 AM over SP500, 1:20 and 1.25 over others. 1:30 an external program if signals in files written from those Strategies exist and email information to people in Europe (= 7:30 or if daylight saving differs between US and here it is 8:30).

It appears that BBFree updates European data way before Yahoo does, but interestingly Yahoo seems to update US EOD data before BBFree does, as my logfile from SP500 informed me that today only 100+ of the 500 were up-to-date (but all European was were updated). This is not represntative as I just start trying to figure out which EOD provider to use to have reliable updates at around 1:00 AM (would wish to have minute granularity not 1 hour in WL - or better to do it manually as it was possible in WL4, and way better as WL6 as individual watchlists could be requested.).

To make a long story short, I added my SP500 script to SM to run over Yahoo and not BBFree and it took those 15 minutes despite every single one of Yahoo's data base being up-to-date (=meaning yesterday's bar; maybe SM is trying to get a today's bar?). Anyway, if the script is not predictable how long it will run, I enter serious problems with my deadlines and/or the email program would not find SP500 signal file when checking/sending at 1:30.

Also, is it better to start all (in this case) 3 scripts at a certain time, e.g. 1:15 (after the update at 1:00), will they run parallel or in which order?

QUOTE:
(would wish to have minute granularity not 1 hour in WL

The Data Manager in v6.2 will have finer control over auto update: the dropdown box will contain localized time (no forced ET) from any hour.

The exact minute when update process triggers will be controlled via the RandomMinute parameter in WealthLabConfig.txt file. It's designed to trigger on a random minute within the specified hour to avoid flooding servers with simultaneous requests, but power users can override the setting.

QUOTE:
I added my SP500 script to SM to run over Yahoo and not BBFree and it took those 15 minutes

Have you noticed which exact step took that time, data collection i.e. "Symbol pending updates: 500" (and rapidly decreasing to 0) in SM Status bar, or the actual execution (when the list is getting filled with recent Alerts)? Were there any error/status messages in the Actions column? Is this a repeatable sequence that you can reproduce right now or was it a one time act probably causing by some symbols not returning updates when they should due to whatever?

Script in SM over Yahoo SP500: 14 minutes (replicable), about 2 to 3 minutes "Symbol pending updates: 500"

Script in SM over BB SP500: less than 1 minute, couple of sec "Symbol pending updates: 500" then creating alarms

Both scripts from editor (oder Yahoo and BBFree SP500): about 20 to 30 sec.

No errors in SM for both.

Additional Remark: SM should not execute a scrpt immediately when added to SM. There is a set time. And one can always manually execute it.

Additional Question: If changes are made to script XY (including compiling), is that being changed when file was added to SM before?

QUOTE:
Additional Question: If changes are made to script XY (including compiling), is that being changed when file was added to SM before?

No. As the User Guide states, SM operates with its own compiled copy of a strategy. Whatever changes being made to the original strategy after activating, are not reflected in its instances across SM while it's running. So to apply edits to the activated strategies, re-open the Strategy Monitor.

QUOTE:
Additional Remark: SM should not execute a scrpt immediately when added to SM. There is a set time. And one can always manually execute it.

What do you mean. This is exactly how it was designed and how it works.

QUOTE:
Script in SM over Yahoo SP500: 14 minutes (replicable), about 2 to 3 minutes "Symbol pending updates: 500"

After successfully requesting data updates, SM starts executing your code and then comes some bottleneck. I'd suggest trying on smaller DataSets, then DataSets made with different symbols, to see if it provides any clue. I'd then focus on profiling/debugging the strategy code, commenting out its sections and running the modified strategy to understand where exactly could be the possible bottleneck when working in SM.

Don't you read what I am saying: The VERY SAME strategy in SM over 500 S&P symbols from BBFree runs in less than a minute while it takes 14 minutes over 500 S&P symbols from Yahoo! Both data were updated! All tests were done several times. This has nothing to do with any code = it is the SAME!

Once again.

After successfully requesting data updates, SM starts executing your code and then comes some bottleneck. Try your code on smaller DataSets, then DataSets made with different symbols, and see if it provides any clue.

Run a vanilla strategy in SM over that S&P 500 DataSet (e.g. the included Trend Following/"Moving Average Crossover"). Any difference?

I don't share the 14-minute experience either - only about 90 seconds for me with the S&P 500. There must be something else at work that's delaying the Y! Provider from returning data quickly at your location.

After running the Strategy, click off of it so that you can see the status color code. Is it red for 14 minutes?

QUOTE:
So still no news on that bug?

Bringing in here some good news after a week since reporting: the disparity will be gone for good in 6.2, SM order of execution to be alphabetical.

I want to add that the Strategy Monitor never guaranteed order of execution, and in fact, for intraday scales less than 10 minutes there is absolutely no alphabetical dependence as indicated in the User Guide: Strategy Monitor > Behavior as a function of Scale.

Z to A is not a bug. Nonetheless, it makes sense to make the SM consistent with the Strategy Window where possible, and this has been done for 6.2.

!gub a ton saw ti taht raeh ot dooG

In case you wonder, for no reason and without telling anyone I often write from right to left. This is not a disorder and of course not my fault if someone cannot read it as he or she would expect to write from left right. So heck, it must be them who are stupid, not me, right? Or shall i say, ?thgir

While I will force to have my next phone book also being sorted from Z to A, I meanwhile wonder if a web request from a strategy run from Strategy Monitor also requires not to type e.g. "www.wealth-lab.com" but "moc.bal-htlaew.www", which clearly would be more intuitive, as I have to to recode my strategies anyways upside down.

As a hint for other users: I now code my strategies either by looking on my monitors via a mirror, and I sometimes decide to do a headstand - but again, only for typing in strategies to be run from SM.

It is good to know that tech support from WL does not see it as a bug, especially as it was them advising me how to code a strategy (for which they have the code, btw) for the SM using the index[0] for the first symbol. Well, thank you for all the days of lost time. And it is not only Z to A, it is index Count-1 to 0 which changes. Also this, of course, starting to count upside down in a computer language/program cannot be called to be a bug.


As you have once again made clear that WL, or lets say MS123 LLC staff is infallible and elbillafni, in case one would import this text into Strategy Monitor, I admit to mistakes I make: I guess it be the fact that I request

CODE:
Please log in to see this code.

which would cause the huge delay requesting S&P 500 Yahoo data via SM. So if it is this, it ofcourse is - as always - my mistake to use a function which works outside the SM okay, like in under a minute, but then may(?) cause the delay in SM?

And yes if this is causing this delay, you do NOT explicitly say that using SM can slow this or that down, or you probably did it in mirror writing somewhere in one of the many help files, like the Wiki, or the User-Guide, or was it in the QuickRef, maybe in the Programming Guide? Well, one can and always has to search the forum, and sometimes one should open a ticket, or should not open one - you never know, depends on MS123 mood of the day. But most likely it is a C# issue, as despite offering native functions WL decides to use .NET (well, the version 2 from the stone age) there is always www.dotnetpearls.com to answer questions for function not being implemented/documented. Glad my grandmother is a C# developer from Redmond and my cousin teaches at MIT. Both now live here next to my desk, to help me with the simple task of debugging my code - or wait maybe not so simple? I only need an extra programming IDE like Visual Studi, lean with only 1 GB, and then I can easily implement to 20 steps from the guide who explains it all.

!remotsuc yppah a ,troppus ruoy rof hcum yrev uoy knahT

P.S.: I better thank Eugene explicitly here as he once complained in a trouble ticket that he invented the Google streaming provider (I guess just for me) and i only did say thank you once in the relating thread, nothing more. So I herewith say thank you, thank you, thank you, that the bug in SM will be gone in 6.2. (even as some of your colleagues will obviously miss this great feature).

QUOTE:
nextEDate = Utility.GetNextEarningsDate(Bars.Symbol);

which would cause the huge delay requesting S&P 500 Yahoo data via SM.

If my understanding is correct, according to your debugging results, the (otherwise correctly working) method GetNextEarningsDate causes runtime delays in SM processing on large Yahoo! DataSets. This is a quite specific scenario, and we definitely are hearing it for the first time, from you. I hope that makes it clear as to why the Wiki online manual lacks any mention of it.

I'll debug that Community.Components piece of code to see if it can have something to do with SM delays in conjunction with large Y! DataSets.

Thank you for looking into it.

One remark as I just checked my code. I only call the method GetNextEarningsDate when a signal is generated. As this happens between 10 and 20 times during a typical scan of the 500 S&Ps, the method only gets called 10 or 20 times (on the than current symbol and only once as all is done on last bar only) and not 500 times which I thought before.

Maybe any call of this method requests the earning dates on all symbols of that particular DataSet, i.e. 10 or 20 x 500 (when done from SM)?

Unfortunately, earnings appear not to be the problem. I just deactivated the earnings request for Y! and it did change nothing, still slow. Still 30sec when run from editor, 1 minute when run over BBFree from SM - but only when run from SM over Y! data I get the delay. Tried it over WL-100 from SM, took 3 mins (which multiplied by 5 wold give me the same 15 mins as for 500 S&Ps).

SMA XOver sample script did execute well from SM over SP500.

The only special thing in my code are: To loop once extra thru all symbols but only on 1st Symbol (or last in SM) but I create some lists and write logfiles there, so this definetly only get exeuted once an not 500 times. And the other thins is I write to files but all streamwriters get closed correctly (as wee also see running the script over other than Y! sources).

This also happens on two different machines with one WL6.1. 64bit being relatively new and on a clean machine (all 6 or 8 GB memory).

But you may remember that I once opened a ticket where I realized that WL did slow down the execution in a progressive way: it runs thru the first symbols fast and then takes more and more time. But nothing changes. This all done by a watchlist loop after clicking a singel symbol. That's why I change the code to go over all symbols by itself which than - on the very same logic - did not slowdown. At least not from the editor.

I could assume that this has something to do with it. And I found another post the other day here where one did complain about the same thing I guess. I could imaagine that there is some hidden problem when accessing Y! data either over a symbol loop or from SM, maybe both have soemthing in common?

I really don't understand this and I have done everything from my side, at least what comes to my mind. I even analysed the binary structure of the data, so Y! and Bloom EOD (and others) are (quite or completly) the same anyway. So why does working with 500 Y!.WL files not work while it does over Bloom.WL, or even over Y!.WL when not executed from SM?


Maybe my only chance is to use other data sources (Y! was reliabel over the last years). I guess BBFree would be good in term of quality but did not update SP500 fast enough. What provider(s) would one recommend to have data quickly after open, and in acceptable quality too?

Can I abuse live data feeds to work as EOD data (not intraday as 500 symbols brought even my 64 bit WL down to its knees, see another ticket)?

Btw, can earnings be requested from any provider if the symbol equals the one where the data is stored?

Thank you

Thank you for your thoughtful analysis and detailed report.

Unfortunately, there's still no clear indication of the culprit in sight so we need more of that trial and error.

1. Indeed all data providers implement the same static provider interface and their .WL files have same structure. The only specific thing about Y! is that it does dividend/split adjustment on-the-fly (if enabled in Data Manager) though this is a less likely hint (performed very fast). You might want to open the Yahoo tab and uncheck all these adjustment options and the Partial Bar option in the first place (if enabled) to see how it affects SM run times.

2. Since SMA XOver on SP500 is welcomed by SM and your strategy is not, its complexity could (or could not) have something to do with this oddity. Commenting out various blocks of code, block by block, line by line, including trading signals / Alert generation, can prove or disprove this theory.

3. I would also suggest to perform a copy/paste of the code to a new strategy window, saving under a different name, and running this version (so we can cut off any potential XML file / saved indicator issues).

4. Since you mentioned 64-bit: it could help if you try it from 32-bit WLD 6.1 and notice possible differences in SM run times.

If all else fails, please create a support ticket for trying out a custom build of the Yahoo! provider. We'll ask the developer to try different build settings.

QUOTE:
Can I abuse live data feeds to work as EOD data

IQFeed has a static provider = an alternative well worth a try. If you don't have a current subscription, they provide free trials.

QUOTE:
Btw, can earnings be requested from any provider if the symbol equals the one where the data is stored?

Earnings as a fundamental item (Yahoo, YCharts) can be shown on any provider's chart if the symbols equals the one the data is stored.

If you mean GetNextEarningsDate, it's also provider agnostic. BTW there are two different, cached variations that would take the date off the hard disk (if present in cache), refreshing it periodically:

Earnings Date Helper
Get symbol data from Finviz.com (company name, sector, industry, next earnings date)

Re: !gub a ton saw ti taht raeh ot dooG
This is by far the most ridiculous argument and off-target analogy that I've read all year... and I keep reading how creating more debt is going to save the world economy.