Get external symbol in a formal indicator without speed penalty
Author: superticker
Creation Date: 8/3/2020 6:21 PM
profile picture

superticker

#1
QUOTE:
You should not use the ExternalSymbol.Series for reason explained in the online user guide. It uses Reflection internally ...
And I agree with you. Reflection is okay for seldom-executed C# "generic" class constructors, but it's dog slow for main-line execution. I just benchmarked it, and ExternalSymbol.Series is 35% slower than calling GetExternalSymbol. :(

Let's talk about faster alternatives. In the code below, the ExternalSymbol.Series line works, but it's 35% slower than the GetExternalSymbol line. In addition, the GetExternalSymbol line needs a reference to the current WealthScript instance, which is inconvenient to capture.

CODE:
Please log in to see this code.

One alternative might be to pull the S&P 500 into the Strategy as a static variable (DataSeries), or Bars.Cache variable, that the indicator class library would have access to. Sounds a bit over the top too, but I'm open to other suggestions.

One way to get the S&P 500 into a static variable is by creating a special utility class that inherits from WealthScript, then have a single member function {Utility.GetIndex("IndexSym")} that calls GetExternalSymbol and saves it as a field variable for the MyStrategy class. Can someone think of an easier or more robust way to get the S&P 500 into an indicator class library? And, of course, we want to avoid calling ExternalSymbol.Series, which uses Reflection, unless it's a one-time call like in the MyStrategy constructor.
profile picture

Eugene

#2
QUOTE:
In the code below, the ExternalSymbol.Series line works, but it's 33% slower than the GetExternalSymbol line.

See ClearExternalSymbols in the QuickRef: Wealth-Lab already caches Bars objects e.g. GetExternalSymbol. The ExternalSymbol is not intended to be used in scripts.

You can use SetGlobal / GetGlobal in your scripts to cache the data into RAM as the fastest alternative. It's a script-based solution so it will impose the same speed penalty if employed in an Indicator.
profile picture

superticker

#3
QUOTE:
Wealth-Lab already caches Bars objects e.g. GetExternalSymbol.
So you're saying I should be using GetExternalSymbol inside my indicator class library? If so, explain how I can call it inside my class library. I can't seem to get that to work for the S&P 500. I want to just pass the symbol index name (e.g. "^SPX") into my indicator. The index "name" may vary with the stock industry.
profile picture

Eugene

#4
GetAllDataForSymbol may be faster, try that.
profile picture

superticker

#5
QUOTE:
GetAllDataForSymbol may be faster, try that.
Same question as in Post# 3, but this time for GetAllDataForSymbol.

The problem is the SetGlobal, GetExternalSymbol, and GetAllDataForSymbol calls all require an instance of WealthScript. But I'm trying to call them inside an indicator library which is entirely static and doesn't have any instance of WealthScript in the first place. And I don't know how to bring an instance of WealthScript into an entirely static library. Perhaps you can enlighten me.
profile picture

Eugene

#6
GetAllDataForSymbol does NOT require an instance of WealthScript. It's your best bet.
profile picture

superticker

#7
QUOTE:
GetAllDataForSymbol does NOT require an instance of WealthScript.
Okay, but how do I get an external symbol like the S&P 500 into the GetAllDataForSymbol call? This again gets back to my question in Post# 3. I'm still not connecting the dots.

Review from Post# 3:
1) I have a static indicator library without an instance of WealthScript.
2) I need to get the external symbol S&P 500 into this static library. I'm willing to employ GetGlobal if it's an option w/o a WealthScript instance.
3) I want to avoid Reflection as much as possible because it is slow, but I'm willing to employ it once in a SetGlobal call even if that means using a third-party strategy (which I would prefer to avoid). I'm willing to check the SetGlobal cache for the S&P 500 to avoid making redundant usage of Reflection.

Perhaps that clarifies Post# 3 better. I'm sorry if Post# 3 was too terse.
profile picture

Eugene

#8
GetAllDataForSymbol is the way to get an external symbol into your indicator (DataSeries) without an instance of WealthScript:

CODE:
Please log in to see this code.
profile picture

superticker

#9
Your solution in Post# 8 works fine; thanks for that. But both GetAllDataForSymbol(...) and ExternalSymbol(...) operate at about the same speed. In contrast, GetExternalSymbol(...) is at least twice as fast and often much faster when employing Yahoo ^GSPC for the S&P500.

But here's the really weird part. When I switch to using IQFeed SPX.XO for the S&P500, all three routines are lighting fast. And that was also the case with Fidelity data. So what we are seeing here has more to do with the speed of the Yahoo provider than which call is being employed. And yes I have on-demand data unchecked when measuring this.

This makes me think we are barking up the wrong tree altogether. So how can I permanently cache the S&P 500 for an indicator library to use? I don't care how I initially grab it (which call employed in unimportant); I just want it cached so I grab it once and only once for the indicator library. I'm guessing GetExternalSymbol(...) somehow caches it, but the two other approaches do not. (And getting data via the Yahoo provider is incredibly slow even when the S&P500 is in disk cache. Kind of weird.)

Honestly, I think the Yahoo provider is broken if it takes that long to fetch an External Symbol out of disk cache during optimization!
profile picture

Eugene

#10
GetAllDataForSymbol is a shortcut to WL's internal method for loading external symbol's data. Unfortunately, it does use Reflection under the hood so a slight speed penalty may take place.

1. Are you sure that Yahoo DataSet is up to date and includes all the symblols?

2. Yahoo provider is programmed to make an extra call to get fundamental items (split and dividend). Could be this.

3. The most speed efficient way could be reading the .WL file directly using Bars.LoadFromFile. But it's cumbersome.
profile picture

superticker

#11
QUOTE:
1. Are you sure that Yahoo DataSet is up to date and includes all the symbols?
Yes, I updated all data on both Saturday and Sunday.

QUOTE:
2. Yahoo provider is programmed to make an extra call to get fundamental items (split and dividend).
Well, I don't think the S&P500 has splits and dividends, and that's the external symbol I'm calling. The dataset being optimized was a Yahoo dataset. I could try moving it to a IQFeed dataset.

QUOTE:
3. The most speed efficient way could be reading the .WL file directly using Bars.LoadFromFile.
I'm trying to think if there's a way to do this once--and no more--in the indicator library. Can I call SetGlobal without a WealthScript instance in the library? Is there an extension method for SetGlobal?

Certainly there should be a way to share a DataSeries.
profile picture

Eugene

#12
Since you cannot call *Global without instantiating WealthScript, Bars.LoadFromFile is the fastest way to get external symbol in a formal indicator.
profile picture

superticker

#13
QUOTE:
... you cannot call *Global without instantiating WealthScript ...
[1] So is the QuickRef wrong when it says you can share *Global data between strategies--which have different instances of WealthScript?

Or [2] are you saying *Global shouldn't be dependent on a WealthScript instance in the first place? In other words, the *Global call is scoped incorrectly.

Which is it, [1] or [2]? You can't have it both ways.
profile picture

Eugene

#14
In the context of this topic, I obviously meant that you cannot call *Global (WealthScript methods) in a custom function library DLL (in a class which doesn't implement WealthScript like a compiled Strategy would) without instantiating WealthScript - as with any other method.
profile picture

superticker

#15
QUOTE:
GetAllDataForSymbol does NOT require an instance of WealthScript. It's your best bet.
Perhaps, but the Synchronize() function does require an instance of WealthScript, and I have to call that before I can use the GetAllDataForSymbol() data. For example,...
CODE:
Please log in to see this code.

The right solution would be to make Set/GetGlobal() core functions of Wealth-Lab core, and not a part of WealthScript. That's possible because they are managing the core heap, not the WealthScript heap. Otherwise, there's no "documented" way to access the core heap storage from within a formal indicator--a big disadvantage here.

If Set/GetGlobal() were core Wealth-Lab functions, the external indexSymbol could be stored in the core heap by calling SetGlobal within the MyStrategy constructor. The formal indicator could then pick it up using GetGlobal.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).