How to create DataSeries "Correlation"

algoapi

#1

3/23/2009 10:33 AM

I tried to generate the correlation dataseries using the following function:

Correlation(double[] x, double[] y, int n)

But the sytem returns the message that this is not recognized. Can anyone show me an example on how to generate Dataseries Correlation (for Pearson correlation)?

Many thanks!

Eugene

#2

3/23/2009 11:08 AM

Here's an example:
Correlation

Requires Community.Components (Extensions > Get Extensions).

algoapi

#3

3/23/2009 11:19 AM

Thanks Eugene.

I did tried to add the following at the beginning:
"using Community.Components;"
but still didn't work.

All I wanted to do is to generate correlation dataseries between two stock time series.

One thing makes me a bit confused is that the other math functions such as LinearRegSlope works well without additional requirements (extensions), while the correlation function requires the extension, which I still couldn't figure out how to get it work. Isn't it more convenient to have all the commonly used math functions built in the WL platform?

Eugene

#4

3/23/2009 11:36 AM

Did you follow the link to install the Extension from this site? You need to restart WL upon installing like instructed here. Upon restarting, the Extension will become available to Strategies.

If something doesn't work, how could we help without at least having a precise error message text?

That's because they are included in the WealthScript/WealthLab.Indicators namespace. And if some function isn't available natively, we can leverage Wealth-Lab with the power of .NET. Extensibility is the key.

For the reason above it's neither possible nor practical to include everything in the package. Look, there are no less than 4 .NET compliant math libraries:

Open Source .Net math libraries

As to correlation, there will always be someone who would prefer some other implementation: DCorrelation, SCorrelation

algoapi

#5

3/23/2009 11:52 AM

Thanks again for the detailed answer :-)
I didn't mean to pick on the platform, actually I think WL is a very good platform.
I didn't post exact message that's because my current PC cannot run WL lively due to firewall. I posted the question based on my experience a few days ago.

For the last point, I still think the Pearson Correlation is the most commonly used math function in many areas. When people mention "correlation" typically that should refer to Pearson Correlation. Actually most other platforms all have this function as one of the basic math funtion built-in.

Anyway, the above is just my advice. I appreciate your hard work, this is a great platform overall.

DrKoch

#6

3/23/2009 12:40 PM

algoapi,
please note: it is rather meaningless to send prices (for example closing prices) into the correlation function. Will result in GIGO.

You'll have to calculate first differences (TA people call this "Momentum" a formidable misnomer) make sure to use percentages or even better logarithmic returns - just be warned...

algoapi

#7

3/23/2009 1:00 PM

Thanks DrKoch for the note, I totally agree.
In my code I actually tried to get the correlation between two stock's percentage return series, not on the prices directly.

Eugene

#8

3/23/2009 5:04 PM

Thanks for the good words. I understand your point.

(Let's ask Fidelity, do they plan to make Correlation available again natively like it was in WL3/4 in a later release?)

algoapi

#9

3/24/2009 7:19 AM

Hi Eugene,

I followed the instruction and installed the community.components extension successfully (copy of screenshot as below)

Community Components Library
Install File: Community.Components.dll
Succeeded

---------------------------

Then I tested the following sample code to calculation correlation:

When I run the code, the system gives me error message as below:

error CS0103 @ (17,22): The name 'Correlation' does not exist in the current context.

Could you advice what I missed here? Thanks

Eugene

#10

3/24/2009 7:30 AM

Have you seen the code example mentioned in my reply @ 3/23/2009 11:08 AM? In short, you haven't created the data series as illustrated in the example.

Please use the CODE tags when posting code.

ss161

#11

3/24/2009 6:45 PM

i needed the function also. the community component function takes an array of doubles and returns the correlation. to calculate the correlation that way for each bar in a price series seems inefficient (each time it is called, it "forgets" all the prior overlapping data items and rebuilds them. i wrote my own utility based on the same logic supplied in the community component and it seems to work -- although i expressly state that this comes with no warranty or assurance that the results are correct. please test it if you are going to use it.

here is the utility class code, followed by a simple program to show how it is used ( i too generally use log(price) series or log(dif(price) series, but am testing it via a simple price series ).

and the program to test it

Size:

Color:

i needed the function also.  the community component function takes an array of doubles and returns the correlation.  to calculate the correlation that way for each bar in a price series seems inefficient (each time it is called, it "forgets" all the prior overlapping data items and rebuilds them.  i wrote my own utility based on the same logic supplied in the community component and it seems to work -- although i expressly state that this comes with no warranty or assurance that the results are correct.  please test it if you are going to use it.

here is the utility class code, followed by a simple program to show how it is used ( i too generally use log(price) series or log(dif(price) series, but am testing it via a simple price series ).

[CODE]	public class WealthLabUtilities {
		
		public WealthScript ws;
		
		public WealthLabUtilities( WealthScript inputWS ) {
			ws = inputWS;
		}
		
		public DataSeries Correl( DataSeries y, DataSeries x, int period ) {

/*
             * this is adapted from Community.Components, which references the following Copyright / License:  
             * 
            Copyright (c) 2007, Sergey Bochkanov (ALGLIB project).

Redistribution and use in source and binary forms, with or without
            modification, are permitted provided that the following conditions are
            met:

- Redistributions of source code must retain the above copyright
              notice, this list of conditions and the following disclaimer.

- Redistributions in binary form must reproduce the above copyright
              notice, this list of conditions and the following disclaimer listed
              in this license in the documentation and/or other materials
              provided with the distribution.

- Neither the name of the copyright holders nor the names of its
              contributors may be used to endorse or promote products derived from
              this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
            "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
            LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
            A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
            OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
            SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
            LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
            DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
            THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
            (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
            OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
            */

DataSeries c		= new DataSeries(ws.Bars, y.Description + ":" + x.Description);
			DataSeries xmean	= SMA.Series( x, period );
			DataSeries ymean	= SMA.Series( y, period );
			DataSeries t1 		= x - xmean;
			DataSeries t2 		= y - ymean;
			DataSeries t1temp	= t1 * t1;
			DataSeries t2temp	= t2 * t2;
			DataSeries stemp	= t1 * t2;
			DataSeries xv		= Sum.Series( t1temp, period );
			DataSeries yv		= Sum.Series( t2temp, period );
			DataSeries s		= Sum.Series( stemp, period );
			
			// could divide out the entire series, but we will most likely be dividing some numbers by 0, so 
			// a for loop helps with this part
			
			for ( int i = 0; i < ws.Bars.Count; i++ ) {
				if ( xv[i] == 0 | yv[i] == 0 )
					c[i] = 0;
				else
					c[i] = s[i]  / ( Math.Sqrt(xv[i]) * Math.Sqrt(yv[i]) );
			}
			
			return c;
        }
			
	}[/CODE]

and the program to test it

[CODE]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;
using TimeSeriesObjects;

namespace WealthLab.Strategies
{
	public class MyStrategy : WealthScript
	{
		protected override void Execute()
		{

SetContext("SPY", true);
			DataSeries spy = Close;
			
			RestoreContext();
			
			WealthLabUtilities u = new WealthLabUtilities( this );
			DataSeries co = u.Correl ( Close, spy, 50 );
			
			ChartPane c = CreatePane( 50, false, true );
			PlotSeries (c, co, Color.Blue, WealthLab.LineStyle.Solid, 3);
			
		}
	}
}
[/CODE]

algoapi

#12

3/24/2009 11:18 PM

Thanks a lot ss161. I will try your program later. Cheers.

Eugene

#13

3/25/2009 2:38 AM

Steve, thanks for sharing. A good idea would be to enhance the Wiki description with your work and include this indicator in Community.Indicators.

ss161

#14

3/25/2009 2:58 PM

you're certainly welcome to do so -- assuming that would be something that you or Cone would do?

Eugene

#15

3/25/2009 3:24 PM

Yes, we'll take care of it.

Eugene

#16

3/30/2009 12:02 PM

Steve, it looks like your data series gives a different value compared to the WL4's Correlation function. Am I missing something?

ss161

#17

3/30/2009 12:54 PM

i'll compare the two later today and come back. thanks for pointing it out.

DartboardTrader

#18

3/30/2009 3:33 PM

The below matches Excel's output, and these are useful for anyone converting from Excel-based lifeforms.

Clean and optimize, if needed.

Usage:

Regards,
--Mike

Eugene

#19

3/30/2009 4:29 PM

Correlation galore! Thanks for sharing, Mike.

ss161

#20

3/31/2009 6:40 AM

after painfully staring at the code for a while, i realize that

does not work; there needs to be a double loop summing all (x - xmean) and (y - ymean)

the following seems to work, i've tested against using the Community.Component to calculate correlation and plot the two on top of each other on a chart.

here is the amended code -- although i'm not sure there is any efficiency gain over Mike's or just using a wrapper around the Community.Component.

Here is a program that tests it, using the input dialogue box that you helped me with yesterday:

Size:

Color:

after painfully staring at the code for a while, i realize that
[CODE]			DataSeries t1 		= x - xmean;
			DataSeries t2 		= y - ymean;
			DataSeries t1temp	= t1 * t1;
			DataSeries t2temp	= t2 * t2;
			DataSeries stemp	= t1 * t2;[/CODE]
does not work;  there needs to be a double loop summing all (x - xmean) and (y - ymean)

the following seems to work, i've tested against using the Community.Component to calculate correlation and plot the two on top of each other on a chart.

here is the amended code -- although i'm not sure there is any efficiency gain over Mike's or just using a wrapper around the Community.Component.

[CODE]			DataSeries c		= new DataSeries(ws.Bars, y.Description + ":" + x.Description);
			DataSeries xmean	= SMA.Series( x, period );
			DataSeries ymean	= SMA.Series( y, period );
			
			DataSeries xv		= new DataSeries( ws.Bars, "" );
			DataSeries yv		= new DataSeries( ws.Bars, "" );
			DataSeries s 		= new DataSeries( ws.Bars, "" );
			for ( int Bar = period -1; Bar < ws.Bars.Count; Bar++ ) {
				double t1sq			= 0;
				double t2sq			= 0;
				double sxy			= 0;
				for ( int i = 0; i < period; i++ ) {
					double t1	= x[Bar - i] - xmean[Bar];
					double t2	= y[Bar - i] - ymean[Bar];
					t1sq		+= t1 * t1;
					t2sq		+= t2 * t2;
					sxy			+= t1 * t2;
				}
				xv[Bar] 		= t1sq;
				yv[Bar]		 	= t2sq;
				s[Bar]			= sxy;

if ( xv[Bar] == 0 | yv[Bar] == 0 )
					c[Bar] = 0;
				else
					c[Bar] = s[Bar] / ( Math.Sqrt(xv[Bar]) * Math.Sqrt(yv[Bar]) );
				
				
			}
[/CODE]

Here is a program that tests it, using the input dialogue box that you helped me with yesterday:

[CODE]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;
using Community.Components;
using TimeSeriesObjects;

namespace WealthLab.Strategies
{
	public class MyStrategy : WealthScript
	{
		protected override void Execute()
		{

int Period = 50;
			
			// add a reference to Microsoft.VisualBasic to use this
			string inputSym = Microsoft.VisualBasic.Interaction.InputBox("Enter a second symbol on which to calculate correl:", "Enter Pair", "", 500, 500);
			SetContext(inputSym, true);
			DataSeries spy = Close;
			
			RestoreContext();
			
			WealthLabUtilities u = new WealthLabUtilities( this );
			DataSeries co = u.Correl ( Close, spy, Period );
			
			ChartPane c = CreatePane( 50, false, true );
			PlotSeries (c, co, Color.Black, WealthLab.LineStyle.Solid, 3);
			
			// the following calculates correlation using the Community.Component Correlation function
			// to compare against the above function
			DataSeries cc = new DataSeries( Bars, "comm comp correl" );
			for (int i = Period -1; i < Bars.Count; i++) {
				
				double[] x;
				double[] y;
				x = new double[Period];
				y = new double[Period];
			
				for (int j = 0; j < Period; j++) {
					x[j] = spy[i-j];
					y[j] = Close[i-j];
				}
				Calculate calc = new Calculate(this); // pass WealthScript
				double crl = calc.Correlation(x, y, Period);
				cc[i] = crl;
//				PrintDebug((i+1).ToString("0") + "," + Bars.Close[i].ToString("0.000000") + "," + spy[i].ToString("0.000000") + "," + co[i].ToString("0.000000") + "," + crl.ToString("0.000000"));
			}
			
			PlotSeries (c, cc, Color.Blue, WealthLab.LineStyle.Histogram, 6);
		}
	}
}[/CODE]

Eugene

#21

3/31/2009 11:05 AM

Perfect. Thanks.

Eugene

#22

4/5/2009 5:50 AM

Both functions added to Community.Indicators 2009.04. Thanks gentlemen.

ajrdwv07

#23

7/28/2016 11:40 AM

quick question. how can I create autocorr? that is the correlation(close, close t-5, 40) for example, that is the corr between the eod and the eod 5 days ago. thanks.

Eugene

#24

7/28/2016 11:52 AM

Please strike F11 in Wealth-Lab to open the QuickRef tool and navigate to "DataSeries Object" to read the description. Here's an example of a shifted Close applied to Correlation DataSeries: