Data Truncation script - Wealth Lab Forum

Sammy_G

#1

10/8/2010 9:46 AM

Fidelity WLPro 6.x
===================

I downloaded the data for symbol KKR from Fidelity. It was a recent IPO on 7/15/10. However, the d/l data extends back to 1/11/07, meaning its a recycled symbol and the data provider hasn't bothered to fix the data.

Rather than delete the data bar-by-bar prior to 7/15/10 (and develop a carpal tunnel in the process), I was wondering if you guys have a handy (or can create one) data truncation utility that will do the job. A safe version would be preferred that will *only* do the job on the clicked symbol, or a version that can let the user decide if the truncation job is to done on the clicked symbol or an entire dataset (I can move the bad symbols to a safe dataset where the utility can merrily chop away at the data w/o affecting the other datasets).

Eugene

#2

10/8/2010 9:52 AM

This should be sufficient for a single symbol :

Data | Removing data using a WL6 Strategy > Request #1

You just have to modify this condition accordingly:

Like this:

P.S. And this line:

Is better changed to:

Sammy_G

#3

10/8/2010 11:35 AM

Many thanks, Eugene. With the right mods, it worked great; I post the code here for benefit of newbies (suggestion: the code could benefit from having the date input from an input box, and 1-2 warnings thrown in):

============================================

Another data-related question: Is there a script or utility that will identify, and optionally delete (actually, much prefer to have it copy the files to another location - either a dataset or a temp directory - just in case), data files with no data?
I have chunk of those; what had happened was that when I started using WL.NET a few months ago, I copied entire datasets over from WL4; quite a few of those symbols don't trade any more so I have a bunch of these symbols scattered over various datasets.
Note: I already tried the Bad History Data Check utility; it doesn't work with zero-data symbols.

Size:

Color:

Many thanks, Eugene. With the right mods, it worked great; I post the code here for benefit of newbies (suggestion: the code could benefit from having the date input from an input box, and 1-2 warnings thrown in):
[CODE]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;

namespace WealthLab.Strategies
{
    public class RemoveDataPriorToADate : WealthScript
    {
        bool NO_AUTO_EXECUTE = true;

protected override void Execute()
        {
            if(NO_AUTO_EXECUTE) goto halt; 
            else 
                goto go;
            halt:
            {
                NO_AUTO_EXECUTE = false;
                return;
            }
            go:
            {
                string sym = Bars.Symbol;
                
                // Specify your base time frame here: 
                Bars bars = new Bars( Bars.Symbol, Bars.Scale, Bars.BarInterval );

// Identify cutoff date
                DateTime cutoff = new DateTime(2010, 7, 15);
            
                // Build the Bars object
                for (int bar = 0; bar < Bars.Count; bar++)
                {
                    if( Bars.Date[bar] >= cutoff )
                        // Save the data excluding data before cutoff date
                        bars.Add( Bars.Date[bar], Bars.Open[bar], Bars.High[bar], 
                            Bars.Low[bar], Bars.Close[bar], Bars.Volume[bar]);
                }

// Save the corrected bars to disk
                string file = @"C:\temp\" + sym + ".WL";
                bars.SaveToFile( file );
            
                // Verify by loading from disk and plotting
                Bars myBars = new Bars( "Bars since cutoff date (" 
                    + cutoff.ToShortDateString() + ")", BarScale.Daily, Bars.BarInterval );
                myBars.LoadFromFile( file );            
                myBars = Synchronize( myBars );            
                ChartPane myPane = CreatePane(50, false, true);
                PlotSymbol( myPane, myBars, Color.DodgerBlue, Color.Red );
                
                // Now manually copy the new data file over to the appropriate directory
            }
        }
    }
}[/CODE]
============================================

Another data-related question: Is there a script or utility that will identify, and optionally delete (actually, much prefer to have it copy the files to another location - either a dataset or a temp directory - just in case), data files with no data?
I have chunk of those; what had happened was that when I started using WL.NET a few months ago, I copied entire datasets over from WL4; quite a few of those symbols don't trade any more so I have a bunch of these symbols scattered over various datasets. 
Note: I already tried the Bad History Data Check utility; it doesn't work with zero-data symbols.

Eugene

#4

10/8/2010 11:45 AM

It's easy:

Now just copy this subset of symbols from the Debug window and paste it in the Modify DataSet window.

Sammy_G

#5

10/8/2010 12:24 PM

I guess one would have to delete symbols from the Quarantine list, and the data files, manually?

hankt

#6

6/16/2012 5:04 AM

Sammy,

I want to just delete bars before my check date rather than write out to a new file. I'm working with Yahoo historical data and they concatenate previously used tickers within 2 companies on 1 ticker.

I can't find any documentation for Bars.Delete

Eugene

#7

6/16/2012 8:20 AM

Bars.Delete is an intentionally undocumented method, so you're using it at your own risk. Furthermore, you won't be able to "just delete" bars because modifying the locked Bars collection is not permitted.

For simplicity's sake, I'd recommend you not process the trading system's logic before the cutoff date for that symbol:

Voila.

Sammy_G

#8

6/17/2012 5:09 PM

Hank,

The following is the code I use to truncate data BEFORE a date. This script pops up with an input box for the date; I find that much easier to use than hard-coding the date into the script and also allows me to use it on multiple symbols. Just make sure you have a C:\temp directory.

Size:

Color:

Hank,

The following is the code I use to truncate data BEFORE a date. This script pops up with an input box for the date; I find that much easier to use than hard-coding the date into the script and also allows me to use it on multiple symbols. Just make sure you have a C:\temp directory.
[CODE]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;
using Community.Components;	// for Utility (input)

namespace WealthLab.Strategies
{
	public class RemoveDataPriorToADate : WealthScript
	{
		bool NO_AUTO_EXECUTE = true;

protected override void Execute()
		{
			if(NO_AUTO_EXECUTE) goto halt; 
			else 
				goto go;
			halt:
			{
				NO_AUTO_EXECUTE = false;
				return;
			}
			go:
			{
				try
				{
					ClearDebug();
					string sym = Bars.Symbol;
				
					// Specify your base time frame here: 
					Bars bars = new Bars( Bars.Symbol, Bars.Scale, Bars.BarInterval );

// Identify cutoff date
					string input = Utility.Input("Input cutoff date in format YYYY,MM,DD (all data prior to this date will be removed)");
					DateTime cutoff = Convert.ToDateTime(input);
			
					// Build the Bars object
					for (int bar = 0; bar < Bars.Count; bar++)
					{
						if( Bars.Date[bar] >= cutoff )
							// Save the data excluding data before cutoff date
							bars.Add( Bars.Date[bar], Bars.Open[bar], Bars.High[bar], 
								Bars.Low[bar], Bars.Close[bar], Bars.Volume[bar]);
					}

// Save the corrected bars to disk
					string file = @"C:\temp\" + sym + ".WL";
					bars.SaveToFile( file );
			
					// Verify by loading from disk and plotting
					Bars myBars = new Bars( "Bars since cutoff date (" 
						+ cutoff.ToShortDateString() + ")", BarScale.Daily, Bars.BarInterval );
					myBars.LoadFromFile( file );			
					myBars = Synchronize( myBars );			
					ChartPane myPane = CreatePane(50, false, true);
					PlotSymbol( myPane, myBars, Color.DodgerBlue, Color.Red );
				
					// Now manually copy the new data file over to the appropriate directory
					PrintDebug(cutoff);
				}
				catch (Exception e)
				{
					PrintDebug(e.Message);
				}
			}
		}
	}
}[/CODE]

hankt

#9

6/17/2012 7:58 PM

Eugene,
I have a portfolio of securities, not a single security, so that idea won't help me.
Check date on previous bar once moved, ignoring the first bar in the dataset possibly?
{EDIT}
Ah, the collection, not current bar object...hmmm

Sammy,
If I use your method, then copy the temp file back into the Yahoo provider directory?

Eugene

#10

6/18/2012 12:14 AM

Incorrect. Try setting up a Dictionary<string,DateTime> where the Key is a security and the Value is its cutoff date:

Sammy_G

#11

6/18/2012 7:17 PM

Yes. Basically, the script truncates the data and plots the corrected data for the symbol in the temp directory; if you are satisfied with the result, then copy the new .WL file over, replacing the old one.