Datasource Truncation utility (Scrub Fidelity) for WLP5?

Sammy_G

#1

5/31/2010 12:58 PM

Hi Cone,

I am a long time WLPro 4.5 user, now just installed v5.6. I notice that there doesn't appear to any equivalent of your very useful ScrubFidelity utility for v5. Are you planning to update it soon for v5? For the record, it saved my goose on more than one occasion in v4.5. You don't need that utility every day, but when you need it, boy you *really* need it!

Cone

#2

5/31/2010 1:55 PM

If I recall, the problem for which that app was built was to correct a temporary server problem that caused weekend bars to be inserted in the data. Since that problem no longer exists, I don't think it anyone needs to develop a solution to work around it.

Sammy_G

#3

5/31/2010 3:22 PM

Cone, I remember the original reason, but later on you expanded the utility to be able to truncate a Fidelity datasource starting on a specified date, and its *that* functionality I was thinking about. You may recall the Flash Crash of 5/6/10. A lot of those prices, dutifully reported by data providers, were completely bogus. Later on, some of the trades were busted (it took some time for that to happen) but data had already been downloaded by users everywhere. Even though sometimes data corrections are made by the provider during data download (e.g. Fidelity may correct upto 1 previous bar of daily data), its an open question when they actually implemented all of the corrections into their own database. So if one is past the data correction window of the provider, you are stuck with the bad data. What do you do in this situation? Its not practical to manually delete one or more bars of data when you have > 10,000 securities in a dataset. Clearly, you have to use a truncation utility like ScrubFidelity.

Cone

#4

5/31/2010 6:06 PM

Here's something quick and dirty for you.

NOTES!
o To avoid inadvertent operation, or multiple runs in an MSB, the code works only by clicking on the first symbol of any DataSet.

1. Works only on Fidelity DataSets that are in the default UserApp location.
2. Backup your data first. Who knows if this will work for all cases?!
3. Edit the truncate data in the Execute procedure.
4. The code will ask to confirm each DataSet before it truncates it. You can Cancel the entire process if required
5. Because of read and write times for large symbol files, it's expected to take a significant amount of time to complete the process for intraday DataSets - it could take hours for those. The code below has a switch to only process Daily DataSets; so you have to edit that to process intraday DataSets.
6. Cross your fingers! (Did you back up your data?)
7. When finished, check the Debug window for messages.

Suggestion: Turn off On-demand data so that charts don't update immediately when you go back to check their last date.

CODE:
Please log in to see this code.

Size:

Color:

Here's something quick and dirty for you.

NOTES!
o To avoid inadvertent operation, or multiple runs in an MSB, the code [b]works only by clicking on the first symbol of any DataSet[/b].

1. Works only on Fidelity DataSets that are in the default UserApp location.
2. Backup your data first.  Who knows if this will work for all cases?!
3. Edit the truncate data in the Execute procedure.
4. The code will ask to confirm each DataSet before it truncates it.  You can Cancel the entire process if required
5. Because of read and write times for large symbol files, it's expected to take a significant amount of time to complete the process for intraday DataSets - it could take hours for those.  The code below has a switch to only process Daily DataSets; so you have to edit that to process intraday DataSets.
6. Cross your fingers! (Did you back up your data?)
7. When finished, check the Debug window for messages.

Suggestion: Turn off On-demand data so that charts don't update immediately when you go back to check their last date.

[CODE]using System;
using System.Windows.Forms;
using System.Collections;
using System.Collections.Generic;
using System.Text;
using System.Windows.Forms;
using WealthLab;

namespace WealthLab.Strategies
{
	public class TruncateFidelityDataSources : WealthScript
	{	
		public string BuildSymbolPath(string sym, DataSource ds )
		{
			// Load all the raw bars from the secondary symbol
			string symPath = Application.UserAppDataPath + @"\Data\FidelityStaticProvider\";
			
			if( ds.BarDataScale.Scale == BarScale.Daily )
				symPath += @"Daily\";
			else 
				symPath += ds.BarInterval.ToString() + " " + ds.BarDataScale.Scale.ToString() + @"\";
			
			
			if (sym[0].ToString() == ".")
				symPath += ConvertSymbolName.ToFileName(sym) + ".WL";	
			else			
				symPath += sym[0] + @"\" + ConvertSymbolName.ToFileName(sym) + ".WL";	
			
			return symPath;
		}
		
		protected override void Execute()
		{
			ClearDebug();
			if (Bars.Symbol != DataSetSymbols[0]) return;			
			
			DataSourceManager dm = new WealthLab.DataSourceManager();
			dm.RootPath = Application.UserAppDataPath +  @"\Data\";						
			
			/* Enter the date of the last day that you want to keep here */
			DateTime dt = new DateTime(2010, 5, 4);
			
			foreach(DataSource ds in dm.DataSources)
			{
				if (ds.Provider.FriendlyName == "Fidelity Investments")
				{
					PrintDebug(ds.Provider.ToString() + "\t" + ds.Name);
										
					/* Use this to process Daily DataSets only */
					if (ds.BarDataScale.Scale != BarScale.Daily) continue;
					
					/* Use this to specify a DataSet to truncate */
					//if (ds.Name != "Dow 30") continue;
					
					// Delete this block if you don't want to confirm Truncating each DataSource
					DialogResult dr = MessageBox.Show("Warning! Use at your own risk!\nBackup your Fidelity Data first!\n\nTruncate \"" + ds.Name + "\" after " + dt.ToShortDateString() + "?",
						"Warning!", MessageBoxButtons.YesNoCancel, MessageBoxIcon.Exclamation, MessageBoxDefaultButton.Button2);
					if (dr.Equals(DialogResult.No))
						continue;	
					else if (dr.Equals(DialogResult.Cancel))
						return;
					
					
					foreach (string symbol in ds.Symbols)
					{	
						string symPath = BuildSymbolPath(symbol, ds);
						// PrintDebug("\t" + symPath );
						PrintStatusBar(ds.Name + ": " + symbol);
																			
						
						Bars rawBars = new Bars("_" + symbol, ds.BarDataScale.Scale, ds.BarInterval);
						try {
							rawBars.LoadFromFile(symPath);
						}
						catch {
							PrintDebug("\tNo data file for " + symPath);
							continue;
						}
						
						if( rawBars.Count == 0 )
						{
							PrintDebug("\tNo bars for " + symbol);
							continue;
						}

bool changed = false;
						for (int bar = rawBars.Count - 1; bar > 0; bar--)
						{
							if (rawBars.Date[bar] > dt)
							{
								rawBars.Delete(bar);
								changed = true;
							}
							else
								break;							
						}
						
						if (changed)
						{
							try {
								rawBars.SaveToFile(symPath);
								PrintDebug("\tSuccess: " + symbol);
							}
							catch {
								PrintDebug("\tError saving file: " + symbol);
							}
						}
						else
							PrintDebug("\tNo change: " + symbol);
					}
				}			
			}
		}		
	}	
	
	
	static class ConvertSymbolName
	{
		private static char[] illegalChars = new char[] { '.', '\\', '/', ':', '*', '?', '"', '<', '>', '|' };

public static string FromFileName(string fileName)
		{
			foreach (char c in illegalChars)
			{
				int v = (int)c;
				string s = v.ToString("X");
				fileName = fileName.Replace("%" + s, c.ToString());
			}			
			return fileName;
		}

public static string ToFileName(string symbolName)
		{
			foreach (char c in illegalChars)
			{
				int v = (int)c;
				string s = v.ToString("X");
				symbolName = symbolName.Replace(c.ToString(), "%" + s);
			}			
			return symbolName;
		}

}	
}[/CODE]

Sammy_G

#5

5/31/2010 9:45 PM

Thanks, Cone, I will test drive it soon.

My ignorance of v5 script language forces me to ask: Does it have a built-in safeguard, like the

CODE:
Please log in to see this code.

argument in v4.5?

Eugene

#6

6/1/2010 1:34 AM

Nice code Robert.

QUOTE:
{$NO_AUTO_EXECUTE}

You will find the answer in the Knowledge Base section of Wealth-Lab Wiki (click Support > Knowledge Base to get there.) There's an article.

Cone

#7

6/1/2010 3:01 AM

Hi Eugene. That code has a little of you and Aleksey in it too. Don't write what you can't borrow or steal :)

As for the safeguard, yes, see Note #4. It asks you to confirm for each DataSet and you can cancel the entire process immediately.