Handling a year of Tick data - Wealth Lab Forum

brizzle351

#1

9/20/2010 6:12 PM

I've purchased a year of tick-by-tick data on the SP500 emini in ASCII format but my computer cannot run strategies on it due to the sheer amount of data. Excel and notepad cannot open all the data either so I can't physically cut and paste a month of data.

Has anyone else had this problem and does anyone have any ideas on how I can simply grab 1 month of data to test? Maybe convert it to minute or 5 minute data rather than 1 tick etc?

Thanks

Eugene

#2

9/21/2010 1:51 AM

If your computer can run any 64-bit Windows, install it and run WLD6 x64 which should be able to handle this amount of data. Otherwise, it's practical to step back to 1-minute bar scale - minutes are less resource intensive and you could backtest a small portfolio of symbols using 1-min data going back several years even on a 32-bit machine.

P.S. To open huge data files (hundreds of MBs or even up to a couple GBs), use an appropriate text editor like Notepad++, TextPad, JujuEdit etc.

brizzle351

#3

9/21/2010 11:11 PM

Great notepad++ works perfectly. Thanks

brizzle351

#4

9/22/2010 12:25 AM

I've uploaded data in tick-by-tick format (example below) so I can convert the scale with flexibility however I cannot convert it to 144 tick or any other format using "scale". It says "no data available". Is there an easy answer to this?

ES 91209 153000 1090.25 3
ES 91209 153000 1090.25 21
ES 91209 153037 1090.25 2
ES 91209 153037 1090.5 1
ES 91209 153039 1090.25 1
ES 91209 153049 1090.5 19
ES 91209 153054 1091.25 1
ES 91209 153206 1090.25 1
ES 91209 153237 1090.5 1
ES 91209 153349 1090.5 1
ES 91209 153428 1090.5 24
ES 91209 153534 1090.25 5
ES 91209 153603 1090.5 1
ES 91209 153747 1090.25 1
ES 91209 155059 1091.25 1
ES 91209 155250 1091.5 1
ES 91209 155722 1092 3
ES 91209 155729 1092 25
ES 91209 155804 1092 2
ES 91209 155944 1091.25 12

Eugene

#5

9/22/2010 2:14 AM

I believe the question has already been answered in your previous support ticket Brian - compressing intraday ASCII data is currently not made possible, sorry. The developer will work on enabling this feature for the next release.

brizzle351

#6

9/22/2010 2:28 AM

Sorry, I missed that. Does the following do it? it's from "Tick to OHLC Bar Converter" in 2004? My C# is still in it's early stages.

CODE:
Please log in to see this code.

Size:

Color:

Sorry, I missed that.  Does the following do it? it's from "Tick to OHLC Bar Converter" in 2004? My C# is still in it's early stages.

[CODE]function CompressTicks(Decimation: integer): integer;
begin
  var O, H, L, V: float;
  var s, t, d: integer;
  var last: integer = BarCount - 1;
  if (BarInterval = 0) or (Decimation <= 1) then exit;

// Decimate (overwriting leftmost bars)
  while s < BarCount do
  begin
    O := PriceOpen(s);
    H := PriceHigh(s);
    L := PriceLow(s);
    V := Volume(s);
    for t := 2 to Decimation do
    begin
      // terminate bar at end of day too
      if LastBar(s) or (s = last) then break;
      s := s + 1;
      H := max(H, PriceHigh(s));
      L := min(L, PriceLow(s));
      V := V + Volume(s);
    end;
    if (s = last) and (t <= Decimation) then break; // incomplete last bar?
    ChangeBar(d, GetDate(s), GetTime(s), O, H, L, PriceClose(s), trunc(V), trunc(OpenInterest(s)));
    d := d + 1;
    s := s + 1;
  end;
  
  // Move decimated bars to the right
  s := BarCount;
  while d > 0 do
  begin
    s := s - 1;
    d := d - 1;
    ChangeBar(s, GetDate(d), GetTime(d), PriceOpen(d), PriceHigh(d), PriceLow(d), PriceClose(d),
              trunc(Volume(d)), trunc(OpenInterest(d)));
  end;
  
  // Let user know where decimated data start
  Result := s;

// Clear (unused) leftmost bars
  O := PriceOpen(0);
  while s > 0 do
  begin
    s := s - 1;
    ChangeBar(s, GetDate(0), GetTime(0), O, O, O, O, 0, 0);
  end;
  
end;

const Decimation = 10;
var FirstBar: integer;
FirstBar := CompressTicks(Decimation);
DrawLabel('Original bars have been decimated by ' + IntToStr(Decimation), 0);
DrawLabel('Now you can loop over compressed bars, starting at Bar number ' + IntToStr(FirstBar), 0);
DrawLabel('Sorry, no way to get uncompressed data back...', 0);
[/CODE]

Eugene

#7

9/22/2010 2:34 AM

Thank you for trying to help.

Wealth-Lab has its own BarScale convertor component so we're all set for this. This is a speculation but from what I recall about the issue, there was a problem elsewhere and disabling intraday ASCII compression probably was a workaround for some 5.x bug. Hopefully, the next build should bring this feature back.