I have a intel duo core XP system with 2G RAM. I was looking at the memory formula in the Wiki:

Required RAM = Number of stocks * number of bars * number of data series (typically 6: DateTime + OHLCV) * 8 bytes

I'm wondering if I upgraded my computer to have 4G RAM could I use all 4G (minus XP etc) to run one backtest or since one backtest is one process would it only use 2G RAM?

This question should have came up before, try to search. It takes a 64-bit OS to allocate more than 3Gb of RAM.

I already searched on memory. Now I searched on 64-bit and found a comment by Cone:

"Incidentally, 4G of memory doesn't buy you anything if Windows only lets you access 2G max per application."

So, I assume I'm limited to less than 2G of RAM...

That's right unless:
1. you get Windows to allow an app to address 3GB. I've never done it, but as Eugene suggests, it's been discussed here before.
2. you've got 64-bit Windows and run Wealth-Lab 5 in 64-bit mode.

Caveats for 2:
1. Until 5.3 is released it takes a work-around to run 64-bit installs.
2. WL Pro 5.3 will be able to be installed on 64-bit systems, but won't run in 64-bit mode due to it's use of a legacy 32-bit C++ app (Fidelity Server).

QUOTE:
1. you get Windows to allow an app to address 3GB. I've never done it, but as Eugene suggests, it's been discussed here before.

XP: the /3GB switch in boot.ini.
Vista: the IncreaseUserVa option in BCEdit (i.e. Bcdedit /Set IncreaseUserVa 3072).

BTW haven't tried them too. ;)

I'm getting an out of memory exception, seemingly prematurely. I know I've done this before and used all 2G of my XP's memory, but it's throwing the exception at 1.7G memory used (on the Performance screen of the task mgr). WealthLapPro 5.x is using 1.4G of memory. I cleared out all other applications and ended nonessential tasks. What gives? I was able to bring up IE and post this message, so I know there's memory left.

XP 32bit can't use 4 Gb of memory fully. it's depends of hardware of your computer. sometimes it can be only 3.5 Gb, rarely 3.9 Gb. So if you set /3GB you limit the kernel with memory and memory shortage can happens (recommended to use /userva=2800).

QUOTE:
WealthLapPro 5.x is using 1.4G of memory.

- but OS allocate for it more memory than reflected in the task mgr
besides, user application cant use more then 2Gb if they compiled without special keys. Lets ask Eugene and Cone - did they use special keys for compiling WLD?

PS: what jimcrist do if he needs so much memory??

The compilation is not special. The LARGEADDRESSAWARE flag was considered and discarded (I think) due to Microsoft's own warning that it could affect memory required by some drivers.

Jimcrist can optimize the strategy in a way it consumes less memory, while international customers can use the x64 build of WLD5.4 which allows allocating much more system memory.

QUOTE:
The LARGEADDRESSAWARE flag was considered

this is what i mean.

But my question is still, why was I only able to use 1.4G with WLP5.4?? Why didn't it use all of my 2G before it threw the memory exception? I think WLP5.1 was able to use all of my 2G. Why is WLP5.4 worse? I can't even do a 1 year backtest anymore using 15-min bars?!? I SHOULD be doing 5 year backtests!!!

The 32-bit version can't use all of your 2G as the memory usage should be topping at 1.7G. RAM is dirt cheap these days, maybe you could consider throwing in another chip?

Assuming you've taken the steps listed in the Wiki to control WLP's memory usage, your 5.4 experience shouldn't be dramatically worse. Only WLD 5.4 x64 - the native 64-bit version - is different in its potential memory consumption.

It took me awhile to determine whether my XP Professional was 32-bit or 64-bit. Here's how:

http://support.microsoft.com/kb/827218

I'm bummed to find out it's 32-bit. But I can still do the /3GB switch thing, I guess.

QUOTE:
But my question is still, why was I only able to use 1.4G with WLP5.4?? ... I think WLP5.1 was able to use all of my 2G.

You're probably remembering incorrectly. Many applications require a contiguous allocation of RAM. It's not surprising that essential OS applications already running would limit your available contiguous RAM to something less than the total available RAM.

QUOTE:
But I can still do the /3GB switch thing, I guess.

To take advantage of the 3 GB, the application itself must be compiled with the /LARGEADDRESSAWARE flag. This is not the case with WL5, for the reason see message by Cone @ 8/7/2009 1:34 AM.

In addition to the /3GB switch, try the 4Gb patch utility. I believe it should patch WL5 to enable the flag.

QUOTE:
But my question is still, why was I only able to use 1.4G with WLP5.4??

bodanker is quite right. I think You use big arrays of data or List classes. There is one feature of Lists in C# lang: when growing it grows twice each time and if List is already Big memory manager cant allocate consistent memory slice for List. So you receive Out of mem Exceprion when used only 1.5 gb of mem. This is my point of view.

I work with 2000000 and more bars of data, computing different things with many List object and WLD utilize only 500 mb and below. I think it would be a right way to optimize your algorithm.

PS: computation with data reading from file lasts nearly 15-20 seconds. And I have not make an optimization yet. computer: 1 core 2500 Mhz + 1gb RAM

Thanks Eugene, I was wondering about that /LARGEADDRESSAWARE flag. So it's the 4Gb patch utility combined with the /3G switch that'll let me use more memory.

RA81's post confuses me. Is there another approach to backtesting that doesn't use as much memory?

RA81 is suggesting that your script is inefficient in its use of memory. There are many ways to program the same thing, and your program might be consuming memory that it doesn't need to. You could be shooting yourself in the foot.

How many symbols are in your 15-minute bar backtest? As a 120-bytes/bar rule of thumb, you can count on about 4MB/symbol for a 5-year backtest. So you should be able to test a 350-400 symbol DataSet with about 1.5G of memory.

As for reducing memory requirements, the only obvious things that come to mind (without seeing your script) are:
1. Use less data
2. Disable Performance Visualizers that you do not need

Yeah, well that's the problem. My system scans alot of symbols. I'm trying to backtest 2,433 symbols...

In addition to what Robert says:

Add more memory. Defrag system memory and unload every extra resident service and/or program you can without sacrificing system stability. Run WLP5 on 2 computers. Call Bars.Cache.Clear.

This is getting annoying. I just tried again. Why does WLP5.4 quit at 1.4G with 1.7G total used and 244M of physical memory left?!? I have 2G of memory. I'm rebooting for a fresh machine. I've tried deleting lots of processes (even explorer.exe!) and I've tried not deleting any processes (in case I'm deleting some memory mgmt process). It doesn't matter 'cuz I'm not using all the memory. My script uses 10 dataseries, but I need them. I've disabled ALL the performance visualizers (am I going to get any useful results?) I call Bars.Cache.Clear at the end. I don't get the defrag memory thing, doesn't rebooting do that? I have tons of disk space available for caching. Run WLP5 on 2 computers? What does that mean? I've got 4G on order, but I wish I knew why it quits early with an out of memory exception?

QUOTE:
Why does WLP5.4 quit at 1.4G with 1.7G total used and 244M of physical memory left?!?

Please read again what we have written to you several times.
how many items contain dataseries?
Give your code, I'll try to say where the error

QUOTE:
how many items contain dataseries?

Please reread my last post:

QUOTE:
My script uses 10 dataseries, but I need them.

How much items does each data series contain? (not "how much dataseries do you have?".)

Here's the initialization of the 10 dataseries:

CODE:
Please log in to see this code.

Seems that he's trying to ask you how many loaded bars are there in your data series, jimcrist.

5539 bars in the last run. 10/1/08 to 8/6/09.

ok. lets go step by step.
1. Start your strategy with long time interval (from 1980 - 2009 for example) and post the interval here
2. if you will receive mem. overflow error, then take a look on bars amount (12345 bars or may be 2000000 bars). Post this number here.
3. do you use portfelio simulation or single symbol? (you talk about 2550 symbols)
4. do you use buy/sell orders? how often?
5. In what scale you start your strategy?? (daily, intraday 10 min/30/60 min ??)

Unbelievable. I upgraded to 4G and it still throws the out of memory exception at 1.4G. I added /3G to my boot.ini as follows:

CODE:
Please log in to see this code.

I downloaded the 4G patch and applied it (twice now) to WealthLapPro.exe. I see that it made a backup and I searched my machine to make sure I didn't have more than one copy of WealthLabPro.exe. The MSDN documentation mentions something about bootcfg, but I'm ignoring that. I've tried /userva=3030 and /userva=2800 and I rebooted after each change. I'm trying to backtest 2433 symbols, 5677 15-min bars, 11 dataseries, so I should only need about 1.2G. I call Bars.Cache.Clear() at the end. I only have one performance visualizer on (performance). I'm running WLP5.4.20.0. The taskmgr.exe shows all the extra memory I'm not using...now what?

I'm not defragging memory, what utility would I use?

QUOTE:
I downloaded the 4G patch and applied it (twice now) to WealthLapPro.exe.

This very little tool patches x86 executables in order to let them have 4GB (instead of only 2) of virtual memory on x64 platforms. So the patch is useless on 32bit OS. You have 32 bit Windows XP. You can't do anything with memory error exept change your code.

PS: change your code - this is the simplest way.

As RA81 says, the 32-bitness of your OS is the limit.

The simplest way is to break down your MSB in, say, two runs - by time or by symbol count. But in the long run, upgrading to an x64 OS and patching the WLP executable file are required.

Reread this thread...you told me to buy more memory, use the /3G switch and try the 4G patch?!? You owe me $50!!!

So now you're telling me, I can't use more than 2G?

Re-read the thread again, 2nd post off the top:

QUOTE:
It takes a 64-bit OS to allocate more than 3Gb of RAM.

How can I use 3G of RAM?

Install an x64 OS on a different partition of your hard drive. For example, Windows 7 x64 RC. Release Candidate will work until it expires next Summer. It's stable and fast, and undoubtedly beats Vista.

64-bit Windows has a larger address space available, so it can give the 32-bit WealthLabPro more memory to utilize. Of course, you still need to apply all the tweaks and patches (here: IncreaseUserVA instead of /3GB).

By the way: at no cost.

Doesn't your Wiki need to be updated starting with "Luckily"?

QUOTE:
Is Wealth-Lab 5.1-5.3 able to use 3 Gb of RAM?

By default, no. Probably, native support could be added in a future release. However, it is possible to utilize 3 Gb of RAM right now:

From the technical standpoint, Wealth-Lab is compiled without the LARGEADDRESSAWARE flag set. On a 32-bit OS, WL5's memory usage is topping out at 1.7GB of virtual memory, subsequently throwing an "OutOfMemoryException". Luckily, it's readily possible to patch WLD/WLP 5.1-5.3 to give it an extra 1 Gigabyte of memory:

1.For XP, enable the /3GB switch in your boot.ini file. For Vista, use the IncreaseUserVA parameter in BCDEdit. See this How-To in MSDN.

After I told you I have a 32-bit OS, why did you say this?

QUOTE:
In addition to the /3GB switch, try the 4Gb patch utility. I believe it should patch WL5 to enable the flag.

1 - Updated like what?
2 - Maybe I overlooked something.

The way I read the Wiki it says: If you have a 32-bit OS, you can use the /3GB switch and the 4GB Patch to access an extra 1G of memory. That's not right is it?

You're right, thanks for the heads-up. Although the text highlighted the usage of NTCore's patch on 64-bit operating systems, it was somewhat misleading overall. Done, thanks.