Has anyone noticed signs of memory leakage in WLP during optimizations? I seem to have many GB of memory usage with WLP after all processes have been terminated and all windows in WLP are closed. It is bad enough that it takes many minutes for Virtual Memory to readjust after I shut down WLP.
Vince
Size:
Color:
It's .NET garbage collector's behavior. Restart WLP to force memory cleanup.
Size:
Color:
Wow! Didn't anybody ever hear of memory management? ;)
Vince
Size:
Color:
Size:
Color:
Windows doesn't seem to do it all that well. :(
Vince
Size:
Color:
If a backtest / optimization windows remains open, the data in memory are required to produce the visualizations. It can be a lot of data. Performance objects remain in use by the processes that may still receive another event to change the visualization or save data or whatever. These objects only go out of scope when the Window is closed. That said, if a new backtest is started, the memory is not recovered quickly as I would expect - the [Win 7] G.C. seems lazy in this respect.
Size:
Color:
In my observations even after all windows in WLP are closed the GC has not been done even if the program is not touched at all, even after an hour.
Is there any way that you can force Windows to perform GC when a window in WLP is closed?
Vince
Size:
Color:
It's possible, but the recommended guidance is to let the GC work as designed. Just because there's a lot of used memory doesn't mean that it won't be "available" when required. In other words, if enough free memory exists for new requests for memory, there's no need for the GC to do anything...the garbage only needs to be taken out when the can is full ;)
Size:
Color:
I suspect that your thought about Windows GC may be incorrect. When I find WLP in this situation and open new windows I get a LOT of disk thrashing (sending this "dead" memory info out to the virtual memory file), so I know that Windows is NOT doing its job.
Vince
Size:
Color:
As a side note, you might want to get rid of that thrashing disk by installing an SSD (at least as system drive).
Size:
Color:
The SSD comes with the next set of machines. For now spinning platters will have to do. That is because WLP is the only program that causes the problem. :(
Vince
Size:
Color:
It's really not just how I think it should work, it's the way Microsoft says it should work:
https://msdn.microsoft.com/en-us/library/0xy59wtx%28v=vs.85%29.aspx As long as address space is available in the managed heap, the runtime continues to allocate space for new objects....The garbage collector's optimizing engine determines the best time to perform a collection, based upon the allocations being made.https://msdn.microsoft.com/en-us/library/s5zscb2d%28v=vs.85%29.aspxIn general, you should avoid calling any of the collect methods and allow the garbage collector to run independently. In most cases, the garbage collector is better at determining the best time to perform a collection. In certain rare situations, however, forcing a collection might improve your application's performance.
Size:
Color:
Cone,
Do you also use an SSD? If so, perhaps the problem I am seeing is common but you and Eugene do not see it in your setups. It maybe more common than you realize for those of us with mechanical drives.
Vince
Size:
Color:
Vince,
What kind of optimizations do you run when memory is not released? (Which optimizer, amount of loaded data, runs etc)
Which operating system and WLP bitness is this?
Size:
Color:
Hi Eugene!
I have been using Len's PSO, using either Clerc Tribes or Basic PSO, for 25-50 generations. I generally test using Watchlists that contain between 400-1000 symbols and use 12-15 years of daily data.
I am using Windows 7 (64 bit) on both my current machines, and one has 8GB of RAM and the other has 16GB. I am using WLP 6.8.10.0 (64 bit) in both cases.
Vince
Size:
Color:
Do you get similar results when using Exhaustive optimization? It's important that what we're focusing on is a WL6 issue, not PSO's.
Size:
Color:
I rarely use Exhaustive Optimization so I cannot say. I have used the Genetic Optimizer however and noticed a similar issue.
Vince
Size:
Color:
In my testing, the PSO optimizer appears to be much more CPU intensive than the Exhaustive optimizer for the same amount of data. I haven't noticed any particular difference in memory usage.
I, too, have had the problem of optimizations hanging, but I've found that closing the Strategy window after a couple of optimizations and closing Wealth-Lab after several optimizations keeps that from happening.
Size:
Color: