Now that the transition from WLP -> WLD has long passed, and the control of WL is now completely in the hands of MS123 LLC, I would like some indications of what the plans are for improvement.

When the development of WL was in the hands of Fidelity over the past several years, there was very little visible effort or attempts to clean up the large number of bugs/anomalies. Furthermore, with the exception of some great efforts by some Forum members to improve the optimization functionality and Gene's continuing effort to add the latest indicators from S&C, little has been done to overhaul some of the limitations that have been part of the package during this time.

In the interim, there has been a HUGE developmental effort for a host of freely available Machine Learning tools. I suspect that for an extremely capable User that these are accessible, but for the vast majority of Users they are not.

Competing backtesting systems have incorporated many capabilities that are not part of WL. These include:
1) Multi-core processor support, especially for parallelized optimizations (processors with 16 cores will soon be very common, and AMD is planning to deliver one with 64 cores next year)
2) Customizable multi-objective optimization (MOO), since all trading systems require a series of trade-offs for real-world applications.
3) Applied constraints during optimization (e.g. minimum number of trades) so that the User can achieve more statistically-valid results.
4) Interfaces for the various freely available statistics/ML packages that can be used by a large contingent of Users.

It would be helpful for me to know what some of the thoughts of MS123 are with regard to the future
of WL. I have been an active user of the SW for MANY years and I truly miss the early days when the development of the system was very active. These last several moribund years have me thinking that this may now be a mature legacy product, but I am still holding out hope. :)

Vince

Hey Vince, the future of Wealth-Lab (no more "Developer") starts today:

The Circle of Life

Something big is in the making but it will take time!

Hi Eugene!

Yes, I saw that post which was the genesis of my post. I, and I am sure many members of the User community, would like SOME idea of where the development is heading. The static nature of WL has gone on way too long.

Vince

Hey Carova! Over the past few years I’ve been developing a new backtesting framework. We’re using this (Quantacula) as the basis for WL7. It’s .NET Core 3.1, so you’ll be able to integrate all of the newest libraries. It’s multi-core enabled for optimization and data retrieval, among other things. And the UI is being done in WPF which is a more modern framework than the old WinForms, so no more weird display issues. We’re all pretty excited and the development is going well, but yes it will take some time. We hope this pushes WL finally into the modern era with a great base to build forward!

Hey Glitch!

It has been a while. It is good to hear that WL is getting a much-needed refreshening. I do know that this will take time but I do hope that you will be able to lay out the improvements as the process unfolds. I do hope that this will not take too long, but I do understand that a major project takes time to write, debug, and package.

Good luck!

Vince

It's really good news to hear the original development team is back together. That's the best news of all.

QUOTE:
new backtesting framework ... as the basis for WL7. It’s .NET Core 3.1 ...

And "part" of the reason for using Core is because it may be more difficult to find replacement programmers for .NET standard in the future. This is a political issue, but it's still important for the life of the product.

QUOTE:
the UI is being done in WPF

Okay, I didn't know Core supported WPF, but perhaps it does on Windows. I learned something new. I'm kind of wondering how portable the WPF GUI is on other non-Windows systems?

I may post some additional suggestions for WL on the Reddit group, but I'm concerned if these suggestions would delay the release of the new product. It would be best to get the new WL released, then implement some of these suggestions later. But the "breaking changes" probably need to be included the initial WL 7.0 release.

Glitch,

One question - Will Multi-Objective Optimization be considered for the initial rollout?

Vince

WPF - it is implemented for Core but it is Windows only.
Multi-Objective Optimizer - was already implemented for Quantacula so yes, we can port that and integrate it into WL7, possibly as an extension. We need to determine that.

Multi-Objective Optimizer - That would be great if it is included. The kludge that I need to do presently to achieve "similar" results to MOO is poor at best.

Vince

QUOTE:
Multi-Objective Optimizer ..., we can port that and integrate it into WL7, possibly as an extension.

If you're willing to publish an API so third-parties can develop their own extensions for this, then I want to cast my support for the extension implementation. Including an example extension would be helpful in understanding this API and what it's capable of.

Some Extension code will be shared on Github. When we get to it we will publish the APIs.

Would I be better served to buy a license to Quantacula or wait for WL7? or will Quantacula become WL7?

If you buy a Quantacula license you'll be able to use the latest Quantacula version perpetually, and will also get a one year license for WL7 once it's released. It could also be useful to help get up to speed with the new coding framework. There will be some major changes in WL6 -> WL7, just like from the old Pascal -> C# days. The main change is the backtest now executes strategies for each symbol in synch, on a bar-by-bar basis, instead of running them for each symbol and compiling the results after the fact. This makes it much more powerful and flexible, but the cost is a new class model for creating Strategies. Since it's the same language (C#) the migration won't be as bad as it was before, and we will help all of you convert your Strategies!

Perfect Dion. I was actually in the process of porting over a rather large codebase from NinjaTrader/WL6/MultiCharts to WL6 now that it can be purchased in the US. I was also writing a TDAmeritrade provider for WL6 which I will stop immediately!

QUOTE:
the backtest now executes strategies for each symbol in synch, on a bar-by-bar basis, instead of running them for each symbol and compiling the results after the fact.

I'm not sure if I follow. Are you saying time (bars) is no longer the fastest moving variable? But when you're analyzing a time series (say with an SMA) one needs time as the fastest moving variable.

... or perhaps you're saying the performance visualizers are computed at the same time as the main simulation, which sounds like it would require much more memory.

QUOTE:
The main change is the backtest executes strategies for each symbol in synch, on a bar-by-bar basis,

That is the paradigm shift I've been hoping for. It likely wasn't feasible on the hardware available years ago. Bravo!

QUOTE:
That is the paradigm shift I've been hoping for.

Exactly. I have once implemented the bar-by-bar symbol context switch in my code but only found that it unrealistically slows down the backtest. So I ended up switching the symbol once per day for each symbol.

I'm just curious but why can we simply migrate from WL6 to Quantacula?
I think I'm missing something but what's the point of developing new WL7 while Quantacula is already working?

QUOTE:
I'm just curious but why can we simply migrate from WL6 to Quantacula?

Dion can answer your next question like no one else but as for this, it's that Wealth-Lab is a very recognized brand. Being a boutique company we need to keep and increase our customer base. While being superior in technology, Quantacula's brand is not known on the market at all. It's a challenge to develop 6.9 but we adhere to the brand and our community so it occurred to me that rebranding QS as the new WL7 would solve that problem. To everyone's luck Dion has agreed to the proposition in general. :)

P.S.
Having learnt lessons of the customer migration of WL4 to .NET, personally I believe that this time we must keep WL6 working not just until WL7 becomes on par but until the last customer decides to migrate. It's only my voice so this is rather going to be a team / business decision.

QUOTE:
I have once implemented the bar-by-bar symbol context switch in my code but only found that it unrealistically slows down the backtest.

And that's the rub. You loose Principle of Locality that way, so you suffer more processor cache misses. But the modern processors have larger caches, so that may offset "some" of that loss. But it still will likely be slower than running the simulation independently from the performance visualizer computation. But there are upgrade pluses:

1) One can now sample "novel relationship" data out of the simulation and include it in the WL performance visualizer computation.
2) You'll get a net speed up with the multi-core optimization, which will offset the speed loss from the additional cache misses.

---
For future purchases, I would recommend buying workstations with as much on-chip processor cache as possible. You don't need the highest processor speed (moderate would be nice), and you certainly don't need lots of processor cores (four maybe nice), but you need tons of processor cache.

And don't use a laptop. Get a workstation so you have as much cache as possible--that's key. Laptops aren't designed for large, demanding, "modern" simulation problems. Save those for word processing, email, and Internet surfing.

Besides retaining the brand, the other reason I didn't want to simply make Quantacula the "new WL" is that in Quantacula I intentionally changed much of the terminology and UI concepts to avoid a direct comparison to Fidelity's WLP. Now that we are developing WL7, I want to retain much of the familiar terms and UI as possible.

Here is my two cents ...

I'm a bit of a software junkie. I use NinjaTrader, MultiCharts, WL6, AmiBroker, TradeStation, AdaptTrade Builder, BuildAlpha, ThinkOrSwim for my trading. In addition to all that I have my own custom software I've created (though I would consider myself more of a hacker than a hardcore programmer). I also worked with the AlgoTrader guys in the early days before they geared themselves toward more institutional clients. I ran a trading desk for many years but retired early in my life to just trade my own account. When I was in the banking world I helped design our trading platform. I'm only sharing this to let you know I'm not a complete dodo bird on this stuff.

All the platforms above are good for specific things but also have glaring weaknesses depending on your use. If you are taking the time to reengineer WL why not broaden the applicability of the product to expand the user base?

mjj3,

Since you are familiar with a large number of backtesting programs, which specific features/capabilities do you believe would broaden WL applicability?

Vince

mjj3,

I (and I think many others) would love to see an unbiased feature comparison chart of all the backtesting software you are familiar with with a heavy emphasis on each program's strengths and WEAKNESSES. This could be very valuable to both users and developers. If you have some spare time...

Dave

Hi Dave,
If I have time, I will go through each platform and outline, in my opinion, the strengths and weaknesses.

Have a good weekend.
Mitch

Hi Mitch!

"If anyone read this far, you have way too much time on your hands..." - When the subject is the tool that I need to accomplish my goal I always find the time! More time is wasted by using a tool ill-suited to the task.

First, "Thanks!" for taking the time to provide your assessment of some desired capabilities for a "wish system". I will agree that WLD never appeared to be designed as an automated live-trading platform, and in many situations (e.g. trading OPM) it is difficult to imagine that many companies would consider this for trading timeframes > a few days, since the client risk-reward ratio is not favorable. The potential errant trades in client accounts is no way to instill confidence! ;)

For shorter-term trades (<1 day) in highly liquid Markets you are often competing against the "Renaissance Technologies" of the trading world, and if you can play with those "big boys", my hat is off to you. :) I will acknowledge that there are many places to find inefficiencies where Renaissance cannot operate, and in those locations automated live-trading might be profitable. However, in order to take advantage of those situations requires your SW system to access both tick-level data and the Bid/Ask, which WLD does not support in Backtesting or Live-trading, and sources for these historical data are rare and costly. Since I am inherently biased against day-trading, I will leave this discussion to others.

I definitely support your suggestion of being able to design and optimize multiple systems for a single portfolio, with control over the position sizing for each individual trade. Anyone who has even limited experience trading (even just Equities) knows that each trade has its own risk profile. In addition, different instruments will be in different modes (trending, consolidating) at different times, so a set of trading systems (trend-following, counter-trend, dipbuying, etc.) needs to available to adapt to these scenarios. WLD has some of the features to enable this capability but, as you point out, more is required.

One thing that you did not mention is an expanded Monte Carlo capability, in which the Minimum Variance Test you mentioned can be incorporated. Again, with the plethora of new "multi-multi-core" processors that are becoming commonplace, an enhanced MC capability is definitely something to consider.

One question that I have for the WL community of HW specialists, is there any value to harnessing the capabilities of 1,000+ core GPUs for WLD? Now that might provide some real acceleration! ;)

Vince

Add: Of course the GPU will be ESSENTIAL if you have an ML extension such as TensorFlow, but the question above is for standard operations such as optimizations.

The offtopic discussion regarding GPU has been moved to:

CUDA, GPU processing to increase speed of optimizations?

I will had my wish list also :-).

I know that everything is possible as Cone says all the time. But I’m a trader and I don’t want to spend my time to develop specific modules or complexify my strategy code to bypass limits. I want to spend my time to research profitable trading strategies. For this reason, WL can be adjusted without a complex redesign.

As I’m trading in European markets only, I would like to have a new version more friendly for non US markets. This means that the Symbol Info Manager must be redesigned to define categories of configuration for ticks, lots, Margins... In Europe, ticks are not limited only to one value (see example here : https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=29612163&lang=en&ib_entity=).

By this way, if you could add a new parameter (the configuration name available in the symbol info manager) in the Market Manager to associate the symbols list, WL could apply the right ticks configuration. Moreover, as the commissions are not the same for products types and exchanges that you trade, it would be good to add the possibility to define in the Market Manager which commissions script need to be apply to the symbol. For a more international versionof the Market Manager, you must manage also the trading breaks session as some markers work like that (ex. : Japon). The last evolution that you can add in the Market Manager is to allow to define the benchmark index associated to the symbol list. By this way, backtest results can apply the right benchmark index compare results instead of to apply all the time the default benchmark index in the WL parameters. Also Quotes tool must be linked also to the Market Manager to start and stop alert generated according to the market hours to be sure to not generate false Alerts due to quotes outside market hours.

Additional to this feature, as I’m trading mutual funds, I would like WL to allow fractions of shares.

For a more advanced redesign, I would like a real build in feature to allow backtest with multicurrency capabilities. Maybe if you add also in the Market manager a parameter to specify for a group of symbol the data serie to apply to convert value to the base currency of your account, you can implement it without a great effort.

With this, I would like to have a real money management logic implemented to facilitate risk management in a multistrategy / portfolio view. This means that you need to apply after the Position sizing logic a specific money management “module” to take into account your global account/position view. This money management module must be allowed to access scorecards values and havespecific functions to simplify to the “trading the equity” strategy also.

Also, for a better trading results comparaison with your backtest, it would be nice to integrate in the Account tool some specific features. I think about a trader journal (to track history events of your trading), an execution analysis to give you slippages (commissions, shares, performances) with the possibility to have it by order types, the expected shortfall and your performances with rolling metrics. See as an example what was QPAS : https://qusma.com/software/qpas/.

At the end, find here some minor changes that you can add in your backlog also :

QUOTE:
Quotes

--> Add in the "Quotes.txt" the Signal name
--> Add the possibility to have a market breaker : define a specific symbol to apply the same logic of the gap filter but cancel scanning for all symbols

QUOTE:
Orders

--> Be able to link Orders by broker interface : This means that WL will be multi brokers capabilities

QUOTE:
Accounts

--> Be able to rename PaperAccount name

QUOTE:
Data Manager

--> Link to the Market Manager to check quotes according to the correct Holliday days
--> Be able to change symbol name
--> Be able to manage Symbol database extra attributes (GICS, Security type…) by products type
--> With the ASCII dataprovider, add the possibility to load security name

QUOTE:
Strategies

--> Be able to have a master folder to define product type and see in the subfolder strategies by categoies (like today)

QUOTE:
Preferencies

--> Add a dark theme

QUOTE:
Editor

--> Integrate a basic debug feature

I hope that Glitsh has more than 2 hands to work on my list :-)

One of Dion's great ideas behind the WL7 user experience is to streamline it. Making it much more simple to use for the end user and developers alike.

Thanks to Expert Mode in WL7 we can integrate more seldom used features while keeping a consistent, integral design at the same time without risking it turning into a complicated patchwork of new tools and buttons that would push away many average users!

>>I hope that Glitsh has more than 2 hands to work on my list :-)<<

No, but I still have 10 good fingers!!

It would be nice if WL7 could build trading systems by itself, iterating over indicators, such as the program AdaptTrade Builder.

I envision a user friendly machine learning extension along those lines after we complete the initial release!

Hi Glitch,
Some questions on WL7:

Is it possible to make the data and backtesting engine callable components? I already have my own custom strategy / data mining search tool like AdaptTrade / BuildAlpha that I've written as well as my own WFO that I would like to incorporate into (or on top of) WL7 . My dream is to get all my tools under one platform where I can focus more on functionality rather than the framework. Along those lines, perhaps you can allow for a 3rd party extensions market place like tradestation?

Hey, we are back... just this time Dion DO NOT SELL the software again. I read this post by pure chance because it this was no longer WLD I knew, but now... How long will it take for the other old timers to show up?

Twenty years ago it was one of the most enriching intellectual experiences spanning from philosophy, music, statistics, economy, and yes... trading. And now the return of the king....

I second most of the things abegy proposes, I would add a real working way to trade the equity curve and be able to choose a base currency.

How about commercial practices, i.e. fees? Escrow to ensure our intelectual investment will not go down the drain? The current license protection scheme is scary in a Business Continuity Planning context.

Cheers,

Like always, the framework is very extendable, with many integration points. And the backtest library can of course be callable. In fact we built a web service to run backtests using the library to service Quantacula and next WL.com. We will also welcome third party extensions and will publish our API docs after release.

It would be nice to expand the SymbolData indicator to plot a bar (OHLC) instead of a single TimeSeries. See attached picture with the NYSE TICK from IQFeed.

What a coincidence, today I was thinking about the same thing.

This thread is a good reason to setup a product feedback tool so good ideas don't get lost and can be prioritized:

https://www.producthunt.com/ask/2195-are-there-any-apps-which-offer-a-feature-request-voting-widget
https://dynamics.microsoft.com/en-us/customer-voice/overview/
https://www.uservoice.com/


Some features I would want to see in future versions:

Backtesting and optimization of option strategies.
- I'm not aware of any platforms that support option optimization. Even if there are, the competition is scarce.
Threading/Gpu support
- Gpu is debatable gains according to other discussion thread but creating an api structure where that could be explored more thoroughly seems worthwhile

Cloud enable the entire platform
- One thing that occurred to me is that for those of us who don't do the fancy non-deterministic strategies, neural nets/machine learning/etc., optimizations only need be done once for a given set of inputs (where the wealthscript is itself an input). While there are multiple marketplaces for algorithms these days, I'm not sure there is one where you can sell/share them with the full spreadsheet of optimization outcomes so the consumer can choose which metrics to trade on.

Automated trading
- Brokerages: TDAmeritrade(thinkorswim), Interactive Brokers, and the up and comer Alpaca.

.Net 5
- Its at release candidate 2 and will almost assuredly be rtm by the time wl7 is ready to go. It has a number of speed improvements and will finally be the sole path forward for .net, might as well rip the Band-Aid off now.
- I think it would also pay dividends later to consider using Uno as the UI framework: https://platform.uno/how-it-works/

QUOTE:
might as well rip the Band-Aid off now

The sore is already dry. WL7 is in .NET Core, and .NET 5 is just the next release of .NET Core.

QUOTE:
This thread is a good reason to setup a product feedback tool so good ideas don't get lost and can be prioritized:

It's a good idea to keep the suggestions in an organized way, but I'd caution everyone to dial down expectations for a first release. You can expect improvements in many areas, but it's a big job just to get where Version 6 is today.

QUOTE:
Backtesting and optimization of option strategies.

If you have the data, what's missing to do that now? WL6 has synthetic options, although using HV in the synth. option pricing is far too inaccurate. The key is obtaining data for historic contracts, which is the real problem as I see it.

QUOTE:
.NET 5 is just the next release of .NET Core.

.net 5 is not just a trivial upgrade of core. Tons of stuff has been refactored and it can show up in weird places, like strings:
https://jimmybogard.com/mind-your-strings-with-net-5-0/

Of course, we will keep up with the ever changing evolution of our beloved .NET :)

My only request is to make WL7 controllable from automation scripts and applications (e.g. AutoIt).
If you draw your application UI by yourself like Fidelity ATP does, it's going to be uncontrollable.

Will WL7 include Neuro-Lab and NNIndicator?

Let me try this again... Will WL7 include Neuro-Lab and NNIndicator?

Not out of the box, but we will port that extension over and make it even better after WL7 is released.

I have just one wish: a stable Interactive Brokers provider (yes, tried the one available but found it unstable).

As far as broker providers go ... i would second IB but also add TDAmeritrade/Schwab as well.

For a WL7 enhancement, you might consider creating a Symbol class that works like the Path class in the .NET framework, thereby making Path specifications portable across all OSes. For the Symbol class, there needs to be a state (field) variable indicating the source provider syntax. Then there needs to be a ConvertTo member function depicting the destination provider. With this information, one should be able to write a regexp to convert symbols from one provider syntax to another that will work 95% of the time. You can add overriding rules for the remaining 5% in a v7.1 release. For example, the conversion ^SPX (yahoo) to SPX.XO (IQFeed) would not work with a single standard RegExp.Replace() expression, although a conditional .NET RegExp could probably be made to handle it in WL v7.1.

I initially felt the Path class in the .NET framework was a hassle, but I confess it has made my code much more portable across OSes.

QUOTE:
For example, the conversion ^SPX (yahoo) to SPX.XO (IQFeed)

If you're interested in coding it, give it a shot. If it would work then it could be part of some Community library 2.0. When you're at it, keep in mind that tickers themselves can be different across data vendors - not just the suffix part.

QUOTE:
If you're interested in coding it, give it a shot. ... it could be part of some Community library 2.0

Well, I can code the RegExp part. But expressions like Bars.Symbol have to return type "Symbol" instead of type "String" to make it work. Type String won't have a state variable indicating whether the "symbol" is from the Yahoo provider, the IQFeed provider, or Tingo provider. Without knowing that, a ConvertTo(...) RegExp cannot be formulated reliably. That's a problem.

And there isn't any reliable way to figure out who the provider is from a String expression "by discovery". So the identity of the source provider needs to be a state variable in the Symbol datatype from the start or you're dead in the water.

Bottom line, all the WL7 coding (including any indicators) needs to be coded as type Symbol instead of type String so the source provider is known. So there would be a big commitment in switching all WL primitives from type String to type Symbol. Coding the RegExp's is the easy part once you know the source and destination providers.

---
I suppose I can code the Symbol class (for US stocks) if you're willing to convert all WL7 primitives and indicators to using type Symbol (instead of type String). Unless you're willing to do that, it's pointless. My initial Symbol class implementation will not have a Dictionary lookup to handle weird cases, so any conversion that cannot be done with a "generalized" if ... else if ... list of RegExp's won't be covered. That's probably okay for US stocks, but for foreign stocks that have special cases, that could be a problem. The overriding Dictionary lookup to handle the weird foreign cases could be added later in v7.1. Perhaps a disk file could be used to populate the Dictionary with weird foreign exceptions.

Wow. Let's keep it simple and use string. :)

Hi Glitch,

Is it possible for Metrics and Performances visualizers to add a new option ? In WL 6.9, you display these information only for the global date range in your backtest.

It would be a great help if we can have an option to display these information at a specific date range also. By this way, you can easly access your OOS data results (or have a look at a specific period of time like during a drawdown or a market crisis period) without to rerun the backtest.

Moreover, as the Byperiod visualizer didn't give you access to all these information, it would be interesting to add the possibility to change the scale (Daily, weekly, monthly...).

I hope you will see theses add-on interesting to implement them ;-)

QUOTE:
without to rerun the backtest

Not saying that the proposed is useful but the program would have to rerun the backtest for a custom data range anyway with all that it implies.

QUOTE:
[For the symbol syntax conversion] Let's keep it simple and use string. :)

Well, I keep it simple today by not trading the weird cases in the first place. That's a good option for US stocks, but may not be a choice for non-US stocks. In order to cover the weird exceptions (which aren't converted with RegExp conversions), one would really need to scan for them in the exception Dictionary for each lookup. Of course, you only need to do the conversion/scan if the source and destination provider don't match, so it's not as bad as it first seems.

What I've typically done is open two strategy windows and run two backtests using different date ranges. I can then tile them horizontally or vertically and compare the OOS with the IS.

The only negative to that is if you want to run analysis of the IS versus the OOS or see combined results ... like correlation to the IS vs OOS, etc (see attached)

QUOTE:
If you buy a Quantacula license you'll be able to use the latest Quantacula version perpetually, and will also get a one year license for WL7 once it's released. It could also be useful to help get up to speed with the new coding framework.

I used WL in the pre-Fidelity days. Now that it's been "liberated" from Fidelity, and you are finally addressing some of the issues that bugged me 15 years ago, maybe it's time to come back. :)

I'll basically be starting over from scratch. Both Quantacula and WL are currently $399, so I'm not sure which is the best answer. Would Quantacula be the best introduction into the "new WL" model? Would that avoid the 6.9 -> 7 conversion headaches?

QUOTE:
Would Quantacula be the best introduction into the "new WL" model?

Yes.

QUOTE:
Would that avoid the 6.9 -> 7 conversion headaches?

Yes. There may be minor name changes and updates that could affect a Quantacula Strategy, but they would easily be resolved.

Will WL7 support a Norgate Data plugin? Thanks!

Most likely, yes. It's up to Norgate of course.

I like Superticker's Symbol class idea. Maybe we can get this integrated in some form or another. I was thinking that a Symbol class would have a property that would determine (from the Source's symbol) what kind of security type it was dealing with. For example, if HistoricalProvider A represents a cryptocurrency as BTC-USD, but streaming provider B represents it as BTC.USD, then the class would first accept the symbol from the provider, Provider A would flag BTC-USD as a currency pair. Provider B would then be able to parse the incoming symbol and reconfigure it appropriately. In my mind it's more of a SymbolMapper class that the Providers would somehow participate in, while we'd leave all the basic symbols as strings.

Certain international data providers are driven by symbol IDs, not symbol names (which are not unique and can be shared by different asset classes of the same provider). With WL's architecture, in Streaming charts there has always been an element of chance in symbol lookup because such provider can pick the wrong symbol. :/

Hi

I am not a programmer, apart from on WL and a bit in R, so can someone please explain why this is important / how it will change the user experience?:

"The main change is the backtest executes strategies for each symbol in synch, on a bar-by-bar basis,"

Thanks
Rod

Hi Rod,

The difference between the existing approach (WL6) and the new one (WL7) is greatly explained in this topic:

What is date sequenced backtesting and does Wealth-Lab meet this criteria?, see post #7 by brucev.

An immediate consequence is having script-based access to the portfolio equity curve or total number of open positions, for example.

QUOTE:
I like Superticker's Symbol class idea. Maybe we can get this integrated in some form or another.... In my mind it's more of a SymbolMapper class that the Providers would somehow participate in, while we'd leave all the basic symbols as strings.

The problem I have is IQFeed calls the ticker symbol PBR.A, but Fidelity's ATP calls it PBR/A, and Zacks calls it something else. And this is with a simple US stock case. If you had a Symbol class that could symbolObj.ConvertTo(destinationProvider) that would take care of this automatically, that would be nice.

My concern is that not every WL user is an oops programmer, so adding a Symbol datatype seems awkward for some. But if you represent a ticker symbol as string, then you won't know what the source provider syntax is, so you can't come up with a RegExp to convert from one provider syntax to another. I guess if you have string, you could do something like this to convert from IQFeed string to Fidelity string:

CODE:
Please log in to see this code.

But there should be an "implicit opps" way of "casting" the above three steps in one step (SymbolMapper class?) if you want to simply convert string to string for a different provider. However, if all tickers were expressed as datatype Symbol, then you wouldn't need a "string" datatype for symbols in the first place.

Symbol mapping is unrealistic because there are too many conventions by the different data vendors out there. Look for ISO 10383 standard, for example. World exchanges sometimes get merged or renamed which changes the suffix/prefix. Even the ticker can vary (not just for indices but for stock symbols across international exchanges). Without accounting for things like that, symbol mapping would have to make assumptions which is out of question. What is the budget? Who would invest time and effort into maintaining all this, and most importantly, what for? The majority of users does not care.

Thanks Eugene

Hi Eugene,

I'm little afraids if there is no possibility to translate symbol code. If I have understood correctly, there will be an IBKR broker extension. IBKR is not a good data provider. There is a lot of limits. For this reason, we need to mix this broker extension with others data provider extension.

Because of IBKR symbolic codifications, we need at least a text file to translate WLD symbols to IBKR. Can you confirm that we will have it for streaming and broker also ?

If yes, why not to use the same logic for all data/broker extensions ? Just give the user the possibility to save a file with the name of the extension where you will find symbols (any providers) and the right symbol for this extension. If the extension find a this file, it will use it. If the symbol is not inside, no translation will be made by the extension. This is simple, isn't it ?

>>This is simple, isn't it ?<<

Yes, that's a simple solution. Each Streaming/Broker Provider will maintain its own simple mapping rules that will be editable by the end user :)

(I haven't read the entire thread above so if it's already addressed I apologize)
Hi Glitch, just in case you're taking requests...

A few suggestions:
1) While there are a lot of good indicators included in 'Community Indicators' it seems some are not robust as they may cause WLDev to crash under some situations e.g. if they get plotted on a symbol with a short history. Perhaps all indicators need to be wrapped inside a code so they act more 'gracefully'.
2) The WL Standard Indicator 'RSquared': Definition - "R-squared explains to what extent the variance of one variable explains the variance of the second variable." As currently implemented it only plots using one series whereas it ought to plot one series (dependent variable) against a second one (independent variable).
3) Community Indicator 'Beta': While it can be correctly implemented inside strategy code, as a drag-and-drop it's essentially useless as it only plots 1 series against itself. Hopefully, this can be fixed by offering a choice of Index. [And why doesn't WL include Beta as a Standard Indicator??]

Hopefully you'll look into this.
I won't be able to follow-up.
Ciao.
P.S. Good to see the original WLab creator back in action!

1 - If you have a specific example of an indicator not acting stable and crashing the program please report it so we could fix it in WL7.

3 - Alpha and Beta will most likely be part of WL7. As other such indicators they'll be usable as drag-and-drop indicators. Check this out:

WL7 Sneak Previews, Alpha in drag and drop mode

I just intalled a WL7 beta build to see how it works with AutoIt.

Unfortunately AutoIt works with WinForms only not WPF which is WL7's UI is implemented.

Does anyone have an experience for automating WPF applications?

Let's start with what exactly are you trying to automate with the external tool in WL7 beta? Maybe we could find a better approach like exposing an interface or something.

QUOTE:
what are you ... trying to automate ...? Maybe we could find a better approach like exposing an interface ...

One thing I really like about WL is having an API to call from my own code to build upon. This is what sets WL apart from the other trading systems (including ATP).

I semi-automate WLD6 for things like these:

1. Monitor WLD process and start the process immediately once it disappears (e.g. application crash).
2. Monitor IQFeed Client error and restart WLD to recover from the data connection issue.
3. Monitor authentication prompt to login automatically for faster and reliable login.
4. Send orders through Trade Ticket so you will never send wrong orders.

There are more scripts but these are essentials for day-trading.

QUOTE:
I semi-automate WLD6 for things like these:

1, 2. OK, a watchdog. Fortunately, both process monitoring tasks aren't WPF specific.

3. There will be no more periodic authentication requirement in WL7 and consequently no dialog prompt to worry about. :-)

QUOTE:
...having an API to call from my own code to build upon

Ever thinking about providing an API we need to keep balance between the flexibility it gives and its other side: unauthorized usage of our libraries in external code by programmers. Just to voice a concern.

QUOTE:
1, 2. OK, a watchdog. Fortunately, both process monitoring tasks aren't WPF specific.

Although these two watchdogs are not WPF specific, there are other scripts which monitor an exception from the strategy code in which case the code pops up a dialog and WPF becomes an issue.

In any case, semi-automation is a must requirement for me to day-trade.

Dear old friends :)
I'm happy to see that WL came back. I bought for the first time WL in version 3/4 ... and now i support , too the product project.
I check last revision it is quite similar the old one ... that i used last time 6-7 years ago.
At the begin the product was cool .. now continue to be innovative respect other one in the market , but the main iusse is that lost the community of ts coder in these years so it's important to attract new ts "inventor" ...
My opinion is that actually a great oppurtinity is in the crypto market . A good TS with a good BOT that work on Crypto market could be a great opportunity .
So , i saw that there is some component for connect some data provider for crypto but for example don't find a data connector fo interconnect directly by API binance . That could be cool .
I search more infor for develop additional component for WL but don't found where it's available ? So i can try to develop it myself and share with the WL community ...
I think that interconnect directly crypto market could be a great opportunity for develop also cool ts for that kind of market.
Best
Roberto

Hi redfox74, wait a little while until WL7 is released. We will welcome third party extensions that could include data providers (crypto) and broker connections. WL7 will be a great environment to build a crypto trading bot!

QUOTE:
I search more infor for develop additional component for WL but don't found where it's available ?

They are available in the Wiki where one has to register and then we'd whitelist his account to access the protected content. However, the WL6 broker API isn't documented at all. Like Glitch said WL7 will have an open, documented modern API for everything so you might want to save yourself some development effort 'cause it will be different from WL6 (read: incompatible)!

Oh that's a great news ,
when will be available v 7 ?
Best
Roberto

So I mentioned this in discord about creating a Worker Service App would solve many of the automation requests. It's very simple to automate almost anything as long as the API hooks are exposed for what you want to do. You can also just have the service ensure that WL7 is running and let WL7 take care of the rest.

As long as unauthorized usage of the libraries in external code could be taken care of, it sounds interesting.

Will it be possible to run multiple copies of WL7 under a single profile?

Vince

Yes.

Hi. Congratulations on your progress and work to the new release, and freeing yourselves from fidelity! I believe the core issues are being addressed,

1. Multi-core processing - (better speed)
2. Working each symbol in parallel within the same time bar - (better reproduction of backtests within a risk framework)

I suspect with this base everything else will take care of itself fairly rapidly!

Hi,

Dion has answered both questions in posts #4 and #13, respectively.

Will WL7 ignore NaN (and other items like infinities) when plotting a custom indicator? Thanks!

Vince

What do you mean? Infinity or NaN can't be plotted in WL6 either.

Eugene, sorry for the poor statement of the issue on my part...

If a DataSeries has a NaN or Infinity in the series it fails to plot currently in WLD. I would like to hope that WL7 will just disregard those pathological data points and plot all of the others.

Vince

QUOTE:
I would like to hope that WL7 will just disregard those pathological data points and plot all of the others.

Yes Vince, it will disregard them:

Thanks! Another "3-point shot" for WL7!!

Vince

Will WL7 have GPU support, to speed up the calculations of things like indicators?

Vince

Indicators are already calculated very fast but the code for some of them is parallelized to take advantage of multiple CPU cores.

But what about including CUDA-support for this activity, especially during operations like optimization, where the impact would be substantial.

Vince

This the default strategy for WLD6.

CODE:
Please log in to see this code.

What does the default strategy for WL7 look like?

WL7 does not have GPU/CUDA support as of now. If we see a demand due to this because of slow calculations we can add it to the roster of features to add post-release.

But I'd be curious if you could use techniques like below in your indicators right now by adding the appropriate references ...
https://developer.nvidia.com/blog/hybridizer-csharp/

Here is the WL7 Strategy default code:

CODE:
Please log in to see this code.

QUOTE:
WL7 does not have GPU/CUDA support as of now. If we see a demand due to this because of slow calculations we can add it to the roster of features to add post-release.

Understand. I am in favor of ANYTHING that speeds up the overall process. I HATE to wait! ;) (insert snarky comment about "instant gratification" here :) )

QUOTE:
But I'd be curious if you could use techniques like below in your indicators right now by adding the appropriate references ...
https://developer.nvidia.com/blog/hybridizer-csharp/

I would be more than interested in learning how to do that, but with my VERY LIMITED knowledge of C# I would need a good example and a bit of "hand-holding", but I am more than willing to give it a go! Count me in!!

Vince

Regarding Post# 98, so there isn't any trading FOR loop with GetTradingLoopStartBar(...) in there. Perhaps, this is early to ask, but how are the parameters for GetTradingLoopStartBar(...) handled then? ... say for the example below.

CODE:
Please log in to see this code.

Because of stochastic differences between stocks, it won't be possible to predict the FirstValidValue--which depends on stock statistics--for each stock until the decorrelatedMacdHisto indicator executes for that particular stock.

Since BarsHistory(Bars) is passed to "Initialize", where strategy DataSeries are developed, one solution is to store the earliest date that all DataSeries are valid in a private variable, then, in the Execute method, just exit if the current bar's date is earlier.

QUOTE:
one solution is to store the earliest date that all DataSeries (TimeSeries) are valid in a private variable

That won't work here. The FirstValidValue varies widely from stock to stock. Some unstable stocks would start trading way too soon loosing money. The FirstValidValue needs to be tailored for each individual stock. And perhaps WL7 can do that now like WL6 does; I just want to be sure; otherwise, I have a problem. The FirstValidValue needs to be stock dependent for a given dataset simulation run. Taking the maximum FirstValidValue for all indicators on a stock-by-stock bases would work. Can this be done?

Alternatively, letting the Initialize(...) method set the FirstValidValue for the simulation on a stock-specific bases would work since at the end of Initialize(...), all the critical FirstValidValues would be known for a specific stock. The Initialize(...) method could then set (return?) the maximum FirstValidValue for that particular stock before exiting Initialize(...). Can that be done?

Initialize is called once for each stock, The private variables, including "earliest date", uniquely exist for each stock. You'll put all the current strategy code above the for loop in Initialize.

QUOTE:
Initialize is called once for each stock, The private variables, including "earliest date", uniquely exist for each stock.

So Initialize(...) can set stock-specific variables like "earliest date" for the trading loop. Sounds like I'm good to go then. Thanks a bunch.

I was concerned that the same "earliest date" would apply to all stocks, and not be applied on a stock-by-stock bases like WL6 can do.

+1 for GPU support :-) ! If you implement this, you will be the first backtesting software with this advantage. At this time, I haven't see another competitors with the possibility.

There are several other optional overloads in UserStrategyBase you can hook into. To set some initial date it would make sense to do this in BacktestBegin, which is called once right at the start. A property called BacktestData gives you a List<BarHistory> which is the complete set of data being tested.

Glitch, I disagree that this would be done in BacktestBegin, I would expect superticker's case to look like this...

CODE:
Please log in to see this code.

I would suggest that users might get a head start, right now, on converting their strategies to WL7, by creating an "Initialize" procedure to encapsulate the work done ahead of the main for loop. The code above compiles in WLD6. WealthSignals authors take note.

There are of course, more than one way to tackle any problem, was just suggesting how I would go about it :) I edited my post above to include a flowchart for the new UserStrategyBase hooks you can override.

Thanks for the helpful flowchart. I was happy see BackTestComplete. I sometimes do some statistical work after the last bar of the last symbol.

Which leads to... Is there a Global feature? That's where I accumulate data.

MS has published some work about using "standard" C# code with GPUs/CUDA to significantly improve speed.
https://www.microsoft.com/en-us/research/publication/accelerator-using-data-parallelism-to-program-gpus-for-general-purpose-uses/
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2005-184.pdf

From the link -

QUOTE:
Some benchmarks significantly outperform C versions on a CPU: they are up to 18 times faster than C code running on a CPU.

Glitch, is this something that would be compatible with the design of WL7? I am too ignorant to judge it myself.

Vince

>>Which leads to... Is there a Global feature? That's where I accumulate data.

You could use static variables, so the variables would be accessible to every symbol in the backtest.
Or, if you really need to save data for use in another run or even another strategy, use the GetGlobal and SetGlobal methods.

I have too much to focus on right now to be able to fully read and digest these papers, but I looked at the example code, below. I can't think of a reason it would not work, you'd just have to reference the proper assembly in the Assembly References tool. I guess there could be a benefit if you code some very complex indicator creation using this method, but it would be problematic if the indicator value depends on its previous values (like RSI for example).

CODE:
Please log in to see this code.

Glitch

QUOTE:
the GetGlobal and SetGlobal methods.

That's all I need, for now. :)

With regard to Post# 106 above, which includes a very nice chart of the programming model for WL7.

1) It's a very good idea to batch the indicator creation one stock at a time as is done with Initialize(...). This provides an opportunity to vectorize the indicators for hardware pipeline acceleration.

2) The Execute(...) step defines "Symbols" as the fastest moving variable and "Bars" as a slower moving variable, which is contrary to WL6. Although there are some analytical advantages to this approach (e.g. a rotational strategy can compare tradeoffs of moving investment monies from one symbol to another on the fly--nice feature), there's significant hardware considerations to be concerned about.

2a) All indicators for all symbols must be cached in memory at the same time. To facilitate this, one now needs to limit the product of (#of_symbols)x(#of_indicators) within a dataset. That's not a problem for a rotational strategy, which is what this design is targeted for. But the days of having 1000 symbols trading with 15+ strategy indicators within a dataset are now over. That would be just to slow to bear.

2b) The Principle of Locality is shot to pieces as far as processor cache hits. That may slow you down by a factor to 10. One might consider storing the indicators as single precision instead of double precision to improve cache hits. One would still get the same answers and your speed (cache hits) would go up somewhat.

2c) It might be fastest to perform some type of "memory transpose" of the Symbols vs Bars vs Indicators before starting the Execute(...) block. That's a three-dimensional transpose. Indicators would have to be the fastest moving variable, followed by symbols, and then followed by bars. Better yet, a "series" of two-dimensional transposes of indicators and symbols for each new bar increment might be doable. Is that what happening? If so, how is that working? (Please include your cache and RAM memory size.)

With this new design, we should all switch to rotational strategies. To not do so would be foolish since we now have to power to pit the performance of one stock against another on a bar-by-bar bases. And that's a major rethinking of everyone's exit code. Rotational strategies, here I come!

I see grate work Glitch to improve WL and make it backtesting more stability

And by talking abut Data provider i am one fo those who use WL to my localal Market using metastock data

I hope that Has been included

How does the memory footprint of WL7 compare to WL6 for a similar item, such as an optimization?

Vince

QUOTE:
Just my opinion - Concentrate on building functionality. Fully exploiting advanced hardware can wait until after WL7 is launched.

I agree ... once WL7 drops I would rather see efforts put towards extensibility (api's documented).

Eventually I think it would benefit the platform to have a simple customizable wpf windowing grid (like ThinkOrSwim) to expand the audience beyond a research tool to a full trading platform. As long as the IHost api, chart control and data manager are exposed this is something I should be able put together.

Grid


Customizable Buttons

to #115 - yes Metastock data file support is included in the base product.
to #116 - I have not run any side by side comparisons of optimizations in WL6 and WL7, maybe that is something vk can do because he's been heavily involved in testing both the standard optimization and the beautifully integrated WFO :)

I can't imagine the optimizations will easily run into memory problems however, because the end result is just an instance of the OptimizationResult class which includes all of the calculated metric values for the various selected ScoreCard metrics. So, each optimization run does not result in a tremendous amount of memory usage.

Glitch,

Re: Memory footprint - I require a significant amount of sophisticated recursive pre-processing on my input data (based on some external inputs), and my calculations for ranking are real memory hogs. A typical training run of 25 years over 500 stocks takes 10-15GB. I am looking at getting a new box (possibly a Ryzen 64-core with 128GB of DDR5 (when available)), but if that is inadequate to step up my capabilities with WL7 I would like to know sooner rather than later. Thanks!

Vince

You could store your pre-processing in the BarHistory Cache or in Global memory. The optimization could then read it each run from the cached items. Or, are you saying that each optimization run needs to generate this 10-15GB of data? If the latter case it true then "it is what it is" and it's your process that's creating the memory footprint, not the optimizer. The footprint you'll wind up with then is your 10-15GB multiplied by the number of parallel threads that are running the optimization. You could, if needed, scale back down to a non-parallel optimizer.

QUOTE:
Or, are you saying that each optimization run needs to generate this 10-15GB of data?

Unfortunately, yes. I have been exploring this new approach the last few months that requires highly adaptive inputs, and I do not know where it will end up, or even if it is worth pursuing. :( But since I need to plan ahead, I want to make sure I am covered IF I continue to pursue it.

BTW, this was also why I am SO INTERESTED in any way to speed up the optimization process.

Vince

The Tail Ratio has been identified as the most highly predictive indicator of OOS results for trading systems (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2745220). It is currently included in the Performance+ Visualizer only.

Will it be included as a Metric for Optimization in WL7?

Vince

Along with other exquisite performance metrics requested by power users, the Tail Ratio will be part of a paid extension which contains the Extended Scorecard.

That is good to hear Eugene!

BTW, when can we expect to see some "snapshots" of what the various pieces and parts of the "WL7 Family" will contain? Also is there a ROUGH ETA for WL7? I suspect that it is currently in Beta, though that is not clear from the Discord website.

And one other question, will all of the current indicators in the TASC and Community collections be ported over to WL7?

Vince

We are in beta and we're shooting for mid to late February for the launch of WL7, which comes with a new website. It's an incredible amount of work for our small team, but we're really excited about it like never before! TASC is already done, Community Indicators will come later.

Thanks Cone! I am sure it has been a LOT of effort! Kudos!!

Any idea when can we expect to see some "snapshots" of the various pieces and parts of the "WL7 Family"?

Vince

QUOTE:
...will all of the current indicators in the ...Community collections be ported over to WL7?

Community Indicators have already been ported for the great part and will take their place in the baseline product. We do plan to keep most frequently used indicators with a view to get rid of many non-essential ones: duplicates, arcane, less known, too jerky, less effective etc. If it happens that a missing indicator has a strong community following and it doesn't duplicate another one then it can be added on demand, of course.

Understand completely Eugene!

Vince

I downloaded the program but it is no more functional as it demands a login. It doesn't accept my login credentials used to access this forum, though it asks for it.

You downloaded it on Jan 22 so the demo period has expired. I've reset it, please follow the link to download the latest beta and then retry.

Will it also expire after some time?

Yes, of course. It will expire in 14 days.