Dear NN-Experts,

could anybody of you explain (in simple words) how the NN-Indicator and Performance Evaluation has to be interpreted?
The guide is not that clear to me.

Regards
Christian

Not an expert by any means but the idea of NNIndicator is to gauge the chance that the condition for which your network was trained will take place. It's normalized in the 0-100 range from a very low (0) to a very high likelihood (100).

That means for a strategy to consider (as much as possible) the "balance" between high number of observations and high NN-indicator ?

The two things I look at in Performance Evaluation are Predicted Output and Actual Output. A well designed NN will have agreement between these columns. Especially, this should be true for the Out-of-sample data, to indicate robustness.

Note these columns have been transformed back to the units of the output variable, giving a big clue that the 0-100 NN range can be translated back to the predicted output value (which disagrees with Eugene's hypothesis).

See sample Performance Output of a good NN.

Sorry for my stupid questions, but i would like to understand the logic and design behind.
I dont want to stress you.

What is the NN-Indicator ranges you have a look at on your Screen shot when you speak about a good NN.
i do hard interpreting this, because in every case the predicted and actual Output differ.

How can i transalte back the NN to the predicted Output.?

By the way, do you have a recommendatin regarding literature of trading and NN?

Regards
Christian

QUOTE:
What is the NN-Indicator ranges you have a look at on your Screen shot when you speak about a good NN.
i do hard interpreting this, because in every case the predicted and actual Output differ.

They differ, but both are more negative at the top and get increasingly positive. This seen in the histogram bars, too, red at the top, green at the bottom.

QUOTE:
How can i transalte back the NN to the predicted Output.?

Translating back is trickier. My example is of a range-limited output, -16 to +16, so in my case 1 corresponds to -16 and 100 to +16. If the output is not range-limited, then you have to remember the range that was shown when you did "Select Training Data." As far I as I know (without Neuro-Lab code) it's a linear transformation.

I have no literature recommendations.

Len

ok, so according your shot i would Focus on NN indicator range 50-55 because there are the most Observation.
So how important is the number of obervations for you?

QUOTE:
So how important is the number of obervations for you?

Very important. I usually aim for at least 35-50K observations. I usually train on ten years years of data, for near 150 symbols. I use "Load every" on "Select Training Data" tab to get near 35K. The most current 60% in-sample, the other 40% out-of sample. It is a problem the way Neuro-Lab splits in-sample/out-of sample on date, because one may be in an up market while the other is in a down market. Can't control that.

I would like to develop a strategy trading the ETF SPY.
So when i train the SPY, so it is only one Symbol.
Even with data for 10 years, i have only 3000 Bars .
You mean that is to less?

In the screenshot i send you the Settings in Network Training i would do.
What do you mean?

With only one specialized symbol, SPY, it is easy to overtrain and difficult to prove robustness. Ten years includes the unusual years 2008 and 2009, and these will be in either the in-sample or out-of sample. I have no experience for what you are doing. I would never trust using a NN (or strategy) that is only proven for one symbol.

To answer your .jpg question: I train by watching the curve of the training error and stop the training manually. The usual case shows, 1. gradual improvement, 2. more rapid improvement, and then 3. return to gradual improvement and eventually no improvement. I stop training soon after the 3rd phase begins, which I feel marks the start of over-fitting. It works for me. Near the end, I may manually clear the graph, then toggle Stop/Continue, hoping to catch a minimum. With only 3K observations, it will run very fast and that may not work for you.

The user interface for Neuro-Lab training is very crude. I had entered tickets to improve training years ago, but they sat for over a year and I eventually cancelled them. Until recently NNIndicator was very slow, making Neuro-Lab unusable for some users. I use my own code which is about 3x faster (formerly 20x faster). There seems to be little interest in Neuro-Lab by the community or developers. With only a little work, it could be a much better tool.

Ok, understood. Worthful answer.
I always thought that generally a NN has to be trained on the portfolio/symbols foreseen to be traded later on.
So when i want to set up a strategy trading the SPY i thought about traing a SPY - NN.

Do you recommend minimizing the number of input neurons to avoid overfitting?
I try to develop a short term ( 3-7 days) )strategy on the SPY and therfore think about using about 15 inputs.
( 8-10 technical indicators with different lock back periods) . Its too many?
Do you have a hint for the input?

... i am wondering the little interest on NN, because the whole machne lesrning stuff is in every mouth.
At least on more NN interested user:-)

QUOTE:
I always thought that generally a NN has to be trained on the portfolio/symbols foreseen to be traded later on.

I agree.

QUOTE:
Its too many?

The problem is not number of inputs. It's that there are only 3,000 Bars of output (observations).

QUOTE:
Do you recommend minimizing the number of input neurons to avoid overfitting?

The best NN technique is sometimes to minimize the number of nodes. One of my best NNs has 11 inputs, and a single hidden layer with only 4 nodes.

Think about this. If you can develop a strategy that can predict SPY 3-7 days in advance, you can become very, very, rich. This tells you it is very difficult. (Remember me if you succeed!)

Len

Hi Len,

attached the performance evaluation of a NN using the parameters from one of my rule-based strategies.
What do you think about that?
The dataset consists of 110 Symbols.

How do you deal generally with training rate and momentum?

Regards
Christian

I can't tell much from your jpgs except that it's been very overtrained - 1500 epochs without an "Error rate low." That's too high by about 1450. I can't tell anything from your Performance jpg. The fact that all results are clustered between 45 and 60 is unusual. I don't Auto-Stop or Auto-Clear. The important info is at the start of training which was erased by Auto-Clear.

Little snippets of screen shots don't help me much.

QUOTE:
How do you deal generally with training rate and momentum?

I use the defaults.

Attached are screenshots of what I consider to be a good training session and performance evaluation. The training shows the curve pattern I mentioned in post #10. That's what you should look for. I would consider training of this NN to be complete. Notice that this is only epoch 804 and also only 14 epochs since an Error Rate low. Any further improvement is overfitting in my opinion.

I leave you to do further learning on your own.

Len

Thanks , Len!
I will continue learning NN.

QUOTE:
There seems to be little interest in Neuro-Lab by the community

I'll volunteer to be another exception (outlier if you will).

Other than the fact that Wealth-Lab Pro has a tool for creating NN's, is there a reason you use NN's rather than support vector machines?

Until your post, I had never heard of support vector machines. Might be some work to integrate that with WL. NN works very well for me.

I originally used Stepwise Linear Regression (1966-68). Then came a long period where I had no need for modeling. In 1999 I started using modeling tools for trading and found a cheap ($50) but robust NN tool which produced easy to use flat file weights. I built my own NNIndicator, if you will, in VBA to calculate NN values. This was effective so I never pursued other options. Because the training interface in my $50 tool is more powerful, I thought about integrating that with WL.

The fact that NN comes with WealthLab also made the choice easy. I was already trying to learn C#.

Len:

Would you care to elaborate on your $50 NN tool?

A link to the source, maybe?

Thanks,

Dave

It's called Qwiknet. This info is from 1999. The software no longer installs on my Windows 10 laptop; I use it under XP.

QUOTE:
Please send any questions, comments or bug reports to:

Craig Jensen
2488 239 Place NE
Redmond, WA 98053
USA
email - cjensen@kagi.com

I Googled QwikNet Neural Network. It may still be available at the same $49 price.