Question about length of time series, persistence and system changes.

How much can we infer from 20 years of stock data? I mean, could we forecast the 100 year flood from only 20 years of data. How far into the future can we forcast from such a narrow window of time.

Would we be better off deriving inferences based on hypothetically 1000 years( if it were practical) of data that has some minute measure of relationship beyond chance then say in an extreme hypothetical example we can find a relationship using 1 minute data such that for the past 20 years every signal ( perhaps 10000s of 100s of 1000s) produced a 2% positive rate of return with a .00002% std. dev. and every signal was distributed evenly throughout the entire time series.

Even with such an unlikely finding, how much time can we assume this can persist? Should I have confidence for 10 years, one year, one month and is there a way to quantify the confidence decay over time?? Is this decay of confidence linear, exponential or does it reach some asymptote? Without further knowledge is it a fallacy to assume with any confidence to any duration of a time series at all?

For example, I think that we can observe a caterpillar to be a caterpillar with near 100% certainty while it remains a caterpillar. Assuming we never had prior experience or knowledge about butterflies, perhaps we would assume caterpillars remain caterpillars indefinitely. So, for the time span we have become aware of a caterpillar(perhaps one minute, one hour or one day)? Is it a not fallacy to assign confidence to it's persistence?

That said, we do observe the sun to come up in the East in the morning and set at West at night. We are aware of the countless times it occurred prior to our birth based on all accounts. We assume persistence and assuming I don't die this year, I expect to be aware of this series of events for this year to come. I'd bet all my savings on that. I expect my favorite beer Guinness, to remain my favorite,or until such time as I've had 3 too many.. :).


In addition, is there some branch of understanding that deals with forecasting system changes and duration with limited data, and is it necessary to consider this generating rules for trading markets? How does one test such a thing? How does one approach such things and is it practical in application?

You can get some good insight by readign "The Black Swann" by Taleb, where you'll find the example of the turkey up until Thanksgiving day. Quite a fitting anecdote for the current bear-market rally, imho.

I thought of "The Black Swan" while I wrote this post.