Hi!

I have a trading strategy with the following process flow:

1. At 8:15 am each morning, the Data Manager will update the needed data (Fidelity Fundamental and insider trading information). This process usually takes around 45 mins to complete.

2. At 9:45 am, the Strategy Monitor executes the strategy which generates all the buying and selling alerts.

This trading strategy of mine used to run correctly. However, very recently (about one week back), all the buying alerts start to show up one day late. For example, the trading system generates one buying alert today morning (1/14/2016) and this alert is with the Alert Time (Time Stamp) of 1/13/2016. This means this alert is supposed to show up yesterday (1/13/2016), but instead of yesterday, it actually shows up today.

This kind of delay has been happening for the past three days. And, this afternoon (1/14/2016), at around 4:20 PM, I did run this strategy again in the Strategy Monitor by using "Run this Strategy now" and three new buying alerts show up with the Alert Time (Time Stamp) of 1/14/2016!

How and why are all these happening? Are all these related to wrong time zone? But, I did check out the Strategy Monitor Time Zone, for my place (Maryland), it is GMT -5 (EST) - which should be correct...

Oh, during all day today (1/14/2016), I have not updated the data again. The only one time data update was done this morning at 8:15 am.

This is puzzling me for some time and any help/suggestion/comment is greatly appreciated.

Thanks,


Alex

Hi Alex,

The strategy log can contain some helpful evidence. Right click on strategy to enable it, then please copy/paste here after it has finished running.

Hi! Eugene,

Here I attached the SCLog,txt which does show today's (1/19/2016) Strategy Monitor's execution.

Again, all the alerts are with alert time (time stamp) of 1/15/2016 (last Friday).

Therefore, all there alerts are one day late again as yesterday (1/18/2016) was a national holiday.

Thank you for your help.


Alex

Thanks Alex. There's whole lot of symbols that trade infrequently with last bar on Jan 11 or 14. Also, requesting updates for such symbols slows the strategy processing. Have you tried excluding them from your SM DataSet? What is the result on SM timestamps?

QUOTE:
This kind of delay has been happening for the past three days. And, this afternoon (1/14/2016), at around 4:20 PM, I did run this strategy again in the Strategy Monitor by using "Run this Strategy now" and three new buying alerts show up with the Alert Time (Time Stamp) of 1/14/2016!

What if the data for these symbols had only become available afternoon?

Hi! Eugene,

Yes and after I review the log file, I see that most data are with 1/14/2016 (last bar). The Strategy Monitor was run at 1/19/2016. With all my alerts timed 1/15/2016 (show up at 1/19/2016). The 1/18/2016 was a National Holiday with 1/16 and 1/17 were Weekends.

The strategy is depending on Fidelity Fundamental data and Open Insider data which are updated right before the strategy under the Strategy Monitor been run.

So the supposed and correct process is: at morning of 1/19/2016, the data (Fidelity Fundamental and Open Insider) was updated to include 1/15/2016 data. Then the Strategy Monitor was run so that alerts timed 1/19/2016 with 1/15/2016 data show up correctly.

However, in reality, when the Strategy Monitor was run and the alerts timed 1/15/2016 with 1/14/2016 data show up (as shown in the log file).

As you point out that the possible cause is that the 1/15/2016 data was not ready until the next business day afternoon (1/19/2016) or at least not ready at the time data got updated and strategy got run…

I think I should update data and run Strategy Monitor at a later time than 8:15 am (my original daily data update time) and 9:45 am (my original daily Strategy Monitor run time)

Do you know how often the Fidelity Fundamental and Open Insider get updated? I am assuming these data are updated daily which is what my strategy depends on...

Oh, thank you for the suggestion to get rid of infrequently traded symbols. How can I get rid of these symbols? I am using the Fidelity Nasdaq dataset with 2904 symbols...

Thanks,


Alex

QUOTE:
Do you know how often the Fidelity Fundamental and Open Insider get updated? I am assuming these data are updated daily which is what my strategy depends on...

Speaking of Open Insider, I'm not a frequent user of this provider but as they seem to have the data for 2016-01-20 21:56:04 and it's only Jan 21st A.M. I believe they are updating it daily.

QUOTE:
Oh, thank you for the suggestion to get rid of infrequently traded symbols. How can I get rid of these symbols?

For example, with the Data Tool's "Remove inactive" feature. Cone will correct me on this but the built-in Symbol Quarantine may also help.

QUOTE:
With all my alerts timed 1/15/2016 (show up at 1/19/2016). The 1/18/2016 was a National Holiday with 1/16 and 1/17 were Weekends.

So... where is the problem? The 15th was the date of the last update, so this will be the date (and data) for the Alerts for trading on the 19th. Many illiquid symbols you're watching didn't trade on the 15th (and some not even on the 14th), consequently the date for their last update will be the Alert date.

Keep this in mind: The trade date/time will always for the bar after the Alert date/time.

Thank you so much Cone and Eugene...

Yes and the OpenInsider data is updated pretty fast and at least once a day for sure.

That leaves the data delay on the Fidelity Fundamental data side. For example, assuming today is a trading day and tomorrow will also be a trading day. Today's Fidelity Fundamental data will not appear on Fidelity's site for Wealth Lab to download until past certain time of tomorrow. And that time is, I believe, after 8:15 am (my original data download time which causes the problem) of tomorrow. That is why there is one day delay on all my alerts. Is this the case? Do you know when these data will be available tomorrow (after certain time)?

QUOTE:
So... where is the problem? The 15th was the date of the last update, so this will be the date (and data) for the Alerts for trading on the 19th. Many illiquid symbols you're watching didn't trade on the 15th (and some not even on the 14th), consequently the date for their last update will be the Alert date.

The data update actually was at morning 8:15 am of 1/19 not 15th... And the Strategy Monitor was run at 9:45 am of 1/19.
I have reviewed the log file further. Actually, none of the symbols were traded on the 15th. But, how can this be? This might not be caused by the liquidity issue. However, if this is caused by the liquidity issue, at least some of them should have been traded on the 15th (there are more than 2000 symbols). I guess this is caused by the reason that these data are not ready until pass certain time of next trading day's morning???

QUOTE:
Keep this in mind: The trade date/time will always for the bar after the Alert date/time.

From my past experience, the trade date is the same as the alert day. But, I could be wrong...


Alex

QUOTE:
But, how can this be?

1. Make sure your fundamental data is up to date by opening up the chart on the date of trading (the "a.m. 19th"). I assume it's part of the equation.
2. Remove the hundreds of illiquid symbols, and let's see what happens.

Good suggestion and thank you, Eugene...

I will make sure the data is up-to-date by opening up the chart than to see what happens.

I will post the result back soon, thanks.


Alex

Re: Fundamental data
These data can definitely arrive late. If you're using fundamental data in your strategies, you should count on 2 to 5 days of delay. In some cases, the delay can be 2 or 3 weeks.

Expanding on Cone's point... My experience was that historical fundamental data was not fixed so a data update today could change past data and trades. Live positions taken may disappear days, weeks or months later. There are big implications from that and your strategies should plan for it.

That's right. Many fundamental data providers are programmed to refresh (reload) the data on each update to account for corrections.

QUOTE:
Expanding on Cone's point... My experience was that historical fundamental data was not fixed so a data update today could change past data and trades. Live positions taken may disappear days, weeks or months later. There are big implications from that and your strategies should plan for it.

I had similar experience. After data update, sometimes, performance result of my strategy changes slightly. This might be caused by changes/updates/corrections within the fundamental data...

My strategy depends on the closing price (certain symbols) of the previous trading day. Is this piece of information part of fundamental data? If not, which data component/package it belongs to? And, how soon will this piece of information become available?

Obviously the closing price is not part of fundamental data (but static). On how soon see this thread.

Thank you, Eugene for the information on Fidelity daily data.

fairone99 asked in a duplicate thread:

Hi!

I have a trading strategy that was working fine before. However, a while back, it started to "behave" strangely.

Basically, my trading strategy was collecting needed data and been run at "Day One" night. Alerts were generated at "Day One" night. Based on these alerts, buying was to be made the very next day morning, "Day Two" morning.

However, a while back, it started to generate alerts exactly one day late (at Day Two night instead of Day One night).

My trading system logic has not been changed at all. And, I am primarily using Open Insider for getting the insider data and Fidelity for getting the pricing and fundamental data.

Since my trading system logic has not been changed and is “intact". This leads me to believe the data that I am getting has somewhat "shifted" by one bar thus caused the one day delay problem on alerts.

Could somebody help on checking if the Open Insider Data (especially) and/or Fidelity Data adapter were modified "incorrectly" thus caused this problem?

Thanks,


Alex

Hi Alex,

No need in the duplicate thread. Click your profile link to see what you've posted so far, or bookmark threads, or prepend your google search with "site:wealth-lab.com/forum/posts", or scan the Most Recent view for traces of your posts moved to other threads etc.

I can assure you that neither the Fidelity provider suite nor OpenInsider extension were modified incorrectly. This is not the reason. Go through this thread to find out possible clues. I believe we provided necessary suggestions before. At this point there's nothing to talk about given that we've never seen your trading strategy code - which is the mother of suspicion in troubleshooting.

Thank you Eugene for the suggestion and I did go through this tread again. And, spend some time investigating it further.

Of course, I will follow your suggestion to use profile to check all the posts/treads that I created (do not know about this until now)...

Let me post part of my strategy code that triggers buying and selling.

CODE:
Please log in to see this code.

As you can see that originally the code for buying is "BuyAtMarket( bar+1 )". This code block for buying always works and provides accurate buy alerts and generate accurate buy trades (historical).

However, one day (within this past January), all the sudden (I did not change any part of my code and configuration), the same code will generate alerts that are exactly one day late. And, when I run the strategy all the historical trades were made (when buying) exactly one day late also.

That is why that I have to modify my code to become "BuyAtMarket( bar )" in order to generate the correct trades (historical) that buy at the right date (not one day late).

All these lead me to think that Open Insider Data might be off by one bar...

Just a hunch and my little two cents...

Any suggestion?

Thanks,


Alex

Alex,

We already pointed to problems caused by delayed fundamental data (I wouldn't be shocked to find that the free data may be delayed) and having illiquid symbols in SM. One of my rules of thumb is to assume that my Strategy code is buggy. I hope you're going to adopt this good rule, too.

The OpenInsider provider (latest version) always returns the data as timestamped by the feed. There's no artificial modification of the date being made. It's up to you to compare the data to those of InsiderTransactionsProvider and Fidelity. Should you find an issue like date mismatch, let me know. Otherwise what we've got here is a false alarm.

As BuyAtMarket(bar) will not produce alerts, this introduces peeking into the future, causes an out-of-sync condition, and will not let the code run in the Strategy Monitor properly. Have you got some proof of OI data being delayed? Please don't hesitate the show me. Include all the details required like the complete code, symbol, data loading range etc.

Eugene,

After further investigation, here are the findings:

1. I have printed out various insider trading information (OpenInsider) by using a small C# program that I wrote within WLP and compare it to Open Insider website. They seem to match each other. And, I always compare OpenInsider data with InsiderCow and Fidelity Insider Trade data. They match with one another as well.

2. I figure out that for a giving insider trading information (OpenInsider), its location within the Bars has changed by one (compare it now with the old one which is before certain day of January). Is it possible that OpenInsider data extension has been changed so that all OpenInsider trading data are now one bar off compare with before?

It is getting confusing to me but thank you for your help, Eugene...


Alex

Alex,

Our conversation is going around in circles because you're approaching the problem from the incorrect angle. If you want me to have a deeper look, give me a factual evidence of the OI data being "delayed" in WL and a procedure to reproduce. This includes details like:

1. the symbol that you think has some one-bar difference,
2. how exactly you've determined that,
3. data loading range, bar scale,
4. the complete code,
5. when and how you update the fundamental data (on demand, Update DataSet, Update All Data)
6. the context the strategy is being run (S.Monitor, S.window)
etc.

You might want to enable hidden file visibility (Windows Control Panel > Folder Options) and attach what you think is a WLF file that demonstrates the issue from:
c:\Users\Windows username\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\OpenInsiderProvider\ ...

Hi! Eugene,

Sure and I will be preparing more "concrete" evidences to back my point and will post the result soon.

One quick question:

When I using MS123 extension which includes Open Insider Data Connector to download daily insider trading data from the Open Insider website:

For example, at each morning at 8:30 am, I am using the "Update only Symbols contained in DataSets" plus "Schedule an Automated Daily Update" (at 8:30 am) to update the data while highlighting the DataSet and checked the needed data Providers. At this time, is this Open Insider Data Connector getting/downloading all insider trading data from the Open Insider website at that specific moment?

If the above is true, whatever the data that I just download should be in sync (exactly match) with the data on the Open Insider website, right?

Thanks.


Alex

QUOTE:
"Update only Symbols contained in DataSets"

This excludes from update any symbols not contained in DataSets.

QUOTE:
while highlighting the DataSet

Highlighting the DataSet doesn't have effect during "Update all data..."

QUOTE:
If the above is true, whatever the data that I just download should be in sync (exactly match) with the data on the Open Insider website, right?

True. Re: "exactly match", only meaningful transactions are included: no gifts, grants etc.

Hi! Eugene,

As always thank you for your prompt response!

Here I provide all the information that you requested before in order to trouble shoot the alert one day late issue... Just to let you know that this alert one day late issue happens on normal strategy run as well...

QUOTE:
1. the symbol that you think has some one-bar difference,

I run my strategy against all Nasdaq Composite symbols (Yahoo).

QUOTE:
2. how exactly you've determined that,

For example: at 3/30 morning around 8:30 am, I started to download all the needed data (Open Insider and Fidelity). Than, after data has been downloaded/updated, my strategy was run at around 9:40 am. Normally, there are alerts dated with 3/30 entry date (base on the 3/29 data) for me to buy. However, these alerts will not appear at that time, these alerts become one day late and appear on the 3/31 morning (Strategy Monitor never show these alerts and only by running the Strategy Window, these alerts show up)...

QUOTE:
3. data loading range, bar scale,

Daily with All Data, Raw Profit

QUOTE:
4. the complete code,

Please see the code attached within the zip file.

QUOTE:
5. when and how you update the fundamental data (on demand, Update DataSet, Update All Data)

Every morning at around 8:30 AM and generally I use the auto update feature with "Update only Symbols that are contained in the Datasets"

QUOTE:
6. the context the strategy is being run (S.Monitor, S.window)
etc.

The delayed alerts never show up in the S. Monitor as they are delayed (disappeared). In the S. Window, I notice these alerts are delayed.

QUOTE:
You might want to enable hidden file visibility (Windows Control Panel > Folder Options) and attach what you think is a WLF file that demonstrates the issue from:
c:\Users\Windows username\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\OpenInsiderProvider\ ...

See the attached and zipped all my OpenInsider WLF...

Thank you again for your help! And, if you think I can improve my code in any way, please also let me know. : )


Alex

Alex,

Thank you for your detailed report. Having looked into the issue, I've just discovered that although the provider returns the correct timestamps, Wealth-Lab processes them in a somewhat unexpected manner that I didn't know before. As you may know, what makes the OpenInsider and InsiderCow providers different is that they rely on the filed date (vs. trade date). In this case, the precise filing date and time is being reported by the feed (website) as yyyy-MM-dd HH:mm:ss e.g. 2016-03-10 16:35:13 for a transaction in DD.

Here's what happens in your case. When a transaction's time is after the market close (04:00 pm) like in the example above, Wealth-Lab would push such fundamental item to... the next trading session e.g. 2016-03-11. On the other hand, another transaction in DD which took place during market hours (e.g. 2014-02-27 12:07:10) is assigned to the Feb 27 bar. The same behavior applies to the InsiderCow provider.

While I understand the premise, I disagree with the way the after-market transaction is being pushed to the next day. IMHO, since EOD signal processing is done after market close and EOD data isn't generally available immediately after, it's quite big a precaution to 'penalize' such lately reported transactions by delaying them onto the next trading session. Fundamentals aren't traded intraday, and there is hardly a "peek into the future" scenario since the feed won't report such transaction until it's actually filed.

Long story short, I will make the providers report the actual filed date and avoid that 1-bar delay. Please update the extension to v2016.04 using the Extension Manager and see if the issue is alleviated. Apart from this, there's no other issue in sight. Hope this helps.

Hi! Eugene,

Great and thanks!

I am eager to give this new OpenInsider Data Connector a try and will post the result back soon.


Alex

Hi! Eugene,

I have tried out the new OpenInsider Extension and it does somehow fix the problem. Now if I run the strategy in the strategy window, the latest trades for buying do show up, thanks!

However, the new version has an issue: it is based on the insider's filing date. And it will generate trades with entry dates exactly the same dates as the insider's filing dates. The entry price is the beginning price of the entry date which is the insider's filing date. The problem is that throughout the insider's filing date, insiders might file their trading to the SEC any time from early in the morning to late evening... And, there is no way, I can buy it at the market opening price at these filing dates/entry dates. I can only buy at the next trading date (filing date plus one bar).

Also, this fix while provides the latest trades in the strategy window, the buying alerts still fail to show up in the strategy monitor.

Is it possible to base the logic not on the filing date, but on the next trading date to the filing date (filing date plus one bar)? I think this will solve this issue.

Oh, also, I did notice the numbers of buying trades that happen within the trading hours are very low. Is it because that insider's filing hours are generally outside of the business hours or is it because of the new fix? I am not sure what causes this and I just bring this up for further discussion and investigation (if possible).

I appreciate your help, thank you.

Alex

QUOTE:
However, the new version has an issue: it is based on the insider's filing date.

This is not an issue, it's by design. It has always been like that. See provider's online guide in the Wiki.

QUOTE:
The problem is that throughout the insider's filing date, insiders might file their trading to the SEC any time from early in the morning to late evening...

Right. If I correctly understand what you're talking about, your trading system must not trade if the corresponding EOD bar for an insider transaction is not available yet. If the filed date is Apr 7 (today) but the market isn't open yet (like when I'm writing this), your system must ignore this fundamental event if no corresponding OHLC is found.

As I already wrote, EOD signal processing is done after market close and I see no issue here in principle. Suppose you update the data in the evening (6pm) but an insider has revealed his transaction at 7pm. You execute your system at any time after 6pm - even the next morning - and your system doesn't know a thing about this transaction. No peeking in the future. Where's the problem if the OI website won't disclose the insider trade until it's actually filed?

QUOTE:
Also, this fix while provides the latest trades in the strategy window, the buying alerts still fail to show up in the strategy monitor.

This is not an OpenInsider provider's issue at all. See posts #4 and #11. To stress, I don't see an issue here and it can not be duplicated. Your Nasdaq Composite DataSet is likely to contain a lot of symbols with data not up-to-date which in our opinion explains what you see. We already gave you suggestions and I've got really nothing else to suggest you on this matter.

QUOTE:
Is it possible to base the logic not on the filing date, but on the next trading date to the filing date (filing date plus one bar)?

No, as there is no "logic". The logic should be placed inside your trading system. My provider just returns the data but I will not make assumptions and guess what next trading date might there be. The provider has no knowledge of future trading dates with all those special hours and market holidays that it implies.

It's possible to revert the April fix and restore provider's previous state, with insider trades appearing on the next day but this seems illogical to me. Please let me know and I'll revert to the traditional behavior that is in line with Wealth-Lab's own assumption.

QUOTE:
Is it because that insider's filing hours are generally outside of the business hours or is it because of the new fix?

Because they report outside the market hours. To prove my words:

QUOTE:
I am not sure what causes this and I just bring this up for further discussion and investigation (if possible).

Just open up the OpenInsider website, filter meaningful transactions by sell or buy side, and notice the distribution of timestamps for yourself.

Hi! Eugene,

One quick question:

QUOTE:
It's possible to revert the April fix and restore provider's previous state, with insider trades appearing on the next day but this seems illogical to me. Please let me know and I'll revert to the traditional behavior that is in line with Wealth-Lab's own assumption.

Do you mean the next date to the insider's actual trading date or the next date to the insider's filing date? I guess it is the next date to the insider's filing date, right?


Alex

Right:

Reported vs. actual dates

QUOTE:
It's possible to revert the April fix and restore provider's previous state, with insider trades appearing on the next day but this seems illogical to me

Clarification: of course not all insider trades were displayed as filed date + 1, only those reported outside the business hours.

Just to clarify:

So the current state/setup of the IO Data Connector is that all insider trades displayed as filed dates.

However, previous state/setup of the IO Data Connector, insider trades within the market trading hours displayed as filed dates and the insider trades outside the market trading hours displayed as filed dates + 1.

Are all these true?

I just checked out OI website in more details by selecting most recent all transactions: there are a lot of insider trades that are outside market trading hours. However, there are still some trades that are within the market trading hours...

QUOTE:
Are all these true?

Right to both. You understood what I told in post #26.

QUOTE:
However, there are still some trades that are within the market trading hours...

To stress my point again, there's no difference between the two types if your data is collected late night or next morning and your system is executed before next open.

Hi! Eugene,

Yes and I always collect my data late night (11:00 pm) or next morning at around 8:30 am. But, do I have to execute my trading system before market opens? What are the differences between executing before or after market opens?


Alex

Alex,

It's up to you i.e. depends on system logic. Just give it some thought: for example, how can you buy next open (aka BuyAtMarket[bar+1]) after market has opened?

Eugene,

However, the bar depends on the data not on the real system time, right?

Since I update the data at 8:30 morning, it is still considered before market opens even I execute my trading strategy after market opens, right?


Alex

Yes to both.

P.S. Expanding the logic in my post #29, if some insider happens to release a transaction in the morning marked "today" (i.e. anywhere from midnight to your update time @ 8:30 am) you would want to ignore it as there's no matching OHLC until >= 4pm "today".

Agree and fortunately, these transactions anywhere from midnight to @ 8:30 am are very very few. So I should be safe. Maybe, I can update at night time around 11:00 pm than run the strategy after that. It might be a bit better?

Oh, I have checked the change log for the OI Data Connector the version 2016.01 (01/05/2016 04:50 AM) for fixing the "no data downloaded due to website changes" issue that I reported. I just notice soon after I updated with this 2016.01 version, than my alerts started to disappear... I am pretty sure that I have updated this right after it was released because it was needed for fixing the "no data downloaded due to website changes" issue. Notice that I reported the alerts started to disappear at 1/14/2016. Could this update have something to do with all these? Is this version 2016.01 (01/05/2016 04:50 AM) somehow different from the version 2015.07 (06/29/2015 07:04 AM) which is the very first version that started to include OI data? I mean the differences in design between them about treating these insider filing dates?

At the risk of repeating, you've been looking in the wrong direction. That version 2016.01 was a simple fix without changes to the logic. In your next reply please indicate how you applied our suggestions from posts #4 and #11 (include the DataSet's XML file and a new SM log).

Hi! Eugene,

You are right about the not frequently traded symbols as they are not frequently traded and somewhat cause some delays on the alert side. I have made a "house cleaning" on all symbols according to your suggestion, thank you!

Also, after the release of OI Data Connector version 2016.04, all the filing dates are correctly"aligned" (all base on filing dates) and this also contributes a lot on fixing the problem...

The alert delay problem is finally fixed and I really appreciate you and Cone's help!


Alex

Alex,

Glad to have helped!