WL6 (Database provider) skips a minute during streaming
Author: haytac
Creation Date: 10/22/2021 12:29 AM
profile picture

haytac

#1
Hi,
I have an app that gets quotes every 5 seconds.
This then is fed into a SQL server table.
WL6 reads from the SQL server and builds its own 1 minute candlestick.
This is during streaming.

This app also creates 1 minute candlesticks from the 5 second data.
This is fed into another table in SQL server.
The after hours runs of WL6 use this 1 minute SQL table.

Yesterday for the first time in one year of running this scheme
I noticed that WL6 did not show the candlestick for 15:20 ET on the price pane.
This was during streaming.

When I run WL6 after hours, I can see the candlestick for 15:20 ET.

Checking 1 minute and 5 second tables I can see data for this minute.

Somehow during streaming WL6 skipped this minute.
Any guesses as to why would be much appreciated.

This error causes a run time error which disables WL6.
My app exits a position if there is no heart beat from WL6.
Nevertheless it would be good to know what may have caused this.

Thanks!
profile picture

Cone

#2
What timestamp does your app use for those 5-second intervals - the start or end of the 5-second period?

QUOTE:
heart beat from WL6

I'm not the SQL adapter guy, but what heartbeat is that? Wealth-Lab itself doesn't have a heartbeat but it should get one from streaming data providers.


QUOTE:
This error causes a run time error which disables WL6.
What are the error's details? Is there a stack trace output?
profile picture

haytac

#3
I have the timer tick tuned so it comes at 0 seconds 100 + milliseconds and then 5 seconds 100 + milliseconds. And so on.
There is some variation but the 100 milliseconds covers it so it is never 44 seconds + for example.

I notice that WL (stream mode) starts running right after the top of the minute.
It seems that it sees the first update of SQL table after the top of the minute and then runs.
I record the OHLC for that minute (the past 60 seconds)
Looks like it is composed of second 0 to second 55.
In other words what gets WL started that SQL reading is not included in the minute OHLC.
It is included in the next minute's OHLC

I was not watching at the time.
It seems WL6 did not run for that minute for some reason.
All I have is the SQL table for 5 second quotes and the `1 minute SQL table.
they look normal in values and times they arrived at.

May be SQL server log file would have some info?

Looks like this event has passed.
Any recommendations to be ready next time this occurs.
In terms of diagnostics.
Thanks!
profile picture

Cone

#4
QUOTE:
I notice that WL (stream mode) starts running right after the top of the minute.
It seems that it sees the first update of SQL table after the top of the minute and then runs.
I don't know what you mean by "WL starts running". WL is an observer of the stream and accumulates what you feed it to build the x-minute bar.

My question about how what I assume to be Minibars (OHLC of the 5 second intervals) are timestamped is important. You MUST timestamp any interval with the end of the interval. e.g., Data for the first 5 seconds of the minute should be timestamped HH:mm:05.000.

Anyway, we're talking about the Static SQL Provider that's "skipping" a minute, right? We really need to get Eugene's attention for this one, but my intuition tells me that it's a timestamp issue. If the data aren't timestamped at the end of the interval, it's won't be correct in Wealth-Lab.
profile picture

haytac

#5
I saw this missing minute in streaming mode.
In the after hours run for the same session I see the missing 15:20 ET candlestick.

WL uses the 5 second quotes during session. It uses 1 minute OHLC after hours. Summarizing my understanding.
With the 15:20 streaming run I started seeing a run time error but WL kept painting the price pane in this and subsequent minutes..
How I saw the missing 15:20 candlestick.

I see what you mean with time stamp. For example unless the 16:00 ET 1 minute OHLC is time stamped at
0 seconds and 0 milliseconds WL skips this last minute each and every time.
So now I fix this up in code. This was another post and I followed your advice there.

Somehow WL accepts the other minute data with 0 or 1 second and 100 plus milliseconds in DateTime column.
WL also accepts 5 second data with non zero milliseconds time stamp in streaming mode.

I have other columns where I record the time SQL server recorded the data.
So I can make sure the 5 second SQL writes all have 0 milliseconds.
And make sure all 1 minute OHLC writes also have 9 seconds and 0 milliseconds.

This problem happened once in the past 12 plus months of this scheme (where I supply SQL data).
This is 250 days of runs with almost 400 minutes in each = 100,000 minutes

Another direction for an explanation is this:
WL does not run when I stop giving it 5 second SQL data.
In this case it did run and caused a run time error. And WL did not write the candlestick to the pane.
In subsequent minutes although the same run time error repeated OHLC was painted for later minutes on the price pane.
It could be that my code had a rare error that stopped the writing of price pane data.

I will let go of this one. Thanks a bunch for your inputs.
One last question is a pointer or two on how I can prepare for a recurrence in terms of better diagnostic data.
Thanks!!
profile picture

haytac

#6
One more thing comes to mind.
It seems that WL paints the candlestick independent of code in a strategy.
Then the strategy adds more picture elements such as sma lines.
It seems that WL not writing a minute's OHLC does not have to do with the code in the strategy.

Here is a question:
- Is the whole picture calculated and painted over and over again at each minute?
- or the image sits in memory and the last minute is added to this location in memory?
profile picture

Eugene

#7
QUOTE:
I see what you mean with time stamp. For example unless the 16:00 ET 1 minute OHLC is time stamped at
0 seconds and 0 milliseconds WL skips this last minute each and every time.

As far as I understand this data item belongs to the previous bar because WL timestamps data at end-of-bar.

QUOTE:
- Is the whole picture calculated and painted over and over again at each minute?

Yes.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).