Hi everyone,
I'm looking for a way to pause the C# script (wait for another program on the pc to update a file) before executing the next bar. I tried

CODE:
Please log in to see this code.

but since it's running inside a bar loop WL shows a blank chart. It's as if WL is so preoccupied with the sleeping to the point that it's frozen and can't display anything. Any suggestions how to pause without freezing WL?


https://docs.microsoft.com/en-us/dotnet/api/system.threading.thread.sleep?view=netframework-4.8

Hi,

Where is the WealthScript Strategy running in, a streaming Strategy window or Strategy Monitor?

What kind of file is being updated and why exactly it has to be waited for?

First, the right way to fix this CPU hogging behavior is by setting the execution priority of Wealth-Lab lower (Below Normal priority) as shown in the attachment. (The attachment employs Process Explorer, but you can use Task Manager instead.) Do not put Wealth-Lab to sleep with a timed wait.

But the more serious question is why is WL hogging so much CPU time in the first place? This needs to be investigated and fixed. The attachment shows my WL process is using 0.39% CPU utilization. Anything over 1.5% utilization during trading-day screaming is a problem. Is your WL strategy running some kind of continuous poling loop? Go into Task Manager and tell us what the actual CPU utilization of Wealth-Lab is on your system.

Now if you're optimizing strategy parameters with a WL optimizer, CPU utilization will be higher than 1.5%, perhaps 20%. Why would you be doing that during the trading day?

QUOTE:
Is your WL strategy running some kind of continuous poling loop?

Yes Mark, topic starter's strategy polls for file input from 3rd party app and of course Thread.Sleep will freeze WL. But setting the execution priority here would be putting the cart before the horse. Because pausing the WealthScript Strategy is the wrong direction we need to understand requirements before making suggestions.

QUOTE:
Because pausing the WealthScript Strategy is the wrong direction we need to understand requirements before making suggestions.

I totally agree. One thing that occurred to me later is what application is actually creating this file? Is it WL itself? If so, then messing with the WL execution priority won't work. We need more information.

And if WL is writing the file and if it's a huge file (We need to know how big it is.), then he needs to be using a StreamWriter class to do the writing. And even using the StreamWriter won't be "fully asynchronous" I/O by itself, but it may be just enough. We need more information.

And if WL is writing this file, perhaps a WL Performance Visualizer could be substituted such that WL doesn't need to write a huge data file in the first place for off-line analysis.

ok, let me clear a few things. The file is not the issue or the cause. Also, it's not a cpu issue. The cpu is just fine. The issue is Thread.Sleep. It freezes the app and prevents it from doing anything, including displaying the chart. So running the code gives a completely blank chart (nada, nothing). So in essence .NET Thread.Sleep just halts completely all app threads. In the old VB days there was a thing called DoEvents() for that.
So, what I'm after is a way for the process to yield to other threads (chart display) while it's waiting for something to happen (in this case, monitor for a file date/time change).

I experimented with things like Task await https://www.codingame.com/playgrounds/4240/your-ultimate-async-await-tutorial-in-c/async-ready-methods-in--net-framework but still getting the screen freeze...

Jake,

Both Mark and I know what you're talking about. Application.DoEvents() won't help your Thread.Sleep call. If I wasn't clear let me rephrase: what you're trying to do with Thread or Task will not succeed in a WealthScript Strategy. So we're back to my original question which is still unanswered:

What kind of file is being updated?
Why exactly it has to be waited for in a loop?

I'm trying to understand the real problem and propose a solution or workaround which won't involve a dead end like Thread.Sleep or Task. The more information about your Strategy design the better.

It's a simple, tiny text file that is being updated with one line data from another app on the pc. In essence it's a synchronous one-way communication method to pass data from one program to another on the pc.

It needs to wait since the update is done each bar and I'd like to use it for both realtime and back-testing scenarios. So, each bar WL will wait for an update (then read the one-line text from the file) before running that bar. And the way it waits is to monitor for new file's date/time with a Thread.sleep loop.

The WealthScript execution model doesn't lend itself well to concepts like event-driven (FileSystemMonitor), threads and tasks.

1. Please clarify why do you think that the file date/time should be monitored with a Thread.Sleep loop? Looks totally extraneous to me.
2. Why won't you simply compare the difference / hash code of file contents?
3. What's in the file? Like I said, the more details the better.

1. I'm not attached to Thread.sleep and if there is a better way than I'd be happy to a adopt. I want to:
a. at a new bar: WL updates a file (call it data.csv) with bar data (text line like: "1,2,3")
b. WL waits with Thread.sleep for a file action.csv to update (this is where the display problem happens, how I verify that it changes is not an issue),
c. the other app processes the data.csv from WL, then updates action.csv with a single text line (like: "1234"). That's the whole file, just 1 line of text.
d. When WL sees that the action.csv was changed it processes the action line from action.csv
e. WL loops to the next bar (back to step a.)

2. Whatever is needed to accomplish (1) is fine by me.

3. The file that WL waits for is just a single text line (like: "1234"). As shown in (1) the file is very simple and I think the focus on it is not the issue but rather the waiting mechanism. All it does is communicate some number back to WL for each bar (and only for that bar). data.csv is a file updated by WL and action.csv is a file updated by the other app.

So, in essence I'm trying to establish a two-way communication between WL and another app that exchange data each bar.

What if you launch the application passing the data file or its contents as an argument and then reading from stdout (standard output)?

https://stackoverflow.com/questions/4291912/process-start-how-to-get-the-output

Does it support such scenario (pretty standard)? If not you may consider asking its author to tweak it.

It's not that simple. The other program, aka, App, keeps track of previous bar data so it has to run alongside WL.

To simplify and focus the issue: Is there a way to pause/sleep/yield/doevents the script for a few seconds without freezing the chart?

The WealthScript Strategy execution model is not compatible with what you're trying it to do i.e. event-driven stuff, threading and tasks. It's not going to change. The chart by design has to freeze (which is purely cosmetic and is not an issue).

Still you haven't convinced me in the need of Thread.Sleep. Is the 3rd party app a long running process or something? Or are you trying to synchronize its output which is not timestamped?

Eugene.
I only used thread.step to wait for the data to come back from the other app, which could take a second or so, every bar step. And as detailed in the steps above, I don't want WL to execute further bars until it gets the action response because that action will dictate to WL how to proceed on that bar.

The main issue is the pause, not the file stuff. Either way, you answered my question in the last reply, thanks. I'd have to look for another mechanism to pause WL.

"Pause" may not be the right word. If you're willing to delay the Strategy processing until a future bar brings the updated action.csv then it may be pretty simple to accomplish:

WealthScript Techniques | Setups, Triggers, Delays, and Timeouts

The disk drive shouldn't even be involved in this problem; it will just slow you down.

What you want to do is "message passing" between the two localhost processes, one being WL and the other being a third-party application. This is an advanced OS operation, and I would read an advanced OS textbook about sockets and message passing before jumping in. It should cover all the issues like synchronous verses asynchronous I/O (i.e. blocking verses non-blocking calls), pipes (i.e. FIFO buffering), unassigned port# ranges, and preserving process state (TCP socket connection) or not (UDP datagram connection).

The next question is where to begin? I would find a good high-level message-passing code package as starting material to install. Do not approach this problem by making WinSock calls directly! That would take me a week to get it running (and I'm a computer engineer). Instead, employ the high-level calls in the code package (via its data type classes) to do all the complex work for you behind the scenes. Remember, work smart, not hard.

I'm not that familiar with Windows communications as I am with Unix communications, so I'm not in a good position to recommend a high-level message passing code package for Windows or the .NET framework. I did find this one on the web by Apache you might take a look at. https://qpid.apache.org/index.html

And here's a C# example employing it you can study. Remember, you're always going to be using "localhost:port#" for all your connections on your system, so imagine that when studying this example. https://qpid.apache.org/releases/qpid-cpp-1.39.0/messaging-api/book/ch01s03.html Your solution should be about the same size.

I googled ".net messaging framework" to get these links. There are other code packages you can check out too.

If you have questions, I would post them to a forum specific to the high-level code package you're employing for message passing.

---
One other thought. If you already have R installed on your system, there's a package called R.NET that will let you perform message passing between WL and R easily. This will let R receive data from WL, compute solutions, and pass them back to WL. But if you're not already using R, then this is a bad idea.

Just out of curiosity, if all this external application does is take one bar of data, perform a calculation, and pass the result back to WL, then why not simply do this calculation on WL in the first place? WL can interface with external numerical analysis libraries, stat libraries, and signal processing libraries. What more could you want?

We may be able to help code this bar-by-bar calculation in WL if we knew more about it.

Yes, I'm quite familer with tcp/socks and was able to create a web services to exchange data with the other app. But the wait for the response shot that idea down as well. So the exchange methods are not the problem. The problem, and I can emphasize it enough, is the wait/pause/delay (call it whatever). That wait for the other app to reply each bar is what freezes WL. It is not the file I/O, it is not winsock, it is not the cpu and it is not anything else. I hope this points the root problem.
The "other app" is Python and it does things I can't do in WL/.NET. It's a given.
The code below highlights the problem. Thread.Sleep loop will freeze the screen. If you compile/run, it will yield a blank screen and freeze the chart until you press <Esc>.

I am not familer with WL's inner plumbing. Maybe Eugene can suggest a command to insert in the loop that will "refresh the chart" or something along those lines before looping again?

CODE:
Please log in to see this code.

QUOTE:
Maybe Eugene can suggest a command to insert in the loop that will "refresh the chart" or something along those lines before looping again?

At the risk of repeating (I don't know how many times), your problem has been clear from the start. To emphasize it, you cannot let WL refresh the chart while the Strategy is being processed. Either it finishes or you press Escape to abort the running thread. In your own terms let's say it's a given.

Out of curiosity, what stuff the Python app does which cannot be done in .NET?

QUOTE:
I'm ... familiar with tcp/socks and was able to create a web services to exchange data with the other app. But the wait for the response shot that idea down as well.

If I'm reading this right, you are using a blocking TCP sockets call in WL which causes the WL Execute block to halt freezing the WL chart update.

My question is, "Why isn't Python replying with a timely answer so the WL Execute block can continue?" Understand, if you're using a blocking TCP call on the WL side, then Python must reply before the WL Execute block can proceed; otherwise, the WL thread will freeze up. You have a problem at the Python end; Python isn't replying with an answer or the blocking TCP connection at the WL end isn't receiving from Python. With a blocking call, if Python remains silent, the WL chart will freeze up.

There may be some things WL can't do, but there's nothing .NET can't do. Entire applications, like Python, are based upon the .NET framework. Keep the problem simple and leave Python out of it. Do what needs to be done exclusively with .NET and WL. Simple is beautiful; and it works faster and better too.

Also, Python is an interpreted language; whereas, C# and WL are JIT compiled. So once you do get this approach running, it's going to execute 10 times slower than if you avoided the Python interpreter altogether--and I think you know that. That's why we don't employ interpreted languages with simulation (e.g. backtesting) problems.

superticker, note above.