Best way to use IPC between node and python?

[philraj]

Hi,

I'm trying to send data to a python script that controls motors on my Raspberry Pi, but I'm confused about the recommended way of doing this. Up until now I was using something like:

function runMotor (speed) {
  var shell = new PythonShell('./script.py');
  shell.send(speed);
  shell.end( function(err) {
      if (err) throw err;
    }
  );
}

But I was getting some latency between sending the command and the motor moving, which I think is because there's a heavy overhead with instantiating a new instance of PythonShell each time and importing the modules every time on the python side.

I tried moving the shell creation outside of the function, and not calling shell.end() immediately, but then it blocks, I assume because it only starts reading the input once you call shell.end(). You also can't instantiate the shell and call send() and end() multiple times, so what is the proper way to do this?

I saw you mention using var shell = PythonShell.run('./script.py') instead in this issue: #16 (comment), but when I try that it complains "Error: write after end".

Is there a way to keep the script 'open' so that I don't incur an overhead from calling the constructor over and over and re-importing the modules the python script needs over and over?

[extrabacon]

You would definitely face latency by spawning that many processes. The overhead is not PythonShell itself (it is just a thin wrapper on Node’s child_process), it’s starting new Python processes over and over.

You want a long-running instance that can communicate back and forth. When doing so, small messages are not received immediately. This is because the OS is buffering both stdin and stdout streams, which is the desired behaviour for most applications. For example, your Python app could emit huge amounts of JSON that can safely stream through your Node application with optimal performance. If you want to send messages and receive them immediately, you need to flush the buffers explicitly.

I originally designed PythonShell to stream content from Node to Python (and vice versa), so stdin/stdout was the best choice. For realtime communication though, you want IPC, which is much more complex than stdin/stdout (something with sockets maybe?). However, I’ve had many requests just like yours to make this work, so it seems like a popular approach.

Take a look here for a discussion on the subject: #18

Also take a look at this PR: #21

[philraj]

I will, thanks. Yes some kind of IPC mechanism would be exactly what I wanted. Thanks for explaining the inner workings. I had some false assumptions in mind.

In the end I found a way to accomplish what I'm doing with pure JavaScript so it's moot. But thanks all the same.