Modbus TCP Server - orphaned connections and possible 'Idle Timeout'?

[JimWiesler]

I have a couple redundant "edge devices" with a containerized Pymodbus AsyncIO server in a large manufacturing plant. The servers are essentially 'data banks' to facilitate communication between 7 isolated Schneider/Foxboro DCSs - the Foxboro system can only act as Modbus TCP Clients. Unfortunately, these clients are not well-behaved, and are frequently resetting the connections ([RST] packet) and starting new connections on different client ports. Additionally, in a number of cases, the clients will fail to properly close the old connection (fail to send either an [RST] or [FIN] packet). These orphaned connections linger in the Pymodbus Async Server, and after a few weeks will crash the container, which essentially runs out of memory.

My question (before I dig too deeply into AsyncIO Transports and Protocols!) - is there a way to set an "idle timeout" on a connection from a client to the Pymodbus Server? If not, any advice on how I can implement this would be greatly appreciated.

My current containers are running Python 3.9.7 and Pymodbus 2.5.2. And by the way - the Pymodbus server itself appears to be exceptionally robust besides this orphaned connection memory issue! I ran it for 6 months in a lab setting with different clients and very high data transfer rates without a single issue.

Many thanks,
Jim

PS - yes, we are working with the client vendor to improve their firmware, but we aren't holding our breath!

[janiversen]

You are a brave man, using asyncio with v2.5.2. I tried it a couple of times for homeassistant, but it is flawed of course it depends on your usage.

The current version v3.5.x is lot more stable, but it requires programming changes in your app.

The never asyncio uses keepalive on the socket, to check if the other end is still alive...but as far as I remember the old ones did not. Apart from the that, the easiest solution is in your app.

The app sends request at intervals if they are not responded to the app receives an error back, and can thus close the connection properly and open a new one....it is quite an easy implementation without side effects (I did it in homeassistant for the same reason long time ago).

It is a lot more problematic if the devices (servers) change their port, because then you need to somehow guess what they will be using next. The connect() will help with that, it will return an error if it cannot connect.

Hope the above helps otherwise please ask.

[JimWiesler]

Thanks for the quick reply! A quick correction - I'm using Pymodbus as a server not as a client. Do servers send 'keepalive' messages? Is there anyway for a server to know that the client-initiated connection is no longer active?

[janiversen]

A server do not send keep alive messages that is solely the client.

In general there are no way for a server to know if the client is still alive, however you could easily amend the datastore to control that it receives a request every X seconds from the client and if not close the connection. It's basically just inherit the base datastore and extend with your functionality.

[JimWiesler]

That looks promising - does the datastore know which client IP address and port receive a request? Is there a specific attribute of the server which holds a list of the active connections?

[janiversen]

No the datastore only knows about registers but it's python so you can reference other parts.

I do not remember how it was in 2.5.2, in 3.5.2 the transport object hold a list.