Cross-platform plugin on TCP

[ndelta0]

This version uses no external libraries. It's implemented with TCP sockets for each of the OSes (Windows and Unix). The server listens on 0.0.0.0:45454. The transmission format is as follows:

1. Send 4 byte (32 bit) integer (Int32) with the size of the data that will follow.
2. Send the telemetry data. The format is unchanged and is compatible with previous releases and clients.

On my computer (i7-4790k) there were no perfomance drops in the game with the telemetry being sent every simulation frame. From what I saw in IO Ninja, the bandwidth that it requires is applicable only for local network data transfer and fast networks (I think it was about 1.5-2 MB/s).

Known Issues:

• If the socket stops receiving data (blocking method call or something) the game can freeze, until the receiving client resumes.

Compatibility:
The project is set to build on Linux x64, Windows x64 and Windows x86. I only tested Windows x64 and Windows x86. It's not confirmed nor said to work on MacOS, but it's possible to work with minor or no tweaks.

[RenCloud]

I will probably make some tests today/tomorrow to check the performance of the implementation.

On first view I had again some problems with TCP, but I think when the performance is fine it allows some feature to build much faster than with shared memory.

In addition, I think it makes the development of other clients simpler.

That means a lot of plus points for TCP if the performance is right.

[ndelta0]

I've also considered using UDP instead of TCP, but it would be too unreliable and you could lose important events

[RenCloud]

Performance seems to be ok. Also, I have some ideas for better performance, so that should not be such a problem.

2 other things are more of a problem:

• I can not change the application when this SDK is loaded. ALT + Tab did not minimize the game.
• when the SDK is unloaded, the game crashes.

[ndelta0]

I'll look into that

[ndelta0]

Fixed the issue with crash on sdk unload. I couldn't reproduce the Alt + Tab issue, it might be related to how the game is made since it doesn't really like to be alt tabbed from. If anyone reproduces it send the game log here.

[alphavector]

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

[ndelta0]

I was able to build a plugin for mac os with minor edits in the code. The game itself runs, but I don't really understand where I should look at the telemetry readings. When I access 0.0.0.0:45454 I get ERR_INVALID_HTTP_RESPONSE, in logs <WARNING> Error sending: 32. @ndelta0, I will send pr to your branch if you are still interested in developing it.

Well, the listening on 0.0.0.0:45454 means that it listens on every interface, i.e. you can access it on localhost:45454 and by using your local device IP from other device.

Regarding ERR_INVALID_HTTP_RESPONSE, it was not meant to be read by a browser (i.e. not JSON or any other plain text data). It's a raw stream of bytes exactly as you would read it from the memory mapped file. You need to open a raw/stream TCP socket to be able to read the data. Tell me what language you're using and I'll send you a sample code.

[alphavector]

If you don't mind, send a sample, in Python, for example

[ndelta0]

If you don't mind, send a sample, in Python, for example

The code for receiving data in Python would look something like this:

import socket # import the socket library


def main():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: # create the socket object with the Internet address family (IPv4) and stream socket type
        s.connect(("127.0.0.1", 45454)) # connect to the plugin listener
        while True:
            data = s.recv(1024) # receive the bytes
            print(data) # print the bytes


if __name__ == '__main__':
    main()

I was able to build a plugin for mac os with minor edits in the code. [...] I will send pr to your branch if you are still interested in developing it.

I'd love to, I don't have any Apple device myself so it would be really useful for me

[ndelta0]

NOTE: Marked the PR as draft since it's not 100% finished and there's still room for improvement

[alphavector]

Yes, we almost synchronously managed to write the code to get the data :D
I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

[ndelta0]

I get a sequence of bytes, but how to deserialize it? I understand correctly that this sequence of bytes is a structure scsTelemetryMap_t?

Exactly, except that every "message" is prefixed with a 4 byte (Int32) value specifying the length of the following data. It can be ignored if you know the data size and have a buffer of the appropriate size (which is required to be pre-allocated before receiving the data in languages like C/C++/C#/etc, I don't this is required in Python). The actual data is exactly as in the telemetry map.

[ndelta0]

Also, since this is a draft pull request, anyone should be able to commit although I don't really know how that works.

[N95JPL]

Is there any kind of "estimated date" for when this will drop?
I am working on a new application which will tie in to the SDK which will make it easier for people to use in their programs/Arduino Connections/WebServers etc.
Currently writing it in C# / ASP.NET with WinForms however, if this is going to be in the near future I will double-back and start working on a leaner, meaner React project for it instead...

Thanks, JPL!

[ndelta0]

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

[N95JPL]

Is there any kind of "estimated date" for when this will drop?

Technically it's ready, you can already use it when you download the artifact on my fork or build it yourself (keep in mind that artifacts are currently x64 only).

I had actually downloaded the src and built with CMake (Wins x64) and then built the Solution and it appears to be working, I didn't notice the artifact!

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

[ndelta0]

I connected via a console and received data (albeit a bunch of scrambled chars/symbols)... i will hook up a JS script later and get some bytes!

Actually it's supposed to be what you received, these bytes are actually exactly what was used before, with addition of an Int32 (4 bytes) at the beggining, which indicate the size of the following payload. Current implementations that receive and parse the data can be adapted by basically receiving the data from a TCP socket instead of a Memory Mapped File and skipping the first 4 bytes when receiving the data.

EDIT: Unlike ETCARS which sends the data in a JSON format, this plugin basically sends the memory that's assigned to the telemetry map type

[RenCloud]

I finally managed to look at the changes in more detail. I'm sorry for delaying this pr so long, but it's a big change, so I need a bit of time which I did not manage to put into this project lately. Beside some smaller support, this big change was sadly too much.

Thanks for that much afford. Also fixing old mistakes and formatting mistakes.

Most parts look good. Here and there I will change code formatting, but that's more or less personal preferences. I will check for a linting GitHub action in the future.

I currently prefer the current event handling implemented with rev 11, but that was later introduced than the start of this pr. ... Maybe neither the new nor the old way will stay, when subscriber pattern is used? At least, I think it would be possible to broadcast the data direct at the gameplay event. Then it would be an event by itself and no extra flag/switch/state/... needed. But that's just an idea at the moment.

Some minor things I need to test, because I may have some questions on them. There are also some small changes I may do. Possibly I can get rid of some commentaries and old to-dos before creating the new dev state out of it.

Beside the other points, I think

Investigate freezing the game when receiving socket is blocking

is one of the important to fix, before merging.

For memory leaks, it will probably code check up and observing application memory while doing a longer test. Perhaps I find some tool that can help here.

Additional to fixing mentioned things, I'm currently thinking if it is useful to add the subscriber pattern in this pr or later with an additional pr same with string and trailer optimizations.

[N95JPL]

ETS2-TCP-Telemetry-Byte-Map.xlsx

This is a Excel document I made to quickly reference where each Value can be found.
Saves having to count and use my calculator all the time!

It's here for anybody to use if they wish too!

You'll notice "End Byte" has 2 columns, this is because some languages want the byte after the range you want (non-inclusive), whereas others are happy with inclusive ranges!

Enjoy,
Thanks JPL!

[RenCloud]

I felt free to directly pushed into your branch, I hope that's fine for you.

Most/all are format changes or changes on comments. And remove the logic from events bits, like in the current main branch.

I tested it a bit with the above python script and displayed in game speed. Seems to still work after my changes. Also, logging seems to work wonderful. The Excel sheet was already helpful for that. Thanks JPL. May I should include a Markdown table with that information.

Performance seems to be ok/good already (windows 10). However, I'm exited to work on the optimizations for strings, trailer and the pattern.

--
Currently, I think getting this stage into dev-branch and after that working on the next steps (creating client lib, subscriber pattern, optimizations on code and transmitted object) is the way to go here to bring this big change to a "production ready" state.

I will check this on Linux and go another time through the code if I overlooked something. If I have then no questions/changes, and you have no changes, I would like to merge it and work than one the next steps

[alphavector]

I would like to suggest a few additional changes:

• I propose to allow you to specify an arbitrary port via environment variables and not be tied strictly to port 45454. This is a small change, I will make it soon.
  The problem I encountered:
• any connection to the server (i.e. the game) blocks the game, for some reason even the asynchronous client in python does not solve the problem. I have to open the connection, quickly read the data and close the connection and reopen it again and so on. This works and has little effect on game performance, but seems like a crutch, ideally I'd like to keep the connection open all the time. I don't know if this is a problem on Windows. Perhaps the problem is due to the fact that in osx you can't specify SOCK_NONBLOCK in the accept function

Further ideas:

• non-blocking sockets on the plugin side
• More documentation, maybe go to https://readthedocs.org/
• I support allowing the client to subscribe to certain types of events

In spite of all the above, I've tested it working in OSX (and even on an Apple Mac M1) -- it works, works great.

[ndelta0]

• I propose to allow you to specify an arbitrary port via environment variables and not be tied strictly to port 45454.

That's a great idea, I've thought about that but never got around to implement it

• any connection to the server (i.e. the game) blocks the game [...] Perhaps the problem is due to the fact that in osx you can't specify SOCK_NONBLOCK in the accept function

I'm aware of this issue, it's just hard to do any asynchronous socket communications on any system, especially in C/C++. I haven't encountered this issue on Windows and Linux (with accepting), but I've encountered a problem with the game hanging when the client does not receive data and fills the buffer.

EDIT: I don't have any way of running ETS2/ATS on a MacOS system (unless you have some way to run MacOS in a virtual machine, DM me then), so any help with MacOS sockets is very appreciated.

[N95JPL]

@alphavector I have sort of worked around the game stutters because of the connection system by using socket.pause() // socket.resume() whilst I'm parsing the last buffer stream.
Admittedly I'm using Node.JS wrapped in Electron with Async functions, even though it's only paused for a slight fraction of a second it stops the client program getting bogged down trying to constantly parse the data when it's being fired at continuously.
I'm hoping to speed things up again and not require the pause but for now it doesn't seem to affect anything.
Also, make sure you skip any data <1000 bytes, I've found this helps.
I've noticed a load of 0 packets getting sent? This might throw it off too!
Hope that helps!
JPL

Edit: Disclaimer, I've not focused on events yet, but everything else works as normal! Truck/Game/Job etc
If I notice issues with events with my current code structure I'll update the thread!

[RenCloud]

The ideas are great. Will come on the list. Some of them are already on that list. The functions will be added when this pr is in dev. At least that is my current plan.

That there is still some blocking is not so great. I also can not test OSX. However, I found this code

int srv = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
int flags = fcntl(srv, F_GETFL);
fcntl(srv, F_SETFL, flags | O_NONBLOCK); // setting the socket nonblocking

This sets the non-blocking with another syscall. Maybe this is possible? Could you try this out?

Tested Code for Windows and non-blocking... added a sleep in python and non-blocking, but totally behind the live values for sleeps bigger than maybe 100ms or so. For 1 sec, the data doesn't seem to update at all. I'm not sure if I did here every thing correct. Need to check that.

set the socket to non-blocking

    u_long iMode = 1;
    int iResult = ioctlsocket(srv, FIONBIO, &iMode);
    if (iResult != NO_ERROR) {
        printf("ioctlsocket failed with error: %ld\n", iResult);
    }

and the error WSAEWOULDBLOCK needs to be handled after accept like for Linux and while sending it needs to be ignored as error.

Just for curiosity, how big was the zeroQ implementation at the end? I need to know how much salt this non-blocking and subscribing will create, because I said to not use an external library.

[alphavector]

@RenCloud, thanks for the idea, I'll try it on the mac. Sorry for being away for a while. :D I had the idea to try to implement a GRPC server, so far I managed to convert the telemetry structure into a proto file and write an asynchronous grpc server. In the near future I plan to finish it. And we will be able to evaluate all advantages and disadvantages of tcp/zeromq/grpc and choose the best way, if nobody minds.

Regarding the implementation of the possibility to specify the user port: there are problems here, if we specify the environment variable in the terminal, it is available only during the session (while the terminal is open). If you specify in ~/.bashrc ~/.zshrc, then again, this variable is only available for the terminal, the game does not see it. I think it would work if you run the game through the terminal, something like SCS_TELEMETRY_PORT=123 zsh ~/path/to/game/executable, but do people often run the game this way? :D
I see 3 ways here:

1. the game itself has a configuration file, if we have access to it from the plugin, we can ask the user to specify the port there
2. Have your own configuration file
3. In steam we can specify additional launch options, is it possible to catch this.

[N95JPL]

@alphavector I use the Terminal for a lot of things but running games isn't one of them 🤣
I can be 99% certain 99% of people will just click "Play Game" in Steam or use the desktop shortcut!
So, unless you provided a shortcut for each OS which in turn launches the Game (IIRC it's a URL type-based system for Steam Shortcuts?) and sets a parameter that a user can change if they wish?
Most people will default to the "standard" port, more advanced users might want different ones but then they will know how to simply edit the shortcut?

--EDIT--
Further, I've built a JS Application which parses all the stream data into JSON and publishes it either via a WebSocket or via a API-Based system (can enable/disable to your liking), I'll upload a demo soon!

JPL

[alphavector]

That's what I meant, it was a rhetorical question :) 99% don't run games through the terminal

[RenCloud]

Sorry for being away for a while. :D I had the idea to try to implement a GRPC server, so far I managed to convert the telemetry structure into a proto file and write an asynchronous grpc server. In the near future I plan to finish it. And we will be able to evaluate all advantages and disadvantages of tcp/zeromq/grpc and choose the best way, if nobody minds.

We can do this. I'm just a little dissatisfied that ndelta0 first created a version with zeromq, then at my request another one without library and now maybe it will be one with library 😅.

But when the end result is better for the user and makes the development so much simpler, I guess I have no real argue against using one header file... 😄

[blitzcaster]

Hi, I'm also interested in Protocol Buffers format. Even without GRPC and using the current memory mapped file it would help a lot. Since protobuf parsers are already written in many language and are production ready codes. @alphavector any chance to open source your proto files? It would help implementing at least for the memory mapped structure.

[alphavector]

Hi, I'm also interested in Protocol Buffers format. Even without GRPC and using the current memory mapped file it would help a lot. Since protobuf parsers are already written in many language and are production ready codes. @alphavector any chance to open source your proto files? It would help implementing at least for the memory mapped structure.

I promise to finish it soon
For now, I'm sharing the proto file https://gist.github.com/alphavector/ea6b427812db418f12d983a2d9e01dc0

[blitzcaster]

I promise to finish it soon For now, I'm sharing the proto file https://gist.github.com/alphavector/ea6b427812db418f12d983a2d9e01dc0

Thank you for answering and sharing the proto files, it would help me quite a lot. Also there's no need for promises, it's an open sourced project and you should finish it at your own pace.

[AltriusRS]

Hi, just curious if there has been any progress on this PR since the last comment.

I am writing an application based on Electron however versions of electron higher than 20.3.8 have issues with memory mapped files being loaded due to some security changes, so I am stuck using an older version of Electron.

If this PR is almost ready to use id be happy to trial it for a few weeks across my application as a development rollout and report back any difficulties I have in getting it set up?

Massive appreciation for you all who have worked on this PR so far, it is a life saver for people trying to use this with JS/TS projects, and anyone trying to integrate this project into a CI/CD pipeline.

[ndelta0]

@AltriusRS as You said it, the PR is basically ready, with exception of few quirks and minro bugs. One big downside is that it's for SDK version 1.17 (ETS2: 1.44, ATS: 1.44), but it will work (just won't have the new available data). Since it's nor merged into the main repo, you need to compile it yourself and redistribute it. If you need, I can provide you with help on how to do that. In that case contact me through any means listed here: https://k4rasudev.me/contact.

[AltriusRS]

sounds good, Ill be busy for a couple more weeks with my current work, but I shall get back to you once I have gotten it working to report my findings and difficulties.

[TheBreekoMan]

Hi there, I'm currently working on a StreamDeck plugin which get telemetry data from your plugin and change display of icons keys according to parsed data. The main limitation is about reading SharedMeMoryFile which is not possible as Streamdeck plugin is running within a closed process environment AFAIK. BTW, I succeed to wrap data from SHM and expose it with a custom python REST server. So the addition of network streaming stack could be really great. Some remarks :

• As the data is going in single way with a quite high frequencyy, maybe the use of UDP socket instead of TCP should be better and could avoid any blocking socket issues (that's what is usually choosen for telemetry system)
• The SHM use remains useful in another scenarios so it should be kept.
• The socket port could be defined as environment variable

Anyway you all made a great job and would like to thank you for that and your effforts.