Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[discovery] RFC2217 Discovery Service #1511

Closed
bodiroga opened this issue Jun 3, 2020 · 28 comments · Fixed by #2519
Closed

[discovery] RFC2217 Discovery Service #1511

bodiroga opened this issue Jun 3, 2020 · 28 comments · Fixed by #2519

Comments

@bodiroga
Copy link

bodiroga commented Jun 3, 2020

After the awesome work done by @wborn in openhab/openhab-addons#7573 and now that many bindings support RFC2217 connections, I propose to add a RFC2217 Discovery Service for openhab-core, giving binding developers similar information to the one provided by org.openhab.core.config.discovery.usbserial.

As no standardized implementation exist for announcing RFC2217 ports, I have created a project called rfc2217-gateway that discovers some well known USB sticks and announces them using mDNS, with the _rfc2217._tpc.local service name. Here you have an example (Enocean USB stick) of the list of properties announced by the service:

{
  "MODEL_ID": "6001", 
  "MODEL": "EnOcean_USB_300_DB", 
  "MODEL_ENC": "EnOcean\\x20USB\\x20300\\x20DB", 
  "MODEL_DB": "FT232 Serial (UART) IC", 
  "VENDOR_ID": "0403", 
  "VENDOR": "Future Technology Devices International, Ltd",
  "VENDOR_ENC": "EnOcean\\x20GmbH", 
  "VENDOR_DB": "Future Technology Devices International, Ltd", 
  "SERIAL": "EnOcean_GmbH_EnOcean_USB_300_DB_FT3MXEB",
  "SERIAL_SHORT": "FT3MXEB"
}

Of course, this should be added to openHAB 3, as openHAB 2.5.x core is frozen.

Would something like this be of any interest for @openhab/core-maintainers? I could try to create the discovery package extending the mdns discovery package, but I may need some help 😄 Anyway, first I need to know if something like this could be accepted and merged.

Many thanks for your work and best regards,

Aitor

@wborn
Copy link
Member

wborn commented Jun 13, 2020

I can imagine support being added to openHAB for some popular RFC2217 port discovery methods. Did you have a look at what protocols are used by other implementations that allow for discovering RFC2217 ports? Is this mDNS implementation already used by another implementation? If there's already another nice and commonly used method it may make sense to use that instead creating a new one.

@bodiroga
Copy link
Author

Hi @wborn!

Many thanks for your answer and for your interest in the topic! 👍

Sadly, as far as I know, there is no announcement mechanism for RFC2217 ports, or at least I have not been able to find one. There was a small talk about the topic in the Eclipse Smarthome repository (eclipse-archived/smarthome#5560 (comment)), but the project mentioned by @gersilex was also started by me, so I think that it doesn't count. Python's pyserial library also has an example that does the same thing and it uses mDNS as the announcement protocol: https://pyserial.readthedocs.io/en/latest/examples.html#multi-port-tcp-ip-serial-bridge-rfc-2217

So... it seems that we are alone. But we can the first ones, can't we? 😉 As @kaikreuzer was also involved in the RFC2217 implementation discussion, it would be awesome if he could express his opinion here.

IMHO, this discovery mechanism can be very interesting for Dockerized environments, because we no longer depend on passing around /dev/tty* devices and we can also rely on TCP/IP.

Thank you for your support and best regards,

Aitor

@wborn
Copy link
Member

wborn commented Jun 22, 2020

Since everyone is already using ser2net it would be nice if it would also implement such a discovery mechanism out of the box. :-)

@bodiroga
Copy link
Author

@wborn, I'm not sure what you mean, but I don't know of any discovery service implemented by ser2net. Have you read anything about it? 😕

@wborn
Copy link
Member

wborn commented Sep 14, 2020

Do you think it makes sense to also add support for a discovery protocol like this in ser2net @cminyard?

@cminyard
Copy link

Are you talking about a telnet discovery protocol, or an RFC2217 discovery protocol? If you have telnet, discovering RFC2217 is easy, doing discovery is built in to the telnet protocol.

However, if you want to do discovery, it really only makes sense on the client side. If the server has it it will always announce it, the client side can choose to enable and use it. Why would you want to discover it on the server side (ser2net)?

The gensio library that ser2net sits on has client code, and it can do this. It does not currently have a way to discover telnet protocol, but that would be pretty easy to add to the client side if you assume that the first thing you receive from a telnet connection is a telnet negotiation.

If you aren't using the gensio library, it might be nice for you to use. It can do all different sorts of connections (serial, tcp, sctp, udp, pty, stdio, etc.) and provides a common interface for them all. It has upper layer protocols for doing things like reliable deliver or serial ports, encryption (ssl), certificate/password based authentication.

@wborn
Copy link
Member

wborn commented Sep 15, 2020

Thanks for your reply @cminyard!

This is about clients discovering all the serial ports provided by servers over telnet streams in the local network. Such a discovery protocol would simplify setting up connections on the client side as users would no longer have to configure details such as IPs and ports.

openHAB is home automation software written in Java. A use case for openHAB users would be that they run openHAB on one machine and setup serial devices on another machine. E.g. because openHAB runs on a server and the machine with the serial device is located at a better location for receiving radio signals. Currently users would run ser2net on the other machine and have to manually configure openHAB to use the hostname/port with a particular technology.

By implementing such a discovery protocol openHAB could discover the ports within the local network. When the discovery protocol contains details such as the USB model/vendor ID, openHAB would also know what particular technology is provided by the port. Based on such discovery results, openHAB users would then only have to approve the discovery results to be able to use the serial device without any hostname/port/technology configuration.

Because openHAB is written in Java we also use a Java RFC2217 client implementation (see TelnetSerialPort) so switching to a library like gensio won't be easy. ;-)

@cminyard
Copy link

cminyard commented Sep 15, 2020 via email

@wborn
Copy link
Member

wborn commented Sep 15, 2020

It would seem better to have some
sort of multicast that says "Here are the things available on my
system" and use that for discovery. Or is that what you are talking
about?

Yes using some multicast protocol for this such as mDNS is exactly what the idea is and I fully agree we don't want to do any port scanning and opening ports because of all the side effects.

And you need to be able to advertise things besides serial ports, I assume.

Note that ser2net can do things besides serial ports now.

Being able to discover more than just serial ports would be even better. I'm sure some of our contributors in the future will be able to utilize the additional (discovery) features of ser2net as well. :-)

@cminyard
Copy link

cminyard commented Sep 15, 2020 via email

@wborn
Copy link
Member

wborn commented Sep 15, 2020

openHAB already supports discovery using both mDNS and UPnP. Maybe @kaikreuzer (as jmdns/jupnp maintainer) is also able to provide some guidance of what might be a suitable protocol for discovering serial (RFC2217) and other network proxied devices nowadays?

@kaikreuzer
Copy link
Member

I think mDNS (or to be more precise: DNS-SD) is the best choice for announcing services on a local network. openHAB itself uses it to announce itself as well.
If ser2net would implement this and announce clients that have a telnet port available to connect to, that would be great - openHAB could then easily discover them.

@bodiroga
Copy link
Author

Thanks to all of you who are participating in this topic and especially to @cminyard for his appearance as the creator of ser2net. I would also choose to use the mDNS protocol to announce the availability of RFC2217 ports, using the _rfc2217._tpc.local service and sending something similar to what was indicated in my first message (MODEL_ID, VENDOR_ID, SERIAL,...) 👍

@cminyard
Copy link

cminyard commented Oct 8, 2020 via email

@wborn
Copy link
Member

wborn commented Oct 10, 2020

That sort of stuff can be added, I suppose, but what would a user do with it?

Based on the model number, vendor, etc. of a device, our code checks if it can add an automatic discovery result for a particular technology. The result will show up in an "Inbox" and then to use it users only need to approve it.

And just about all of it is irrelevant for non-USB/non-PCI serial ports. For those it would be completely empty.

That should be fine too. We can still use such results to populate a list of available serial ports when a user manually adds a device.

rfc2217 can be in some sort of capability field

Sounds good to me. As long as the information is available in some way we should be able to use it.

For instance, in a racked system, you could provide the rack number/shelf/slot and server name for a serial port. Now that would be useful.

Yes that would be useful too. We can use that information when setting a label for the discovery result so the device can be more easily identified by the user. I guess that info will be based on some details in a configuration file?

So can you explain why this is needed?

We already have a mechanism for discovering devices based on sysfs on the system openHAB itself is running on. It provides the information in this UsbSerialDeviceInformation class to discovery participants. It's for instance used to discover an EnOcean USB300 in this EnOceanUsbSerialDiscoveryParticipant class. So that kind of information would be really useful.

@cminyard
Copy link

cminyard commented Oct 17, 2020 via email

@wborn
Copy link
Member

wborn commented Oct 18, 2020

That sounds like very good progress @cminyard ! :-)

I had a look at how we currently obtain all this info using sysfs.

The process itself is actually pretty well documented in the SysfsUsbSerialScanner that we use for obtaining the information from the local system.

So I had a look at the sysfs files for a USB stick connected as ttyUSB0 on my Ubuntu 20.04 desktop.

Following the symlink you would end up at:

$ readlink -f /sys/class/tty/ttyUSB0
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/ttyUSB0/tty/ttyUSB0

After adding some more logging to the SysfsUsbSerialScanner I saw it was reading these details from the following files:

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/idVendor
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/idProduct
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/serial
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/manufacturer
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/product
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/bInterfaceNumber
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/interface

@cminyard
Copy link

cminyard commented Oct 18, 2020 via email

@wborn
Copy link
Member

wborn commented Oct 18, 2020

Ok, there are things in your original list that are not in the
sysfs; I will send what I can get. Any preferences as to txt names?

The original list was provided by @bodiroga. Perhaps he can tell how this data was obtained/generated? It looks like it is part of a data structure within that rfc2217-gateway implementation.

The sysfs details would be enough for discovery and we might be able to rework our existing sysfs discovery mechanism such that it becomes possible to use the same logic for discovering both local and remote ports. As long as the naming is clear and can be related to the sysfs values it should be fine.

I will add a "devicetype=serialusb" so other things can be added later
and so you can tell if it's that or some other type of serial port.

Sounds good to me!

@cminyard
Copy link

cminyard commented Oct 23, 2020 via email

@wborn
Copy link
Member

wborn commented Oct 25, 2020

Many thanks for your efforts @cminyard! The docs look good and with some trial and error I was even able to build both gensio and ser2net in a Debian 10 Docker container. I will give it a test to see if my build efforts were successful. :-)

It failed to compile gensio on Debian 10 when libopenipmi-dev was installed. Is that a known issue?

  CC       gensio_ll_ipmisol.lo
gensio_ll_ipmisol.c: In function 'sol_disable':
gensio_ll_ipmisol.c:1217:2: warning: implicit declaration of function 'ipmi_sol_force_close_wsend'; did you mean 'ipmi_sol_force_close'? [-Wimplicit-function-declaration]
  ipmi_sol_force_close_wsend(solll->sol, 0);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~
  ipmi_sol_force_close
gensio_ll_ipmisol.c:1218:13: error: 'ipmi_con_t' {aka 'struct ipmi_con_s'} has no member named 'disable'
  solll->ipmi->disable(solll->ipmi);
             ^~
make[2]: *** [Makefile:599: gensio_ll_ipmisol.lo] Error 1
make[2]: Leaving directory '/root/gensio/lib'
make[1]: *** [Makefile:449: all-recursive] Error 1
make[1]: Leaving directory '/root/gensio'
make: *** [Makefile:381: all] Error 2

After removing it the build did succeed. Maybe I need to install something more/else for that.

@cminyard
Copy link

cminyard commented Oct 25, 2020 via email

@bodiroga
Copy link
Author

Amazing work @cminyard, thank you very much!

The original list was provided by @bodiroga. Perhaps he can tell how this data was obtained/generated? It looks like it is part of a data structure within that rfc2217-gateway implementation.

All the information is collected by the pyudev module and the predefined values that you see in https://github.com/bodiroga/rfc2217-gateway/tree/master/src/gateway_devices are just to check which device type has been discovered, in case one device needs to read more specific information (the Home ID of the Z-Wave network, for example). So, in my case, the key is the UDEV service.

@wborn
Copy link
Member

wborn commented Oct 27, 2020

I did some testing with my own build and I was still able to make rfc2217 connections. I also had a look at the mdns configuration and it is nice that there is a lot that can be configured! But for some reason my build has issues to start the mdns @cminyard.

When I launch ser2net it logs:
Unable to start mdns: Out of memory

Do you know what might be causing this? It could also be due to that I run it in a Docker container or I am missing some kind of package. Does it need the Avahi daemon to be running? I tried that but it doesn't help.

@cminyard
Copy link

cminyard commented Oct 27, 2020 via email

@wborn
Copy link
Member

wborn commented Oct 27, 2020

hopefully that will give some more information.

I recompiled the code but unfortunately it didn't give any more info. But I think it was some kind of Docker related issue because after I setup a regular Debian 10 VM it does work as expected. :-)

This configuration worked nicely to setup a FTDI stick:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: mydevicename

After that I was able to query all the details using mDNS:

$ avahi-browse --ignore-local --resolve --terminate _iostream._tcp
+  wlan0 IPv4 my-device-name                                _iostream._tcp       local
=  wlan0 IPv4 my-device-name                                _iostream._tcp       local
   hostname = [debian.local]
   address = [192.168.1.226]
   port = [2222]
   txt = ["product=FT232R USB UART" "manufacturer=FTDI" "serial=XXXXXXXX" "idVendor=0403" "idProduct=6001" "interface=FT232R USB UART" "bInterfaceNumber=00" "devicetype=serialusb" "gensiostack=telnet(rfc2217),tcp" "provider=ser2net"]

So many thanks for helping us @cminyard! 👍

Now we only have to add some Java code for this and hope all the distros quickly update to this version when it is released. :-)

wborn added a commit to wborn/openhab-core that referenced this issue Oct 11, 2021
* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes openhab#1511

Signed-off-by: Wouter Born <[email protected]>
wborn added a commit to wborn/openhab-core that referenced this issue Oct 13, 2021
* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes openhab#1511

Signed-off-by: Wouter Born <[email protected]>
wborn added a commit to wborn/openhab-core that referenced this issue Oct 13, 2021
* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes openhab#1511

Signed-off-by: Wouter Born <[email protected]>
wborn added a commit to wborn/openhab-core that referenced this issue Nov 3, 2021
* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes openhab#1511

Signed-off-by: Wouter Born <[email protected]>
cweitkamp pushed a commit that referenced this issue Nov 12, 2021
* Add ser2net mDNS USB serial discovery

* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes #1511

Signed-off-by: Wouter Born <[email protected]>
@williamblair333
Copy link

williamblair333 commented Sep 19, 2022

hopefully that will give some more information.

I recompiled the code but unfortunately it didn't give any more info. But I think it was some kind of Docker related issue because after I setup a regular Debian 10 VM it does work as expected. :-)

This configuration worked nicely to setup a FTDI stick:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: mydevicename

After that I was able to query all the details using mDNS:

$ avahi-browse --ignore-local --resolve --terminate _iostream._tcp
+  wlan0 IPv4 my-device-name                                _iostream._tcp       local
=  wlan0 IPv4 my-device-name                                _iostream._tcp       local
   hostname = [debian.local]
   address = [192.168.1.226]
   port = [2222]
   txt = ["product=FT232R USB UART" "manufacturer=FTDI" "serial=XXXXXXXX" "idVendor=0403" "idProduct=6001" "interface=FT232R USB UART" "bInterfaceNumber=00" "devicetype=serialusb" "gensiostack=telnet(rfc2217),tcp" "provider=ser2net"]

So many thanks for helping us @cminyard! 👍

Now we only have to add some Java code for this and hope all the distros quickly update to this version when it is released. :-)

Did anyone find a workaround so that ser2net will run in a Docker container?

edit: I think i have it working after I went back and re-read this page and another page... I'll post a github repo of how I set it up after testing. edit again.. I think it's working now after udev is setup: https://github.com/williamblair333/ser2net

@cminyard
Copy link

cminyard commented Sep 20, 2022 via email

splatch pushed a commit to ConnectorIO/copybara-hab-core that referenced this issue Jul 12, 2023
* Add ser2net mDNS USB serial discovery

* Add support for using multiple UsbSerialDiscovery services
* Add Ser2NetUsbSerialDiscovery that can use mDNS to discover ser2net RFC2217 serial ports
* Use discovered USB ports in SerialConfigOptionProvider

mDNS discovery is supported in ser2net 4.3.0 and newer.
E.g. you can install a ser2net version that provides this using APT in Ubuntu 21.04 and Debian 11.

Example ser2net YAML configuration that allows a serial port to be discovered using mDNS discovery:

%YAML 1.1
---
connection: &con01
  accepter: telnet(rfc2217),tcp,2222
  connector: serialdev,/dev/ttyUSB0
  options:
    mdns: true
    mdns-sysattrs: true
    mdns-name: devicename

Closes openhab#1511

Signed-off-by: Wouter Born <[email protected]>
GitOrigin-RevId: 917e268
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants