[WIP] Some needed Improvements

[toni-moreno]

Hi @paulstuart .

I've forked your project and I'm trying to do some needed improvements for me.( I'm new in goland world so I'm not sure how long will take).

These are the scope of a the forked influxsnmp version:

• Extended config file based on TOML (better than just ini file)
• Add extra tags per device
• Add SNMP v3 support.
• Upgrade influxdb client to v2.
• Add ability to send self monitoring metrics ( memory/gc/etc).
• Add configurable measurement names.
• Add arbitrary OID's as measurement sources ( not only switch ports).
• Improved message logging
• Ability to auto-detect config changes

I will send you some PR's ASAP. I will be happy if we can work together in these new Features.

[toni-moreno]

Hi to all.

I've finally finished the fist refactor and is ready to test here.

https://github.com/toni-moreno/influxsnmp/tree/first_big_refactor

The new features are.

• Extended config file based on TOML
• Add extra tags per device
• Add SNMP v3 support.
• Upgrade influxdb client to v2.
• Add ability to send self monitoring metrics ( memory/gc/etc).
• Add configurable measurement /Fields names.
• Add arbitrary OID's as measurement sources ( not only switch ports).
• ability to compute rates and differences on counter metrics
• Improved message logging global and per device

A sample config here:

https://github.com/toni-moreno/influxsnmp/blob/first_big_refactor/conf/sample.config.toml

Any comment will be appreciated.

[topranks]

Thanks. I've just set up influxsnmp today (sorry I am a newb to both this and Influx I have to say).

Great work on the refactor it seems. I take it with the custom fields per device I may be able to capture the interface description values as stored on our Cisco routers? I pulled the latest GIT to install but the version I got seemed to have the older style sample conf file.

Can I just use the newer conf format? Or do I need to get a different version of the code? Apologies for my ignorance if you could point me in the right direction I'll give it a shot ;)

[paulstuart]

Wow, that's a big list. I haven't been active in the project because I
wasn't able to use it myself, this has fortunately changed.
I'll review and reply in a day or two.

On Sun, Apr 17, 2016 at 3:09 PM, Toni Moreno [email protected]
wrote:

Hi to all.

I've finally finished the fist refactor and is ready to test here.

https://github.com/toni-moreno/influxsnmp/tree/first_big_refactor

The new features are.

• Extended config file based on TOML
• Add extra tags per device
• Add SNMP v3 support.
• Upgrade influxdb client to v2.
• Add ability to send self monitoring metrics ( memory/gc/etc).
• Add configurable measurement /Fields names.
• Add arbitrary OID's as measurement sources ( not only switch ports).
• Improved message logging global and per device

A sample config here:

https://github.com/toni-moreno/influxsnmp/blob/first_big_refactor/conf/sample.config.toml

Any comment will be appreciated.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#16 (comment)

[toni-moreno]

Hi @topranks you should clone directly from the branch

git clone [email protected]:toni-moreno/influxsnmp.git --branch first_big_refactor

I don't know if you could use http url format, can you try?

git clone https://github.com/toni-moreno/influxsnmp.git --branch first_big_refactor

@paulstuart have you seen the new version?

[paulstuart]

Hi Toni,

I've looked over the code and there's a lot of changes in there.

The original project had started off as a simple poller and then had more
functionality glued into it but was then neglected as I wasn't able to use
it. There were some "design decisions" that were quick and dirty but upon
returning to them I've seen how they could be done differently. I got a
chance to return to this and had applied some lessons learned (I hope) in a
test project that I'm going to be applying now.

There's some functionality added that I'm not sure about in snmpmetrics.go,
specifically in "cooking" the data, as influx has derivatives and
continuous queries that will do that. Because of the significance of the
changes I'm going to be pulling in bits in a manner I can wrap my head
around. I hope to have the changes complete within the next day or so.

-Paul

On Wed, Apr 20, 2016 at 7:19 AM, Toni Moreno [email protected]
wrote:

Hi @topranks https://github.com/topranks you should clone directly from
the branch

git clone [email protected]:toni-moreno/influxsnmp.git --branch first_big_refactor

I don't know if you could use http url format, can you try?

git clone https://github.com/toni-moreno/influxsnmp.git --branch first_big_refactor

@paulstuart https://github.com/paulstuart have you seen the new version?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#16 (comment)

[toni-moreno]

Hi @paulstuart , cooked data is for me the most important new feature, I've coded

a) Most users who are using other tools to gather metrics from snmp devices and are used to work with "datasource" type definitios to get cooked data, by example

• rrdtools
• cacti
• collectd

b) , there is no need to send raw data to backend db if you could send "ready to draw" data, IMHO most of snmp metrics are counters and if you send all this data as "raw data" , you are adding unneeded load to the backend .

I use grafana with dashboards designed to compute online consolidations / agregations and other filters too much load when added too to derivative on each serie.( working with some millons of metrics)

c) there are known issues with derivatives and influxdb, so is better send cooked data.

influxdata/influxdb#6118
influxdata/influxdb#4679
influxdata/influxdb#4675
influxdata/influxdb#3247

d) You can also send raw data if you define Datasrctype as INTEGER instead of COUNTER64

It is true that there is need to define all posible datasourceTypes, ( righ now INTEGER and GAUGE has the same behavior) and needs to define how absolute datasource will work ( if really needed). But we can work in the future to close all these types.

I hope you can maintain all these new features. And perhaps begin to work in a "second big refactor" ( with a lot of new features !!)

Thanks a lot @paulstuart -

[topranks]

Thanks @toni-moreno for the tip. I'll have a look at that over the weekend.

Regarding the 'cooked' data I can sort of see both sides. But I've already come across an issue where requiring InfluxDB to perform the derivative() function has caused me issues. For instance I wanted a 'percentage discards' graph for switch ports (using Grafana,) but it won't let me run a MATH operation on two fields if I try to add the derivitive function to them (as per the issue #6118 above). So having just the delta recorded in the DB would allow me to overcome this limitation.

Thanks both for all the hard work!

[paulstuart]

Ok, I've finally merged in my own refactor and have updated the readme with noted functionality. A key architectural change is to move actual SNMP processing to a separate library that is backend agnostic.

I have a couple philosophical differences with the approach taken in your big refactor, and how the core issues should be addressed.

The first is the handling of OID name lookup. I fall into the camp of "use the symbolic name and let the computer look it up for me". For a smaller number of OIDs, manually establishing that info in the config is trivial, but I'm dealing with multiple tables with lots of entries that I'd rather not bother having to lookup up and add to the config.

Then there is the issue of cooking the data. I believe that if one is using influxdb, than the "proper approach" is to apply those data transformations as continuous queries. It seems like they've finally got that sorted out. That said, I added the ability to calculate data on the fly in the library if sending data to a different backend.

Let me know if this works for you are you are missing any functionality in the changes I've applied.

[paulstuart]

In the interest of cleanliness, I'm going to close this issue. The only issues not addressed here:

• fix config file format (will do but secondary to core functionality)
• reloading config file (nice to have, but dynamic updates of polling time and available ports deemed as more important for core operations)
• memory/gc data are more of a debugging realm, and the data is exposed via the exposed net/http/pprof interface.

@toni-moreno, I appreciate your enthusiastic contributions - I know that you have a somewhat different take on how to get the job done but I hope I've addressed the key points of how that can happen. Please let me know if this addresses your needs, and if not, how we can make that happen.

[toni-moreno]

hi @paulstuart , sorry by the late response .

I like a lot your great work, but I would like to be able to use a snmp agent with more configurable options. ( measurement names, field names , device names, device tags, ) .Also I will need cooked data (counter to differences and rates) .

In the future I would like to have a web frontend as an easy way to configure the snmp agent and easy way to import/export device/metric templates like cacti does (http://docs.cacti.net/templates).

I have forked a new project from the influxsnmp "big refactor" a did some weeks ago.(https://github.com/toni-moreno/influxsnmp/tree/first_big_refactor)

in an attempt to achieve these objectives and this is a very early result.( the front-end is not working yet but you can see a preview here)

https://github.com/toni-moreno/snmpcollector

Thank you again.