Skip to content
This repository has been archived by the owner on Nov 8, 2022. It is now read-only.

Designing glossary of terms #1352

Merged
merged 1 commit into from
Nov 18, 2016
Merged

Designing glossary of terms #1352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,16 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# **snap** <sup><sub>_the open telemetry framework_</sub></sup>
# **The Snap Telemetry Framework** [![Build Status](https://travis-ci.org/intelsdi-x/snap.svg?branch=master)](https://travis-ci.org/intelsdi-x/snap) [![Join the chat on Slack](https://intelsdi-x.herokuapp.com/badge.svg)](https://intelsdi-x.herokuapp.com/) [![Go Report Card](https://goreportcard.com/badge/intelsdi-x/snap)](https://goreportcard.com/report/intelsdi-x/snap)

[![Join the chat on Slack](https://intelsdi-x.herokuapp.com/badge.svg)](https://intelsdi-x.herokuapp.com/)
[![Build Status](https://travis-ci.org/intelsdi-x/snap.svg?branch=master)](https://travis-ci.org/intelsdi-x/snap)
[![Go Report Card](https://goreportcard.com/badge/intelsdi-x/snap)](https://goreportcard.com/report/intelsdi-x/snap)
<img src="https://cloud.githubusercontent.com/assets/1744971/20331694/e07e9148-ab5b-11e6-856a-e4e956540077.png" width="70%">

<img src="https://cloud.githubusercontent.com/assets/1744971/16677455/f1d4e9de-448a-11e6-9afb-c31dcc7e3274.png" width="50%">
**Snap** is an open telemetry framework designed to simplify the collection, processing and publishing of system data through a single API. The goals of this project are to:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I liked the feel of the previous README header. It was very simple and clean. Maybe try putting these words under the Snappy picture and see how that looks.


* Empower systems to expose a consistent set of telemetry data
* Simplify telemetry ingestion across ubiquitous storage systems
* Allow flexible processing of telemetry data on agent (e.g. filtering and decoration)
* Provide powerful clustered control of telemetry workflows across small or large clusters

----

Expand Down Expand Up @@ -51,33 +54,32 @@ limitations under the License.

## Overview

![workflow-collect-process-publish](https://cloud.githubusercontent.com/assets/1744971/14644683/be49a6b6-0607-11e6-8621-14f7b54e2192.png)
**The Snap Telemetry Framework** is a project made up of multiple parts:

**Snap** is an open telemetry framework designed to simplify the collection, processing and publishing of system data through a single API. The goals of this project are to:
* A hardened, extensively tested daemon, `snapteld`, and CLI, `snaptel` (in this repo)
* A growing number of maturing `plugins` (found in the [Plugin Catalog](#plugin-catalog))
* Lots of example `tasks` to gather and publish metrics (found in the [Examples folder](examples/))

* Empower systems to expose a consistent set of telemetry data
* Simplify telemetry ingestion across ubiquitous storage systems
* Improve the deployment model, packaging and flexibility for collecting telemetry
* Allow flexible processing of telemetry data on agent (e.g. filtering and decoration)
* Provide powerful clustered control of telemetry workflows across small or large clusters
These and other terminology are explained in the [glossary](docs/GLOSSARY.md).

![workflow-collect-process-publish](https://cloud.githubusercontent.com/assets/1744971/14644683/be49a6b6-0607-11e6-8621-14f7b54e2192.png)

The key features of Snap are:

* **Plugin Architecture**: Snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow Snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The **open plugin model** allows for loading built-in, community, or proprietary plugins into Snap.
* **Collectors** - Collectors consume telemetry data. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#author-a-plugin)). Telemetry data is organized into a dynamically generated catalog of available data points.
* **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
* **Collectors** - Collectors consume telemetry data. Collectors are plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#author-a-plugin)). Telemetry data is organized into a dynamically generated catalog of available data points.
* **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing systems. Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
* **Publishers** - Store telemetry into a wide array of systems. Snap decouples the collection of telemetry from the implementation of where to send it. Snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows Snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.


* **Dynamic Updates**: Snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get /intel/psutil/load/load1/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.
* **Dynamic Updates**: Snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get `/intel/psutil/load/load1/v1`). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.

* **Snap tribe**: Snap is designed for ease of administration. With Snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all Snap nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks and plugins from nodes within a given tribe. It is cluster configuration management made simple.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should Tribe be capitalized in this header and paragraph.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope.


Some additionally important notes about how Snap works:

* Multiple management modules including: [CLI](docs/SNAPCTL.md) (snapctl) and [REST API](docs/REST_API.md) (each of which can be turned on or off)
* Secure validation occurs via plugin signing, SSL encryption for APIs and payload encryption for communication between components
* CLI control from Linux or OS X
* CLI control from Linux or MacOS

**Snap** is not intended to:

Expand Down Expand Up @@ -229,11 +231,11 @@ NAMESPACE VERSIONS
...
```

NOTE: Plugin bundles are available for convenience in the Snap [GitHub release page](https://github.com/intelsdi-x/snap/releases), for the latest up to date version use the release/download in the [plugin catalog](#plugin-catalog).
NOTE: Plugin bundles are available for convenience in the Snap [GitHub release page](https://github.com/intelsdi-x/snap/releases), for the latest up-to-date version use the release/download in the [plugin catalog](#plugin-catalog).

### Running Tasks

To collect data, you need to create a [task](docs/TASKS.md) by loading a `Task Manifest`. The manifest contains a specification for what interval a set of metrics are gathered, how the data is transformed, and where the information is published. For more information see [task](docs/TASKS.md) documentation.
To collect data, you need to create a task by loading a `Task Manifest`. The Task Manifest contains a specification for what interval a set of metrics are gathered, how the data is transformed, and where the information is published. For more information see [task](docs/TASKS.md) documentation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a word is referred to for the first time, like here 'Task Manifest' is being introduced for the first time, we should add a link to that entry in the glossary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That feels redundant here - we describe a Task Manifest in this paragraph while the glossary is more a overall reference and doesn't need regular linking. We should link to deeper dive topics inside docs/.


Now, download and load the [psutil example](examples/tasks/psutil-file.yaml):
```
Expand Down Expand Up @@ -316,28 +318,26 @@ We have a few known features we want to take on from here while we remain open f
If you would like to propose a feature, please [open an Issue](https://github.com/intelsdi-x/snap/issues)) that includes RFC in it (for [request for comments](https://en.wikipedia.org/wiki/Request_for_Comments)).

## Community Support
This repository is one of **many** projects in the **Snap framework**. Discuss your questions about Snap by reaching out to us:

* Through GitHub Issues. Issues is our home for **all** needs: Q&A on everything - installation, request for events, integrations, bug issues, futures. [Open up an Issue](https://github.com/intelsdi-x/snap/issues) and know there's no wrong question for us.
* We also have a [Slack team](https://intelsdi-x.herokuapp.com/) where we hang out
* Submit a blog post on your use of Snap to [our Medium.com publication](https://medium.com/intel-sdi)
This repository is one of many in the Snap framework and [has maintainers supporting it](docs/MAINTAINERS.md). We love contributions from our community along the way. No improvement is too small.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this deleted information being said somewhere else in the documentation? I feel like it's important to invite others to contact us on github, slack or post a Snap blog. I can incorporate some of it into the Contact Us section of Maintainers.md. It may fit better there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call - it's redundant to the information covered below in Contributing


The full project lives here, at http://github.com/intelsdi-x/snap.
This note is especially important for plugins. While the Snap framework is hardened through tons of use, **plugins mature at their own pace**. If you have subject matter expertise related to a plugin, please share your feedback on that repository.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


## Contributing
We encourage contributions from the community. No improvement is too small. Snap needs:
We encourage contributions from the community. Snap needs:

* _Feedback_: try it and tell us about it through issues, blog posts or Twitter
* _Contributors_: We can always use more eyes on the core framework and its testing
* _Contributors_: We always appreciate more eyes on the core framework and plugins
* _Feedback_: try it and tell us about it on [our Slack team](https://intelsdi-x.herokuapp.com/), through [a blog posts](https://medium.com/intel-sdi/) or Twitter with #SnapTelemetry
* _Integrations_: Snap can collect from and publish to almost anything by [authoring a plugin](#author-a-plugin)

To contribute to the Snap framework, see [our CONTRIBUTING file](CONTRIBUTING.md). To give back to a specific plugin, open an issue on its repository.


### Author a Plugin
The power of Snap comes from its open architecture. Add to the ecosystem by building your own plugins to collect, process or publish telemetry.
The power of Snap comes from its open architecture and its growing community of contributors. You can be one of them:

* The definitive how-to is in [PLUGIN_AUTHORING.md](docs/PLUGIN_AUTHORING.md)
* Recommendations to make effective, well-designed plugins are in [PLUGIN_BEST_PRACTICES.md](docs/PLUGIN_BEST_PRACTICES.md)

Add to the ecosystem by building your own plugins to collect, process or publish telemetry.

## Security Disclosure

Expand Down
98 changes: 98 additions & 0 deletions docs/GLOSSARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Glossary
Snap is simple in scope and it becomes more simple when you know the terminology we use throughout the project. Here they are:

* [Glossary](#glossary)
* [Config: Global Config](#config-global-config)
* [Config: Global Options](#config-global-options)
* [Config: Metric Config](#config-metric-config)
* [Config: Plugin Config](#config-plugin-config)
* [Metric Catalog](#metric-catalog)
* [Metric: Dynamic](#metric-dynamic)
* [Metric: Namespace](#metric-namespace)
* [Metric Namespace: Dynamic Element](#metric-namespace-dynamic-element)
* [Metric Namespace: Static Element](#metric-namespace-static-element)
* [Plugin](#plugin)
* [Plugin Type: Collector](#plugin-type-collector)
* [Plugin Type: Processor](#plugin-type-processor)
* [Plugin Type: Publisher](#plugin-type-publisher)
* [Snap](#snap)
* [Snap Telemetry](#snap-telemetry)
* [snapctl](#snapctl)
* [snapd](#snapd)
* [Task](#task)
* [Task Manifest](#task-manifest)
* [Tribe](#tribe)
* [Workflow](#workflow)
* [Workflow: Distributed](#workflow-distrubted)
* [Workflow Manifest](#workflow-manifest)

### Config: Global Config
* Values loaded at runtime of the daemon ([reference](SNAPD_CONFIGURATION.md))

### Config: Global Options
* Values passed as command-line parameters or environmental variables ([reference](SNAPCTL.md#global-options))

### Config: Metric Config
* key/value pairs shared by collector namespace in a Task Manifest ([example](https://github.com/intelsdi-x/snap-plugin-collector-meminfo/blob/master/examples/tasks/task-mem.json#L15))

### Config: Plugin Config
* key/value pairs stored in the `config` block within a Task Manifest ([example](https://github.com/intelsdi-x/snap-plugin-collector-meminfo/blob/master/examples/tasks/task-mem.json#L24))

### Metric Catalog
* List of all available metrics exposed by a running instance of the Snap daemon ([reference](PLUGIN_LIFECYCLE.md#what-happens-when-a-plugin-is-loaded))

### Metric: Dynamic
* A metric is described as dynamic when it includes one or more wildcards in its namespace ([reference](METRICS.md#dynamic-metrics))

### Metric: Namespace
* Namespaces are a series of namespace elements that uniquely identify a metric in Snap ([reference](METRICS.md))

#### Metric: Dynamic Namespace Element
* An element of a metric whose value is set at runtime ([reference](METRICS.md))

#### Metric: Static Namespace Element
* An element of a metric whose value is set at load time ([reference](METRICS.md))

### Plugin
* An independent [binary][binary] that is compatible with Snap (see [Plugin Lifecycle](PLUGIN_LIFECYCLE.md))

### Plugin Type: Collector
* Gathers data and presents as a dynamically-generated namespaced metric catalog ([reference](PLUGIN_AUTHORING.md#plugin-type))

### Plugin Type: Processor
* Extends or filters collected metrics ([reference](PLUGIN_AUTHORING.md#plugin-type))

### Plugin Type: Publisher
* Persists metrics into a target endpoint ([reference](PLUGIN_AUTHORING.md#plugin-type))

### Snap
* The project name, focused on the Snap daemon and the plugins that power its collection, processing and publishing of telemetry

### Snap Telemetry
* The full name of the Snap project, used mostly for easy searching (like snap-telemetry.io) or hashtag (#SnapTelemetry)

### `snapctl`
* The command-line interface (CLI) for Snap, released as a [binary][binary]

### `snapd`
* The [daemon process](http://www.linfo.org/daemon.html) for Snap, released as a [binary][binary]

### Task
* A job running within Snap, including the API version, schedule and workflow (all documented [here](TASKS.md))

### Task Manifest
* A file that includes the API version, schedule and workflow of a Task in a declarative form ([reference](TASKS.md#task-manifest))

### Tribe
* The clustering feature of Snap, documented [here](TRIBE.md)

### Workflow
* The explicit map of how collectors, processors and publishers are used in Snap ([reference](TASKS.md#the-workflow))

### Workflow: Distributed
* A workflow where one or more steps have a remote target specified ([reference](DISTRIBUTED_WORKFLOW_ARCHITECTURE.md))

### Workflow Manifest
* A file that describes only the workflow of a Task ([example at the bottom](SNAPCTL.md#load-and-unload-plugins-create-and-start-a-task))

[binary]: https://www.quora.com/Whats-the-difference-between-an-installer-source-code-and-a-binary-package-when-installing-software
34 changes: 21 additions & 13 deletions docs/PLUGIN_AUTHORING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,25 +137,33 @@ We recommend releasing new binaries to Github Release page whenever the plugin v

In the plugin repo root directory, the `metadata.yml` file provides Snap project additional information about your plugin when we generate the [plugin catalog](./PLUGIN_CATALOG.md) page.

* name: plugin full name
* type: plugin type
* maintainer: your github org or username
* license: the plugin software licence
* description: paragraph describing the plugin's purpose
* badge: a list of [badges](https://shields.io/) to display
* ci: a list of ci services running for this repo
* **name**: plugin full name
* **type**: plugin type
* **maintainer**: your github org or username
* **license**: the plugin software license
* **description**: paragraph describing the plugin's purpose
* **badge**: a list of [badges](https://shields.io/) to display
* **ci**: a list of ci services running for this repo
* **status**: one of the three statuses [described below](#plugin-status)

All metadata fields are optional, but recommended to help users discover your plugin. Please check out the file plugin's [metadata.yml](https://github.com/intelsdi-x/snap-plugin-publisher-file/blob/master/metadata.yml) file for a working example.

To list your plugin in the catalog, please submit a PR and update [plugins.yml](./plugins.yml) file to include the plugin's github `organization/repo_name`.
We recommend sharing your plugins early and often by adding them to the list of known plugins. To list your plugin in the plugin catalog, please submit a PR and update [plugins.yml](./plugins.yml) file to include the plugin's github `organization/repo_name`.

### Plugin Status

While the Snap framework is hardened through tons of testing, **plugins mature at their own pace**. We want our community to share plugins early and update them often. We are defining categories of maturity of a plugin and will roll them out with the resolution of [#1322](https://github.com/intelsdi-x/snap/issues/1322).

### Documentation

All plugins should include a README with the following information:

1. Supported Platforms
2. Snap Version dependencies
3. Installation
4. Usage
5. Contributors
6. License
1. Known Issues
1. Snap Version dependencies
1. Installation
1. Usage
1. Contributors
1. License

You are welcome to copy an existing README.md (and CONTRIBUTING.md) to get started. I recommend looking at [psutil](https://github.com/intelsdi-x/snap-plugin-collector-psutil).
2 changes: 1 addition & 1 deletion docs/PLUGIN_CATALOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Plugin Catalog
This is the master catalog of plugins for Snap. The plugins in this list may be written by multiple sources. Please examine the license and documentation of each plugin for more information.
This is the master catalog of plugins for Snap. While the Snap framework is hardened through tons of testing, **plugins mature at their own pace**. Review the documentation for each plugin before considering it ready for use.

## All Plugins
This file is automatically generated. If you would like to add to the plugin list, [add your plugin to this list](plugins.yml) and it will be added (usually within 24 hours).
Expand Down