Skip to content

Latest commit

 

History

History
374 lines (320 loc) · 9.73 KB

put-transform.asciidoc

File metadata and controls

374 lines (320 loc) · 9.73 KB

Create {transform} API

Create {transform}

Instantiates a {transform}.

{api-request-title}

PUT _transform/<transform_id>

{api-prereq-title}

Requires the following privileges:

  • cluster: manage_transform (the transform_admin built-in role grants this privilege)

  • source indices: read, view_index_metadata

  • destination index: read, create_index, index. If a retention_policy is configured, the delete privilege is also required.

    Note
    		 If you provide secondary authorization headers, those credentials are used.

{api-description-title}

This API defines a {transform}, which copies data from source indices, transforms it, and persists it into an entity-centric destination index. If you choose to use the pivot method for your {transform}, the entities are defined by the set of group_by fields in the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values in the latest object.

You can also think of the destination index as a two-dimensional tabular data structure (known as a {dataframe}). The ID for each document in the {dataframe} is generated from a hash of the entity, so there is a unique row per entity. For more information, see [transforms].

When the {transform} is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use the defer_validation parameter to skip these checks.

Deferred validations are always run when the {transform} is started, with the exception of privilege checks. When {es} {security-features} are enabled, the {transform} remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the {transform} fails when it attempts unauthorized operations.

Important
 You must use {kib} or this API to create a {transform}. Do not add a {transform} directly into any .transform-internal* indices using the {es} index API. If {es} {security-features} are enabled, do not give users any privileges on .transform-internal* indices. If you used {transforms} prior to 7.5, also do not give users any privileges on .data-frame-internal* indices.

You must choose either the latest or pivot method for your {transform}; you cannot use both in a single {transform}.

{api-path-parms-title}

<transform_id>

(Required, string) Identifier for the {transform}. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64 character limit and must start and end with alphanumeric characters.

{api-query-parms-title}

defer_validation

(Optional, Boolean) When true, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the {transform} is created.

timeout

(Optional, time) Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. Defaults to 30s.

{api-request-body-title}

description

(Optional, string) Free text description of the {transform}.

dest

(Required, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of dest
frequency

(Optional, time units) {es-repo-dir}/rest-api/common-parms.asciidoc

latest

(Required*, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of latest
sort

(Required, string) {es-repo-dir}/rest-api/common-parms.asciidoc

unique_key

(Required, array of strings) {es-repo-dir}/rest-api/common-parms.asciidoc

pivot

(Required*, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of pivot
aggregations or aggs

(Required, object) {es-repo-dir}/rest-api/common-parms.asciidoc

group_by

(Required, object) {es-repo-dir}/rest-api/common-parms.asciidoc

retention_policy

(Optional, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of retention_policy
settings

(Optional, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of settings
dates_as_epoch_millis

(Optional, boolean) {es-repo-dir}/rest-api/common-parms.asciidoc

docs_per_second

(Optional, float) {es-repo-dir}/rest-api/common-parms.asciidoc

align_checkpoints

(Optional, boolean) {es-repo-dir}/rest-api/common-parms.asciidoc

deduce_mappings

(Optional, boolean) {es-repo-dir}/rest-api/common-parms.asciidoc

max_page_search_size

(Optional, integer) {es-repo-dir}/rest-api/common-parms.asciidoc

source

(Required, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of source
index

(Required, string or array) {es-repo-dir}/rest-api/common-parms.asciidoc

query

(Optional, object) {es-repo-dir}/rest-api/common-parms.asciidoc

runtime_mappings

(Optional, object) {es-repo-dir}/rest-api/common-parms.asciidoc

sync

(Optional, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of sync
time

(Required, object) {es-repo-dir}/rest-api/common-parms.asciidoc

Properties of time
delay

(Optional, time units) {es-repo-dir}/rest-api/common-parms.asciidoc

field

(Required, string) {es-repo-dir}/rest-api/common-parms.asciidoc

Tip
 In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

{api-examples-title}

The following {transform} uses the pivot method:

PUT _transform/ecommerce_transform1
{
  "source": {
    "index": "kibana_sample_data_ecommerce",
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id"
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform1",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  },
  "retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "30d"
    }
  }
}

When the {transform} is created, you receive the following results:

{
  "acknowledged" : true
}

The following {transform} uses the latest method:

PUT _transform/ecommerce_transform2
{
  "source": {
    "index": "kibana_sample_data_ecommerce"
  },
  "latest": {
    "unique_key": ["customer_id"],
    "sort": "order_date"
  },
  "description": "Latest order for each customer",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform2"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  }
}