apm-data: Use representative count as event.success_count if available

[rubvs]

Closes elastic/apm-data#388

The pipeline changes copies the representative count calculated in APM Data (see Code), into the event.success_count metric. The default representative count is 1.

If an event, which can only be a transaction or span in this context, failed for some reason, the event.success_count metric is set to zero. To my understanding, we are not capturing any data related to failed events. Nor do I see any usage of something like an event.failure_count metric. Please correct me if I'm wrong.

Along with the unit test provided, the pipeline changes was also test manually in Kibana Console:

Test

1. Create a cloud deployment with any version.
2. In Dev Tools paste the below pipeline simulation and run.
3. Verify that the representative_count value is the same as that of event.success_count.

POST _ingest/pipeline/_simulate

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "Test APM pipeline",
    "processors": [
      {
        "set": {
          "if": "ctx.transaction?.type != null",
          "field": "processor.event",
          "value": "transaction"
        }
      },
      {
        "set": {
          "field": "processor.event",
          "value": "span",
          "override": false
        }
      },
      {
        "script": {
          "if": "ctx[ctx.processor.event]?.duration == null",
          "source": """
            def eventDuration = ctx.event?.duration ?: 0;
            def rootObjectName = ctx.processor?.event;
            def rootObject = ctx[rootObjectName];
            if (rootObject == null) {
              rootObject = [:];
              ctx[rootObjectName] = rootObject;
            }
            rootObject.duration = ["us": (long)(eventDuration/1000)];
          """
        }
      },
      {
        "remove": {
          "field": ["event.duration"],
          "ignore_failure": true,
          "ignore_missing": true
        }
      },
      {
        "set": {
          "if": "ctx.event?.outcome == 'failure'",
          "field": "event.success_count",
          "value": 0
        }
      },
      {
        "set": {
          "if": "ctx.event?.outcome == 'success'",
          "field": "event.success_count",
          "value": 1
        }
      },
      {
        "script": {
          "if": "ctx.event?.outcome == 'success' && ctx[ctx.processor?.event]?.representative_count != null",
          "source": "ctx.event.success_count = ctx[ctx.processor?.event].representative_count;"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "processor": {
          "event": "transaction"
        },
        "transaction": {
          "representative_count": 2,
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "success"
        }
      }
    },
    {
      "_source": {
        "processor": {
          "event": "span"
        },
        "span": {
          "representative_count": 3,
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "success"
        }
      }
    },
    {
      "_source": {
        "processor": {
            "event": "transaction"
        },
        "transaction": {
          "representative_count": 2,
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "failure"
        }
      }
    },
    {
      "_source": {
        "processor": {
          "event": "span"
        },
        "span": {
          "representative_count": 3,
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "failure"
        }
      }
    },
    {
      "_source": {
        "processor": {
          "event": "span"
        },
        "span": {
          "representative_count": null,
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "success"
        }
      }
    },
    {
      "_source": {
        "processor": {
          "event": "transaction"
        },
        "span": {
          "type": "request",
          "duration": {
            "us": 1234567
          }
        },
        "event": {
          "duration": 1234567000,
          "outcome": "success"
        }
      }
    }
  ]
}

[elasticsearchmachine]

Hi @rubvs, I've updated the changelog YAML for you.

[carsonip]

thanks for covering the case I missed and also adding the corresponding test case

[simitt]

LGTM once the changelog is fixed

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 119995

[dakrone]

This PR doesn't appear to have bumped the version. Was the version bumped elsewhere so that it will still be installed, or do we need a separate PR so that the version is incremented and the new pipeline installed?

[rubvs]

@dakrone I believe we will need to do this manually with a new PR. But I'm not sure, this is my first merge into ES. How can I check if it was bumped elsewhere?

[dakrone]

You can check the git blame for this line:

elasticsearch/x-pack/plugin/apm-data/src/main/resources/resources.yaml

Lines 1 to 4 in b3959b6

	# "version" holds the version of the templates and ingest pipelines installed
	# by xpack-plugin apm-data. This must be increased whenever an existing template or
	# pipeline is changed, in order for it to be updated on Elasticsearch upgrade.
	version: 12

(which is also the value you would need to increment).

It appears it hasn't been bumped since November of last year, so incrementing it would be necessary (just change it from 12 -> 13)

[rubvs]

Thanks, I'm also busy fixing the merge issue by checky-picking my commit, but I don't see a v8.19 tag in ES repo?

[dakrone]

That's because v8.19 hasn't been released yet, if you're trying to backport, you should backport to the 8.x branch.

[rubvs]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation