[Question]: AWS Dynamodb data not generating under Data Streams tab.

[amolnater-qasource]

Description:
We added AWS dynamodb integration to default policy, however we are not able to see any data related to same under Data Streams tab.
Do we need any AWS setup/credentials to get AWS Dynamodb data?
Please let us know if are missing anything.

Reference Ticket:
elastic/kibana#93315

Build details 7.14.0 Snapshot:

Build: 42333
Commit: d2904ca073e91582a2c5fa6ef6e712491cf3521f

Screenshots:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & assigned to @jen-huang

CC: @EricDavisX

[jen-huang]

Do we need any AWS setup/credentials to get AWS Dynamodb data?

Yes, @kaiyan-sheng, can you provide guidance on getting AWS data?

[kaiyan-sheng]

I created a 7.14 cloud instance and added AWS Dynamodb integration. I was able to see data under data streams:

Here is the agent policy I used:

inputs:
  - id: ad02c2de-86b9-4359-bf99-860e49e37b1f
    name: aws-1
    revision: 2
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 0.8.0
    data_stream:
      namespace: default
    streams:
      - id: aws/metrics-aws.dynamodb-ad02c2de-86b9-4359-bf99-860e49e37b1f
        data_stream:
          dataset: aws.dynamodb
          type: metrics
        access_key_id: foo
        session_token: >-
          bar
        period: 5m
        tags_filter: null
        metricsets:
          - dynamodb
        secret_access_key: foobar

[jen-huang]

@kaiyan-sheng Just to confirm, this doesn't need a real set of AWS credentials?

[amolnater-qasource]

Hi @jen-huang
We have re-attempted to use aws-dynamodb on 7.14.0 Snapshot.
Observations:

• Agent went Unhealthy.
• Observed below errors on Logs tab:

Build details:

Build: 42405
Commit: d449412c8ca14a87a10527ffdc3967d42c12f34a
Artifact Link: https://snapshots.elastic.co/7.14.0-6fb865c2/downloads/beats/elastic-agent/elastic-agent-7.14.0-SNAPSHOT-windows-x86_64.zip

Thanks
QAS

[kaiyan-sheng]

@jen-huang @amolnater-qasource Sorry for the confusion. Real AWS credentials are required to collect metrics from any AWS services here. The error message from elastic-agent also is complaining about credentials are not valid.

Do you have an AWS environment/account to use for testing purposes?

[amolnater-qasource]

Hi @kaiyan-sheng
Thanks for the feedback.

Do you have an AWS environment/account to use for testing purposes?

We don't have any AWS environment. Though we access to AWS-dev with our elastic gmail accounts.
Please let us know which credentials are required to be used in AWS integration for testing purposes.

Thanks
QAS

[kaiyan-sheng]

Hi @amolnater-qasource I just pinged you privately, let's try to get an AWS test account set up for you.

[amolnater-qasource]

Hi @kaiyan-sheng
Thanks for the feedback.
@EricDavisX we don't have permissions to create IAM user, could you please look into this.
We need this info to validate AWS Dynamodb data sets under Data Streams tab.

Thanks
QAS

[EricDavisX]

Let's review and discuss testing options offline tomorrow.

[amolnater-qasource]

Hi @EricDavisX
As per feedback, we will be skipping Data Streams test for AWS dynamoDB integration since we can’t fully configure it currently..

Hence we are closing this.
Thanks
QAS