[FEATURE] Add IAM authentication option

[aetter]

If people want to connect to an Amazon OpenSearch Service cluster, they have to figure out how to use this client alongside some third-party signing library and/or the AWS SDK, which is non-trivial due to how we (by design) conceal the underlying HTTP requests that the client makes. We should offer IAM signing as an option, just like the OpenSearch CLI does.

Is your feature request related to a problem?

If you want to use this client with an Amazon OpenSearch Service cluster that has IAM authentication rather than basic authentication, good luck.

What solution would you like?

When initializing the client, accept an iam_auth option in place of the current basic auth option. If basic, accept username and password. If IAM, accept options for access key, secret key, session token, region, and service.

What alternatives have you considered?

An additional JS signing library for use on top of the client, such as https://github.com/compwright/aws-elasticsearch-connector. But this is third-party, might not use the latest version of the JS SDK, and given that it would only work with this client, it seems better and easier to just add it to the client and maintain it ourselves.

Do you have any additional context?

opensearch-project/OpenSearch#1400

[Ghazgkull]

Until OpenSearch adds authentication support for hosted offerings like AWS OpenSearch, is there a workaround available for users of those hosted offerings? I'm confused about the current situation... how is anyone using the OpenSearch JS client today, without support for authentication?

[rawpixel-vincent]

until the role authentication is supported, we're stuck into the latest supported version of elasticsearch with the latest elasticsearch node.js client compatible with a library that wrap that client to sign the requests for aws.
we're looking forward to get this support so we can upgrade to opensearch 1.0

[rawpixel-vincent]

I think it should take around 1 day to get it done and hopefully around 3 weeks to fix/review/release/document it? that would be amazing and aws customers could put all the pain behind and enjoy an amazing truly open source service, what do you think?

[ChristopheBougere]

This is an important feature for us as well, I hope this will be implemented soon 🙏

[alexsasharegan]

I've been trying to extend the Connection and/or Transport classes to support signing, but when you need to work with the v3 aws sdk, they have all the new credentials resolvers and signers that are promise-based. Hooking into the Connection#makeRequest and Transport#request means running into the fundamentally synchronous API. I can think of a few ways past this, but not without breaking the existing interface contracts between Connection and Transport.

[alexsasharegan]

A follow up to my previous comment, because the Connection.request method synchronously returns a mere Abortable (just an interface with an abort method to cancel the request), I found I can use an AbortController and queue up all the asynchronous signing work along with the rest of request procedure. Because the node ClientRequest accepts an abort signal, the request is properly connected to the Abortable that gets returned before the request is initialized.

This solution is terrible for me because I have to control the entire implementation of Connection.request, but it perhaps demonstrates a proof of concept for how an async prepare or signRequestParams method might be added that would let users like me hook in and sign the request params without taking ownership of the whole request method.

The end usage here for me is to call this function as a mixin. I feed it my AWS config, and I mixin its return with my opensearch config when instantiating a client.

Connection extension

import { Credentials, Provider } from "@aws-sdk/types";
import { defaultProvider } from "@aws-sdk/credential-provider-node";
import { SignatureV4 } from "@aws-sdk/signature-v4";
import { Sha256 } from "@aws-crypto/sha256-browser";
import {
  ClientOptions,
  Connection,
  errors,
} from "@opensearch-project/opensearch";
const { pipeline } = require("stream");

const INVALID_PATH_REGEX = /[^\u0021-\u00ff]/;
const { TimeoutError, ConnectionError, RequestAbortedError } = errors;

const [vMajor, _vMinor, _vPatch] = process.versions.node.split(".");
const hasNativeAbortController = Number(vMajor) >= 15;
if (!hasNativeAbortController) {
  global.AbortController =
    require("../helpers/AbortController").AbortController;
}

// https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-node
// https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

export function createAWSConnection(config: {
  credentials?: Credentials | Provider<Credentials>;
  region: string;
  clientInit: ClientOptions;
}): ClientOptions {
  const { credentials, region, clientInit: init } = config;

  const host = init.node ?? init.nodes[0];
  let hostUrl: null | URL = null;
  if (host != null) {
    hostUrl = new URL(host);
  }

  return {
    Connection: class AWSConnection extends Connection {
      /**
       *
       * @param {Parameters<Connection['request']>[0]} params
       * @param {Parameters<Connection['request']>[1]} callback
       * @returns
       */
      request(params, callback) {
        this._openRequests++;
        let cleanedListeners = false;

        const requestParams = this.buildRequestObject(params);

        if (requestParams.headers["content-type"] == null) {
          requestParams.headers["content-type"] = "application/json";
        }
        if (
          requestParams.headers["host"] == null &&
          hostUrl?.hostname != null
        ) {
          requestParams.headers["host"] = hostUrl.hostname;
        }

        // https://github.com/nodejs/node/commit/b961d9fd83
        if (INVALID_PATH_REGEX.test(requestParams.path) === true) {
          callback(
            new TypeError(`ERR_UNESCAPED_CHARACTERS: ${requestParams.path}`),
            null
          );
          /* istanbul ignore next */
          return { abort: () => {} };
        }

        const abortCtl = new AbortController();
        const request = {
          abort() {
            abortCtl.abort();
          },
        };

        const signer = new SignatureV4({
          credentials: credentials ?? defaultProvider(),
          region,
          service: "es",
          sha256: Sha256,
          applyChecksum: true,
        });

        const payloadToSign = {
          headers: requestParams.headers,
          query: searchStringToQueryBag(params.querystring),
          body: params.body,
          method: requestParams.method ?? "GET",
          path: params.path,
        };

        signer.sign(payloadToSign).then((signedRequestParams) => {
          const teardown = [];

          /**
           * @type {import("http").ClientRequest}
           */
          const request = this.makeRequest({
            ...requestParams,
            headers: signedRequestParams.headers,
            signal: abortCtl.signal,
          });

          // Without support, the request won't listen to the signal natively.
          if (!hasNativeAbortController) {
            const abortFromSignal = () => {
              request.abort();
            };
            abortCtl.signal.addEventListener("abort", abortFromSignal);
            teardown.push(() => {
              abortCtl.signal.removeEventListener("abort", abortFromSignal);
            });
          }

          const onResponse = (response) => {
            cleanListeners();
            this._openRequests--;
            callback(null, response);
          };

          const onTimeout = () => {
            cleanListeners();
            this._openRequests--;
            request.once("error", () => {}); // we need to catch the request aborted error
            request.abort();
            callback(new TimeoutError("Request timed out", params), null);
          };

          const onError = (err) => {
            cleanListeners();
            this._openRequests--;
            callback(new ConnectionError(err.message), null);
          };

          const onAbort = () => {
            cleanListeners();
            request.once("error", () => {}); // we need to catch the request aborted error
            this._openRequests--;
            callback(new RequestAbortedError(), null);
          };

          request.on("response", onResponse);
          request.on("timeout", onTimeout);
          request.on("error", onError);
          request.on("abort", onAbort);

          // Disables the Nagle algorithm
          request.setNoDelay(true);

          // starts the request
          if (isStream(params.body) === true) {
            pipeline(params.body, request, (err) => {
              /* istanbul ignore if  */
              if (err != null && cleanedListeners === false) {
                cleanListeners();
                this._openRequests--;
                callback(err, null);
              }
            });
          } else {
            request.end(params.body);
          }

          function cleanListeners() {
            request.removeListener("response", onResponse);
            request.removeListener("timeout", onTimeout);
            request.removeListener("error", onError);
            request.removeListener("abort", onAbort);
            for (const fn of teardown) {
              fn();
            }
            cleanedListeners = true;
          }
        });
        // TODO: Sign the request params

        return request;
      }
    },
  };
}

function isStream(obj) {
  return obj != null && typeof obj.pipe === "function";
}

/**
 *
 * @param {string} search
 * @returns {import("@aws-sdk/protocol-http").HttpRequest['query'] | undefined}
 */
function searchStringToQueryBag(search) {
  const query = {};
  const usp = new URLSearchParams(search);

  for (const [name] of usp) {
    const list = usp.getAll(name);
    if (list.length > 1) {
      query[name] = list;
    } else {
      query[name] = usp.get(name);
    }
  }

  return query;
}

[rawpixel-vincent]

are we gonna be left with an unsupported version of elasticsearch before aws implement a client for their fork of opensearch no customer either asked about?

that's a real question from a customer

[yatanasov]

any update on this please?

[lizsnyder]

@Shivamdhar any idea if this is on the roadmap, similar to https://github.com/opensearch-project/opensearch-py#using-iam-credentials-for-authentication?

[Shivamdhar]

I do not have much information about this one, but may be @ananzh can help us here.

[brijos]

The team does not have the bandwidth to commit to the supporting IAM in OpenSearch JS client in the next few months. Does anyone has the capacity to assist?

Below are a two work arounds proposed by engineers to work in the meantime:

1. Use a proxy to do the signing. One can setup a proxy server which can sign the request and then forward the request to AWS OpenSearch domain. One available proxy is https://github.com/awslabs/aws-sigv4-proxy, and there are other options as well: https://www.google.com/search?q=aws+sigv4+proxy.

2. One can choose to not using IAM auth for their AWS OpenSearch domain and instead use fine-grained accessed control (FGAC). Set the AWS OpenSearch domain access policy to allow only certain IP address using https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-ip and secure the domain with FGAC user name and password combination.

Feedback on the workarounds is welcomed.

[AlexMabry]

I was able to get it working using the following code:

  import { Client } from '@opensearch-project/opensearch';
  import { defaultProvider } from '@aws-sdk/credential-provider-node';
  const createAwsOpensearchConnector = require('aws-opensearch-connector');

  const awsCredentials = await defaultProvider()();
  const connector = createAwsOpensearchConnector({
    credentials: awsCredentials,
    region: process.env.AWS_REGION ?? 'us-east-1',
    getCredentials: (cb: () => void) => cb(),
  });
  const client = new Client({
    ...connector,
    node: ES_URL,
  });

[dblock]

@AlexMabry Nice! Care to contribute this to the docs at least? I still think we need native options in the client, but it looks a lot simpler to implement using @aws-sdk/credential-provider-node. PRs welcome.

[AlexMabry]

Here is another solution that doesn't require the aws-opensearch-connector package.

aws-connector.ts

import { Credentials } from '@aws-sdk/types';
import { ClientOptions, Connection } from '@opensearch-project/opensearch';
import { Request, sign } from 'aws4';

export function createAwsConnector(
  credentials: Credentials,
  region: string,
): ClientOptions {
  class AmazonConnection extends Connection {
    override buildRequestObject(params: any) {
      const req: Request = super.buildRequestObject(params) as Request;

      req.service = 'es';
      req.region = region;
      req.headers = req.headers || {};
      req.headers['host'] = req.hostname;

      return sign(req, credentials);
    }
  }

  return {
    Connection: AmazonConnection,
  };
}

os-client.ts

import { Client } from '@opensearch-project/opensearch';
import { createAwsConnector } from './aws-connector';  // from above
import { defaultProvider } from '@aws-sdk/credential-provider-node';

export async function getClient() {
  const credentials = await defaultProvider()();

  return new Client({
    ...createAwsConnector(credentials, 'us-east-1'),
    node: ES_URL,
  });
}

[AlexMabry]

@AlexMabry Nice! Care to contribute this to the docs at least? I still think we need native options in the client, but it looks a lot simpler to implement using @aws-sdk/credential-provider-node. PRs welcome.

Do you mean update the README or some other documentation?

[lizsnyder]

I am working on testing them and plan to add them here https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-node

I will make a TS section, but I need to get these working in Node as well.

@AlexMabry you're more than welcome to contribute yourself, if you have time https://github.com/awsdocs/amazon-opensearch-service-developer-guide/blob/master/doc_source/request-signing.md

[AlexMabry]

I am actually using TypeScript in my Node project, but here is a JavaScript version of the aws-connector file from above:

const { Connection } = require('@opensearch-project/opensearch');
const aws4 = require('aws4');

module.exports = (credentials, region) => {
  class AmazonConnection extends Connection {
    buildRequestObject(params) {
      const req = super.buildRequestObject(params);

      req.service = 'es';
      req.region = region;
      req.headers = req.headers || {};
      req.headers['host'] = req.hostname;

      return aws4.sign(req, credentials);
    }
  }

  return { Connection: AmazonConnection };
};

[lizsnyder]

Thanks a lot @AlexMabry. I tested these out, simplified them a bit, and added them to the AWS docs: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-node

[moltar]

@lizsnyder thanks for the final solution docs addition!

Btw, SDK v3, has the signing package as well, which can be used instead of the unofficial aws4 package:

https://github.com/aws/aws-sdk-js-v3/tree/main/packages/signature-v4

It's not documented, unfortunately, but you can find use cases this way:

https://github.com/aws/aws-sdk-js-v3/search?l=TypeScript&q=%22signature-v4%22&type=code

[moltar]

Oh, I think I understand the problem better now, the SDK v3 package signing interface is async (returning Promise) which is incompatible with the Connector interface. :(

[alexsasharegan]

Oh, I think I understand the problem better now, the SDK v3 package signing interface is async (returning Promise) which is incompatible with the Connector interface. :(

That's what I was getting at here: #187 (comment)

[ChristopherGillis]

I'm trying @AlexMabry 's solution, but seeing this error:

Error:
TypeError: Cannot read properties of undefined (reading 'username')
    at getAuth (/x/node_modules/@opensearch-project/opensearch/index.js:299:12)
    at new Client (/x/node_modules/@opensearch-project/opensearch/index.js:91:25)

I'm not sure why opensearch is looking for basic auth properties when I'm passing a connection... Is there something else I need to pass to get it to not look for basic auth?

[ChristopherGillis]

Ahh, nvm. I wasn't passing a string as a "node".

[ChristopherGillis]

I'm still getting 403. Is there anything I need to configure on my cluster to allow this? (IAM instead of basic auth)

[MattReimer]

Here is another solution that doesn't require the aws-opensearch-connector package.

with @AlexMabry 's solution I'm getting:

Class constructor Connection cannot be invoked without 'new

export async function getOpenSearchClient(region: string): Promise<Client> {
  const credentials = await defaultProvider()()
  return new Client({
    ...createAwsConnector(credentials, region),
    node: ES_NODE_STRING,
  })
}

Looks the same to me. I don't know what's going on...

[rawpixel-vincent]

The example provived by the community and copy/pasted by aws engineers into their documentation doesn't refresh the credentials (token) when they expires, this will cause 4xx errors.
We're using https://github.com/yosefbs/aws-opensearch-connector for now.

Looking forward to the built-in support in the client, any ETA from aws @brijos @lizsnyder ?

Edit: for example, if you use role authentication in ecs tasks, if a task run for more than 8 hours, all the request will fail.

[dblock]

@rawpixel-vincent I don't think anyone is actively working on this, would you like to contribute what you have? I can help correct any documentation as well separately

[rawpixel-vincent]

Hi @dblock,

we're testing https://github.com/yosefbs/aws-opensearch-connector with yosefbs/aws-opensearch-connector#7 applied
and https://github.com/opensearch-project/opensearch-js/ with #254 applied

it seems to work fine so far.

edit: you should probably do something about the documentation, because the current example is failling after a certain time, and it's difficult to detect that issue on ephemeral dev/staging environments before sending to production - which might cause your consumers to end up with outage.

[dblock]

@rawpixel-vincent which documentation specifically and what's the fix (link?)

I'd like this project to have first class support for Sigv4 signing and make it dead easy to do Sigv4 without having to think connector/wrapper or work too hard. Something similar to how curl supports --aws-sigv4 as a first class citizen. At the same time I am definitely not suggesting everyone be dragging a bunch of vendor-specific dependencies into their projects unnecessarily. So, stepping back a little, what would you do?

[moltar]

I think transport interface needs to be amended to support async.

Then use the official @aws-sdk/signature-v4 package for signing.

[dblock]

@moltar Reading through the history here looks like you know what we need to do to make the transport interface async. Short of doing it, would you mind opening a new issue and describing what you're suggesting?

[rawpixel-vincent]

@rawpixel-vincent which documentation specifically and what's the fix (link?)

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-node

First block, signing with aws4.

I do not have the fix, I switched to the solution I described before.

[adri1wald]

The example provived by the community and copy/pasted by aws engineers into their documentation doesn't refresh the credentials (token) when they expires, this will cause 4xx errors. We're using https://github.com/yosefbs/aws-opensearch-connector for now.

Looking forward to the built-in support in the client, any ETA from aws @brijos @lizsnyder ?

Edit: for example, if you use role authentication in ecs tasks, if a task run for more than 8 hours, all the request will fail.

From what I can tell unless you call the credential provider on every request (which you can't due to sync nature of buildRequestObject), using defaultProvider or fromNodeProviderChain the credentials won't actually be refreshed if they are expired. Therefore, I wrote this auto-refreshing credential provider that refreshes the credentials before they expire. Note this mutates the credentials object, so it won't work if you create a copy before buildRequestObject.

const THIRTY_SECONDS = 10 * 1000
const MAX_SIGNED_32_BIT_INT = ~(1 << 31)

function getWaitMs(expiration: Date, refreshBeforeExpirationMs: number) {
  let waitMs = expiration.getTime() - Date.now()
  waitMs = waitMs - refreshBeforeExpirationMs
  waitMs = Math.max(waitMs, 0)
  waitMs = Math.min(waitMs, MAX_SIGNED_32_BIT_INT)
  return waitMs
}

function mutablyUpdateCredentials(credentials: Credentials, newCredentials: Credentials) {
  for (const key of Object.keys(credentials)) {
    // @ts-ignore
    delete credentials[key]
  }
  for (const [key, value] of Object.entries(newCredentials)) {
    // @ts-ignore
    credentials[key] = value
  }
}

interface AutoRefreshingProvider {
  (): Promise<Credentials>
  readonly refresh: () => Promise<void>
  readonly isRefreshing: boolean
}

export function createAutoRefreshingCredentialProvider(
  provider:  MemoizedProvider<Credentials>,
  refreshBeforeExpirationMs: number = THIRTY_SECONDS,
  maxRetriesBeforeStopping: number = 3,
  retryBackoffMsFn: (retryCount: number) => number = () => 0
): AutoRefreshingProvider {
  let credentialsPromise: Promise<Credentials> | undefined
  let refreshPromise: Promise<void> | undefined

  const getOrCreateCredentialsPromise = () => {
    if (credentialsPromise) {
      return credentialsPromise
    }
    credentialsPromise = provider()
      // don't want to mutate the credentials within the provider
      .then((credentials) => ({ ...credentials }))
    return credentialsPromise
  }

  const autoRefresh = async (retry: number = 0) => {
    const credentials = await getOrCreateCredentialsPromise()
    if (!(credentials.expiration instanceof Date)) {
      return
    }
    const waitMs = getWaitMs(credentials.expiration, refreshBeforeExpirationMs)
    if (waitMs > 0) {
      await wait(waitMs)
    }
    let error: Error | undefined
    try {
      const newCredentials = await provider({ forceRefresh: true })
      mutablyUpdateCredentials(credentials, newCredentials)
    } catch (error) {
      error = error
      console.error(error)
    }
    if (error) {
      retry = retry + 1
      if (retry <= maxRetriesBeforeStopping) {
        const backoffMs = retryBackoffMsFn(retry)
        if (backoffMs > 0) {
          await wait(backoffMs)
        }
        await autoRefresh(retry)
      } else {
        throw error
      }
    }
    await autoRefresh()
  }

  const startRefresh = async () => {
    if (refreshPromise) {
      return refreshPromise
    }
    try {
      refreshPromise = autoRefresh()
      await refreshPromise
    } finally {
      refreshPromise = undefined
    }
  }

  const intermediateProvider = async () => {
    const credentials = await getOrCreateCredentialsPromise()
    startRefresh().catch(() => {})
    return credentials
  }

  const autoRefreshingProvider = Object.assign(intermediateProvider, {
    refresh: startRefresh,
    get isRefreshing() {
      return !!refreshPromise
    }
  })

  return autoRefreshingProvider
}

[dblock]

Thanks @adri1wald, this is a great point. I see a lot of bugs where clients get instantiated, work for a while, then start failing after a refresh period.

We're still looking for a PR that would implement this feature as a 1st class citizen in the library. Appreciate if anyone could give it a stab!

#279 please comment!

[rawpixel-vincent]

I didn't get any notifications about the PR for this feature, mentioning it to get feedback for more people interested to this issue.
please comment on #279

[wbeckler]

I'm closing this as per the last comment unless there are objections.

[rawpixel-vincent]

any plan to release this? @dblock

[dblock]

any plan to release this? @dblock

We should release quickly. I heard there are some automation things @VachaShah and @gaiksaya would like to drive here before we do more "manual" releases, but I don't think that should stop us?

I opened #303, add your +1.

[Prophet32j]

@dblock @wbeckler any reason this is still open? Looks like everything is merged and available in 2.1.x

[wbeckler]

This is released. Closing

[akleiber]

Just for reference - here is described how you use it now: https://github.com/opensearch-project/opensearch-js/blob/2.1.0/USER_GUIDE.md#using-aws-v3-sdk