feat: add interactive captcha handler with Slack and Discord

docs/help/troubleshoot.md

@@ -2,6 +2,9 @@
 
 ## Captcha issues
 
+???+ info
+    A new interactive captcha handler has been implemented. You can learn more about how to use it [here](../reference/captcha.md)! Otherwise, feel free to still try the below options.
+
 ### Option 1
 
 If you're running into problems, try running in headful mode: `HEADLESS="false"`.

docs/reference/captcha.md

@@ -0,0 +1,84 @@
+# Captcha
+
+A mechanism has been implemented to allow users to interactively handle captcha challenges without being directly connected to their streetmerchant instance. This works by sending the captcha challenge to the user directly via their preferred messaging service and waiting for their response, which is then input to the captcha page, allowing streetmerchant to proceed with its processing. 
+
+???+ attention
+    This implementation has only been tested/used on Amazon (North America). Please submit an issue if you're facing captcha on other stores so we can get it integrated.
+
+## How to use
+
+To use this feature, simply configure the variables shown below and then start your streetmerchant instance.
+
+## Testing
+
+You can test your notification configuration by running `npm run test:captcha`.
+
+???+ info
+    The test command will allow the user up to 30 seconds to enter a response before timing out. This is not directly configurable.
+
+## Configuration variables
+
+| Environment variable | Description |
+|---|---|
+| `CAPTCHA_HANDLER_POLL_INTERVAL` | Interval (in milliseconds) at which streetmerchant will check if the user has responded. Default: `5000` (5 seconds) |
+| `CAPTCHA_HANDLER_RESPONSE_TIMEOUT` | Timeout (in milliseconds) duration, after which streetmerchant will assume the user is unavailable and continue to the next page. Default: `300000` (5 minutes) |
+| `CAPTCHA_HANDLER_SERVICE` | [Supported messaging service](#supported-messaging-services) to use for the captcha handler |
+| `CAPTCHA_HANDLER_TOKEN` | Token to identify the bot user of the selected messaging service. See the [FAQ](#faq) for information on where to obtain this. |
+| `CAPTCHA_HANDLER_USER_ID` | ID representing _your account_ in the selected messaging service. The account specified here will receive the bot's DMs. See the [FAQ](#faq) for information on where to obtain this. |
+
+???+ info
+    The poll interval is 5 seconds so that the bot doesn't get rate-limited trying to check for responses (plus let's be honest, it's only 5 seconds at most).
+
+???+ info
+    While you can obviously adjust the response timeout to your liking, setting it to a high value is better. If you set it too low, you likely won't have time to respond before the bot moves on, and you will also get bombarded with DM notifications. If your bot runs into captcha pages without solving them, it will start to get flagged more frequently and eventually only get captcha pages. It's better to set a high timeout and solve it once, even if it stops the processing for a few minutes, rather than have to deal with multiple captchas anyway, but that's your call to make.
+
+## Supported messaging services
+
+| Service | Environment variable |
+|---|---|
+| Discord | `discord` |
+| Slack | `slack` |
+
+## FAQ
+
+### How do I obtain my user ID for <service name>?
+
+#### Slack
+
+In the top-right corner of the app, click your avatar and select "View Profile" from the dropdown. In the profile pane that appears, click the More button (3 dots) and then click "Copy member ID" from the dropdown.
+
+???+ attention
+    You have to use the desktop or web app to get this ID.
+
+#### Discord
+
+You have to enable Developer Mode in the Advanced settings. Once that's enabled, you can simply right-click your avatar in a thread or the member's list (on right side of screen) and click "Copy ID."
+
+### How do I obtain a token for <service name>?
+
+#### Slack
+
+Create an app [here](https://api.slack.com/apps) and copy the token you get once the setup is complete. Put the token in the dotenv file.
+
+???+ info
+    The app will need `chat:write`, `im:history`, `im:write`, and `reactions:write` permissions.
+
+#### Discord
+
+Create an app [here](https://discord.com/developers/applications) and copy the token, client ID, and permissions integer (I used `518208`). Then use the url [here](https://discord.com/developers/docs/topics/oauth2#bot-authorization-flow-url-example), replacing the `client_id` and `permissions` values with your own to add the bot to your server. Paste the token into your dotenv file.
+
+### The bot didn't send a message when I got a captcha page.
+
+That isn't a question. This is an FAQ.
+
+### The bot didn't send a message when I got a captcha page?
+
+Much better. This could either be a configuration error in streetmerchant (not completed, wrong values, etc) or the bot user isn't configured correctly in your messaging service. Double-check the configuration variables you've entered and use `npm run test:captcha` to help find out the root cause.
+
+### Why isn't captcha being detected on some of the stores I'm monitoring?
+
+Not sure, but we'll want to get that fixed! Submit an issue and we can look into it.
+
+### Will this work on every store?
+
+As it is, no. Currently this has only been tested/used for Amazon (North America) and this is where the vast majority of captcha complaints come from. If a store you're monitoring is giving your bot captcha pages, submit an issue so we can work on getting it integrated.

dotenv-example

@@ -13,6 +13,11 @@ APNS_PRODUCTION=
 APNS_TEAMID=
 AUTO_ADD_TO_CART=
 BROWSER_TRUSTED=
+CAPTCHA_HANDLER_POLL_INTERVAL=
+CAPTCHA_HANDLER_RESPONSE_TIMEOUT=
+CAPTCHA_HANDLER_SERVICE=
+CAPTCHA_HANDLER_TOKEN=
+CAPTCHA_HANDLER_USER_ID=
 DESKTOP_NOTIFICATIONS=
 DISCORD_NOTIFY_GROUP=
 DISCORD_NOTIFY_GROUP_3060=

package.json

@@ -16,7 +16,9 @@
     "start:production": "node build/src/index.js",
     "test": "c8 mocha 'build/test/**/test-*.js' --exclude 'build/test/functional/**/test-*.js'",
     "test:notification": "npm run compile && node build/test/functional/test-notification.js",
-    "test:notification:production": "node build/test/functional/test-notification.js"
+    "test:notification:production": "node build/test/functional/test-notification.js",
+    "test:captcha": "npm run compile &&  node build/test/functional/test-captcha.js",
+    "test:captcha:production": "node build/test/functional/test-captcha.js"
   },
   "engines": {
     "node": ">=12.0.0"

src/config.ts

@@ -16,6 +16,8 @@ if (process.env.npm_config_conf) {
   }
 } else if (existsSync(path.resolve(__dirname, '../../dotenv'))) {
   dotenv.config({path: path.resolve(__dirname, '../../dotenv')});
+} else if (existsSync(path.resolve(__dirname, '../dotenv'))) {
+  dotenv.config({path: path.resolve(__dirname, '../dotenv')});
 } else {
   dotenv.config({path: path.resolve(__dirname, '../../.env')});
 }
@@ -199,6 +201,17 @@ const browser = {
   userAgent: '',
 };
 
+const captchaHandler = {
+  pollInterval: envOrNumber(process.env.CAPTCHA_HANDLER_POLL_INTERVAL, 5000),
+  responseTimeout: envOrNumber(
+    process.env.CAPTCHA_HANDLER_RESPONSE_TIMEOUT,
+    300000
+  ),
+  service: envOrString(process.env.CAPTCHA_HANDLER_SERVICE),
+  token: envOrString(process.env.CAPTCHA_HANDLER_TOKEN),
+  userId: envOrString(process.env.CAPTCHA_HANDLER_USER_ID),
+};
+
 const docker = envOrBoolean(process.env.DOCKER, false);
 
 const logLevel = envOrString(process.env.LOG_LEVEL, 'info');
@@ -475,6 +488,7 @@ export const defaultStoreData = {
 
 export const config = {
   browser,
+  captchaHandler,
   docker,
   logLevel,
   notifications,

src/messaging/captcha.ts

@@ -0,0 +1,19 @@
+import {config} from '../config';
+import {getDiscordCaptchaInputAsync} from './discord';
+import {getSlackCaptchaInputAsync} from './slack';
+
+const {service} = config.captchaHandler;
+
+export async function getCaptchaInputAsync(
+  payload: string,
+  timeout?: number
+): Promise<string> {
+  switch (service) {
+    case 'discord':
+      return await getDiscordCaptchaInputAsync(payload, timeout);
+    case 'slack':
+      return await getSlackCaptchaInputAsync(payload, timeout);
+    default:
+      return '';
+  }
+}

src/messaging/discord.ts

@@ -0,0 +1,205 @@
+import {Link, Store} from '../store/model';
+import Discord from 'discord.js';
+import {config} from '../config';
+import {logger} from '../logger';
+
+const {notifyGroup, webhooks, notifyGroupSeries} = config.notifications.discord;
+const {pollInterval, responseTimeout, token, userId} = config.captchaHandler;
+
+let clientInstance: Discord.Client | undefined;
+let dmChannelInstance: Discord.DMChannel | undefined;
+
+function getIdAndToken(webhook: string) {
+  const match = /.*\/webhooks\/(\d+)\/(.+)/.exec(webhook);
+
+  if (!match) {
+    throw new Error('could not get discord webhook');
+  }
+
+  return {
+    id: match[1],
+    token: match[2],
+  };
+}
+
+export function sendDiscordMessage(link: Link, store: Store) {
+  if (webhooks.length > 0) {
+    logger.debug('↗ sending discord message');
+
+    (async () => {
+      try {
+        const embed = new Discord.MessageEmbed()
+          .setTitle('_**Stock alert!**_')
+          .setDescription(
+            '> provided by [streetmerchant](https://github.com/jef/streetmerchant) with :heart:'
+          )
+          .setThumbnail(
+            'https://raw.githubusercontent.com/jef/streetmerchant/main/docs/assets/images/streetmerchant-logo.png'
+          )
+          .setColor('#52b788')
+          .setTimestamp();
+
+        embed.addField('Store', store.name, true);
+        if (link.price)
+          embed.addField('Price', `${store.currency}${link.price}`, true);
+        embed.addField('Product Page', link.url);
+        if (link.cartUrl) embed.addField('Add to Cart', link.cartUrl);
+        embed.addField('Brand', link.brand, true);
+        embed.addField('Model', link.model, true);
+        embed.addField('Series', link.series, true);
+
+        embed.setTimestamp();
+
+        let notifyText: string[] = [];
+
+        if (notifyGroup) {
+          notifyText = notifyText.concat(notifyGroup);
+        }
+
+        const notifyKeys = Object.keys(notifyGroupSeries);
+        const notifyIndex = notifyKeys.indexOf(link.series);
+        if (notifyIndex !== -1) {
+          notifyText = notifyText.concat(
+            Object.values(notifyGroupSeries)[notifyIndex]
+          );
+        }
+
+        const promises = [];
+        for (const webhook of webhooks) {
+          const {id, token} = getIdAndToken(webhook);
+          const client = new Discord.WebhookClient(id, token);
+
+          promises.push(
+            new Promise((resolve, reject) => {
+              client
+                .send(notifyText.join(' '), {
+                  embeds: [embed],
+                  username: 'streetmerchant',
+                })
+                .then(resp => {
+                  logger.info('✔ discord message sent resp.id: ' + resp.id);
+                  resolve(resp);
+                })
+                .catch(err => reject(err))
+                .finally(() => client.destroy());
+            })
+          );
+        }
+
+        await Promise.all(promises).catch(err =>
+          logger.error("✖ couldn't send discord message", err)
+        );
+      } catch (error: unknown) {
+        logger.error("✖ couldn't send discord message", error);
+      }
+    })();
+  }
+}
+
+export async function sendDMAsync(
+  payload: string
+): Promise<Discord.Message | undefined> {
+  if (userId && token) {
+    logger.debug('↗ sending discord DM');
+    try {
+      const client = await getDiscordClientAsync();
+      const dmChannel = await getDMChannelAsync(client);
+      if (!dmChannel) {
+        logger.error('unable to get discord DM channel');
+        return;
+      }
+      const result = await dmChannel.send(payload);
+      logger.info('✔ discord DM sent');
+      return result;
+    } catch (error: unknown) {
+      logger.error("✖ couldn't send discord DM", error);
+    }
+  }
+  return;
+}
+
+export async function getDMResponseAsync(
+  botMessage: Discord.Message | undefined,
+  timeout: number
+): Promise<string> {
+  const iterations = Math.max(timeout / pollInterval, 1);
+  let iteration = 0;
+  const client = await getDiscordClientAsync();
+  const dmChannel = await getDMChannelAsync(client);
+  if (!dmChannel) {
+    logger.error('unable to get discord DM channel');
+    return '';
+  }
+  return new Promise(resolve => {
+    let response = '';
+    const intervalId = setInterval(async () => {
+      const finish = (result: string) => {
+        clearInterval(intervalId);
+        resolve(result);
+      };
+      try {
+        iteration++;
+        const messages = await dmChannel.messages.fetch({
+          after: botMessage?.id,
+        });
+        const lastUserMessage = messages
+          .filter(message => message.reference?.messageID === botMessage?.id)
+          .last();
+        if (!lastUserMessage) {
+          if (iteration >= iterations) {
+            await dmChannel.send('Timed out waiting for response... 😿');
+            logger.error('✖ no response from user');
+            return finish(response);
+          }
+        } else {
+          response = lastUserMessage.cleanContent;
+          lastUserMessage.react('✅');
+          logger.info(`✔ got captcha response: ${response}`);
+          return finish(response);
+        }
+      } catch (error: unknown) {
+        logger.error("✖ couldn't get captcha response", error);
+        return finish(response);
+      }
+    }, pollInterval);
+  });
+}
+
+export async function getDiscordCaptchaInputAsync(
+  payload: string,
+  timeout?: number
+): Promise<string> {
+  const message = await sendDMAsync(payload);
+  const response = await getDMResponseAsync(
+    message,
+    timeout || responseTimeout
+  );
+  closeClient();
+  return response;
+}
+
+function closeClient() {
+  if (clientInstance) {
+    clientInstance.destroy();
+    clientInstance = undefined;
+    dmChannelInstance = undefined;
+  }
+}
+
+async function getDiscordClientAsync() {
+  if (!clientInstance && token) {
+    clientInstance = new Discord.Client();
+    await clientInstance.login(token);
+  }
+  return clientInstance;
+}
+
+async function getDMChannelAsync(client?: Discord.Client) {
+  if (!dmChannelInstance && userId && !!client) {
+    const user = await new Discord.User(client, {
+      id: userId,
+    }).fetch();
+    dmChannelInstance = await user.createDM();
+  }
+  return dmChannelInstance;
+}

src/notification/index.ts → src/messaging/index.ts

@@ -1 +1,2 @@
+export * from './captcha';
 export * from './notification';