RemixJS - Cloudflare Server Adapter - Sentry Wrapper

[jcpaulco]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which package are you using?

@sentry/remix

SDK Version

7.11.0

Framework Version

Remix-Run: 1.6.7 & React 17.0.2

Link to Sentry event

No response

Steps to Reproduce

1. Opted to use Cloudflare workers as opposed to Remix's built in app server
2. Followed Sentry integration instructions
3. Sentry.captureException(new Error("Error) is only being reported to Sentry if in primary tsx component, Not in the loader / action functions.

It looks like if Express is used as the server adapter as opposed to cloudflare-workers then there exists the wrapExpressCreateRequestHandler to handle it but nothing exists for the cloudflare-workers.

See current server.js below utilizing cloudflare-workers:

import { createEventHandler } from "@remix-run/cloudflare-workers";
import * as build from "@remix-run/dev/server-build";

addEventListener(
  "fetch",
  createEventHandler({ build, mode: process.env.NODE_ENV })
);

Current Sentry.init in app/entry.server.tsx:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  tracesSampleRate: 0.2,
});

Expected Result

Sentry capture exceptions should be reported in the server code as well.

Actual Result

No sentry event is reported from the server, only client.

[AbhiPrasad]

Hey, There is currently no support for cloudflare workers since the Sentry SDK does not officially support the cloudflare worker runtime. Backlogging for the team, but PRs welcome in the meantime!

[AbhiPrasad]

Related: #2484

[huw]

Spent the morning on a workaround until we have proper WinterCG support. It’s a first pass, largely hacky, and relying on what we have in Toucan today.

First, copy-paste this into server/instrumentBuild.ts (create it next to server/index.ts):

import { type EntryContext, type HandleDocumentRequestFunction, type ServerBuild } from "@remix-run/cloudflare";
import { extractData, type AppData } from "@remix-run/react/dist/data";
import { isResponse } from "@remix-run/server-runtime/dist/responses";
import { type ActionFunction, type LoaderFunction } from "@remix-run/server-runtime/dist/routeModules";
import { type Exception, type WrappedFunction } from "@sentry/types";
import { fill, normalize } from "@sentry/utils";
import * as cookie from "cookie";
import type Toucan from "toucan-js";
import { type Options } from "toucan-js";

const extractResponseError = async (response: Response): Promise<unknown> => {
  const responseData = (await extractData(response)) as unknown;

  if (typeof responseData === "string") {
    return responseData;
  }

  if (response.statusText) {
    return response.statusText;
  }

  return responseData;
};

const captureRemixServerException = async (
  sentry: Toucan,
  error: unknown,
  name: string,
  request: Request
): Promise<void> => {
  // Skip capturing if the thrown error is not a 5xx response
  // https://remix.run/docs/en/v1/api/conventions#throwing-responses-in-loaders
  if (isResponse(error) && error.status < 500) {
    return;
  }

  const exception = isResponse(error) ? await extractResponseError(error) : error;

  sentry.withScope((scope) => {
    // Transformed & handled later in `beforeSend`
    // Attempts to match https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/remix/src/utils/instrumentServer.ts#L107
    scope.setExtra("request", request);
    scope.setExtra("exceptionMechanism", {
      type: "instrument",
      handled: true,
      data: {
        function: name,
      },
    });

    sentry.captureException(exception);
  });
};

const makeWrappedDocumentRequestFunction =
  (sentry: Toucan) =>
  (origDocumentRequestFunction: HandleDocumentRequestFunction): HandleDocumentRequestFunction => {
    return async function (
      this: unknown,
      request: Request,
      responseStatusCode: number,
      responseHeaders: Headers,
      context: EntryContext
    ): Promise<Response> {
      let response: Response;

      try {
        // eslint-disable-next-line @babel/no-invalid-this -- We need to be able to pass `this` to the original function to wrap it correctly.
        response = await origDocumentRequestFunction.call(this, request, responseStatusCode, responseHeaders, context);
      } catch (error) {
        await captureRemixServerException(sentry, error, "documentRequest", request);
        throw error;
      }

      return response;
    };
  };

interface MakeWrappedDataFunction {
  (sentry: Toucan, dataFunctionType: "action", origFunction: ActionFunction): ActionFunction;
  (sentry: Toucan, dataFunctionType: "loader", origFunction: LoaderFunction): LoaderFunction;
}

const makeWrappedDataFunction: MakeWrappedDataFunction = (sentry: Toucan, dataFunctionType, origFunction) => {
  return async function (this: unknown, args: Parameters<typeof origFunction>[0]): Promise<Response | AppData> {
    let response: unknown;

    try {
      // eslint-disable-next-line @babel/no-invalid-this -- We need to be able to pass `this` to the original function to wrap it correctly.
      response = (await origFunction.call(this, args)) as unknown;
    } catch (error) {
      await captureRemixServerException(sentry, error, dataFunctionType, args.request);
      throw error;
    }

    return response;
  };
};

const makeWrappedAction =
  (sentry: Toucan) =>
  (origAction: ActionFunction): ActionFunction => {
    return makeWrappedDataFunction(sentry, "action", origAction);
  };

const makeWrappedLoader =
  (sentry: Toucan) =>
  (origLoader: LoaderFunction): LoaderFunction => {
    return makeWrappedDataFunction(sentry, "loader", origLoader);
  };

export const beforeSend: Options["beforeSend"] = (event) => {
  // Add data from the `request` extra to the event request object.
  // Mostly adapted from https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/node/src/requestdata.ts#L140
  if (event.extra?.request) {
    const request = event.extra.request as Request;
    const headers = Object.fromEntries(request.headers.entries());
    const cookieHeader = headers.cookie;
    delete headers.cookie;

    event.request = {
      ...event.request,
      url: request.url,
      method: request.method,
      data: normalize(request.body) as unknown,
      query_string: new URL(request.url).search.replace("?", ""),
      cookies: (cookieHeader && cookie.parse(cookieHeader)) || {},
      // Add more to the allowlist later if you'd like.
      headers: { user_agent: headers["user-agent"] },
    };

    delete event.extra.request;
  }

  // Add the exception mechanism from extra to the event object.
  // Adapted from https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/utils/src/misc.ts#L93
  if (event.extra?.exceptionMechanism) {
    const firstException = event.exception?.values?.[0];
    const newMechanism = event.extra?.exceptionMechanism as Exception["mechanism"];

    if (firstException) {
      const defaultMechanism = { type: "generic", handled: true };
      const currentMechanism = firstException.mechanism;
      firstException.mechanism = { ...defaultMechanism, ...currentMechanism, ...newMechanism };

      if (newMechanism && "data" in newMechanism) {
        const mergedData = { ...currentMechanism?.data, ...newMechanism?.data };
        firstException.mechanism.data = mergedData;
      }
    }

    delete event.extra.exceptionMechanism;
  }

  return event;
};

/**
 * Instruments `remix` ServerBuild for performance tracing and error tracking.
 */
export const instrumentBuild = (sentry: Toucan, build: ServerBuild): ServerBuild => {
  const routes: ServerBuild["routes"] = {};

  const wrappedEntry = { ...build.entry, module: { ...build.entry.module } };

  // Not keeping boolean flags like it's done for `requestHandler` functions,
  // Because the build can change between build and runtime.
  // So if there is a new `loader` or`action` or `documentRequest` after build.
  // We should be able to wrap them, as they may not be wrapped before.
  if (!(wrappedEntry.module.default as WrappedFunction).__sentry_original__) {
    fill(wrappedEntry.module, "default", makeWrappedDocumentRequestFunction(sentry));
  }

  for (const [id, route] of Object.entries(build.routes)) {
    const wrappedRoute = { ...route, module: { ...route.module } };

    if (wrappedRoute.module.action && !(wrappedRoute.module.action as WrappedFunction).__sentry_original__) {
      fill(wrappedRoute.module, "action", makeWrappedAction(sentry));
    }

    if (wrappedRoute.module.loader && !(wrappedRoute.module.loader as WrappedFunction).__sentry_original__) {
      fill(wrappedRoute.module, "loader", makeWrappedLoader(sentry));
    }

    routes[id] = wrappedRoute;
  }

  return { ...build, routes, entry: wrappedEntry };
};

export default instrumentBuild;

If you’re keeping score at home, this modifies instrumentServer.ts to use Toucan instead of @sentry/node. It drops anything that Toucan doesn’t natively support, which means you’ll lose some metadata about what action/loader was run as well as all tracing support.

Then modify server/index.ts:

import { createEventHandler } from "@remix-run/cloudflare-workers";
import * as build from "@remix-run/dev/server-build";
import Toucan from "toucan-js";
import instrumentBuild, { beforeSend } from "./instrumentBuild";

addEventListener("fetch", (event: FetchEvent) => {
  const sentry = new Toucan({
    dsn: SENTRY_DSN,
    context: event,
    beforeSend,
    // `allowedHeaders` and `allowedSearchParams` are ignored, see `beforeSend` in `instrumentBuild.ts`
    // allowedHeaders: ["user-agent"],
    // allowedSearchParams: /(.*)/u,
  });

  createEventHandler({
    build: instrumentBuild(sentry, build),
    mode: process.env.NODE_ENV,
  })(event);
});

Hope this helps anyone stuck on this one ^_^ I’ll try and patch it up a bit further to get metadata & tracing in.

[huw]

I finally got around to updating the above thanks to Toucan v3. Everything’s in this gist, but essentially you need to:

1. Rewrite instrumentBuild to accept a passed-through Hub instance, rather than using getCurrentHub() (including, perniciously, in helper functions such as hasTracingEnabled())
2. In server/index.ts, create a root transaction from the current route’s name and wrap the loaders and actions in spans via instrumentBuild
3. In root.tsx, pass through your tracing & baggage into the meta function
4. In root.tsx, include the current domain in tracePropagationTargets (because Remix fetchers will fetch from the entire URL rather than / root, which will confuse Sentry)

One major caveat is that server-side timing will be extremely inaccurate. Cloudflare freeze Date.now() on the server to avoid side-channel timing attacks. This means that within Workers, the time will never increment during a single request, and in Durable Objects, it will increment with the time of the last I/O. Relative times of multiple worker invocations during a single trace will be accurate, and so will anything measured on the client-side.

If anyone stumbles across this in the future and is having trouble with the code gimme a yell. (And if Sentry are interested I’ll happily modify the existing code to support Workers, but it’ll be a decent refactor because of getCurrentHub()).

[huw]

I’ve updated my Gist for Toucan v3.2.0 and Remix v2. It’s a lot more complicated & hacky now thanks to Remix v2, but still very possible. Please reply there if you have any questions :)

[huw]

@AbhiPrasad (or others) I’d like to start work on contributing back some stuff that should make non-Node runtime support a bit easier, but I wanted to run it by the team first in case you have better ideas than me :) (I know you’re working on Deno next—unsure how much of that would be reusable here)

The main change I’d like to start with would be (1) exporting the utility functions such as instrumentBuild and (2) refactoring the utility functions to accept getCurrentHub as an argument. This would allow Cloudflare Workers & Toucan users to run ex.:

const hub = new Toucan({ … });
const getCurrentHub = () => hub;
const newBuild = instrumentBuild(getCurrentHub, build);

This would go 90% of the way to preventing me from having to vendor & maintain your code in my repo while remaining fully backwards-compatible with the existing implementation. Would you be open to this?

[AbhiPrasad]

We have #9225 - feel free to take a look at that.

(2) refactoring the utility functions to accept getCurrentHub as an argument

What might be the easiest is just to change the behaviour of getCurrentHub by defining your own async context strategy. Something that looks like so https://github.com/getsentry/sentry-javascript/blob/develop/packages/vercel-edge/src/async.ts

[AbhiPrasad]

@geemanjs amazing writeup! We probably will go ahead an add workers export condition to the remix sdk to make this work.

@onurtemizkan I know you touched upon this before, now that we have @sentry/cloudflare how complex do you think it'll be to move forward with this?

[charlie-bud]

We probably will go ahead an add workers export condition to the remix sdk to make this work.

@AbhiPrasad Would this also cover fixing the sourcemaps, mentioned in @geemanjs's write up?

[AbhiPrasad]

Would this also cover fixing the sourcemaps

Nope, but that requires help from cloudflare, there's not much we can do alone here.

[charlie-bud]

Hey @AbhiPrasad - Happy New Year 🎊

I appreciate you mention there is not much Sentry can do alone here. In the interest of trying to reach out to the right people to get a resolution, how do you feel this issue would be best resolved? Is is something that would require patching by Sentry and other package maintainers e.g. Cloudflare? Or just Cloudflare? Any insight or direction you can give would be great, our Sentry errors have long been reporting incorrect stack traces which makes it very difficult to debug issues. 🙏

[JoshMatthew]

@AbhiPrasad I am trying to run Sentry with Shopify Hydrogen + Remix + Vite. I tried following this docs https://docs.sentry.io/platforms/javascript/guides/remix/frameworks/hydrogen/ to allow CloudFlare Workers. I believe the docs is outdated or have a typo because it tries to tell us to use this command twice npx @sentry/wizard@latest -i remix which I think is not correct. Also the removeAsyncHooksPlugin() custom plugin seems to not work as I still get this error Error: [o2:runtime] External modules are not supported: "node:async_hooks"

[s1gr1d]

Hy, Abhi is currently not available and will be back end of February. But thanks for pointing out the issue in the docs, we will take a look at this!

edit: fixed it getsentry/sentry-docs#12608

[charlie-bud]

Hey @s1gr1d 👋 Was wondering if you might be able to take a look into this comment? 🙏

It would be great to at least understand a path to resolution as server-side errors continue to be a blind spot for monitoring Cloudflare-deployed Remix/React Router V7 apps 🤞

[s1gr1d]

From what I can understand reading the thread here:

• fixing source maps: will need help from Cloudflare (comment here). But as I am not too deep into the topic of what Cloudflare would provide, I cannot say much more here.
• fixing error reporting: as @AbhiPrasad mentioned, we could add workers export condition (comment here)

We have this on our mind and will come back to you with more information once we know more ourselves.