How to return async

[cyberwombat]

I've been successful wrapping my eval'd return but not if it's a promise.

const fn = `(async () => { return Promise.resolve(123);})()`
let s = await isolate.compileScript(
    `new _ivm.Reference(eval(${JSON.stringify(fn)}))`
 )
 const b = await s.run(context)
 const r = await b.copy()
 console.log(r) //  TypeError: #<Promise> could not be cloned.

How can I achieve this? Or is it even possible if functions are not cloneable? One alternative I thought of if that is the case is to handle the return within the isolated code and use an isolated function to return.

Suggestions?

[cyberwombat]

Ok ended up doing that. Created a reply function similar to the log example and it let's me pass the promise return.

[lokey]

@cyberwombat I am stuck in a similar place. Can you please share your code on how you made it work?

[cyberwombat]

@lokey - It's still a work in progress but if you look at the example code the author has with the log module I did it the same way

So I set a reply with a reference to an function which for is just a promise handler

 jail.setSync(
      '_reply',
      new ivm.Reference(async function(args) {
        if (args instanceof Error) {
          reject(args)
        } else resolve(args)
      })
    )

Then inside the script compile I set the reply as a global - I may change this part because I actually want to abstract the 'reply' function from user running the code but not sure how yet. This is taken from the 'log' example.

    let bootstrap = await isolate.compileScript(
      'new ' +
        function() {
          let ivm = _ivm
           _ivm = null

          //let log = _log
          //_log = null

          let reply = _reply
          _reply = null

          global.reply = function(...args) {
            reply.apply(
              undefined,
              args.map(arg => new ivm.ExternalCopy(arg).copyInto())
            )
          }
        }
    )

    ....

Then further down I compile my code to run in vm as a module

    // Some code
    const source = `export default async function(event) {
      return 'foo'
    }`

    const handlerModule =  isolate.compileModuleSync(
      source
    )

    handlerModule.evaluateSync()
    
    // I do other things here like loading other modules - see one of the tests for how to do modules

    // Then I wrap it all - so the handler is the untrusted code. This abstracts the return handling allowing the code to just return a promise and not worry about 'reply'
    const code = `
      import handler from './handler';
     ;(async () => {
       try {
        reply( await handler(${JSON.stringify(event)}))
       } catch(e) {
         reply(e.toString())
       }
     })();
    `

Hope that helps. If you look at the example with the log and the test with the modules I think it will help. I am still not sure this is the best way but it works.

[laverdet]

You could do something like this:

const ivm = require('ivm');
const isolate = new ivm.Isolate;
const context = isolate.createContextSync();
isolate.compileScriptSync('function untrusted() { return Promise.resolve(123); }').runSync(context);
context.setSync('_ivm', ivm);
const forward = isolate.compileScriptSync(`(function(ivm) {
  delete _ivm;
  return ivm.Reference(function forward(promise, resolve, reject) {
    promise.deref().then(
      value => resolve.applyIgnored(undefined, [ value ]),
      error => reject.applyIgnored(undefined, [ ivm.ExternalCopy(error).copyInto() ])
    );
  });
})(_ivm);`).runSync(context);
const promise = new Promise(async (resolve, reject) => {
  const fn = await context.global.get('untrusted');
  const promiseRef = await context.global.apply(undefined, []);
  forward.applyIgnored(undefined, [ promiseRef, new ivm.Reference(resolve), new ivm.Reference(reject) ]);
});

promise.then(value => console.log(value));

[cyberwombat]

@laverdet thanks for sample code. I'm having a little issue running it. I fixed a few things - like getting ref to context.global instead of context and applying the fn function which I think was a typo but while this runs without error it doesn't return the value.

const ivm = require('isolated-vm')
const isolate = new ivm.Isolate()
const context = isolate.createContextSync()

async function runCode() {
  isolate
    .compileScriptSync('function untrusted() { return Promise.resolve(123); }')
    .runSync(context)

  let jail = context.global
  jail.setSync('_ivm', ivm)

  const forward = isolate
    .compileScriptSync(
      `(function(ivm) {
  delete _ivm;
  return new ivm.Reference(function forward(promise, resolve, reject) {
    
    promise.deref().then(
      value => resolve.applyIgnored(undefined, [ value ]),
      error => reject.applyIgnored(undefined, [ ivm.ExternalCopy(error).copyInto() ])
    );
  });
})(_ivm);`
    )
    .runSync(context)

  const fn = await jail.get('untrusted')
  const promiseRef = await fn.apply(undefined, [])
  const promise = new Promise((resolve, reject) => {
    forward.applyIgnored(undefined, [
      promiseRef,
      new ivm.Reference(resolve),
      new ivm.Reference(reject)
    ])
  })

  return promise
}
;(async () => {
  try {
    console.log(await runCode())
  } catch (e) {
    console.log(e)
  }
})()

[cyberwombat]

Weird - so I rebuild my node_modules and now it gives an error. TypeError: A non-transferable value was passed. Tried running latest gh code w no success. I am on node 12.6.0 and Catalina.

[cyberwombat]

@lokey - were you able to get example working?

[FranklinWaller]

I ran into the same issue but after an hour of tinkering and understanding the mechanics, I got all my use cases working. (Running an asynchronous entry point & calling an asynchronous method outside isolate and returning its value) For the people who are also looking on how to use asynchronous methods here you go:

const code = byteArrayToString(binary); is executing this:

const bin = `
        async function main() {
            log("Hello world");

            const result = await storageStore('Key', 'Value');

            log("storageStore message: ", result);

            return 123;
        }
    `;

import { Results } from "../context";
import byteArrayToString from "../../../utils/byteArrayToString";
// Did not yet look into the TypeScript issues i was having with my own build setup
import Ivm from 'isolated-vm';
// __non_webpack_require___ can be replaced with require
const ivm = __non_webpack_require__('isolated-vm');

export default async function executeJsCode(binary: Uint8Array): Promise<Results> {
    const code = byteArrayToString(binary);
    const isolate: Ivm.Isolate = new ivm.Isolate({ memoryLimit: 128 });
    const context: Ivm.Context = await isolate.createContext();
    const jail = context.global;

    jail.setSync('global', jail.derefInto());
    jail.setSync('_log', new ivm.Reference((...args: any[]) => {
        console.log('[Log]: ', ...args);
    }));

    jail.setSync('_storageStore', new ivm.Reference(async (key: string, value: string, resolve: Ivm.Reference<any>) => {
        console.log('Setting key & value pair ->', key, value);

        setTimeout(() => {
            resolve.applyIgnored(undefined, [
                new ivm.ExternalCopy('I came from storageStore').copyInto(),
            ]);
        }, 1000);
    }));

    jail.setSync('_ivm', ivm);

    const bootstrap = await isolate.compileScript(`new function() {
        let ivm = _ivm;
        delete _ivm;

        let log = _log;
        delete _log;

        let storageStore = _storageStore;
        delete _storageStore;

        global.log = (...args) => {
            log.applyIgnored(undefined, args.map(arg => new ivm.ExternalCopy(arg).copyInto()));
        }

        global.storageStore = (key, value) => {
            return new Promise((resolve) => {
                storageStore.applyIgnored(
                    undefined,
                    [new ivm.ExternalCopy(key).copyInto(), new ivm.ExternalCopy(value).copyInto(), new ivm.Reference(resolve)]
                );
            });
        }

        return new ivm.Reference(function forwardMainPromise(mainFunc, resolve) {
            const derefMainFunc = mainFunc.deref();

            derefMainFunc().then((value) => {
                resolve.applyIgnored(undefined, [
                    new ivm.ExternalCopy(value).copyInto(),
                ]);
            });
        });
    }`);

    const bootstrapScriptResult: Ivm.Reference<any> = await bootstrap.run(context);

    const script = await isolate.compileScript(code);
    await script.run(context);
    const mainFunc = await jail.get('main');

    const executionPromise = new Promise(async (resolve) => {
        await bootstrapScriptResult.apply(undefined, [
            mainFunc,
            new ivm.Reference(resolve),
        ]);
    });

    const result = await executionPromise;

    console.log('Done -> ', result);

    return null;
}

Which gave me the following output:

[Log]:  Hello world
Setting key & value pair -> Key Value
[Log]:  storageStore message:  I came from storageStore
Done ->  123

[OrkhanAlikhanov]

After a sleepless night here is what I ended up doing.

Update: I couldn't implement proper timeouts around this, will update if I achieve.

Let's say I want to call main method of following thrid-party script:
AsyncRunner is a global object with a run async method on it.

async function main(sleeper, { sleep }) {
  await sleeper;
  console.log(await AsyncRunner.run('test'));
  await sleep(2000);
  return 'hello from main';
}

I call executeJsCode(code) where code is above script.
executeJsCode is defined as follows:

import ivm from 'isolated-vm';
import { serialize, serializationScript, deserialize } from './serialization';

// language=JavaScript
const bootsrap = `(function () {
  const ivm = _ivm;
  ${serializationScript}
  console = deserialize(console);
  AsyncRunner = deserialize(AsyncRunner);
  
  delete _ivm;
  delete globalThis;
})()`;

async function sleep(amount) {
  return new Promise(resolve => {
    console.log(`Sleeping for ${amount}ms`);
    setTimeout(() => {
      console.log(`Slept for ${amount}ms`);
      resolve();
    }, amount);
  });
}

export default async function executeJsCode(code) {
  const inspector = false;
  const isolate = new ivm.Isolate({
    memoryLimit: 128,
    inspector
  });

  const context = isolate.createContextSync({ inspector });
  const jail = context.global;
  jail.setSync('_ivm', ivm);
  jail.setSync('console', serialize({
    log(...args) {
      console.log('[Log]: ', ...args);
    }
  }));

  jail.setSync('AsyncRunner', serialize({
    async run(...args) {
      console.log('[AsyncRunner Running]: ', ...args);
      await sleep(1000);
      console.log('[AsyncRunner Done]: ', ...args);
      return 'AsyncRunner did it work?';
    }
  }));

  /// bootstrap
  isolate.compileScriptSync(bootsrap).runSync(context);

  /// compile script
  isolate.compileScriptSync(code).runSync(context);

  const obtainRefToMain = `(function(){ const ivm = _ivm; ${serializationScript}; return serialize(main)})()`;
  jail.setSync('_ivm', ivm);
  /// obtain reference to main
  const main = deserialize(isolate.compileScriptSync(obtainRefToMain).runSync(context));

  /// Run it just like any other function
  const result = await main(sleep(1000), { sleep });
  console.log('Done -> ', result);
  return result;
}

Click to see serialization.js

/// serialization.js
import ivm from 'isolated-vm';

const errorKey = '______isError____';
export function serialize(v) {
  if (typeof v === 'function') {
    return new ivm.Reference((...args) => serialize(v.apply(undefined, args.map(deserialize))));
  }

  if (Array.isArray(v)) {
    return new ivm.ExternalCopy(v.map(serialize)).copyInto();
  }

  /// represent promise as thenable object
  /// no need to deserialize, it's possible to await a thenable
  if (v instanceof Promise) {
    return serialize({
      then: v.then.bind(v)
    });
  }

  /// represent error object.
  if (v instanceof Error) {
    return serialize({
      [errorKey]: true,
      ...v,
      stack: v.stack,
      message: v.message
    });
  }

  if (typeof v === 'object') {
    return new ivm.ExternalCopy(Object.entries(v).reduce((obj, [key, v]) => (obj[key] = serialize(v), obj), {})).copyInto();
  }

  /// primitive?
  return v;
}

export function deserialize(v) {
  if (v instanceof ivm.Reference && v.typeof === 'function') {
    return (...args) => deserialize(v.applySync(undefined, args.map(serialize)));
  }

  if (Array.isArray(v)) {
    return v.map(deserialize);
  }

  if (typeof v === 'object') {
    /// Deserialize error into Error instance
    if (v && errorKey in v) {
      delete v[errorKey];
      return Object.assign(new Error(), deserialize(v));
    }

    /// no need to deserialize Promises
    return Object.entries(v).reduce((obj, [key, v]) => (obj[key] = deserialize(v), obj), {});
  }

  /// primitive?
  return v;
}

// just string version of the exact same code above
// language=JavaScript
export const serializationScript = `
const errorKey = '______isError____';
function serialize(v) {
  if (typeof v === 'function') {
    return new ivm.Reference((...args) => serialize(v.apply(undefined, args.map(deserialize))));
  }

  if (Array.isArray(v)) {
    return new ivm.ExternalCopy(v.map(serialize)).copyInto();
  }

  /// represent promise as thenable object
  /// no need to deserialize, it's possible to await a thenable
  if (v instanceof Promise) {
    return serialize({
      then: v.then.bind(v)
    });
  }

  /// represent error object.
  if (v instanceof Error) {
    return serialize({
      [errorKey]: true,
      ...v,
      stack: v.stack,
      message: v.message
    });
  }

  if (typeof v === 'object') {
    return new ivm.ExternalCopy(Object.entries(v).reduce((obj, [key, v]) => (obj[key] = serialize(v), obj), {})).copyInto();
  }

  /// primitive?
  return v;
}

function deserialize(v) {
  if (v instanceof ivm.Reference && v.typeof === 'function') {
    return (...args) => deserialize(v.applySync(undefined, args.map(serialize)));
  }

  if (Array.isArray(v)) {
    return v.map(deserialize);
  }

  if (typeof v === 'object') {
    /// Deserialize error into Error instance
    if (v && errorKey in v) {
      delete v[errorKey];
      return Object.assign(new Error(), deserialize(v));
    }

    /// no need to deserialize Promises
    return Object.entries(v).reduce((obj, [key, v]) => (obj[key] = deserialize(v), obj), {});
  }

  /// primitive?
  return v;
}`;

The log after running script

Sleeping for 1000ms
Slept for 1000ms
[AsyncRunner Running]:  test
Sleeping for 1000ms
Slept for 1000ms
[AsyncRunner Done]:  test
[Log]:  AsyncRunner did it work?
Sleeping for 2000ms
Slept for 2000ms
Done ->  hello from main

---

Explanation

serialize and deserialize are the methods that should be available in both sides, they are essentially helping us to pass objects, functions, promises, errors between isolates. Promise and Error are not transferable, so we need to pass around objects that represent them. Error or its subclass instances are serialized into object with a key where we deserialize an object having that key into new Error instance on the other side. Promises are only serialized into thenables, there is no need to deserialize them (do deserialize it into a new Promise otherwise, catch method will not be available) as await keyword also works with thenable objects.

[laverdet]

I wanted to comment on this with some thoughts. The API in place right now is very powerful and fully expressive, but it's also cumbersome. I'd like to make it easier to get started with things like this so I'm leaving this open as reminder to put work into that.

Specifically I'd like some options to automatically forward promises, and an option to automatically wrap and unwrap ExternalCopy and Reference instances.

[cyberwombat]

@laverdet thank you. And if you get a chance I would love for you to take a look at the sample code you provided earlier - I never got it to work so thats been on pause for me. Probably missing something simple but can't figure it out. Just not returning values.

[laverdet]

You can modify the example to this and it will work as expected. Note that if your promise resolves a non-transferable value you'll need to wrap it in ExternalCopy or Reference

const ivm = require('isolated-vm')
const isolate = new ivm.Isolate()
const context = isolate.createContextSync()

async function runCode() {
  isolate
    .compileScriptSync('function untrusted() { return Promise.resolve(123); }')
    .runSync(context)

  let jail = context.global
  jail.setSync('_ivm', ivm)

  const forward = isolate
    .compileScriptSync(
      `(function(ivm) {
  delete _ivm;
  return new ivm.Reference(function forward(fn, resolve, reject) {
    
    fn().then(
      value => resolve.applyIgnored(undefined, [ value ]),
      error => reject.applyIgnored(undefined, [ ivm.ExternalCopy(error).copyInto() ])
    );
  });
})(_ivm);`
    )
    .runSync(context)

  const fn = await jail.get('untrusted')
  const promise = new Promise((resolve, reject) => {
    forward.apply(undefined, [
      fn.derefInto(),
      new ivm.Reference(resolve),
      new ivm.Reference(reject)
    ])
  })

  return promise
}
;(async () => {
  try {
    console.log(await runCode())
  } catch (e) {
    console.log(e)
  }
})()

[cyberwombat]

Wow I would not have gotten that. Thank you!

[laverdet]

I just pushed 3.0.0 to npm with some new features which improve situations like this. The previous example could be rewritten like this:

const ivm = require('isolated-vm')
const isolate = new ivm.Isolate()
const context = isolate.createContextSync()

async function runCode() {
  const fn = await context.eval('(function untrusted() { return Promise.resolve(123); })', { reference: true })
  return fn.result.apply(undefined, [], { result: { promise: true } })
}
runCode().then(value => console.log(value))
  .catch(error => console.error(error))

Let me know if it works out for you~

[cyberwombat]

Thank you @laverdet that works great. I see that eval is a sort of one stop for compile/run and was able to recreate your example with both methods. In my project i however need modules and I am having difficulties recreating the above with modules. Basically I would like this to be my code:

'export default async function() { return Promise.resolve('123'); }) .}

How can I run this function when exporting? When I evaluate this after compile/instantiate i returns undefined and not sure how to get access to the function.