Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix concurrent writes in the update operation #384

Merged
merged 5 commits into from
Oct 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ dist/

# IDEs
.idea
.vscode

# Decrypted private key we do not want to commit
.github/OSBotify-private-key.asc
67 changes: 41 additions & 26 deletions lib/Onyx.js
Original file line number Diff line number Diff line change
Expand Up @@ -1392,6 +1392,42 @@ function mergeCollection(collectionKey, collection) {
});
}

/**
* Internal recursive function to execute the functions in the correct order
*
* @param {Array} data An array of objects with shape {onyxMethod: oneOf('set', 'merge', 'mergeCollection', 'multiSet', 'clear'), key: string, value: *}
* @returns {Promise<Void>}
*/
function innerUpdate(data) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add JSDoc

if (data.length === 0) {
return Promise.resolve();
}

const {onyxMethod, key, value} = data.shift();
let promise = Promise.resolve();
switch (onyxMethod) {
case METHOD.SET:
promise = set(key, value);
break;
case METHOD.MERGE:
promise = merge(key, value);
break;
case METHOD.MERGE_COLLECTION:
promise = mergeCollection(key, value);
break;
case METHOD.MULTI_SET:
promise = multiSet(value);
break;
case METHOD.CLEAR:
promise = clear();
break;
default:
break;
}
Comment on lines +1408 to +1426
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this change possibly led to some regressions spotted in this PR and discussed in this Slack thread that updated the Onyx version


return promise.then(() => innerUpdate(data));
}

/**
* Insert API responses and lifecycle data into Onyx
*
Expand All @@ -1400,8 +1436,9 @@ function mergeCollection(collectionKey, collection) {
*/
function update(data) {
// First, validate the Onyx object is in the format we expect
const validMethods = [METHOD.CLEAR, METHOD.SET, METHOD.MERGE, METHOD.MERGE_COLLECTION, METHOD.MULTI_SET];
_.each(data, ({onyxMethod, key, value}) => {
if (!_.contains([METHOD.CLEAR, METHOD.SET, METHOD.MERGE, METHOD.MERGE_COLLECTION, METHOD.MULTI_SET], onyxMethod)) {
if (!_.contains(validMethods, onyxMethod)) {
throw new Error(`Invalid onyxMethod ${onyxMethod} in Onyx update.`);
}
if (onyxMethod === METHOD.MULTI_SET) {
Expand All @@ -1414,32 +1451,10 @@ function update(data) {
}
});

const promises = [];
let clearPromise = Promise.resolve();

_.each(data, ({onyxMethod, key, value}) => {
switch (onyxMethod) {
case METHOD.SET:
promises.push(() => set(key, value));
break;
case METHOD.MERGE:
promises.push(() => merge(key, value));
break;
case METHOD.MERGE_COLLECTION:
promises.push(() => mergeCollection(key, value));
break;
case METHOD.MULTI_SET:
promises.push(() => multiSet(value));
break;
case METHOD.CLEAR:
clearPromise = clear();
break;
default:
break;
}
});
// Put clear operation on top
data.sort(a => (a.onyxMethod === METHOD.CLEAR ? -1 : 1));

return clearPromise.then(() => Promise.all(_.map(promises, p => p())));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see we are undoing the previous logic to move the clear() to the front of the update queue. Does running the updates sync solve the issue that caused us to make that change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, probably not. I'll take care of it.

Copy link
Contributor Author

@ospfranco ospfranco Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure making sure clear runs first is a desired behavior but I guess now it is too late to change it, other parts of the system might rely on it. I added sorting to the list to make sure clear operations get executed first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree it is a bit of a weird hack. But changing it could have unexpected consequences. Thanks for updating.

return innerUpdate(data);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a data point, I tried the proposed solution by Marc which would try to resolve earlier while keeping the writes async, this however is not correct and broke a bunch of other tests.

}

/**
Expand Down
71 changes: 68 additions & 3 deletions tests/unit/onyxTest.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import _ from 'underscore';
import Onyx from '../../lib';
import waitForPromisesToResolve from '../utils/waitForPromisesToResolve';
import Storage from '../../lib/storage';

const ONYX_KEYS = {
TEST_KEY: 'test',
Expand Down Expand Up @@ -899,9 +900,9 @@ describe('Onyx', () => {
{onyxMethod: Onyx.METHOD.MERGE_COLLECTION, key: ONYX_KEYS.COLLECTION.TEST_UPDATE, value: {[itemKey]: {a: 'a'}}},
])
.then(() => {
expect(collectionCallback).toHaveBeenNthCalledWith(1, {[itemKey]: {a: 'a'}});
expect(testCallback).toHaveBeenNthCalledWith(1, 'taco', ONYX_KEYS.TEST_KEY);
expect(otherTestCallback).toHaveBeenNthCalledWith(1, 'pizza', ONYX_KEYS.OTHER_TEST);
expect(collectionCallback).toHaveBeenNthCalledWith(2, {[itemKey]: {a: 'a'}});
expect(testCallback).toHaveBeenNthCalledWith(2, 'taco', ONYX_KEYS.TEST_KEY);
expect(otherTestCallback).toHaveBeenNthCalledWith(2, 'pizza', ONYX_KEYS.OTHER_TEST);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This previously was receiving the write updates before the hydration callback, but now that the write operations take a little longer the order is correct.

Onyx.disconnect(connectionIDs);
});
});
Expand Down Expand Up @@ -960,4 +961,68 @@ describe('Onyx', () => {
});
});
});

it('should persist data in the correct order', () => {
const key = `${ONYX_KEYS.TEST_KEY}123`;
const callback = jest.fn();
connectionID = Onyx.connect({
key,
initWithStoredValues: false,
callback,
});

return waitForPromisesToResolve()
.then(() => Onyx.update([
{
onyxMethod: 'set',
key,
value: 'one',
},
{
onyxMethod: 'merge',
key,
value: 'two',
},
{
onyxMethod: 'set',
key,
value: 'three',
},
]))
.then(() => Storage.getItem(key))
.then((value) => {
expect(callback).toHaveBeenNthCalledWith(1, 'one', key);
expect(callback).toHaveBeenNthCalledWith(2, 'two', key);
expect(callback).toHaveBeenNthCalledWith(3, 'three', key);
expect(value).toBe('three');
});
});

it('should persist data in the correct order', () => {
const key = `${ONYX_KEYS.TEST_KEY}123`;
const callback = jest.fn();
connectionID = Onyx.connect({
key,
initWithStoredValues: false,
callback,
});

return waitForPromisesToResolve()
.then(() => Onyx.update([
{
onyxMethod: 'set',
key,
value: 'one',
},
{
onyxMethod: 'clear',
},

]))
.then(() => Storage.getItem(key))
.then((value) => {
expect(callback).toHaveBeenNthCalledWith(1, 'one', key);
expect(value).toBe('one');
});
});
});