[BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships #5230

runspired · 2017-10-15T19:51:49Z

Once reviewed/approved, this should be backported to 2.14, 2.15, and 2.16 beta.

Resolves #5037 #5055 #5208 #5209 #5211 #5267 #5268

Problem

Ember Data 2.14 introduced lazy-relationship parsing; however, this parsing used left-side/right-side keying that was incompatible with polymorphic relationships. This resulted in payloads for newly encountered polymorphic types causing all previous payloads to be dereferenced.

In addition, a number of bugs were discovered where we were incorrectly setting up RelationshipPayloads as if there was an inverse when there was none, or where the inverse resolved to the key :. This PR fixes that from the perspective of RelationshipPayloads but part of the problem was a bug in our calculation of inverseFor which is resolved with #5261

Solution

This PR resolves these issues with three primary changes:

relationship-info is now calculated and stored separately from the RelationshipPayloads objects. This not only should be a performance win for us by enabling us to avoid calculating out the info in several scenarios where we would have needed to repeatedly redo the work before, but helps to resolve lookup for scenarios where there is no inverse or the inverse is polymorphic. Additionally it cleans up many areas of the lazy-relationship code substantially to have this separation of concerns.
RelationshipPayloads are now always cached on the manager by the resolved base-model-name instead of model-name. This enables us to confidently always utilize the correct RelationshipPayloads even when encountering a polymorphic type.
The payloads held for a relationship in RelationshipPayloads are now stored in a two-level cache by model-name and id instead of a single-level cache of just id. This enables situations in which polymorphic siblings do not share a common unique-id constraint to work properly.

Tests

one-side polymorphic belongsTo / hasMany
both-sides polymorphic belongsTo / hasMany
multi-table polymorphism (e.g. when IDs are not unique across all polymorphic types)
reflexive polymorphic relationships
self-referential non-reflexive polymorphic relationships

bmac

lgtm

hjdivad

Looks great @runspired

Just noting the todos we talked about OOB

hjdivad · 2017-10-19T19:06:08Z

addon/-private/system/relationships/relationship-payloads.js

+      inverseRelationshipMeta.options !== undefined &&
+      inverseRelationshipMeta.options.polymorphic === true;
+
+    assert(`Both side of a relationship cannot be polymorphic`, !this._lhsIsPolymorphic || !this._rhsIsPolymorphic);


as we discussed out of band, it's fine for both sides to be polymorphic

hjdivad · 2017-10-19T19:11:28Z

addon/-private/system/relationships/relationship-payloads-manager.js

@@ -205,6 +209,22 @@ export default class RelationshipPayloadsManager {
    let lhsKey = `${modelName}:${relationshipName}`;
    let rhsKey = `${inverseModelName}:${inverseRelationshipName}`;


as discussed out of band, in both cases we want the key to be the model name defined by the inverse relationship, when an inverse relationship exists

so we should always create RelationshipPayloads with the base name and then add the actual model name

hjdivad · 2017-10-19T19:12:19Z

addon/-private/system/relationships/relationship-payloads.js

@@ -44,10 +44,32 @@ export default class RelationshipPayloads {
  constructor(store, modelName, relationshipName, relationshipMeta, inverseModelName, inverseRelationshipName, inverseRelationshipMeta) {


maybe s/modelName/baseModelName or something here as we want this to always be the base model name

hjdivad · 2017-10-19T19:14:11Z

addon/-private/system/relationships/relationship-payloads.js

@@ -126,14 +159,28 @@ export default class RelationshipPayloads {
    }
  }

+  _isSide(potentialMatches, modelName, relationshipName) {


hjdivad · 2017-10-19T19:26:32Z

tests/unit/system/relationships/relationship-payload-manager-test.js

@@ -674,3 +678,72 @@ test('get can retrieve payloads with self-links in reflexive relationships', fun
  let entry = this.relationshipPayloadsManager.get('user', 1, 'friends');
  assert.deepEqual(entry, friendsPayload, 'self-link in reflexive relationship');
 });
+
+test('handles relationships where one side is polymorphic', function(assert) {


as discussed out of band, need more tests

push subtype then push basetype

push basetype then push subtype

both sides poly

Any way I can keep this moving? Is it possible for me to merge tests into this feature branch? Never tried that before

@danielspaniel you can open pull requests against runspired/data to branch fix-polymporphic-payloads and when they're merged it'll update this pr. I've done it before and it works surprisingly well.

@danielspaniel if you have time to contribute tests for this that would be super helpful. Thanks ^_^

I have all tests ready but @runspired added this to RelationshipPayloads constructor:

assert(`Both side of a relationship cannot be polymorphic`, !this._lhsIsPolymorphic || !this._rhsIsPolymorphic);

That is showstopper, since previously in ember data you could do that .. so I think that assertion is perhaps guarding agains something I don't know what.

But I removed that assertion and all ember-data tests passed EXCEPT the new test I wrote for "both sides polymorphic", so maybe that had something to do with it .. not sure

This feature ( lazy relationship loading ) is super complex and was implemented with very limited tests in the first place.
I am thinking it might be better to dump the feature, rather than keep patching it.

Was this your feature @hjdivad ? Is there a great value to it. If there is I am not seeing it.
Would you be up for dumping it/ removing it?

So, if a feature is not done properly and introduces very serious bugs, are you telling me you are reluctant to back out the feature because you would lose the nice performance gains?
That makes no sense.
First off, whoever wrote the feature should take responsibility and either fix it or remove it.
I am willing to help out as much as I can, but the way it was done really looks like spaghetti to me, so I can't do much to save it.
Secondly, if I wrote this feature, I would be embarrassed to keep it, if it broke ember-data. Does not really matter what kind of fun it offers if it is broken.

How long should this broken feature sit in ember-data and just kill off polymorphic relationship users?
Is it right to just wait around for some other work someone else is doing? Who knows how long that other thing Igor is working on will be done. Seems irrelevant to me, especially since he is not involved in this discussion much. It has only been you, me , Chris and Stephan ( saying he liked the original bug posting )

@danielspaniel Sorry for being a bit quiet. Last week when @hjdivad and I discussed this PR OOB we discussed that assert. Polymorphic to Polymorphic in ED has always been very buggy, and I wasn't sure it really worked at all and I thought it'd be too difficult to achieve here; however, in that conversation David and I also arrived at a solution for supporting it. So long as we always resolve to the baseModelName on both sides, the approach this PR outlines will work for both sides being polymorphic as well. I will update the PR with that approach.

@danielspaniel can you PR the tests you've added to my branch?

sure thing .. gimme a few moments

I submitted PR to your branch of ED just now

hjdivad · 2017-10-24T14:33:36Z

@runspired possibly also #5208

runspired · 2017-10-27T05:01:15Z

Updated to handle polymorphic-to-polymorphic relationships and with the help of @danielspaniel we now have more tests. I would like to clean up the testing a bit more but this is ready for another review pass by @hjdivad.

The only piece that concerns me is that we now MUST lookup the inverse greedily in a situation we did not need to before here: https://github.com/emberjs/data/pull/5230/files#diff-1c53e096d92ba3a647478f3f51185a01R178

I'm also sure now that we've got an implementation that covers the appropriate cases that this is due for some further refactoring and simplification.

BryanCrotaz · 2017-10-27T06:06:02Z

Haven't looked at source in detail but have you tested with multiple levels of model extension?

E.g. A extends B extends C, with relationships introduced at each level?

runspired · 2017-10-27T06:14:18Z

@BryanCrotaz those cases introduce no additional complexity or scenarios. A relationship must explicitly declare that it is polymorphic, and it must use the base type's type when doing so.

BryanCrotaz · 2017-10-27T10:51:48Z

@runspired :) will try it out

danielspaniel · 2017-10-29T15:58:22Z

This fix is still not complete enough for polymorphic relationships. I found that polymorphic self referential relationships don't work .. 2 options

I can add the failing tests to this branch, @runspired can send more brainwaves at the problem
or
we can just give up now, rip out this lazy relationships manager and save everyone who loves polymorphism from dying an early death from alcoholism

runspired · 2017-10-29T16:07:57Z

Added more failing tests for the polymorphic self-reflexive case both with and without full-linkage in the push. Full linkage already worked, incomplete linkage was fixed by fixing the isReflexive test to use the baseModelName instead of the given modelName when setting up the RelationshipPayloads instance.

…elationships

…verage for it

danielspaniel · 2017-10-29T18:55:41Z

I guess you beat me to it .. I was working on it but you added a whole ton of self-reflexive tests that were the ones that were breaking for me

danielspaniel · 2017-10-29T19:03:21Z

actually check out my test ( in the PR ) .. it still fails ( and I just rebased )

runspired · 2017-10-29T22:59:01Z

I'm afk but reading those asserts I believe they assert the incorrect thing.

runspired · 2017-10-30T15:41:09Z

@danielspaniel ported and fixed up your test asserts, found the issue was with self-referential but non-reflexive relationships (as you setup). Added fix.

danielspaniel · 2017-10-30T18:54:19Z

Your getting ever so very close to fixing all this @runspired .. you are still hanging in there which is pretty awesome.

ok .. how's about this for some philosophical ramblings.

lets say we have polymorphic hats and the data is like this:

  const bigHatData = { data: { id: 1, type: "big-hat", attributes: { type: "big-hat" }, relationships: { "user": { "data": { "type": "user", "id": "1" } } } } };
  const smallHatData = { data: { id: 1, type: "small-hat", attributes: { type: "small-hat" }, relationships: { "user": { "data": { "type": "user", "id": "1" } } } } };

upon close inspection one sees that the two models have the same id ( 1 ) but they are different sub-types.

If you push the user and then these two hats .. the test to get the user hats fails ( only finds 1 .. not 2 )

yours/our tests are doing this

  const bigHatData = { data: { id: 1, type: "big-hat", attributes: { type: "big-hat" }, relationships: { "user": { "data": { "type": "user", "id": "1" } } } } };
  const smallHatData = { data: { id: 2, type: "small-hat", attributes: { type: "small-hat" }, relationships: { "user": { "data": { "type": "user", "id": "1" } } } } };

using a different id for every subtype and therefore they pass .. but I think that the first way should pass as well .. because it used to in the past and I think polymorphic models can have same id for different type

you want me to PR this failing test ?

or you want to give up now and get a free carton of fig newtons and whatever is behind curtain number 2

runspired · 2017-10-30T19:25:59Z

This is a hard one because this case has actually never worked properly. It seems to work on the surface but has long plagued us as a source of bug reports.

danielspaniel · 2017-10-30T19:29:45Z

I hear you, but I not sure what to tell the databases that are storing the polymorphic data ( different tables ) this way .. since they going to laugh at me if I ask for different id's tomorrow

runspired · 2017-10-30T19:31:16Z

@danielspaniel I would push some tests for this case, I will see what I can do but can't promise much atm.

runspired · 2017-10-30T19:35:02Z

@danielspaniel just a thought, but worst case it would be pretty easy for you to rectify the ID conflicts client-side by normalizing them to ${modelName}-${id} on the way in and then stripping the modelName while serializing on the way out.

danielspaniel · 2017-10-30T19:39:05Z

I guess I could, but so would everyone else .. and currently without the lazy relationship thing, if you just create stuff like that , it works .. since create bypasses the loading lazy thing-a-majigger.

igorT · 2017-11-22T07:44:10Z

addon/-private/system/relationships/relationship-payloads.js

 /**
-  Manages the payloads for both sides of a single relationship, across all model
-  instances.
+ Manages the payloads for both sides of a single relationship, across all model


Are the whitespace changes on purpose?

BryanCrotaz · 2017-11-30T02:20:18Z

Does this overlap with my PR #5268? It seems as though we're fixing the same problem in very different ways.

runspired · 2017-11-30T02:32:59Z

@BryanCrotaz 100% overlap

BryanCrotaz · 2017-11-30T02:57:52Z

My solution seems simpler but then I think I must therefore be missing something...

runspired · 2017-11-30T02:58:38Z

Check the various test cases added :P

runspired · 2017-12-05T22:36:50Z

addon/-private/system/relationships/relationship-payloads.js

  unload(modelName, id, relationshipName) {
    this._flushPending();

    if (this._isLHS(modelName, relationshipName)) {
-      delete this._lhsPayloads[id];
+      delete this.lhs_payloads.delete(modelName, id);


this is a mistake?

seems sketch this passes tests

runspired · 2017-12-05T22:38:57Z

addon/-private/system/relationships/relationship-payloads.js

-    @method
-  */
+   @method
+   */
  _isLHS(modelName, relationshipName) {


Can you please rewrite/add to the comments here

runspired · 2017-12-05T22:39:15Z

addon/-private/system/relationships/relationship-payloads.js

+    if (isRelationship === true) {
+      return isSelfReferential === true || // itself
+        modelName === relInfo.lhs_baseModelName || // base or non-polymorphic
+        relInfo.lhs_modelNames.indexOf(modelName) !== -1; // polymorphic


please add a comment here

danielspaniel · 2017-12-31T22:08:17Z

I guess this never made it to 2.18 .. any idea when this might get merged ?

cibernox · 2018-01-03T17:09:03Z

I believe I'm hitting this. Is there any more work to be done?

BryanCrotaz · 2018-02-07T21:37:29Z

Has this stalled? What’s left to do? Can we all help?

BryanCrotaz · 2017-11-30T12:02:04Z

addon/-private/system/relationships/relationship-payloads-manager.js

+    inverseCache.set(modelName, relationshipName, info);
+
+    // Greedily populate the inverse
+    inverseCache.set(inverseBaseModelName, inverseRelationshipName, info);


in #5268 I did this (line 284-288) by only using baseModelName as a cache key - I think that would mean that you don't have to greedily populate the inverse - all polymorphic types would use the same key.

@BryanCrotaz that turns out to be error prone

(I originally did that too)

Why can't the world just be perfect? Gah.

BryanCrotaz · 2018-02-28T21:17:54Z

Congrats. This is a biggie.

…elationships (emberjs#5230) * [BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships (cherry picked from commit b93b3e7)

…elationships (#5230) * [BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships (cherry picked from commit b93b3e7)

bmac · 2018-03-05T14:44:14Z

I've back-ported this fix to 2.14.11, 2.15.4, 2.16.4, 2.17.1, 2.18.2, and 3.0.2.

danielspaniel · 2018-03-05T14:48:54Z

This was a good weekend. Thanks for the merge and backports.

bmac approved these changes Oct 16, 2017

View reviewed changes

hjdivad suggested changes Oct 19, 2017

View reviewed changes

runspired mentioned this pull request Oct 27, 2017

test: pushing data with included payload, multiple has-many pointing … #5208

Closed

runspired and others added 6 commits October 29, 2017 10:53

[BUGFIX] enable lazy-relationship payloads to work with polymorphic r…

6ce71be

…elationships

remove trailing comma

5fd015d

added more tests for lazy relationship push with polymorphics

65b2562

add more tests and fix things

2ba9af4

fix tests

009c125

[FIX] enable polymorphic self-reflexive relationships and add test co…

b8589d6

…verage for it

runspired force-pushed the fix-polymporphic-payloads branch from 4c24628 to b8589d6 Compare October 29, 2017 18:43

runspired added 3 commits October 30, 2017 08:41

[FIX] self-referential polymorphic but non-reflexive relationships

e137112

Begin test cleanup for polymorphic tests

f3f0323

cleanup tests for self-referential non-reflexive relationships

b953a79

runspired added 3 commits November 20, 2017 20:25

fix the things

0f22ccb

cleanup some naming

cb4c533

fix prod tests

cdb7b17

igorT reviewed Nov 22, 2017

View reviewed changes

This was referenced Nov 30, 2017

inverse relationships not set up when pushing jsonapi blob with polymorphic model #5267

Closed

Fix missing inverse #5268

Closed

runspired commented Dec 5, 2017

View reviewed changes

kjhangiani mentioned this pull request Jan 4, 2018

Polymorphic relationships that are sideloaded do not materialize the relationship the first time they are pushed into the store (unless the relationship has been "primed" with a .get beforehand) #5313

Closed

runspired mentioned this pull request Feb 7, 2018

[BUGFIX] Fix issues with links #5257

Closed

igorT merged commit b93b3e7 into emberjs:master Feb 28, 2018

BryanCrotaz reviewed Feb 28, 2018

View reviewed changes

cursedquail mentioned this pull request Mar 1, 2018

Possible race condition in relationships #5295

Closed

bmac pushed a commit that referenced this pull request Mar 1, 2018

[BUGFIX] enable lazy-relationship payloads to work with polymorphic r…

c8332ca

…elationships (#5230) * [BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships (cherry picked from commit b93b3e7)

This was referenced Mar 5, 2018

Links related relationship on modelA nulls belongsTo on modelB #5209

Closed

A bidirectional relationship parent.children becomes hasLoaded: true when a child that references the parent is pushed to the store #5211

Closed

kennethlarsen mentioned this pull request Mar 26, 2018

Ember 3.1 Release Blog Post emberjs/website#3230

Merged

13 tasks

		@@ -205,6 +209,22 @@ export default class RelationshipPayloadsManager {
		let lhsKey = `${modelName}:${relationshipName}`;
		let rhsKey = `${inverseModelName}:${inverseRelationshipName}`;

		@@ -44,10 +44,32 @@ export default class RelationshipPayloads {
		constructor(store, modelName, relationshipName, relationshipMeta, inverseModelName, inverseRelationshipName, inverseRelationshipMeta) {

[BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships #5230

[BUGFIX] enable lazy-relationship payloads to work with polymorphic relationships #5230

Conversation

runspired commented Oct 15, 2017 • edited Loading

bmac left a comment

Choose a reason for hiding this comment

hjdivad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielspaniel Oct 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjdivad commented Oct 24, 2017

runspired commented Oct 27, 2017

BryanCrotaz commented Oct 27, 2017

runspired commented Oct 27, 2017

BryanCrotaz commented Oct 27, 2017

danielspaniel commented Oct 29, 2017

runspired commented Oct 29, 2017 • edited Loading

danielspaniel commented Oct 29, 2017

danielspaniel commented Oct 29, 2017 • edited Loading

runspired commented Oct 29, 2017

runspired commented Oct 30, 2017

danielspaniel commented Oct 30, 2017

runspired commented Oct 30, 2017

danielspaniel commented Oct 30, 2017 • edited Loading

runspired commented Oct 30, 2017

runspired commented Oct 30, 2017

danielspaniel commented Oct 30, 2017

Choose a reason for hiding this comment

BryanCrotaz commented Nov 30, 2017

runspired commented Nov 30, 2017

BryanCrotaz commented Nov 30, 2017

runspired commented Nov 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielspaniel commented Dec 31, 2017

cibernox commented Jan 3, 2018

BryanCrotaz commented Feb 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BryanCrotaz commented Feb 28, 2018

bmac commented Mar 5, 2018

danielspaniel commented Mar 5, 2018

runspired commented Oct 15, 2017 •

edited

Loading

danielspaniel Oct 26, 2017 •

edited

Loading

runspired commented Oct 29, 2017 •

edited

Loading

danielspaniel commented Oct 29, 2017 •

edited

Loading

danielspaniel commented Oct 30, 2017 •

edited

Loading