Batch link expansion queries #3106
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brucebolt
force-pushed
the
towers/batch-graphql-edition-queries
branch
3 times, most recently
from
3de40b0 to
179143e
Compare
brucebolt
force-pushed
the
towers/batch-graphql-edition-queries
branch
from
179143e to
fc6d38f
Compare
|
I tried combining the link set and edition links into a single query .... but ended up making 19 SQL queries instead of 15! I don't think the performance improvement will be significant by combining the two, so have left them as separate queries. |
brucebolt
force-pushed
the
towers/batch-graphql-edition-queries
branch
from
fc6d38f to
de1628b
Compare
yndajas
reviewed
Jan 30, 2025
We are filtering the links by the content store of the target document's edition, but no test was included for this. Adding a test prior to making changes that could affect this filtering.
Previously LinkedToEditionsSource would make one query per edition (getting linked editions, given a list of link types). This change makes one query per parent edition, by getting the dataloader to batch queries across multiple editions and link types. For the prime minister page, the number of database queries reduces from 93 to 15, and the ActiveRecord execution time decreases from ~100ms to ~20ms, when run locally. Co-authored-by: Richard Towers <[email protected]>
brucebolt
force-pushed
the
towers/batch-graphql-edition-queries
branch
from
de1628b to
cff1b0c
Compare
yndajas
approved these changes
Feb 3, 2025
| @@ -10,7 +10,7 @@ | |||
| create(:link, link_set: link_set, target_content_id: target_edition_3.content_id, link_type: "test_link") | |||
|
|
|||
| GraphQL::Dataloader.with_dataloading do |dataloader| | |||
| request = dataloader.with(described_class, parent_object: source_edition).request("test_link") | |||
| request = dataloader.with(described_class, content_store: source_edition.content_store).request([source_edition, "test_link"]) | |||
There was a problem hiding this comment.
question: are .request in this spec and .load in BaseObject both ending up at the fetch method in LinkedToEditionsSource? 🤯
There was a problem hiding this comment.
It appears to be, but I'm not sure why. The same is done is the other dataloader tests too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously
LinkedToEditionsSourcewould make one query per edition (getting linked editions, given a list of link types).We can do better than this by making it take a pair of edition and link type, and then execute a query like:
This allows the dataloader to batch up all the editions and requested link types at each level of depth before sending one big query, which dramatically reduces the number of SQL queries needed.
For the /government/prime-minister page on @brucebolt's machine, this reduces ActiveRecord time from ~100ms to ~20ms.
ActiveRecord doesn't seem to support the kind of tuple-inclusion query I'm making here. It does if all the columns are on the same table, but we need columns in different tables, so I've had to drop down to using Arel.
Trello card