Fix inefficient queries in legacy search API #1878
Conversation
Improve legacy search API performance by replacing filters against many-to-many relationships from `SELECT DISTINCT` and `JOIN` query to `WHERE .. IN` query. Signed-off-by: Alexander Saprykin <[email protected]>
| platforms__in=models.Platform.objects.filter(name__in=platforms)) | ||
| platforms_qs = models.Content.objects.only('pk').filter( | ||
| platforms__name__in=platforms) | ||
| return queryset.filter(pk__in=platforms_qs) |
There was a problem hiding this comment.
Is writing it this appreciably different in the resulting SQL from
return queryset.filter(platforms__name__in=platforms)
There was a problem hiding this comment.
This call queryset.filter(platforms__name__in=platforms) generates SQL JOIN query, for example:
SELECT ...
FROM main_content
JOIN main_content_platforms ON ...
JOIN main_platforms ON ...
WHERE main_platforms.name IN (...)
Such JOIN query on many-to-many relationship creates row duplicates, that then can be eliminated by DISTINCT \ DISTINCT ON. Applying dumb .distinct() call on a queryset leads to generating SELECT DISTINCT over all fields in query, which for this select query can be large (considering selecting fields from related tables).
One possible solution is to select records and join with subquery that returns unique content ids, but Django ORM doesn't provide an obvious interface for custom joins.
Another solution, we use in search, is making WHERE 'id' IN (SELECT ...) query, which has similar performance to JOIN with unique IDs.
Improve legacy search API performance by replacing
filters against many-to-many relationships from
SELECT DISTINCTandJOINqueryto
WHERE .. INquery.Signed-off-by: Alexander Saprykin [email protected]
Fixes: #1876