[PP-2233] Prevent deadlocks in axis.import_identifiers and axis.reap_…

[dbernstein]

…collection

Description

This update does the same two things for the reap collections and import collections methods to try to address the deadlock issue.

The deadlocks seem to be arising because all the CM's axis collections are being reaped and imported in parallel. As a result when the data they are updating overlap - such us updates to an edition's publish - it is easy for one of many collections to get into a database deadlock conflict with another.

To try to address this problem I've done 3 things:

1. I've introduced a time delay of 10 seconds between firings of import_collection and reap_collection tasks in order to prevent them from executing too close to the same moment.
2. I've reduced the batch size from 25 to 10 items per task. So both types of task will process at most 10 items and then requeue the task.
3. I also added an OperationalError to ensure when there is a deadlock that the task is retried.

Also added some logging to see the elapsed time and the number of items processed in one message.

We may need to fine tune the subtask_execution_interval_in_secs and/or the batch size in order to find the right balance.

Motivation and Context

https://ebce-lyrasis.atlassian.net/browse/PP-2233

How Has This Been Tested?

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[codecov]

Codecov Report

Attention: Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.

Project coverage is 91.27%. Comparing base (a389d5a) to head (36ddbe7).

Files with missing lines	Patch %	Lines
src/palace/manager/celery/tasks/axis.py	94.11%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2330      +/-   ##
==========================================
- Coverage   91.28%   91.27%   -0.01%     
==========================================
  Files         364      364              
  Lines       41317    41313       -4     
  Branches     8870     8869       -1     
==========================================
- Hits        37715    37710       -5     
  Misses       2340     2340              
- Partials     1262     1263       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.