[Feature][All] Truncate non-domain tables

[sstojak1]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

As a developer, I want to avoid having excessively large raw/tool tables in DevLake, as they consume significant storage and may impact performance. I would prefer a mechanism to manage these tables efficiently, such as automatic deletion or truncation of old records, to keep the system optimized.

Description

Is it possible for DevLake to automatically delete or truncate non-domain tables? My understanding is that raw/tool tables are only required to transform data into domain tables. Once the data is transformed and pushed to domain tables, older records (beyond the most recent import) are no longer used.
We’re importing a large number of Jira boards (around 100) frequently, which is causing these raw tables to grow significantly. This issue isn't limited to Jira but applies to all tools supported by DevLake.
I’ve attached a screenshot showing an overview of the sizes for all our tables for reference.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct