Reducing Archivist content to increase performance and creating an archive

[HayleyMills]

The performance of the main Archivist it slow because of the amount of content in there. The plan is to have one Archivist act as an archive which contains everything which is in Discovery, then have smaller instances which only contain work-in-progress.

• Clone main Archivist instance to hold Discovery content as an archive @spuddybike
• Load all Discovery instruments from all instances into new archive instance
    - [ ] check list of exported content from Archivist (Jenny's script output)
    - [ ] use Jenny's script to load instruments, (do we want to also move the datasets and mappings, the final version of these are on the dev server?) @jli755
• Rename main Archivist to 'ucl' for wip (~80) @spuddybike
• Rename alspac to 'lps' for wip (~95) @spuddybike
• Export and re-load remaining wip instruments, datasets and mappings @HayleyMills
    - [ ] from genscotland, heaf, onsls, wirral to 'lps'
    - [ ] from alspac to 'lps'
    - [ ] BCS, NCDS, NS, MCS, NSHD to remain in 'ucl'
    - [ ] ALSPAC, WHII, SWS, HCS, from main to 'lps'
• Update Airtable with correct IDs @HayleyMills (need a list of before and after IDs)

[spuddybike]

is there a problem with this approach with the update questionnaires into Discovery

[HayleyMills]

@spuddybike can I check this process above is still the plan or are we simply saving all files in Discovery, rather than moving them to another instance?

[spuddybike]

I think we should export the files and have that as a backup  as we will just end up having an unusable instance if we have an Archivist archive instance. We should I think put it on the S drive. JonOn 21 May 2024, at 09:56, HayleyMills ***@***.***> wrote: @spuddybike can I check this process above is still the plan or are we simply saving all files in Discovery, rather than moving them to another instance? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[HayleyMills]

• Export all questionnaire and mapping .txt files from main Archivists as a backup - Jenny using script
• Export all questionnaire and mapping .txt files from other Archivists as a backup - Jenny using script
• Create list of Discovery questionnaires to be deleted
• Delete all Discovery questionnaires from main Archivist - Jenny using script
• Delete all Discovery questionnaires from other Archivists - @jli755 using script
• Rename main Archivist to 'ucl' for wip (~80) @spuddybike
• Rename alspac to 'lps' for wip (~95) @spuddybike
• Export and re-load remaining wip instruments, datasets and mappings @HayleyMills
  • from genscotland to 'lps' @beckyoldroyd
  • from heaf to 'lps'
  • from onsls, to 'lps'
  • from wirral to 'lps' @beckyoldroyd
  • BCS, NCDS, NS, MCS, NSHD to remain in 'ucl'
  • ALSPAC, WHII, SWS, HCS, from main to 'lps'
  • BIB from heaf to 'lps' @beckyoldroyd
• Update Airtable with correct IDs @HayleyMills @beckyoldroyd

@spuddybike just checking we are only doing this for the instrument xml - No also need to export the .txt files too.

[HayleyMills]

To archive (S:\IOECLOSER_Share\USP\Software\Archivist_instrument_backup), then delete after content update

delete_after_content_update.txt

[spuddybike]

performance still not great - looking at database bloat/vacuum

[simonreed]

@spuddybike a significant amount of the database space is the storage of documents used for import/export. I wonder if we should look to delete those beyond a certain timeframe?