Changes to reduce memory footprint in Aggregator

[snehal-das]

Changes made:

• Explicit deletion of local variables reduces the run-time footprint as the garbage collector
• Moved the measurement of memory to after tensor_db::clean_up() to get a clearer picture of memory usage.
• Added garbage collection at the end of round post db clean_up to free up unused memory.

[theakshaypant]

Would also suggest running formatting script as the pipeline is failing.

sh scripts/format.sh

[rahulga1]

@snehal-das can you run it with memory_callback with large number of rounds to see the impact of this change?

[MasterSkepticista]

@snehal-das Does this PR "fix" memory leak once and for all, or it only optimizes memory usage?

[snehal-das]

can you run it with memory_callback with large number of rounds to see the impact of this change?

@rahulga1 I've run these changes for 100-200 rounds with torch/mnist and torch/histology tests. I've also tested it with 32 collaborators to evaluate the impact of the change.

Does this PR "fix" memory leak once and for all, or it only optimizes memory usage?

@MasterSkepticista The current state of code does not fix memory issues. It only reduces the footprint of the aggregator, but there is still a growth in overall memory usage with time, albeit at a slower rate. There are a couple more approaches being considered for this:

1. Reducing memory fragmentation: Tracking fordblks show that memory fragmentation is increasing proportional to the number of rounds.
2. Delegating memory intensive tasks to child processes - Return the memory allocated at the time of execution back to the allocator.