WIP: Accelerator refactor

[justusschock]

What does this PR do?

Fixes #4510

This PR separates the Accelerator (Hardware Part) from the actual different training routines.

Remaining TODOs:

• TPUAccelerator (Thomas)
• DDP2 Plugin (Adrian)
• Shared Training Plugin (Sean)
• RPC Plugin (Sean)
• Add Layers for parallel: SingleProcessMultipleDevice (SPMD), SingleProcessSingleDevice (SPSD)
• Rebase on release branch
• Testing DDP2 and DDP Slurm
• Remove old plugins (pl/plugins/old)
• Make Tuner work (requires setting some attrs through trainer) (Adrian)

So far this PR was co-authored with @awaelchli !

cc @awaelchli @tchaton @SeanNaren who will likely work on this!

Slides for motivation and high-level overview

List of PRs to look out for (when rebasing, code we need to manually copy over to new files):
#5221, #5195, #5300, #5388

[Borda]

👍 ... similar to trainer new attributes _distrib_type and _device_type
on the other hand, it will be breaking change... :/

[Borda]

this is refactored to _distrib_type but we can rename it if you want...

[awaelchli]

Yes I know, we need to update the branch with these changes. It's a bit challenging now with keeping uptodate. But we will keep it in mind thanks

[Borda]

probably no need for an update as you will cut this work to multiple PRs, just for you to be aware of it... :]

[Borda]

could we have some high-level summary, I see that accelerators are for HW (as it is in the description) but some summary of how the plugins (I would move them on the same level as accelerators as there re rather partners not sub-category) are linked, and can you have "any" combination?

[awaelchli]

We made some slides (see link in description above) for a high level overview. For some plugins and accelerators multiple combinations are possible, but some configurations are obviously not. So for our built-in modes, we take care of valid configuration. But the user can define their own accelerator and compatible plugins if they want to, or re-use our built-in ones.

[pep8speaks]

Hello @justusschock! Thanks for updating this PR.

• In the file pytorch_lightning/accelerators/accelerator.py:

Line 212:26: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/accelerator_connector.py:

Line 63:18: W291 trailing whitespace
Line 186:33: E128 continuation line under-indented for visual indent
Line 187:33: E128 continuation line under-indented for visual indent

• In the file pytorch_lightning/accelerators/cpu.py:

Line 14:45: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/gpu.py:

Line 25:37: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/base_plugin.py:

Line 4:1: E302 expected 2 blank lines, found 1
Line 31:14: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/precision/apex_amp.py:

Line 12:1: E302 expected 2 blank lines, found 1
Line 115:53: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/precision/mixed.py:

Line 4:1: E302 expected 2 blank lines, found 1
Line 7:24: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/precision/native_amp.py:

Line 48:40: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/training_type/ddp2.py:

Line 4:1: E302 expected 2 blank lines, found 1
Line 5:9: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/training_type/dp.py:

Line 8:1: E302 expected 2 blank lines, found 1
Line 44:27: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/training_type/parallel.py:

Line 17:1: E302 expected 2 blank lines, found 1
Line 91:23: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/training_type/sharded_spawn.py:

Line 39:5: E303 too many blank lines (2)

• In the file pytorch_lightning/accelerators/plugins/training_type/single_device.py:

Line 20:1: W293 blank line contains whitespace
Line 40:19: W292 no newline at end of file

• In the file pytorch_lightning/accelerators/plugins/training_type/training_type_plugin.py:

Line 11:1: E302 expected 2 blank lines, found 1

• In the file pytorch_lightning/plugins/__init__.py:

Line 1:53: W292 no newline at end of file

• In the file pytorch_lightning/plugins/old/plugin_connector.py:

Line 38:9: E265 block comment should start with '# '

• In the file pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:

Line 224:1: E305 expected 2 blank lines after class or function definition, found 0
Line 224:3: E227 missing whitespace around bitwise or shift operator
Line 224:7: E225 missing whitespace around operator
Line 227:3: E225 missing whitespace around operator
Line 227:5: E225 missing whitespace around operator
Line 227:7: E225 missing whitespace around operator
Line 228:9: E113 unexpected indentation
Line 230:7: E225 missing whitespace around operator

• In the file pytorch_lightning/trainer/properties.py:

Line 427:121: E501 line too long (121 > 120 characters)

• In the file tests/callbacks/test_callbacks.py:

Line 112:1: E122 continuation line missing indentation or outdented
Line 112:3: E225 missing whitespace around operator
Line 112:5: E227 missing whitespace around bitwise or shift operator
Line 115:1: E122 continuation line missing indentation or outdented
Line 115:3: E225 missing whitespace around operator
Line 115:5: E225 missing whitespace around operator
Line 115:7: E225 missing whitespace around operator
Line 115:8: E251 unexpected spaces around keyword / parameter equals
Line 116:1: E122 continuation line missing indentation or outdented
Line 116:1: W503 line break before binary operator
Line 116:7: E225 missing whitespace around operator

• In the file tests/models/test_amp.py:

Line 152:121: E501 line too long (121 > 120 characters)

Comment last updated at 2021-01-22 09:54:14 UTC