forked from checkpoint-restore/criu
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathcriu-amdgpu-plugin.txt
108 lines (82 loc) · 2.92 KB
/
criu-amdgpu-plugin.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
ROCM Support(1)
===============
NAME
----
criu-amdgpu-plugin - A plugin extension to CRIU to support checkpoint/restore in
userspace for AMD GPUs.
CURRENT SUPPORT
---------------
Single and Multi GPU systems (Gfx9)
Checkpoint / Restore on different system
Checkpoint / Restore inside a docker container
Pytorch
Tensorflow
Using CRIU Image Streamer
DESCRIPTION
-----------
Though *criu* is a great tool for checkpointing and restoring running
applications, it has certain limitations such as it cannot handle
applications that have device files open. In order to support *ROCm* based
workloads with *criu* we need to augment criu's core functionality with a
plugin based extension mechanism. *criu-amdgpu-plugin* provides the necessary support
to criu to allow Checkpoint / Restore with ROCm.
Dependencies
~~~~~~~~~~~~~~
*amdkfd support*::
In order to snapshot the *VRAM* and other *GPU* device states, we require
an updated version of amdkfd(amdgpu) driver. The kernel patches are under
review currently.
*criu 3.16*::
This work is rebased on latest criu release available at this time.
OPTIONS
-------
Optional parameters can be passed in as environment variables before
executing criu command.
*KFD_FW_VER_CHECK*::
Enable or disable firmware version check.
If enabled, firmware version on restored gpu needs to be greater than or
equal firmware version on checkpointed GPU. Default:Enabled
E.g:
KFD_FW_VER_CHECK=0
*KFD_SDMA_FW_VER_CHECK*::
Enable or disable SDMA firmware version check.
If enabled, SDMA firmware version on restored gpu needs to be greater than or
equal firmware version on checkpointed GPU. Default:Enabled
E.g:
KFD_SDMA_FW_VER_CHECK=0
*KFD_CACHES_COUNT_CHECK*::
Enable or disable caches count check. If enabled, the caches count on
restored GPU needs to be greater than or equal caches count on checkpointed
GPU. Default:Enabled
E.g:
KFD_CACHES_COUNT_CHECK=0
*KFD_NUM_GWS_CHECK*::
Enable or disable num_gws check. If enabled, the num_gws on
restored GPU needs to be greater than or equal num_gws on checkpointed
GPU. Default:Enabled
E.g:
KFD_NUM_GWS_CHECK=0
*KFD_VRAM_SIZE_CHECK*::
Enable or disable VRAM size check. If enabled, the VRAM size on
restored GPU needs to be greater than or equal VRAM size on checkpointed
GPU. Default:Enabled
E.g:
KFD_VRAM_SIZE_CHECK=0
*KFD_NUMA_CHECK*::
Enable or disable NUMA CPU region check. If enabled, the plugin will restore
GPUs that belong to one CPU NUMA region to the same CPU NUMA region.
Default:Enabled
E.g:
KFD_NUMA_CHECK=1
*KFD_CAPABILITY_CHECK*::
Enable or disable capability check. If enabled, the capability on
restored GPU needs to be equal to the capability on the checkpointed GPU.
Default:Enabled
E.g:
KFD_CAPABILITY_CHECK=1
AUTHOR
------
The AMDKFD team.
COPYRIGHT
---------
Copyright \(C) 2020-2021, Advanced Micro Devices, Inc. (AMD)