Bdellabe/awq modifier v3 #1177
Bdellabe/awq modifier v3 #1177
Conversation
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
| ) | ||
| with align_module_device(fc): | ||
| fc.weight.mul_(scales_view) | ||
| fc.weight.data = ( |
There was a problem hiding this comment.
Here you're updating a parameter, so you'll have to use update_offload_parameter
There was a problem hiding this comment.
updated
…LENGTH are very low Signed-off-by: Brian Dellabetta <[email protected]>
| if module in balance_layers: | ||
| module.weight.mul_(scales.view(1, -1).to(module.weight.device)) | ||
| elif module == smooth_layer: | ||
| if module.weight.ndim == 1: |
There was a problem hiding this comment.
Here you're updating a weight and bias, so you'll have to use update_offload_parameter
There was a problem hiding this comment.
we have to use update_offload_parameter for weights and biases too?
There was a problem hiding this comment.
weights and biases are parameters, so yes
There was a problem hiding this comment.
You can keep your inplace modification and just call
update_offload_parameter(module, "weight")
update_offload_parameter(module, "bias")
brian-dellabetta
force-pushed
the
bdellabe/awq-modifier-v3
branch
from
052ed7e to
9273ef3
Compare
Signed-off-by: Brian Dellabetta <[email protected]>
brian-dellabetta
force-pushed
the
bdellabe/awq-modifier-v3
branch
from
9273ef3 to
28f8bca
Compare
Signed-off-by: Brian Dellabetta <[email protected]>
|
|
||
| samples = [batch["input_ids"] for batch in dataloader] | ||
|
|
||
| samples = torch.cat(samples, dim=0) |
There was a problem hiding this comment.
Why are you putting all samples into the same batch. This is likely the source of your memory issues
There was a problem hiding this comment.
thanks for pointing that out! will toggle this and see if it affects anything
SUMMARY:
Draft of AWQModifier, replaces #181 and #824 (hence v3)
TEST PLAN:
"please outline how the changes were tested"