Post training quantization support in TRTorch #44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
CIFAR10 for ptq example Gets about 90-91% accuracy, initial LR 0.01, dropout 0.15 Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
- Couple assorted fixes in conversion implementation - Set up the space to have phase specific settings inside the compiler - PTQ Calibrator implementation moved to the public API, means Python will need its own but it probably did anyway - PTQ now works with dataloader and all the overrides for Calibration algorithm work - CIFAR10 Dataloader implementation - Application still has bugs in reporting accuracy and reading from calibration cache Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
responsibility of the user to transfer data to GPU and ensure types are correct. Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
test set for calibration Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
narendasan
added
the
WIP
- Now creates output tensors of the correct type to accept data - There still may be a data race in the creation of the dataloader iterator - Quantization and Dynamic Shape right now don't play well together, potential subsequent release of TRT may address this Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
address datarace issue Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
narendasan
removed
the
WIP
narendasan
changed the title
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
narendasan
commented
Apr 24, 2020
core/conversion/conversion.cpp
Outdated
| @@ -235,7 +235,7 @@ bool VerifyConverterSupportForBlock(const torch::jit::Block* b) { | |||
| if (!OpSupported(n)) { | |||
| auto schema = n->maybeSchema(); | |||
| TRTORCH_CHECK(schema, "Unable to get schema for Node " << util::node_info(n) \ | |||
| << " (conversion.AddLayer)"); | |||
| << " (conversion.VerifyCoverterSupportForBloxk"); | |||
Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
HACK: WYA tracing without being in eval mode and ignoring the warning, will follow up with the PyTorch Team and test after script mode support lands Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will add PTQ support in TRTorch using the TensorRT INT8 Calibrator. Adds a new application to demonstrate PTQ functionality where the program will quantize a VGG16 network trained on CIFAR10 (training recipe provided)
API Changes
make_int8_calibrator: Generates a class that uses a provided dataloader and a TensorRT calibrator algorithm to run quantizationextra_info.max_batch_size: Setsbuilder.max_batch_sizeextra_info.strict_type->extra_info.strict_typesContract Changes
Bug Fixes
Will close #35, should fix #41