Merge pull request #575 from Nota-NetsPresso/dev

Micro update: v1.0.3 release
Nota-NetsPresso · Oct 29, 2024 · f0e0c1f · f0e0c1f
2 parents 1bdd58e + 1cdf318
commit f0e0c1f
Show file tree

Hide file tree

Showing 56 changed files with 1,313 additions and 329 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,36 @@ No changes to highlight.
 
 No changes to highlight.
 
+# v1.0.3
+
+## New Features:
+
+- Add RandomResize2 by `@illian01` in [PR 550](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/550)
+- Add confidence score on detection visualization by `@hglee98` in [PR 552](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/552)
+- Add save_best_only option for saving model by `@hglee98` in [PR 555](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/555), [PR 567](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/567)
+- Add option to select best model saving criterion by `@hglee98` and `@illian01` in [PR 557](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/557), [PR 573](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/573), [PR 574](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/574)
+- Add MultiStepLR scheduler by `@hglee98` in [PR 559](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/559)
+- Add class-wise metric analysis option by `@illian01` in [PR 568](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/568)
+- Add ReLU6 by `@hglee98` in [PR 566](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/566)
+- Add TFLite model evaluation feature by `@hglee98` in [PR 563](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/563)
+- Add YOLO-Fastest-v2 by `@hglee98` in [PR 548](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/548)
+- Add tabulating step for metric standard outputs by `@illian01` in [PR 570](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/570)
+
+## Bug Fixes:
+
+- Fix keyword error of segmentation training by `@illian01` in [PR 551](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/551)
+- Fix typo when saving optimizer state_dict by `@hglee98` in [PR 553](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/553)
+- Fix not initialized save_dtype error by `@hglee98` in [PR 565](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/565)
+- Fix mAP error in case of certain classes object is not in the dataset `@hglee98` in [PR 571](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/571), [PR 572](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/572)
+
+## Breaking Changes:
+
+- Massive refactoring metric modules and add flexible metric selecting option by `@illian01` in [PR 564](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/564)
+
+## Other Changes:
+
+No changes to highlight.
+
 # v1.0.2
 
 ## New Features:

diff --git a/README.md b/README.md
@@ -58,7 +58,7 @@ pip install -e netspresso-trainer
 ### Set-up with docker
 
 Please clone this repository and refer to [`Dockerfile`](./Dockerfile) and [`docker-compose-example.yml`](./docker-compose-example.yml).  
-For docker users, we provide more detailed guide in our [Docs](https://nota-netspresso.github.io/netspresso-trainer).
+For docker users, we provide more detailed guide in our [Docs](https://nota-netspresso.github.io/netspresso-trainer/getting_started/installation/docker_installation).
 
 ## Getting started
 
@@ -142,4 +142,4 @@ tensorboard --logdir ./outputs --port 50001 --bind_all
 ```
 
 where `PORT` for tensorboard is 50001.  
-Note that the default directory of saving result will be `./outputs` directory.
+Note that the default directory of saving result will be `./outputs` directory.
diff --git a/config/benchmark_examples/classification-imagenet1k-resnet18/logging.yaml b/config/benchmark_examples/classification-imagenet1k-resnet18/logging.yaml
@@ -4,8 +4,14 @@ logging:
   tensorboard: true
   image: true
   stdout: true
-  save_optimizer_state: true
-  sample_input_size: [224, 224] # Used for flops and onnx export
-  onnx_export_opset: 13 # Recommend in range [13, 17]
-  validation_epoch: &validation_epoch 5
-  save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [224, 224] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 5
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/benchmark_examples/classification-imagenet1k-resnet34/logging.yaml b/config/benchmark_examples/classification-imagenet1k-resnet34/logging.yaml
@@ -4,8 +4,14 @@ logging:
   tensorboard: true
   image: true
   stdout: true
-  save_optimizer_state: true
-  sample_input_size: [224, 224] # Used for flops and onnx export
-  onnx_export_opset: 13 # Recommend in range [13, 17]
-  validation_epoch: &validation_epoch 5
-  save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [224, 224] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 5
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/benchmark_examples/classification-imagenet1k-resnet50/logging.yaml b/config/benchmark_examples/classification-imagenet1k-resnet50/logging.yaml
@@ -4,8 +4,14 @@ logging:
   tensorboard: true
   image: true
   stdout: true
-  save_optimizer_state: true
-  sample_input_size: [224, 224] # Used for flops and onnx export
-  onnx_export_opset: 13 # Recommend in range [13, 17]
-  validation_epoch: &validation_epoch 10
-  save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [224, 224] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 10
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/augmentation.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/augmentation.yaml
@@ -0,0 +1,56 @@
+augmentation:
+  train:
+    - 
+      name: mosaicdetection
+      size: [640, 640]
+      mosaic_prob: 1.0
+      affine_scale: [0.5, 1.5]
+      degrees: 10.0
+      translate: 0.1
+      shear: 2.0
+      enable_mixup: True
+      mixup_prob: 1.0
+      mixup_scale: [0.1, 2.0]
+      fill: 0
+      mosaic_off_epoch: 285
+    -
+      name: hsvjitter
+      h_mag: 5
+      s_mag: 30
+      v_mag: 30
+    -
+      name: randomhorizontalflip
+      p: 0.5
+    -
+      name: resize
+      size: 640
+      interpolation: bilinear
+      max_size: ~
+      resize_criteria: long
+    -
+      name: pad
+      size: 640
+      fill: 0
+    -
+      name: randomresize
+      base_size: [640, 640]
+      stride: 32
+      random_range: 5
+      interpolation: bilinear
+    -
+      name: totensor
+      pixel_range: 1.0
+  inference:
+    - 
+      name: resize
+      size: 640
+      interpolation: bilinear
+      max_size: ~
+      resize_criteria: long
+    - 
+      name: pad
+      size: 640
+      fill: 0
+    -
+      name: totensor
+      pixel_range: 1.0
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/data.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/data.yaml
@@ -0,0 +1,21 @@
+data:
+  name: coco2017
+  task: detection
+  format: local # local, huggingface
+  path:
+    root: ~/data/coco2017 # dataset root
+    train:
+      image: images/train # directory for training images
+      label: labels/train # directory for training labels
+    valid:
+      image: images/valid # directory for valid images
+      label: labels/valid # directory for valid labels
+    test:
+      image: ~
+      label: ~
+    pattern:
+      image: ~
+      label: ~
+  id_mapping: id_mapping.json
+  # id_mapping: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']
+  pallete: ~
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/environment.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/environment.yaml
@@ -0,0 +1,5 @@
+environment: 
+  seed: 1
+  num_workers: 8
+  gpus: 0, 1, 2, 3
+  batch_size: 64 # Batch size per gpu
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/logging.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/logging.yaml
@@ -0,0 +1,17 @@
+logging:
+  project_id: ~
+  output_dir: ./outputs
+  tensorboard: true
+  image: true
+  stdout: true
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [640, 640] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 10
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/model.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/model.yaml
@@ -0,0 +1,41 @@
+model:
+  task: detection
+  name: yolo_fastest_v2
+  checkpoint:
+    use_pretrained: False
+    load_head: False
+    path: ~
+    optimizer_path: ~
+  freeze_backbone: False
+  architecture:
+    full: ~ # auto
+    backbone:
+      name: shufflenetv2
+      params: 
+        model_size: 0.5x
+      stage_params: 
+        ~
+    neck:
+      name: lightfpn
+      params:
+        out_channels: 72 
+    head:
+      name: yolo_fastest_head_v2
+      params:
+        anchors:
+          &anchors
+          - [12.,18., 37.,49., 52.,132.]  # P2
+          - [115.,73., 119.,199., 242.,238.]  # P3
+  postprocessor: 
+    params:
+      # postprocessor - decode
+      score_thresh: 0.01
+      # postprocessor - nms
+      nms_thresh: 0.65
+      anchors: *anchors
+      class_agnostic: False
+  losses:
+    - criterion: yolofastest_loss
+      anchors: *anchors 
+      l1_activate_epoch: ~
+      weight: ~
diff --git a/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/training.yaml b/config/benchmark_examples/detection-coco2017-yolo_fastest_v2/training.yaml
@@ -0,0 +1,21 @@
+training:
+  epochs: 300
+  mixed_precision: True
+  max_norm: ~
+  ema:
+    name: exp_decay
+    decay: 0.9999
+    beta: 2000
+  optimizer:
+    name: sgd
+    lr: 0.001
+    momentum: 0.949
+    weight_decay: 0.0005 # No bias and norm decay
+    nesterov: True
+    no_bias_decay: True
+    no_norm_weight_decay: True
+    overwrite: ~
+  scheduler:
+    name: multi_step
+    milestones: [150, 250]
+    gamma: 0.1
diff --git a/config/benchmark_examples/detection-coco2017-yolox_s/logging.yaml b/config/benchmark_examples/detection-coco2017-yolox_s/logging.yaml
@@ -4,8 +4,14 @@ logging:
   tensorboard: true
   image: true
   stdout: true
-  save_optimizer_state: true
-  sample_input_size: [640, 640] # Used for flops and onnx export
-  onnx_export_opset: 13 # Recommend in range [13, 17]
-  validation_epoch: &validation_epoch 10
-  save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [640, 640] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 10
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/logging.yaml b/config/logging.yaml
@@ -4,8 +4,14 @@ logging:
   tensorboard: true
   image: true
   stdout: true
-  save_optimizer_state: true
-  sample_input_size: [512, 512] # Used for flops and onnx export
-  onnx_export_opset: 13 # Recommend in range [13, 17]
-  validation_epoch: &validation_epoch 10
-  save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [512, 512] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 10
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings
diff --git a/config/model/yolo/yolo-fastest-v2-detection.yaml b/config/model/yolo/yolo-fastest-v2-detection.yaml
@@ -1,8 +1,8 @@
 model:
   task: detection
-  name: yolofastest
+  name: yolo_fastest_v2
   checkpoint:
-    use_pretrained: False
+    use_pretrained: True
     load_head: False
     path: ~ 
     optimizer_path: ~
@@ -23,17 +23,19 @@ model:
       name: yolo_fastest_head_v2
       params:
         anchors:
-          - [12,18, 37,49, 52,132]  # P2
-          - [115,73, 119,199, 242,238]  # P3
+          &anchors
+          - [12.,18., 37.,49., 52.,132.]  # P2
+          - [115.,73., 119.,199., 242.,238.]  # P3
   postprocessor: 
     params:
       # postprocessor - decode
-      topk_candidates: 1000
-      score_thresh: 0.05
+      score_thresh: 0.01
       # postprocessor - nms
-      nms_thresh: 0.45
+      nms_thresh: 0.65
+      anchors: *anchors
       class_agnostic: False
-  # Temporary loss to test the full YOLOFastestV2 model to work right
   losses:
-    - criterion: retinanet_loss
+    - criterion: yolofastest_loss
+      anchors: *anchors 
+      l1_activate_epoch: ~
       weight: ~
diff --git a/docs/benchmarks/benchmarks.md b/docs/benchmarks/benchmarks.md
@@ -45,4 +45,5 @@ If you have a better recipe, please share with us anytime. We appreciate all eff
 | COCO-val | [YOLOX-s](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-s-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_s_coco.safetensors?versionId=QRLqHKqhv8TSYBrmsQ3M8lCR8w7HEZyA) | (640, 640) | 58.56 | 44.10 | 40.63 | 8.97M | 26.81G | Supported | conf_thresh=0.01, nms_thresh=0.65 |
 | COCO-val | [YOLOX-m*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-m-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_m_coco.safetensors?versionId=xVUySP8xgVTpa6NhCMQpulqmYeRUAhpS) | (640, 640) | 65.00 | 51.34 | 47.04 | 25.33M | 73.76G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
 | COCO-val | [YOLOX-l*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-l-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_l_coco.safetensors?versionId=1GR6YNRu.yUfnjq8hKPgARyZ6YejdxMB) | (640, 640) | 68.07 | 55.18 | 50.68 | 54.21M | 155.65G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
-| COCO-val | [YOLOX-x*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-x-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_x_coco.safetensors?versionId=NWskUEbSGviBWskHQ3P1dQZXnRXOR1WN) | (640, 640) | 69.13 | 56.46 | 51.79 | 99.07M | 281.94G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
+| COCO-val | [YOLOX-x*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-x-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_x_coco.safetensors?versionId=NWskUEbSGviBWskHQ3P1dQZXnRXOR1WN) | (640, 640) | 69.13 | 56.46 | 51.79 | 99.07M | 281.94G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
+| COCO-val | [YOLO-Fastest-v2](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolo/yolo-fastest-v2-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolofastest/yolo_fastest_v2_coco.safetensors?versionId=CGhNjiZygGVjtHm0M586DzQ6.2FqWvl1) | (640, 640) | 25.03 | 11.60 | 12.78 | 0.25M | 0.74G | Supported | conf_thresh=0.01, nms_thresh=0.65 |