Merge pull request #561 from bmaltais/dev

v21.5.0

README.md

@@ -91,6 +91,7 @@ Options:
   -p, --public                  Expose public URL in runpod mode. Won't have an effect in other modes.
   -r, --runpod                  Forces a runpod installation. Useful if detection fails for any reason.
   -s, --skip-space-check        Skip the 10Gb minimum storage space check.
+  -u, --no-gui                  Skips launching the GUI.
   -v, --verbose                 Increase verbosity levels up to 3.
 ```
 
@@ -257,12 +258,47 @@ This will store a backup file with your current locally installed pip packages a
 
 ## Change History
 
-* 2024/04/02 (v21.4.2)
+* 2023/04/07 (v21.5.0)
+    - Update MacOS and Linux install scripts. Thanks @jstayco
+    - Update windows upgrade ps1 and bat
+    - Update kohya_ss sd-script code to latest release... this is a big one so it might cause some training issue. If you find that this release is causing issues for you you can go back to the previous release with `git checkout v21.4.2` and then run the upgrade script for your platform. Here is the list of changes in the new sd-scripts:
+        - There may be bugs because I changed a lot. If you cannot revert the script to the previous version when a problem occurs, please wait for the update for a while.
+        - The learning rate and dim (rank) of each block may not work with other modules (LyCORIS, etc.) because the module needs to be changed.
+
+        - Fix some bugs and add some features.
+            - Fix an issue that `.json` format dataset config files cannot be read.  [issue #351](https://github.com/kohya-ss/sd-scripts/issues/351) Thanks to rockerBOO!
+            - Raise an error when an invalid `--lr_warmup_steps` option is specified (when warmup is not valid for the specified scheduler).  [PR #364](https://github.com/kohya-ss/sd-scripts/pull/364)  Thanks to shirayu!
+            - Add `min_snr_gamma` to metadata in `train_network.py`. [PR #373](https://github.com/kohya-ss/sd-scripts/pull/373) Thanks to rockerBOO!
+            - Fix the data type handling in `fine_tune.py`. This may fix an error that occurs in some environments when using xformers, npz format cache, and mixed_precision.
+
+        - Add options to `train_network.py` to specify block weights for learning rates. [PR #355](https://github.com/kohya-ss/sd-scripts/pull/355) Thanks to u-haru for the great contribution!
+            - Specify the weights of 25 blocks for the full model.
+            - No LoRA corresponds to the first block, but 25 blocks are specified for compatibility with 'LoRA block weight' etc. Also, if you do not expand to conv2d3x3, some blocks do not have LoRA, but please specify 25 values ​​for the argument for consistency.
+            - Specify the following arguments with `--network_args`.
+            - `down_lr_weight` : Specify the learning rate weight of the down blocks of U-Net. The following can be specified.
+            - The weight for each block: Specify 12 numbers such as `"down_lr_weight=0,0,0,0,0,0,1,1,1,1,1,1"`.
+            - Specify from preset: Specify such as `"down_lr_weight=sine"` (the weights by sine curve). sine, cosine, linear, reverse_linear, zeros can be specified. Also, if you add `+number` such as `"down_lr_weight=cosine+.25"`, the specified number is added (such as 0.25~1.25).
+            - `mid_lr_weight` : Specify the learning rate weight of the mid block of U-Net. Specify one number such as `"down_lr_weight=0.5"`.
+            - `up_lr_weight` : Specify the learning rate weight of the up blocks of U-Net. The same as down_lr_weight.
+            - If you omit the some arguments, the 1.0 is used. Also, if you set the weight to 0, the LoRA modules of that block are not created.
+            - `block_lr_zero_threshold` : If the weight is not more than this value, the LoRA module is not created. The default is 0.
+
+        - Add options to `train_network.py` to specify block dims (ranks) for variable rank.
+            - Specify 25 values ​​for the full model of 25 blocks. Some blocks do not have LoRA, but specify 25 values ​​always.
+            - Specify the following arguments with `--network_args`.
+            - `block_dims` : Specify the dim (rank) of each block. Specify 25 numbers such as `"block_dims=2,2,2,2,4,4,4,4,6,6,6,6,8,6,6,6,6,4,4,4,4,2,2,2,2"`.
+            - `block_alphas` : Specify the alpha of each block. Specify 25 numbers as with block_dims. If omitted, the value of network_alpha is used.
+            - `conv_block_dims` : Expand LoRA to Conv2d 3x3 and specify the dim (rank) of each block.
+            - `conv_block_alphas` : Specify the alpha of each block when expanding LoRA to Conv2d 3x3. If omitted, the value of conv_alpha is used.
+    - Add GUI support for new features introduced above by kohya_ss. Those will be visible only if the LoRA is of type `Standard` or `kohya LoCon`. You will find the new parameters under the `Advanced Configuration` accordion in the `Training parameters` tab.
+    - Various improvements to linux and macos srtup scripts thanks to @Oceanswave and @derVedro
+    - Integrated sd-scripts commits into commit history. Thanks to @Cauldrath
+* 2023/04/02 (v21.4.2)
     - removes TensorFlow from requirements.txt for Darwin platforms as pip does not support advanced conditionals like CPU architecture. The logic is now defined in setup.sh to avoid version bump headaches, and the selection logic is in the pre-existing pip function. Additionally, the release includes the addition of the tensorflow-metal package for M1+ Macs, which enables GPU acceleration per Apple's documentation. Thanks @jstayco
 * 2023/04/01 (v21.4.1)
     - Fix type for linux install by @bmaltais in https://github.com/bmaltais/kohya_ss/pull/517
     - Fix .gitignore by @bmaltais in https://github.com/bmaltais/kohya_ss/pull/518
-* 2024/04/01 (v21.4.0)
+* 2023/04/01 (v21.4.0)
     - Improved linux and macos installation and updates script. See README for more details. Many thanks to @jstayco and @Galunid for the great PR!
     - Fix issue with "missing library" error.
 * 2023/04/01 (v21.3.9)

fine_tune.py

@@ -275,7 +275,7 @@ def fn_recursive_set_mem_eff(module: torch.nn.Module):
             with accelerator.accumulate(training_models[0]):  # 複数モデルに対応していない模様だがとりあえずこうしておく
                 with torch.no_grad():
                     if "latents" in batch and batch["latents"] is not None:
-                        latents = batch["latents"].to(accelerator.device)
+                        latents = batch["latents"].to(accelerator.device).to(dtype=weight_dtype)
                     else:
                         # latentに変換
                         latents = vae.encode(batch["images"].to(dtype=weight_dtype)).latent_dist.sample()

gen_img_diffusers.py

@@ -2275,7 +2275,7 @@ def __getattr__(self, item):
                     if metadata is not None:
                         print(f"metadata for: {network_weight}: {metadata}")
 
-                network = imported_module.create_network_from_weights(
+                network, weights_sd = imported_module.create_network_from_weights(
                     network_mul, network_weight, vae, text_encoder, unet, **net_kwargs
                 )
             else:
@@ -2285,14 +2285,16 @@ def __getattr__(self, item):
 
             if not args.network_merge:
                 network.apply_to(text_encoder, unet)
+                info = network.load_state_dict(weights_sd, False)       # network.load_weightsを使うようにするとよい
+                print(f"weights are loaded: {info}")
 
                 if args.opt_channels_last:
                     network.to(memory_format=torch.channels_last)
                 network.to(dtype).to(device)
 
                 networks.append(network)
             else:
-                network.merge_to(text_encoder, unet, dtype, device)
+                network.merge_to(text_encoder, unet, weights_sd, dtype, device)
 
     else:
         networks = []

gui.sh

@@ -1,9 +1,12 @@
 #!/usr/bin/env bash
 
+# This gets the directory the script is run from so pathing can work relative to the script where needed.
+SCRIPT_DIR=$(cd -- "$(dirname -- "$0")" && pwd)
+
 # Activate the virtual environment
-source ./venv/bin/activate
+source "$SCRIPT_DIR/venv/bin/activate"
 
 # If the requirements are validated, run the kohya_gui.py script with the command-line arguments
 if python tools/validate_requirements.py; then
-    python kohya_gui.py "$@"
-fi
+    python "$SCRIPT_DIR/kohya_gui.py" "$@"
+fi

library/common_gui.py

@@ -4,6 +4,7 @@
 import gradio as gr
 import easygui
 import shutil
+import sys
 
 folder_symbol = '\U0001f4c2'  # 📂
 refresh_symbol = '\U0001f504'  # 🔄
@@ -31,7 +32,7 @@
 # define a list of substrings to search for
 ALL_PRESET_MODELS = V2_BASE_MODELS + V_PARAMETERIZATION_MODELS + V1_MODELS
 
-FILE_ENV_EXCLUSION = ['COLAB_GPU', 'RUNPOD_POD_ID']
+ENV_EXCLUSION = ['COLAB_GPU', 'RUNPOD_POD_ID']
 
 
 def check_if_model_exist(output_name, output_dir, save_model_as):
@@ -120,7 +121,7 @@ def get_dir_and_file(file_path):
 def get_file_path(
     file_path='', default_extension='.json', extension_name='Config files'
 ):
-    if not any(var in os.environ for var in FILE_ENV_EXCLUSION):
+    if not any(var in os.environ for var in ENV_EXCLUSION) and sys.platform != 'darwin':
         current_file_path = file_path
         # print(f'current file path: {current_file_path}')
 
@@ -155,7 +156,7 @@ def get_file_path(
 
 
 def get_any_file_path(file_path=''):
-    if not any(var in os.environ for var in FILE_ENV_EXCLUSION):
+    if not any(var in os.environ for var in ENV_EXCLUSION) and sys.platform != 'darwin':
         current_file_path = file_path
         # print(f'current file path: {current_file_path}')
 
@@ -197,7 +198,7 @@ def remove_doublequote(file_path):
 
 
 def get_folder_path(folder_path=''):
-    if not any(var in os.environ for var in FILE_ENV_EXCLUSION):
+    if not any(var in os.environ for var in ENV_EXCLUSION) and sys.platform != 'darwin':
         current_folder_path = folder_path
 
         initial_dir, initial_file = get_dir_and_file(folder_path)
@@ -217,7 +218,7 @@ def get_folder_path(folder_path=''):
 def get_saveasfile_path(
     file_path='', defaultextension='.json', extension_name='Config files'
 ):
-    if not any(var in os.environ for var in FILE_ENV_EXCLUSION):
+    if not any(var in os.environ for var in ENV_EXCLUSION) and sys.platform != 'darwin':
         current_file_path = file_path
         # print(f'current file path: {current_file_path}')
 
@@ -253,7 +254,7 @@ def get_saveasfile_path(
 def get_saveasfilename_path(
     file_path='', extensions='*', extension_name='Config files'
 ):
-    if not any(var in os.environ for var in FILE_ENV_EXCLUSION):
+    if not any(var in os.environ for var in ENV_EXCLUSION) and sys.platform != 'darwin':
         current_file_path = file_path
         # print(f'current file path: {current_file_path}')
 

library/config_util.py

@@ -445,7 +445,7 @@ def extract_dreambooth_params(name: str) -> Tuple[int, str]:
     try:
       n_repeats = int(tokens[0])
     except ValueError as e:
-      print(f"ignore directory without repeats / 繰り返し回数のないディレクトリを無視します: {dir}")
+      print(f"ignore directory without repeats / 繰り返し回数のないディレクトリを無視します: {name}")
       return 0, ""
     caption_by_folder = '_'.join(tokens[1:])
     return n_repeats, caption_by_folder
@@ -486,7 +486,8 @@ def load_user_config(file: str) -> dict:
 
   if file.name.lower().endswith('.json'):
     try:
-      config = json.load(file)
+      with open(file, 'r') as f:
+        config = json.load(f)
     except Exception:
       print(f"Error on parsing JSON config file. Please check the format. / JSON 形式の設定ファイルの読み込みに失敗しました。文法が正しいか確認してください。: {file}")
       raise

library/dataset_balancing_gui.py

@@ -44,6 +44,9 @@ def dataset_balancing(concept_repeats, folder, insecure):
 
             # Count the number of image files
             images = len(image_files)
+
+            if images == 0:
+                print(f'No images of type .jpg, .jpeg, .png, .gif, .webp were found in {os.listdir(os.path.join(folder, subdir))}')
 
             # Check if the subdirectory name starts with a number inside braces,
             # indicating that the repeats value should be multiplied

library/git_caption_gui.py

@@ -28,7 +28,7 @@ def caption_images(
 
     print(f'GIT captioning files in {train_data_dir}...')
     run_cmd = (
-        f'.\\venv\\Scripts\\python.exe "finetune/make_captions_by_git.py"'
+        f'{PYTHON} finetune/make_captions_by_git.py'
     )
     if not model_id == '':
         run_cmd += f' --model_id="{model_id}"'
@@ -44,7 +44,10 @@ def caption_images(
     print(run_cmd)
 
     # Run the command
-    subprocess.run(run_cmd)
+    if os.name == 'posix':
+        os.system(run_cmd)
+    else:
+        subprocess.run(run_cmd)
 
     # Add prefix and postfix
     add_pre_postfix(

library/train_util.py

@@ -2460,7 +2460,7 @@ def get_scheduler_fix(args, optimizer: Optimizer, num_processes: int):
     Unified API to get any scheduler from its name.
     """
     name = args.lr_scheduler
-    num_warmup_steps = args.lr_warmup_steps
+    num_warmup_steps: Optional[int] = args.lr_warmup_steps
     num_training_steps = args.max_train_steps * num_processes * args.gradient_accumulation_steps
     num_cycles = args.lr_scheduler_num_cycles
     power = args.lr_scheduler_power
@@ -2484,6 +2484,11 @@ def get_scheduler_fix(args, optimizer: Optimizer, num_processes: int):
 
             lr_scheduler_kwargs[key] = value
 
+    def wrap_check_needless_num_warmup_steps(return_vals):
+        if num_warmup_steps is not None and num_warmup_steps != 0:
+            raise ValueError(f"{name} does not require `num_warmup_steps`. Set None or 0.")
+        return return_vals
+
     # using any lr_scheduler from other library
     if args.lr_scheduler_type:
         lr_scheduler_type = args.lr_scheduler_type
@@ -2496,20 +2501,20 @@ def get_scheduler_fix(args, optimizer: Optimizer, num_processes: int):
             lr_scheduler_type = values[-1]
         lr_scheduler_class = getattr(lr_scheduler_module, lr_scheduler_type)
         lr_scheduler = lr_scheduler_class(optimizer, **lr_scheduler_kwargs)
-        return lr_scheduler
+        return wrap_check_needless_num_warmup_steps(lr_scheduler)
 
     if name.startswith("adafactor"):
         assert (
             type(optimizer) == transformers.optimization.Adafactor
         ), f"adafactor scheduler must be used with Adafactor optimizer / adafactor schedulerはAdafactorオプティマイザと同時に使ってください"
         initial_lr = float(name.split(":")[1])
         # print("adafactor scheduler init lr", initial_lr)
-        return transformers.optimization.AdafactorSchedule(optimizer, initial_lr)
+        return wrap_check_needless_num_warmup_steps(transformers.optimization.AdafactorSchedule(optimizer, initial_lr))
 
     name = SchedulerType(name)
     schedule_func = TYPE_TO_SCHEDULER_FUNCTION[name]
     if name == SchedulerType.CONSTANT:
-        return schedule_func(optimizer)
+        return wrap_check_needless_num_warmup_steps(schedule_func(optimizer))
 
     # All other schedulers require `num_warmup_steps`
     if num_warmup_steps is None: