MMDAutoTrace2_en.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8oCs7zcAvs5p"
      },
      "source": [
        "# Welcome to the colab version of MMD Autotrace v2! (Execution)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9GsUrjROxoDK"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "The operating status and maintenance information of this tool is available on Twitter（[@miu200521358](https://twitter.com/miu200521358)）.\n",
        "\n",
        "If an error occurs or the product does not start, first check the current distribution status.\n",
        "\n",
        "We also accept inquiries by mail or DM.\n",
        "\n",
        "Also, v2 requires learning data that can be downloaded after membership registration, so we ask you to register as a member."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oe_4TOdDx37F"
      },
      "source": [
        "# table of contents\n",
        "\n",
        "This notebook prepares and runs MMD automatic traces.\n",
        "\n",
        "There is a habit of using notebooks, so please familiarize yourself with how to use them in \\[[Preparation](https://colab.research.google.com/github/miu200521358/motion_trace_colab/blob/master/AutoTraceIntroduction_en.ipynb)\\].\n",
        "\n",
        "Click on the three lines at the top left of the screen. The table of contents opens. (If it is already open, proceed to the next)\n",
        "\n",
        "![目次](https://drive.google.com/uc?export=view&id=1HGk4sJmcPtMbMwcJOvE3aU1GjvKinwA_)\n",
        "\n",
        "Create a `autotrace` folder directly under Google Drive.\n",
        "\n",
        "(If it has already been created in the preparation section, it does not need to be recreated)\n",
        "\n",
        "Check your notebook from top to bottom and follow the steps below one by one.\n",
        "\n",
        "The cells that need to be executed are numbered. Please execute in order from ①.\n",
        "\n",
        "- **\"① Data download\"**\n",
        "  - Links with Google Drive and Google SDK (tool)\n",
        "  - Please check the introduction for how to cooperate.\n",
        "  - Download the learning data required for person estimation in advance.\n",
        "- **\"② Preferences\"**\n",
        "  - Confirm that the runtime could be changed to GPU\n",
        "    - For how to change, please check the introduction\n",
        "  - Change the version of Tensorflow to 1.x\n",
        "  - Download sound effects from \\[[Sound Effect Lab](https://soundeffect-lab.info/)\\]\n",
        "  - Sounds during long processing such as preparation and actual tracing processing.\n",
        "  - Mute your browser volume if you don't need it\n",
        "- **\"③ Preparation\"**\n",
        "  - Build the code needed for MMD autotrace on colab.\n",
        "  - It takes about 15 minutes.\n",
        "- **\"④ Place the data\"**\n",
        "  - Place the training data required for MMD automatic tracing.\n",
        "- **\"⑤ Place the video\"**\n",
        "  - Place the video data that is the target of MMD automatic tracing.\n",
        "- **\"⑥ MMD automatic trace execution\"**\n",
        "  - Perform MMD automatic tracing.\n",
        "\n",
        "---\n",
        "\n",
        "The following warning may appear during work, but there is no problem if you proceed as it is.\n",
        "\n",
        "(GPU is mainly used for part of person estimation and posture estimation, so other cells may appear during work.)\n",
        "\n",
        "![警告](https://drive.google.com/uc?export=view&id=1XQYDn1jE9Qw7a9q3m6C9c3cOIKbmxX6t)\n",
        "\n",
        "---\n",
        "\n",
        "If the following error occurs, the environment has already been destroyed.\n",
        "\n",
        "Go back to cell ① from the table of contents and try all over again.\n",
        "\n",
        "![Googleドライブ連携](https://drive.google.com/uc?export=view&id=1Tsjzs2QAECPMfUTaTh7PYX2cMEypEIF2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYif75w8vuDT"
      },
      "source": [
        "# MMD automatic trace execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0zesaONCvUaQ"
      },
      "source": [
        "## Advance preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Stvjb4Icsq-A"
      },
      "source": [
        "Please register two data required for person estimation on each site and download them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7_RGODpJYhK"
      },
      "source": [
        "1. Go to the \\[[ExPose Project Page](https://expose.is.tue.mpg.de)\\].\n",
        "\n",
        "2. Click the \"Sign In\" button.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1__io7EjqFQnhq08Zi_YfxKUeSottZWJl)\n",
        "\n",
        "3. Click the \"Register\" button.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1B8mJPJuTdHANo0UJ71Gue6CmznSDCdbw)\n",
        "\n",
        "4. Enter your \"email address\" and \"password\", turn on all the switches, and click the \"Register\" button.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1R2jCZHKJu71hXLW2KHhSLswDjwJEWEiZ)\n",
        "\n",
        "5. Click the link in the email you received.\n",
        "\n",
        "6. Enter the registered \"email address\" and \"password\" and click the \"Log In\" button.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1loay_bPetEBQfaOQIpS_JCLhZrFHolyF)\n",
        "\n",
        "7. Click the \"Download\" button.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1GFhczKTl5Mc6E6PSRXT0hznCVfo-FWDr)\n",
        "\n",
        "8. Click the \"ExPose Model\" button in the middle of the screen.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1FjaHb9yGr52nI27xBLHAQBBz4gOEnR6x)\n",
        "\n",
        "9. Place the downloaded zip file (expose_data.zip) in the `autotrace` folder.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zf_DKKFJjK2R"
      },
      "source": [
        "1. Go to the \\[[SMPL-X Project Page](https://smpl-x.is.tue.mpg.de)\\].\n",
        "\n",
        "2. Register as a member in the same way as Expose.\n",
        "\n",
        "3. Click the \"download SMPL-X v1.1\" button at the top of the screen.\n",
        "\n",
        "![SignIn](https://drive.google.com/uc?export=view&id=1cGOHUdhDtSOfIqIHYUXhQTc5ki5pgl85)\n",
        "\n",
        "4. Place the downloaded zip file (models_smplx_v1_1.zip) in the `autotrace` folder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sj7qFjn46EK3"
      },
      "source": [
        "## ① Data DL\n",
        "\n",
        "Works with the `autotrace` folder in Google Drive.\n",
        "\n",
        "Execute the cell [①-A] below and paste the key for permission to link into the input field with \"Ctrl + V\".\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZreTNgwipgE"
      },
      "source": [
        "### ①-A Cooperation with Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L6UsSufU6WBv",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown [Cell ①-A] Cooperation with Google Drive\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "exec_dict = {}\n",
        "exec_dict['1'] = True\n",
        "\n",
        "from google.colab import drive\n",
        "import os\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "base_path = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "if os.path.exists(base_path):\n",
        "    ! echo \"autotraceフォルダの中身 -----------\"\n",
        "    ! ls -l \"$base_path\"\n",
        "    ! echo \"--------------------\"\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　The cooperation with the autotrace folder was successful.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    exec_dict['1-A'] = True\n",
        "else:\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The autotrace folder was not found directly under Google Drive.\")\n",
        "    print(\"■　Please re-execute from the preparation section.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ebEeC4TE7NFE"
      },
      "source": [
        "**【OK】**\n",
        "\n",
        "If the contents of the `autotrace` folder are displayed as shown below, it is successful.\n",
        "\n",
        "![Googleドライブ連携](https://drive.google.com/uc?export=view&id=19OhLvcyP-CN90KWDmkmBuSRR3BpO5GzV)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hLRXMSkHx4pe"
      },
      "source": [
        "### ①-B Cooperation with Google SDK\n",
        "\n",
        "It works with `Google SDK` to download the necessary data from miu's Google Drive.\n",
        "\n",
        "As with cell ①-A, give permission from the URL.\n",
        "\n",
        "Execute the cell [①-B] below and paste the key for permission to link into the input field with \"Ctrl + V\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lwhFQNTqxoOK",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown [Cell ①-B] Cooperation with Google SDK\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "# Googleドライブアクセスライブラリ\n",
        "!pip install -U -q PyDrive\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        " \n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "exec_dict['1-B'] = True\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　The integration with Google SDK was successful.\")\n",
        "print(\"■■■■■■■■■■■■■■■\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FpsJrV-VpC29"
      },
      "source": [
        "### ①-C Expose learning data placement"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9rvPkHvTdNrU",
        "cellView": "form"
      },
      "source": [
        "#@markdown Execute the cell [①-C] below.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown [Cell ①-C] Placement of expose_data\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "expose_data_path = f\"{base_path}/expose_data.zip\"\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not os.path.exists(expose_data_path):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The expose_data.zip was not found directly under autotrace.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "else:\n",
        "    if os.path.exists('./expose_data.zip'):\n",
        "        os.remove('./expose_data.zip')\n",
        "\n",
        "    ! cp \"$expose_data_path\" .\n",
        "\n",
        "    import hashlib\n",
        "    filechecksum = 'ac8e8a53922364ed7bd04e8f9722f937b987f146f9b53f6b4317a0e69a0d055b'\n",
        "\n",
        "    with open('./expose_data.zip','rb') as f:\n",
        "        checksum = hashlib.sha256(f.read()).hexdigest()\n",
        "    if checksum == filechecksum:\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　** OK **\")\n",
        "        print(\"■　Placement of expose_data was successful.\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "        exec_dict['1-C'] = True        \n",
        "    else:\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　** ERROR **\")\n",
        "        print(\"■　The hash value of expose_data in the autotrace folder does not match.\")\n",
        "        print(\"■　The placement may have failed, so try running the cell again.\")\n",
        "        print(\"■　If an error occurs even after executing 3 times, please try again from uploading.\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b43Zh2-FjAeO"
      },
      "source": [
        "### ①-D SMPL-X Learning data placement\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jLK03OXBkOuW",
        "cellView": "form"
      },
      "source": [
        "#@markdown Execute the cell [①-D] below.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown [Cell ①-D] Arrangement of SMPL-X data\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "smplx_data_path = f\"{base_path}/models_smplx_v1_1.zip\"\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not os.path.exists(smplx_data_path):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Models_smplx_v1_1.zip was not found directly under autotrace.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    \n",
        "else:\n",
        "    if os.path.exists('./models_smplx_v1_1.zip'):\n",
        "        os.remove('./models_smplx_v1_1.zip')\n",
        "\n",
        "    ! cp \"$smplx_data_path\" .\n",
        "\n",
        "    import hashlib\n",
        "    filechecksum = 'cb593838a5d602395735081c7a8fd1d6b04fba261042b84e24644d875039be61'\n",
        "\n",
        "    with open('./models_smplx_v1_1.zip','rb') as f:\n",
        "        checksum = hashlib.sha256(f.read()).hexdigest()\n",
        "    if checksum == filechecksum:\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　** OK **\")\n",
        "        print(\"■　The placement of models_smplx_v1_1 was successful.\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        exec_dict['1-D'] = True\n",
        "    else:\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　** ERROR **\")\n",
        "        print(\"■　The hash values for models_smplx_v1_1 in the autotrace folder do not match.\")\n",
        "        print(\"■　The placement may have failed, so try running the cell again.\")\n",
        "        print(\"■　If an error occurs even after executing 3 times, please try again from uploading.\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uVf7oqzN0KJC"
      },
      "source": [
        "## **②　Environmental setting**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wv4gj64t0APu"
      },
      "source": [
        "Select \"Runtime\"> \"Change runtime type\"> \"GPU\" in the header.\n",
        "\n",
        "If you can change it, execute the cell [②] below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGYV6obv0FFv",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ②】　\n",
        "#@markdown - Make sure the runtime is GPU\n",
        "#@markdown - Download sound effects from [Sound Effect Lab](https://soundeffect-lab.info/)\n",
        "#@markdown - Tensorflow version change moved to cell ③ (2022/08/06)\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-D' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-D may not be executed\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "exec_dict['2'] = True\n",
        "\n",
        "! echo --------------\n",
        "! echo 【A】 Change runtime to GPU\n",
        "! echo --------------\n",
        "\n",
        "import os\n",
        "\n",
        "CUDA_PATH = '/usr/local/cuda-10.1'\n",
        "\n",
        "if not os.path.exists(CUDA_PATH):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　No viable CUDA was found.\")\n",
        "    print(\"■　Please contact the author.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "p = os.getenv('PATH')\n",
        "ld = os.getenv('LD_LIBRARY_PATH')\n",
        "os.environ['PATH'] = f\"/usr/local/cuda-10.1/bin:{p}\"\n",
        "os.environ['LD_LIBRARY_PATH'] = f\"/usr/local/cuda-10.1/lib64:{ld}\"\n",
        "! nvcc --version\n",
        "\n",
        "! nvcc --version\n",
        "! nvidia-smi\n",
        "\n",
        "! echo --------------\n",
        "! echo 【B】 Download sound effects\n",
        "! echo --------------\n",
        "\n",
        "! wget --no-check-certificate -c \"https://soundeffect-lab.info/sound/anime/mp3/sceneswitch1.mp3\"\n",
        "\n",
        "from IPython.display import Audio, display\n",
        "def play_sound():\n",
        "    try:\n",
        "        display(Audio(\"sceneswitch1.mp3\", autoplay=True))\n",
        "    except:\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　The sound effect could not be played\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFtzYGbg4zhJ"
      },
      "source": [
        "**【OK】**\n",
        "\n",
        "If the following is displayed and the last sound is heard, it is successful.\n",
        "\n",
        "![GPU切り替え失敗](https://drive.google.com/uc?export=view&id=1WK0Oo3qQESB8hVlkvyWfsq17LUn3N308)\n",
        "\n",
        "---\n",
        "\n",
        "**【NG】**\n",
        "\n",
        "If the following is displayed, the runtime change has failed, so check the introductory part again and change the runtime.\n",
        "\n",
        "![GPU変更成功](https://drive.google.com/uc?export=view&id=1E230eOWMctbD2F7qTvn_fR-3gBRQQ_uQ)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w0p_WxUc-jz7"
      },
      "source": [
        "## **③　Code preparation**\n",
        "\n",
        "Build MMD autotrace code on Colab.\n",
        "\n",
        "Please execute the cell of [③] below. I get a lot of messages. It takes about 15 minutes.\n",
        "\n",
        "**Since the compilation runs on the way, there are some parts that seem to stop working.**\n",
        "\n",
        "**Please do not interrupt the cell because you are surprised or worried. If you wait as it is, it will end.**\n",
        "\n",
        "The character string `ERROR` may appear, but there is a high possibility that there is no problem in executing automatic tracing, so ignore it and proceed as it is.\n",
        "\n",
        "(As of 2021 / 08/15, 1 line of ERROR and 2 lines of WARNING appear, but none of them interferes with tracing.\n",
        "\n",
        " If you get more ERROR, there is a high possibility that the installation has failed, so please try cell ③ again.）\n",
        "\n",
        "When the DL sound is heard in [②], the process is complete."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PyGvHSTo-uPb",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ③】　Code preparation execution\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "else:\n",
        "    import shutil\n",
        "    \n",
        "    if os.path.exists('./expose_mmd'):\n",
        "        shutil.rmtree('./expose_mmd')\n",
        "\n",
        "    # MMD自動トレースキットバージョンタグ\n",
        "    ver_tag = \"ver2.05.07.00\"\n",
        "\n",
        "    ! git clone  --depth 1 -b \"$ver_tag\" \"https://github.com/miu200521358/expose_mmd.git\"\n",
        "    \n",
        "    # tensorflow-gpu を入替\n",
        "    !pip uninstall -y tensorflow-gpu tensorflow kapre abslg numpyg wraptg gastg protobufg kerasg optg sixg wheelg tensorflowg googleg termcolorg astorg grpciog kerasg tensorboardg h5pyg markdowng werkzeugg setuptoolsg importlibg zippg typingg cachedg\n",
        "    !pip install tensorflow==1.15.2 tensorflow-gpu==1.15.2 tensorflow-probability==0.7\n",
        "\n",
        "    # %tensorflow_version 1.x\n",
        "\n",
        "    # %tensorflow_version\n",
        "\n",
        "    # 使わないtorchライブラリを削除\n",
        "    ! pip uninstall -y torchtext\n",
        "    ! pip uninstall -y torchaudio \n",
        "\n",
        "    ! pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "    ! pip install --upgrade -r \"expose_mmd/requirements.txt\"\n",
        "\n",
        "    import os \n",
        "    p = os.getenv('PATH')\n",
        "    ld = os.getenv('LD_LIBRARY_PATH')\n",
        "    os.environ['PATH'] = f\"/usr/local/cuda-10.1/bin:{p}\"\n",
        "    os.environ['LD_LIBRARY_PATH'] = f\"/usr/local/cuda-10.1/lib64:{ld}\"\n",
        "\n",
        "    ! cd expose_mmd/lighttrack/lib/ && python setup.py install\n",
        "    ! pip3 install expose_mmd/lighttrack/lib/\n",
        "    \n",
        "    ! cd expose_mmd/lighttrack/graph/torchlight && python setup.py install\n",
        "    ! pip3 install expose_mmd/lighttrack/graph/torchlight/\n",
        "\n",
        "    exec_dict['3'] = True\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　The code construction is complete.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXZDL-JDilSe"
      },
      "source": [
        "## **④　Place the data**\n",
        "\n",
        "Place the data required for MMD automatic tracing on Colab.\n",
        "\n",
        "Please execute the cell of [④] below. I get a lot of messages.\n",
        "\n",
        "When the DL sound is heard in [②], the process is complete."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mtIz8_ixmZSj",
        "cellView": "form"
      },
      "source": [
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ④】　Place the data\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '3' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ③ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "else:\n",
        "    import os\n",
        "    import shutil\n",
        "    if os.path.exists('./data'):\n",
        "        shutil.rmtree('./data')\n",
        "\n",
        "    if os.path.exists('./models'):\n",
        "        shutil.rmtree('./models')\n",
        "\n",
        "    if os.path.exists('./expose_mmd/data/'):\n",
        "        shutil.rmtree('./expose_mmd/data/')\n",
        "\n",
        "    if os.path.exists('./expose_mmd/lighttrack/weights/'):\n",
        "        shutil.rmtree('./expose_mmd/lighttrack/weights/')\n",
        "    os.makedirs('./expose_mmd/lighttrack/weights/', exist_ok=True)\n",
        "\n",
        "    # expose関連データ移動\n",
        "    ! unzip ./expose_data.zip && mv -f data ./expose_mmd/\n",
        "    ! unzip ./models_smplx_v1_1.zip && mv -f models ./expose_mmd/data/\n",
        "\n",
        "    # backbones.zip\n",
        "    downloaded = drive.CreateFile({'id': '17Pq_Yjx2TaZeSVxE1kLkhTs8UlDgAAMd'})\n",
        "    downloaded.GetContentFile('/content/backbones.zip')\n",
        "    ! mv -f ./backbones.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./backbones.zip\n",
        "\n",
        "    # CPN101.zip\n",
        "    downloaded = drive.CreateFile({'id': '10pOvWra9GysGJQxEhOj4j5VsOIWRFtlm'})\n",
        "    downloaded.GetContentFile('/content/CPN101.zip')\n",
        "    ! mv -f ./CPN101.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./CPN101.zip\n",
        "\n",
        "    # GCN.zip\n",
        "    downloaded = drive.CreateFile({'id': '1KuL2uLfu-GLZLFHq7XKjjJWOjE_18Vzn'})\n",
        "    downloaded.GetContentFile('/content/GCN.zip')\n",
        "    ! mv -f ./GCN.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./GCN.zip\n",
        "\n",
        "    # mobile-deconv.zip\n",
        "    downloaded = drive.CreateFile({'id': '11cMlBVuNJon_2XjXhFTXGLZXAO32k7Q3'})\n",
        "    downloaded.GetContentFile('/content/mobile-deconv.zip')\n",
        "    ! mv -f ./mobile-deconv.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./mobile-deconv.zip\n",
        "\n",
        "    # MSRA152.zip\n",
        "    downloaded = drive.CreateFile({'id': '14Yz3Z_5avWFxu8MZ7vhWP2M-xPxUEQKh'})\n",
        "    downloaded.GetContentFile('/content/MSRA152.zip')\n",
        "    ! mv -f ./MSRA152.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./MSRA152.zip\n",
        "\n",
        "    # YOLOv3.zip\n",
        "    downloaded = drive.CreateFile({'id': '1s9Rs90jKrlCvwM9oWgituKlo-SmPhp4q'})\n",
        "    downloaded.GetContentFile('/content/YOLOv3.zip')\n",
        "    ! mv -f ./YOLOv3.zip expose_mmd/lighttrack/weights/ && cd expose_mmd/lighttrack/weights && unzip -o ./YOLOv3.zip\n",
        "\n",
        "    # # shape_predictor_68_face_landmarks.dat.bz2\n",
        "    # downloaded = drive.CreateFile({'id': '1Gai5UVpqYJ0aWU9itFPDBu7iy53JqdV6'})\n",
        "    # downloaded.GetContentFile('/content/shape_predictor_68_face_landmarks.dat.bz2')\n",
        "    # ! mv -f ./shape_predictor_68_face_landmarks.dat.bz2 expose_mmd/data/ && cd expose_mmd/data && bunzip2 -f ./shape_predictor_68_face_landmarks.dat.bz2\n",
        "\n",
        "    exec_dict['4'] = True\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　The placement of other data was successful.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MX_AtRSywDhD"
      },
      "source": [
        "## **⑤　Input video upload**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PLrxplXwQJW"
      },
      "source": [
        "Prepare the video file you want to process.\n",
        "\n",
        " - The file name should be **only alphanumeric characters**. opencv cannot read double-byte characters.\n",
        " - If the video is too long, the disk space will be insufficient. If it exceeds 6000f, divide it.\n",
        " - Place it directly under the **autotrace** folder on Google Drive.\n",
        " - **Overwriting or updating files on the Goole drive after mounting will not be recognized correctly.** \n",
        " - Please upload the new file with a new name before processing.\n",
        " - Enter only the file name (including extension).\n",
        " - Please execute the cell of [⑤] below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jcveeryDv6sl",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑤】　Input video file upload\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "\n",
        "#@markdown Enter the name of the video file to be analyzed and execute the cell.\n",
        "\n",
        "input_video_name = \"input.mp4\"  #@param {type: \"string\"}\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '3' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ③ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '4' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ④ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "exec_dict['5'] = True\n",
        "\n",
        "input_video_path = f\"{base_path}/{input_video_name}\"\n",
        "\n",
        "if not os.path.exists(input_video_path):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The specified video was not found directly under autotrace.\")\n",
        "    print(\"■　File Path: \" + input_video_path)\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "else:\n",
        "    ! cp \"$input_video_path\" .\n",
        "\n",
        "    input_video_path = f\"/content/{input_video_name}\"\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　The input video settings are complete.\")\n",
        "    print(\"■　File Path: \" + input_video_path)\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KoprSvp4xuXu"
      },
      "source": [
        "## **⑥　MMD automatic trace execution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0g0Disv6xsmp"
      },
      "source": [
        "MMD automatic tracing is divided into person estimation, tracking specification, and motion output.\n",
        "\n",
        "Execute the cell [⑥-A] below. The final conversion process takes time.\n",
        "\n",
        "※ If the following message appears, you can ignore the first one.\n",
        "\n",
        "When the second time comes out, divide the input video at a good point and process it separately.\n",
        "\n",
        "![容量オーバー](https://drive.google.com/uc?export=view&id=1fe1Fn6NX8gYUiYs9ukWG4p4C1pj6pZ0P)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcPBwOqB4ULh"
      },
      "source": [
        "### ⑥-A　Person estimation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2EAjUCpyyLMq",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-A】　Person estimation\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "import shutil\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '3' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ③ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '4' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ④ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '5' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑤ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not os.path.exists('/content/expose_mmd/data') or not os.path.exists('/content/expose_mmd/data/models'):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The training data placement may have failed.\")\n",
        "    print(\"■　Go back to ①-C and try again.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    \n",
        "else:\n",
        "    if os.path.exists('/content/output'):\n",
        "        shutil.rmtree('/content/output')\n",
        "\n",
        "    if os.path.exists('./out.webm'):\n",
        "        os.remove('./out.webm')\n",
        "\n",
        "    if os.path.exists('./out2.webm'):\n",
        "        os.remove('./out2.webm')\n",
        "\n",
        "    import datetime\n",
        "    process_img_dir_name = \"{0}_{1:%Y%m%d_%H%M%S}\".format(os.path.basename(input_video_path).replace('.', '_'), datetime.datetime.now())\n",
        "    process_img_dir_path = os.path.join('/content/output', process_img_dir_name)\n",
        "    os.makedirs(process_img_dir_path, exist_ok=True)\n",
        "\n",
        "    ! cd expose_mmd/ && python executor.py --video-file \"$input_video_path\" --parent-dir \"$process_img_dir_path\" --process prepare,expose,tracking --verbose 20 --log-mode 0\n",
        "\n",
        "    # mp4に変換\n",
        "    bbox_file_path = f\"{process_img_dir_path}/bbox.mp4\"\n",
        "\n",
        "    if not os.path.exists(bbox_file_path):\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "        print(\"■　** ERROR **\")\n",
        "        print(\"■　Person estimation or person tracking failed.\")\n",
        "        print(\"■　Check if there is a block that says \\\"ERROR\\\".\")\n",
        "        print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "        import sys\n",
        "        sys.exit(-1)\n",
        "\n",
        "    # from base64 import b64encode\n",
        "    # mp4 = open(bbox_file_path,'rb').read()\n",
        "    # data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** INFO **\")\n",
        "    print(\"■　Start preparing to execute cells ⑥-B.\")\n",
        "    print(\"■　Please wait.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    # !ffmpeg -i \"$bbox_file_path\" -vcodec vp9 ./out.webm\n",
        "    # ! ffmpeg -i \"$bbox_file_path\" -r 30 -c:v libvpx-vp9 -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1 -c:a libopus -b:a 64k -f webm ./out.webm\n",
        "    ! ffmpeg -i \"$bbox_file_path\" -c:v libvpx-vp9 -b:v 1000K -r 30 -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1 -lag-in-frames 25 -c:a libopus -b:a 64k -f webm -y out.webm\n",
        "\n",
        "    # # webm形式で別途出力\n",
        "    # import os\n",
        "    # import glob\n",
        "    # import cv2\n",
        "    # from tqdm import tqdm\n",
        "\n",
        "    # def save_webm(process_img_pathes, out_path):\n",
        "    #     fourcc = cv2.VideoWriter_fourcc(*\"vp80\")\n",
        "    #     out = cv2.VideoWriter(out_path, fourcc, 30.0, (1280, 720))\n",
        "\n",
        "    #     for iidx, process_img_path in enumerate(tqdm(process_img_pathes)):\n",
        "    #         out_frame = cv2.imread(process_img_path)\n",
        "    #         out.write(out_frame)\n",
        "\n",
        "    #     out.release()\n",
        "    #     cv2.destroyAllWindows()\n",
        "\n",
        "    # process_img_pathes = sorted(glob.glob(os.path.join(process_img_dir_path, \"frames\", \"**\", \"bbox_*.png\")))\n",
        "    # process_bbox_path = \"./out.webm\"\n",
        "    # save_webm(process_img_pathes, process_bbox_path)\n",
        "\n",
        "    exec_dict['6-A'] = True\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　Cell ⑥-B is ready to run. Please proceed to the next.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZ9PIgPIGtbZ"
      },
      "source": [
        "Check the result prepared in \"⑥-A\" from the following viewpoints.\n",
        "\n",
        " - A frame will be placed where it is recognized as a person.\n",
        " - Check the INDEX at the top left of the frame for chasing one person.\n",
        " - The current frame number is displayed in the upper left.\n",
        " - If the person is not clearly taken, it will be better to exclude it even if the frame is taken.\n",
        "     - Use the frame that has all the trunk and the elbows and knees as a guide.\n",
        "     - If the same person has a double frame, please use only the person who can get to the end.\n",
        " - How to specify\n",
        "     - When importing INDEX from the beginning to the end: [INDEX number]\n",
        "     - When importing INDEX from the beginning to the middle: [INDEX number]:-[End frame]\n",
        "     - When importing INDEX from the middle to the end: [INDEX number]: [Start frame]-\n",
        "     - When importing INDEX from the middle to the middle: [INDEX number]: [Start frame]-[End frame]\n",
        "\n",
        "---\n",
        "\n",
        "**INDEX specification example**\n",
        "\n",
        " ```\n",
        "0,44:263-,94,126,145,148,159,161:-3858,182,244,246\n",
        "1,123,157,163:3572-,170,180,208:4427-,213,239:5580-\n",
        "2,44:-261,49,61,64,71,95,135,139,146,156,188,197,205,215,247\n",
        " ```\n",
        "\n",
        "**How to read the above INDEX (1st line: 1st person)**\n",
        "\n",
        "1. From the beginning to the end of the `INDEX: 0` frame\n",
        "2. From 263F to the end of the `INDEX: 44` frame\n",
        "3. From the beginning to the end of the `INDEX: 94` frame\n",
        "3. From the beginning to the end of the `INDEX: 126` frame\n",
        "3. From the beginning to the end of the `INDEX: 145` frame\n",
        "3. From the beginning to the end of the `INDEX: 148` frame\n",
        "3. From the beginning to the end of the `INDEX: 159` frame\n",
        "3. From the beginning to 3858F in the frame of `INDEX: 161`\n",
        "3. From the beginning to the end of the `INDEX: 182` frame\n",
        "3. From the beginning to the end of the `INDEX: 244` frame\n",
        "3. From the beginning to the end of the `INDEX: 246` frame\n",
        "\n",
        "---\n",
        "\n",
        "Execute the cell [⑥-B] below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uToXQits4asb"
      },
      "source": [
        "### ⑥-B　Confirmation of person estimation result"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7lzNX_akFF-r",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-B】　Confirmation of person estimation result\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "exec_dict['6-B'] = True\n",
        "\n",
        "from IPython.display import HTML\n",
        "import base64\n",
        "import io\n",
        "\n",
        "video = io.open('./out.webm', 'r+b').read()\n",
        "encoded = base64.b64encode(video)\n",
        "data  = '''\n",
        "<video id=\"out\" width=\"1280\" height=\"720\" controls><source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" /></video>\n",
        "'''.format(encoded.decode('ascii')) + '''\n",
        "<br>\n",
        "<button id=\"prev-10frame\">Back 10F</button>\n",
        "<button id=\"prev-1frame\">Back 1F</button>\n",
        "<button id=\"prev-05frame\">Back 0.5F</button>\n",
        "<button id=\"next-05frame\">Forward 0.5F</button>\n",
        "<button id=\"next-1frame\">Forward 1F</button>\n",
        "<button id=\"next-10frame\">Forward 10F</button>\n",
        "(If it is misaligned, move it by 0.5F to adjust it.)\n",
        "<script type=\"text/javascript\">\n",
        "    var video = document.getElementById('out');\n",
        "    var prev05Frame = document.getElementById('prev-05frame');\n",
        "    var next05Frame = document.getElementById('next-05frame');\n",
        "    var prev1Frame = document.getElementById('prev-1frame');\n",
        "    var next1Frame = document.getElementById('next-1frame');\n",
        "    var prev10Frame = document.getElementById('prev-10frame');\n",
        "    var next10Frame = document.getElementById('next-10frame');\n",
        "    var frameRate = 1/30;\n",
        "\n",
        "    video.onloadedmetadata = function(){\n",
        "        //next\n",
        "        next05Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate / 2);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev05Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate / 2);\n",
        "        });\n",
        "\n",
        "        //next\n",
        "        next1Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev1Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate);\n",
        "        });\n",
        "\n",
        "        //next\n",
        "        next10Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate * 10);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev10Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate * 10);\n",
        "        });\n",
        "    };\n",
        "</script>\n",
        "\n",
        "'''\n",
        "\n",
        "HTML(data)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1T-gKAkC3AXH",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown ※If you want to download and check the estimation result, please execute this cell.\n",
        "\n",
        "#@markdown Copy and place it in Google Drive with the file name \"bbox.mp4\".\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "drive_dir_path = os.path.join(base_path, process_img_dir_name)\n",
        "\n",
        "import os\n",
        "os.makedirs(drive_dir_path, exist_ok=True)\n",
        "\n",
        "! cp -f  \"$bbox_file_path\" \"$drive_dir_path\"\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　Copying bbox.mp4 to Google Drive is complete.\")\n",
        "print(\"■　File path: \"+ drive_dir_path +\" /bbox.mp4\")\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9KasFu8GVLrs",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown Tracking timer\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "\n",
        "#@markdown When creating a person tracking designation, I think you can have some time without touching Colab for a certain period of time.\n",
        "\n",
        "#@markdown If you don't touch it for about 20 minutes, you will get a warning.\n",
        "\n",
        "#@markdown If you execute this cell, it will be a timer for any number of minutes, so the warning will not be issued.\n",
        "\n",
        "timer_minutes = 10  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown You can stop the timer in the middle with the keyboard shortcut of \"Ctrl + M + I\".\n",
        "\n",
        "from time import sleep\n",
        "from tqdm import tqdm\n",
        "\n",
        "try:\n",
        "    for _ in tqdm(range(timer_minutes * 60)):\n",
        "        sleep(1)\n",
        "except:\n",
        "    print(\"\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** INFO **\")\n",
        "    print(\"■　The timer has been interrupted.\")\n",
        "    print(\"■　Execute the following cell.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o1yUMzZp4fYq"
      },
      "source": [
        "### ⑥-C　Person tracking designation\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydpLN3F0Gk7Z",
        "cellView": "form"
      },
      "source": [
        "#@markdown Please specify INDEX so that each person can connect to the end of the video one by one.\n",
        "\n",
        "#@markdown Outputs only the specified ID.\n",
        "\n",
        "#@markdown From top to bottom, the first person, the second person, and so on. (Assuming up to 10 people)\n",
        "\n",
        "#@markdown After specifying, execute this [⑥-C] cell. This cell can be executed repeatedly.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-C】　Person tracking designation\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "order_1 = \"0\"  #@param {type: \"string\"}\n",
        "\n",
        "order_2 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_3 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_4 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_5 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_6 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_7 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_8 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_9 = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "order_data = [order_1, order_2, order_3, order_4, order_5, order_6, order_7, order_8, order_9]\n",
        "order_file_path = f'{process_img_dir_path}/order.csv'\n",
        "\n",
        "with open(order_file_path, 'w') as f:\n",
        "    for order in order_data:\n",
        "        f.write(order)\n",
        "        f.write(\"\\n\")\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　The person INDEX was specified successfully.\")\n",
        "print(\"■　File Path: \" + order_file_path)\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "exec_dict['6-C'] = True\n",
        "\n",
        "play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fxxGD35U0OkC"
      },
      "source": [
        "### ⑥-D　Tracking specified execution"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zt1_MhU50X3J",
        "cellView": "form"
      },
      "source": [
        "#@markdown Executes person tracking with the INDEX specified earlier.\n",
        "\n",
        "#@markdown Execute this [⑥-D] cell. The final conversion process takes time.\n",
        "\n",
        "#@markdown This cell can be executed repeatedly.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-D】　Tracking specified execution\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '3' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ③ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '4' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ④ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '5' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑤ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not os.path.exists('/content/expose_mmd/data') or not os.path.exists('/content/expose_mmd/data/models'):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The training data placement may have failed.\")\n",
        "    print(\"■　① Return to -C and try again.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    \n",
        "else:\n",
        "    ! cd expose_mmd/ && python executor.py --img-dir \"$process_img_dir_path\" --order-file \"$order_file_path\" --process order --verbose 20 --log-mode 0\n",
        "\n",
        "    # mp4に変換\n",
        "    ordered_file_path = f\"{process_img_dir_path}/ordered_bbox.mp4\"\n",
        "\n",
        "    # from base64 import b64encode\n",
        "    # mp4 = open(ordered_file_path,'rb').read()\n",
        "    # data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "\n",
        "    # ! rm './out2.webm'\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** INFO **\")\n",
        "    print(\"■　Preparations for executing cell ⑥-E will start.\")\n",
        "    print(\"■　Please wait.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    # !ffmpeg -i \"$ordered_file_path\" -vcodec vp9 ./out2.webm\n",
        "    # ! ffmpeg -i \"$ordered_file_path\" -c:v libvpx-vp9 -b:v 1000K -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1 -lag-in-frames 25 -c:a libopus -b:a 64k -f webm ./out2.webm\n",
        "    ! ffmpeg -i \"$ordered_file_path\" -c:v libvpx-vp9 -b:v 1000K -r 30 -threads 8 -speed 1 -tile-columns 6 -frame-parallel 1 -auto-alt-ref 1 -lag-in-frames 25 -c:a libopus -b:a 64k -f webm -y out2.webm\n",
        "\n",
        "    # process_img_pathes = sorted(glob.glob(os.path.join(process_img_dir_path, \"frames\", \"**\", \"order_*.png\")))\n",
        "    # process_bbox_path = \"./out2.webm\"\n",
        "    # save_webm(process_img_pathes, process_bbox_path)\n",
        "\n",
        "    exec_dict['6-D'] = True\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　Cell ⑥-E is ready to run. Please proceed to the next.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fF-DJps63lr8"
      },
      "source": [
        "### ⑥-E　Tracking designation confirmation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Riv5Hvmm3RTh",
        "cellView": "form"
      },
      "source": [
        "#@markdown Make sure you are tracking the person as you intended.\n",
        "\n",
        "#@markdown Execute this [⑥-E] cell.\n",
        "\n",
        "#@markdown If it is not as expected, go back to ⑥-C and specify INDEX again.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-E】　Tracking designation confirmation\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "exec_dict['6-E'] = True\n",
        "\n",
        "from IPython.display import HTML\n",
        "import base64\n",
        "import io\n",
        "\n",
        "video = io.open('./out2.webm', 'r+b').read()\n",
        "encoded = base64.b64encode(video)\n",
        "data  = '''\n",
        "<video id=\"out\" width=\"1280\" height=\"720\" controls><source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" /></video>\n",
        "'''.format(encoded.decode('ascii')) + '''\n",
        "<br>\n",
        "<button id=\"prev-10frame\">Back 10F</button>\n",
        "<button id=\"prev-1frame\">Back 1F</button>\n",
        "<button id=\"prev-05frame\">Back 0.5F</button>\n",
        "<button id=\"next-05frame\">Forward 0.5F</button>\n",
        "<button id=\"next-1frame\">Forward 1F</button>\n",
        "<button id=\"next-10frame\">Forward 10F</button>\n",
        "(If it is misaligned, move it by 0.5F to adjust it.)\n",
        "<script type=\"text/javascript\">\n",
        "    var video = document.getElementById('out');\n",
        "    var prev05Frame = document.getElementById('prev-05frame');\n",
        "    var next05Frame = document.getElementById('next-05frame');\n",
        "    var prev1Frame = document.getElementById('prev-1frame');\n",
        "    var next1Frame = document.getElementById('next-1frame');\n",
        "    var prev10Frame = document.getElementById('prev-10frame');\n",
        "    var next10Frame = document.getElementById('next-10frame');\n",
        "    var frameRate = 1/30;\n",
        "\n",
        "    video.onloadedmetadata = function(){\n",
        "        //next\n",
        "        next05Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate / 2);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev05Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate / 2);\n",
        "        });\n",
        "\n",
        "        //next\n",
        "        next1Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev1Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate);\n",
        "        });\n",
        "\n",
        "        //next\n",
        "        next10Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.min(video.duration, video.currentTime + frameRate * 10);\n",
        "        });\n",
        "\n",
        "        //prev\n",
        "        prev10Frame.addEventListener('click',function(){\n",
        "            video.currentTime = Math.max(0, video.currentTime - frameRate * 10);\n",
        "        });\n",
        "    };\n",
        "</script>\n",
        "\n",
        "'''\n",
        "\n",
        "HTML(data)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P2BtROyJ4yCD",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown ※If you want to download and check the tracking specification result, please execute this cell.\n",
        "\n",
        "#@markdown Copy and place it in Google Drive with the file name \"ordered_bbox.mp4\".\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "drive_dir_path = os.path.join(base_path, process_img_dir_name)\n",
        "\n",
        "import os\n",
        "os.makedirs(drive_dir_path, exist_ok=True)\n",
        "\n",
        "! cp -f  \"$ordered_file_path\" \"$drive_dir_path\"\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　The copy of \\\"ordered_bbox.mp4\\\" to Google Drive is complete.\")\n",
        "print(\"■　File Path: \" + drive_dir_path + \"/ordered_bbox.mp4\")\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NizDhi8h4nG3"
      },
      "source": [
        "### ⑥-F　Output Motion"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "10ssIxUAMUl8",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown 【Cell ⑥-F】　Motion output\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown Check any function you want to execute.\n",
        "\n",
        "#@markdown This cell can be executed repeatedly.\n",
        "\n",
        "# #@markdown **身体モーション出力**　（表情のみを出力したい場合などはOFFにしてください）\n",
        "\n",
        "# process_motion_body = True #@param {type:\"boolean\"}\n",
        "\n",
        "# #@markdown **上半身のみのモーション出力**　（グルーブを出力しません）\n",
        "\n",
        "# process_motion_upper = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown **Hand motion output**　（It will take some time）\n",
        "\n",
        "process_motion_hand = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown **Facial expression output**　（It doesn't take much time）\n",
        "\n",
        "process_motion_face = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown If you can check it, execute this [⑥-F] cell.\n",
        "\n",
        "#@markdown The following file is output under the folder of \"` <video file name>_<execution date, time, minute, second (US time)> `\" under the `autotrace` folder of Google Drive, so download it. please.\n",
        "\n",
        "#@markdown     - Stickman model（`trace_<Execution date, day, hour, minute, second (US time)>_rot_model_no<Tracker No>.pmx`）\n",
        "\n",
        "#@markdown     - Stickman motion（`trace_<Execution date, day, hour, minute, second (US time)>_rot_no<Tracker No>.vmd`）\n",
        "\n",
        "#@markdown     - Motion for Animasa Miku（`trace_<Execution date, day, hour, minute, second (US time)>_miku_smooth_no<Tracker No>.vmd`）\n",
        "\n",
        "#@markdown     - Stickman expose Moving motion JSON data（`trace_<Execution date, day, hour, minute, second (US time)>_rot_no<Tracker No>.json`）\n",
        "\n",
        "#@markdown     - Stickman mediapipe Moving motion JSON data（`mp_trace_<Execution date, day, hour, minute, second (US time)>_rot_no<Tracker No>.vmd`）\n",
        "\n",
        "#@markdown     - HTML for viewing JSON data（`viz_pos.html`）\n",
        "\n",
        "#@markdown The motion conforms to the model in which the groove and upper2 bones are added with a semi-standard plug-in to the \"Animasa Miku (Hatsune Miku Ver2.pmd)\" included in MMD.\n",
        "\n",
        "if not exec_dict or '1-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '1-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ①-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '2' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ② may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '3' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ③ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "if not exec_dict or '4' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ④ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '5' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑤ may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-A' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-A may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-B' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-B may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-C' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-C may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-D' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-D may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not exec_dict or '6-E' not in exec_dict.keys():\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　Cell ⑥-E may not be executed.\")\n",
        "    print(\"■　Please go back from the table of contents and execute.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "elif not os.path.exists('/content/expose_mmd/data') or not os.path.exists('/content/expose_mmd/data/models'):\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** ERROR **\")\n",
        "    print(\"■　The training data placement may have failed.\")\n",
        "    print(\"■　Go back to ①-C and try again.\")\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    \n",
        "else:\n",
        "    process_txt = \"mediapipe,smooth,motion\"\n",
        "\n",
        "    motion_hand_flg = \"1\" if process_motion_hand else \"0\"\n",
        "    motion_face_flg = \"1\" if process_motion_face else \"0\"\n",
        "\n",
        "    ! cd expose_mmd/ && python executor.py --img-dir \"$process_img_dir_path\" --order-file \"$order_file_path\" --process $process_txt --hand-motion $motion_hand_flg --face-motion $motion_face_flg --verbose 20 --log-mode 0\n",
        "\n",
        "    import shutil\n",
        "\n",
        "    # 日本語対策でpythonコピー\n",
        "    output_name = os.path.basename(input_video_name).replace('.', '_')\n",
        "    output_path = os.path.join(base_path, process_img_dir_name)\n",
        "\n",
        "    os.makedirs(output_path, exist_ok=True)\n",
        "\n",
        "    import glob\n",
        "    for target_file_name in [\"trace_*_mov_model_no*.pmx\", \"trace_*_rot_model_no*.pmx\", \"trace_*_mov_no*.vmd\", \"trace_*_rot_no*.vmd\", \"trace_*_miku_smooth_no*.vmd\", \"*.json\"]:\n",
        "        for vmd_path in glob.glob(os.path.join(process_img_dir_path, \"motion\", target_file_name)):\n",
        "            out_vmd_path = os.path.join(output_path, os.path.basename(vmd_path))\n",
        "            shutil.copy(vmd_path, out_vmd_path)\n",
        "\n",
        "    shutil.copy(f'{process_img_dir_path}/ordered_bbox.mp4', output_path)\n",
        "    shutil.copy('expose_mmd/viz_pos.html', output_path)\n",
        "\n",
        "    exec_dict['6-F'] = True\n",
        "\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "    print(\"■　** OK **\")\n",
        "    print(\"■　The motion output was successful.\")\n",
        "    print(\"■　File Path: \" + output_path)\n",
        "    print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "    play_sound()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LnAUBEOeHoPe"
      },
      "source": [
        "# Work after automatic tracing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vu4_v1zqH2gM"
      },
      "source": [
        "## ①　If VMD is not output to Google Drive"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BFA_3plH8Y7"
      },
      "source": [
        " - Check if the message **ERROR**　**CRITICAL** is displayed.\n",
        " - If no message is displayed, check if the VMD file is output to the processing output folder.\n",
        "   1. Click the folder button on the left side of the screen.\n",
        "\n",
        " ![フォルダーボタン](https://drive.google.com/uc?export=view&id=1AoTQjFHaz2uY4jGJttmVRkWk2nct2kb3)\n",
        "\n",
        "   2. Check if the file is created under the output folder (`output/<Video file name>_<Execution date, day, hour, minute, second>/motion`). \n",
        "\n",
        " ![motionフォルダ](https://drive.google.com/uc?export=view&id=1wJsomaHWFwSyt6lmt7Zuli8AJ-cxi3uX)\n",
        "\n",
        "     - If you have a PMX or VMD file, download it.\n",
        "        - Required files are listed in [⑥-F].\n",
        "        - Select the file and right-click to display the menu.\n",
        "     - If you don't have a PMX or VMD file, you're getting an error somewhere.\n",
        "     - If you don't know the cause of the error, share your notebook.\n",
        "       -  Please check「[Preparation](https://colab.research.google.com/github/miu200521358/motion_trace_colab/blob/master/AutoTraceIntroduction.ipynb)」for how to share.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qxwONV2ULFc_"
      },
      "source": [
        "## ② 　If you want joint position data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vDutzsxhGKcy"
      },
      "source": [
        "When joint position data (result data of posture estimation, person tracking, facial expression recognition) is required\n",
        "\n",
        "When you execute the cell below, a zip containing the relevant data will be copied to Google Drive, so please download it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B4GLtUJIGMNu",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown If you want raw data, run this cell.\n",
        "\n",
        "#@markdown It will take a few minutes after the copy is complete to be visible on Google Drive.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "json_org_path = os.path.join(process_img_dir_path, 'ordered')\n",
        "zip_name = f'{os.path.basename(process_img_dir_path)}_ordered.zip'\n",
        "zip_path = os.path.join(process_img_dir_path, zip_name)\n",
        "drive_zip_path = os.path.join(base_path, zip_name)\n",
        "\n",
        "! zip -r \"$drive_zip_path\" \"$json_org_path\"\n",
        "\n",
        "# ! echo \"$drive_zip_path\"\n",
        "# ! echo \"$zip_path\"\n",
        "\n",
        "# ! cp \"$zip_path\" \"$drive_zip_path\"\n",
        "\n",
        "# import shutil\n",
        "# shutil.copyfile(zip_name, )\n",
        "\n",
        "# import shutil\n",
        "# shutil.copytree(f'{process_img_dir_path}/ordered', json_path)\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　The raw data has been copied to Google Drive.\")\n",
        "print(\"■　File Path: \" + drive_zip_path)\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2uIu8L07HMMz",
        "cellView": "form"
      },
      "source": [
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "#@markdown If you want the data after smoothing, please execute this cell.\n",
        "\n",
        "#@markdown It will take a few minutes after the copy is complete to be visible on Google Drive.\n",
        "\n",
        "#@markdown ■■■■■■■■■■■■■■■■■■\n",
        "\n",
        "json_org_path = os.path.join(process_img_dir_path, 'smooth')\n",
        "zip_name = f'{os.path.basename(process_img_dir_path)}_smooth.zip'\n",
        "zip_path = os.path.join(process_img_dir_path, zip_name)\n",
        "drive_zip_path = os.path.join(base_path, zip_name)\n",
        "\n",
        "! zip -r \"$drive_zip_path\" \"$json_org_path\"\n",
        "\n",
        "# ! echo \"$drive_zip_path\"\n",
        "# ! echo \"$zip_path\"\n",
        "\n",
        "# ! cp \"$zip_path\" \"$drive_zip_path\"\n",
        "\n",
        "# import shutil\n",
        "# shutil.copyfile(zip_name, )\n",
        "\n",
        "# import shutil\n",
        "# shutil.copytree(f'{process_img_dir_path}/smooth', json_path)\n",
        "\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "print(\"■　** OK **\")\n",
        "print(\"■　After smoothing, the data has been copied to Google Drive.\")\n",
        "print(\"■　File Path: \" + drive_zip_path)\n",
        "print(\"■■■■■■■■■■■■■■■\")\n",
        "\n",
        "play_sound()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O1U9nW7Fl6Bv"
      },
      "source": [
        "## ③　If you want to modify the depth or jump\n",
        "\n",
        "After entering the approximate depth (center Z) and lateral movement amount (center X) for all parents, use \"全親移植\" of [モーションサポーター](https://seiga.nicovideo.jp/seiga/im10581987).\n",
        "\n",
        "Center Y is divided into grooves, so I think it's better to manually correct jumps."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IkwK61mP32Tn"
      },
      "source": [
        "## ④　When you want to convert foot FK data to foot IK data\n",
        "\n",
        "Please use 「足FKtoIK」 of [モーションサポーター](https://seiga.nicovideo.jp/seiga/im10581987).\n",
        "\n",
        "**Recommended settings**\n",
        "\n",
        " 1. 「足首水平化：ON」「かかと・つま先Y=0：ON」 -> 「足FKtoIK」Execute\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o5BxR5tGM2k-"
      },
      "source": [
        "## ⑤　When you want to remove unnecessary keys from all hit data\n",
        "\n",
        "Please use 「スムージング」 of [モーションサポーター](https://seiga.nicovideo.jp/seiga/im10581987).\n",
        "\n",
        "※If you delete unnecessary keys for smoothing, it will deviate slightly from the original movement.\n",
        "\n",
        "**Recommended settings**\n",
        "\n",
        " 1. 「処理対象：全ボーン・モーフ」「処理回数：4」「補間方法：補間曲線に従う」「不要キー削除処理を追加実行する：チェックON」 -> 「スムージング」Execute\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-SMqIYJMa2p"
      },
      "source": [
        "## ⑥　When you want to convert from Animasa Miku to motion data that matches other models\n",
        "\n",
        "Please use [VMDサイジング](https://seiga.nicovideo.jp/seiga/im9755721) .\n",
        "\n",
        "**How to create the original model**\n",
        "\n",
        "1. Introduce Soboro's [Semi-standard plug-in](http://www.nicovideo.jp/watch/sm14956092) into PmxEditor\n",
        "2. Start PmxEditor\n",
        "3. Load \"Hatsune Miku Ver2.pmd\" included in MMD\n",
        "4. Execute the semi-standard plug-in and introduce \"Root\" and \"Groove\" and \"Upper2\"\n",
        "\n",
        "**Recommended settings**\n",
        "\n",
        "1. 「元モデル：あにまさ式ミク準標準」「スタンス追加補正：ON」「捩り分散：ON」 -> Sizing to your favorite model\n",
        "\n",
        "**Recommended work order**\n",
        "\n",
        "1. Center position adjustment (Animasa Miku: manual)\n",
        "2. All parent transplant ③ (Animasa Miku: Motion supporter)\n",
        "3. Foot FKtoIK④ (Animasa Miku: Motion Supporter)\n",
        "4. Smoothing ⑤ (Animasa Miku: Motion Supporter)\n",
        "3. Sizing ⑥ (Animasa Miku → Arbitrary model: VMD sizing)\n",
        "4. Motion correction (any model: manual)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HDG_5rZFaiha"
      },
      "source": [
        "## ⑥　If you want to trace another video\n",
        "\n",
        "The Colab environment will be retained for 12 hours if there is no operation for 90 minutes.\n",
        "\n",
        "You can upload another video from ④ and start over.\n",
        "\n",
        "※ If you get an error like the one below, your environment has already been destroyed.\n",
        "\n",
        "Go back to cell ① from the table of contents and try all over again.\n",
        "\n",
        "![Googleドライブ連携](https://drive.google.com/uc?export=view&id=1Tsjzs2QAECPMfUTaTh7PYX2cMEypEIF2)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nm28k-hcifr"
      },
      "source": [
        "## ⑦　If you want to start over\n",
        "\n",
        "1. Click \"Runtime\"> \"Reset Runtime\"> \"Yes\" from the menu.\n",
        "\n",
        " ![リセット](https://drive.google.com/uc?export=view&id=1YNxEPW1KISxWKbGn7DntfFYfGcgCd0q2)\n",
        "\n",
        "2. Click \"Reconnect\" at the top right of the menu.\n",
        "\n",
        " ![再接続](https://drive.google.com/uc?export=view&id=1pRXjasT_B1Ep0-6oLCB6FRsbwa4OTkHt)\n",
        "\n",
        "3. The reset is complete when the green check mark is displayed.\n",
        "\n",
        " ![リセット完了](https://drive.google.com/uc?export=view&id=1QN6jY6ApcYQVkxY8ISLGDYLTEnQlyqTm)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kbsMHQ7QM8zm"
      },
      "source": [
        "# License & credit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KV5FmXRDdKwN"
      },
      "source": [
        "Please be sure to check the license when publishing or distributing the results of MMD automatic tracing. The same applies when applying to other applications such as Unity.\n",
        "\n",
        "[MMD Auto Trace v2 License & Credit](https://github.com/miu200521358/expose_mmd/wiki/MMD自動トレースv2-ライセンス・クレジット)"
      ]
    }
  ]
}