My codebase for Stanford/EPFL NIPS2017 RL Competition w/ OpenSim

you know, to walk the skeleton.

original DDPG/PPO code from https://github.com/ctmakro/gymnastics

more details at https://github.com/stanfordnmbl/osim-rl

What on earth is EPFL

École polytechnique fédérale de Lausanne (School Polytechnical Federal of Lausanne), the MIT of Switzerland. They seem to speak French.

Dependencies

• Python 3.5 (for training with TF on Windows)
• Python 2.7 (for running osim-rl on Windows)
• Anaconda/Miniconda (for the harmony of the two above)
• TF
• matplotlib
• pymsgbox (act as a stop button instead of using ctrl-c)
• Canton (No.1 DL library on earth)
• numpy, gym
• osim-rl (the competition)
• OpenCV3 (waterfall graph)
• Pyro4 (RPC)
• parse
• IPython (for convenience)

To Run

$ ipython -i ddpg2.py

then enter r(100) to train the agent for 100 episodes.

For PPO implementation, just change the filename:

$ ipython -i ppo3.py

Parallelism / Farming

Current version requires farming (which is VERY IMPORTANT! Please read stanfordnmbl/osim-rl#58). Before starting ddpg2.py, you should first start one or more farm, preferably with Python 2.7, by running python farm.py on each SLAVE machine you own. Then create a farmlist.py in the working directory (on the HOST machine) with the following content:

farmlist_base = [('127.0.0.1', 4),('192.168.1.33',8)]
# a farm of 4 cores is available on localhost, while a farm of 8 available on another machine.
# expand the list if you have more machines.
# this file will be consumed by the host to find the slaves.

Then start ipython -i ddpg2.py. The farmer should be able to reach the farm on your local machine. You can now type r(100) to train.

if you want to use the farming code for other algorithms/purposes, please see farming_demo.py for an example.

Note for users on Win7 x64 + Python 3.5 (2017-07-06)

Assume you want to run osim-rl on Windows w/py35, since TensorFlow support only Python 3.5 on Windows.

Edit: this is not encouraged. Start your env in Py27 and talk to it from Py35 is more sustainable.

1. Build OpenSim yourself (since @kidzik didn't do this for us)

   • Install VC++ 2015 build tools. Should not take long

   • CMD

     • Clone the opensim conda builder, then run the build command:

       (chinese users) > set "HTTPS_PROXY=whatever_you_use"
       > git clone https://github.com/opensim-org/conda-opensim.git
       > cd conda-opensim
       > conda build opensim --python 3.5

       Above will fail on Chinese Windows due to file encoding error. If so, don't run conda build again, as that will create a new build environment and pull the whole OpenSim from GitHub again.

       Instead, open D:\Anaconda3\conda-bld\opensim_1499279773305\work\dependencies\BTK\Utilities\Open3DMotion\src\Open3DMotion\MotionFile\Formats\MDF\FileFormatMDF.cpp and replace every occurence of the square symbol (superscript "2") with "^2".

       The file should now look like:

       //...
       { "Marker Acceleration", "m/s^2", "%.3fm/s2", "%.3f", (float)0.05},
       //...

     • Run the build script generated by the build process above again:

       > cd D:\Anaconda3\conda-bld\opensim_1499279773305(actual-path-may-vary)\work
       > bld.bat

       The compilation should run smoothly this time.

     • Since CMD was used to run bld.bat, you may notice that some commands, such as an cp (copy) did not run successfully. It should produce errors like that:

       (d:\Anaconda3\conda-bld\opensim_1499279773305\_b_env) Nonecp d:\Anaconda3\conda-bld\opensim_1499279773305\_b_env\Library
       \simbody\bin\simbody-visualizer.exe   d:\Anaconda3\conda-bld\opensim_1499279773305\_b_env\simbody-visualizer.exe
       'cp' is not recognized as an internal or external command,
       operable program or batch file.`

       you can do the copy manually, though.

     • also the python setup.py install might not run smoothly:

       (d:\Anaconda3\conda-bld\opensim_1499279773305\_b_env) Nonepython setup.py install
       Traceback (most recent call last):
         File "setup.py", line 8, in <module>
           execfile('opensim/version.py')
       NameError: name 'execfile' is not defined

       • cause

         setup.py is written in py2, so it seems that i built the py2(default) branch of OpenSim.

       • solution

         in setup.py, replace execfile(name) with exec(open(name).read()).

       Then cd to the correct path (D:\Anaconda3\conda-bld\opensim_1499279773305\_b_env\Library\sdk\python) and manually run pip install -e .

     • Now test the OpenSim installation:

       • try import opensim in python.

         upon import opensim python may fail to find simbody's libraries.

       • solution

         add D:\Anaconda3\conda-bld\opensim_1499279773305\work\opensim_build\Release to your PATH.

2. install osim-rl

   • CMD

     (chinese users) > set HTTPS_PROXY=whatever_you_use
     > git clone https://github.com/stanfordnmbl/osim-rl.git
     > cd osim-rl
     > pip install -e .
     > cd tests
     > python test.manager.py

     You should now see the skeleton swinging!

The simulation is too slow

Update: this is not the goto choice since it introduce bias. your agent may not peform well during submission.

• modify D:\Anaconda3\conda-bld\opensim_1499279773305\work\OpenSim\Simulation\Manager\Manager.cpp as follows:

  Manager::Manager(Model& model) : Manager(model, true)
  {
      SimTK::Integrator *vi = new SimTK::RungeKutta2Integrator(_model->getMultibodySystem());
      vi->setAccuracy(3e-2); // reduce accuracy saves time
      _defaultInteg.reset(vi);
      _integ = *_defaultInteg;
  }

  then build the whole thing again by running bld.bat mentioned above.

• parallelize the training environment (see code)

Visualize in environments without a graphic card (i.e. on cloud)

• Windows users:

  • download precompiled Mesa3D for Windows at https://github.com/pal1000/mesa-dist-win/releases.

  • Self-extract that on your remote machine, in my case Windows Server 2008 R2.

  • Run the extracted quickdeploy.cmd. I installed opensim for Py27 via a conda package built by kidzik (as per instructions on github/stanfordnmbl/osim-rl/README), into a conda virtual environment called osrl, therefore the simbody-visualizer.exe is located in c:\programdata\miniconda3\envs\osrl\. Paste that path into the CMD window and let the quickdeploy.cmd do the rest for you.

  • Now you can start RunEnv(visualize=True) without getting any errors! Mesa3D just emulated OpenGL for you.

• Linux users:

  • Not much I could help, but the approach should be the same: if OpenGL is not supported, emulate it.