Untangling the ThreeJS animation structure

[bhouston]

Been looking at modernizing/formalizing the ThreeJS animation framework to be more similar to Unity/Unreal Engine 4. There is a lot of great code in ThreeJS but there is a bit of missing top level organization of animation structures, rather there is a lot of duplication.

Here are the classes I've been studying today:

• SkinnedMesh - derived from Mesh, adds support for array of Bone and Skeleton

The AnimationHandler/Animation pair:

• AnimationHandler - a global object for creating and advancing playing Animation
• Animation - a hierarchy of keyframe animation tracks usually used for bones on a skinned mesh, closely linked to AnimationHandler.

The following is not compatible with the Animation/AnimationHandler framework:

• KeyframeAnimation - AnimationHandler independent Keyframe track, only used by Collada example.
• MorphAnimation - AnimationHandler independent MorphTarget animation, only used by *morphtargets_horse.html examples.

The following combine scheduling of animations with meshes/mesh hierarchies:

• BlendCharacter - SkinnedMesh + Animation Scheduler (sometimes used with manually animated morph targets, such as in webgl_animation_skinning_morph.html)
• MorphAnimMesh - single Morph Target animation (for simple repeating things.)
• MorphBlendMesh - multiple Morph Target animation blending (meshes that change between morph target animations.)
• UCSCharacter - SkinnedMesh + Animation (only used in webgl_morphtargets_human.html)
• MD2Character - A bunch of MorphAnimMeshes (used only in webgl_morphtargets_md2.html)
• MD2Charactercomplex - A bunch of MorphBlendMeshes (used only in webgl_morphtargets_md2_control.html)

I'm trying to think of a way of reducing all of these classes into a clearer structure.

I think a structure similar to this might be the way forward:

• Track - a keyframe animation that contains a name, a set of keys that are composed of times and values. Keys can be weights, or pos/rot/scale. The name of the track can refer to morphTargets or bones/nodes or possibly other values within a node. (We could even abstract Track to not be keyframed, it could be equation based as well if we get the interface right.)
• Clip - a set of tracks with a name, duration. An animation sequence, such as blink, walk, jump, etc. Can be a morph or a keyframe or maybe both. Similar to Unity's AnimationClip class
• Action - a clip that is being played. Would be associated with a root node to which the Track names would be relative. Would have a blend weight associated with it and can be queued into animation system. Could optionally loop and have a play speed. (Will likely require more parameters for various edge cases.) Similar to Unity's AnimationState class
• Mixer - a container for a number of actions. Would be responsible for applying the actions to the hierarchy -- it would be sort of like AnimationHandler, it could be global or there could be multiple -- I'm open on this case. It would be responsible for blending between actions.
• Animator - a higher level structure on top of Mixer that would contain the logic to do things like walk, run, jump, it would do that by managing what actions are queued up and how they are blended between in order to create a smooth result. (Will have to be easy to extend to support new entity behaviors.) Similar to Unity's Animator class

I would get rid of AnimationHandler, Animation (replaced by Clip/Track), KeyframeAnimation (replaced by Track), MorphAnimation (replaced by Track), BlendCharacter (replaced by Mixer/Clip/Track), MorphAnimMesh (replaced by Mixer/Clip/Track), MorphBlendMesh (replaced by Animator/Mixer/Clip/Track), USCCharacter, MD2Character, MD2CharacterComplex (replaced by Animator/Mixer/Clip/Track), or at least deprecate them in favor for the Track/Clip/Action/Mixer/Animator design, and this design could drive any objects in ThreeJS, including Materials, Lights and Cameras, if you so desired.

Basically this is a more flexible and actually simpler design.

The above is very similar to Unity/Unreal Engine.

[mrdoob]

Agreed! The animation side is a bit of a mess at the moment.
Your proposal sounds good to me!

[titansoftime]

This would be amazing =]

[MasterJames]

I just wanted to suggest we consider to import or maybe natively adopt the http://alembic.io/ format (way of thinking) at this time.
+1 to making a smarter animation structure here.
I also wondered if there's some c to js transpiling possible from any existing code maybe from blender etc.

[MasterJames]

Actually alembic is ment for baking procedural stuff I think so maybe not so good for animation.

[bhouston]

@MasterJames, Alembic is great -- we have a product for Alembic that is fairly popular: http://exocortex.com/alembic And you are correct, it is used in games, but mostly only in the cut sequences part where everything is baked.

[MasterJames]

It seems FBX is the best but not officially documented. Here I found something possibly useful...
https://code.blender.org/2013/08/fbx-binary-file-format-specification/

[bhouston]

@MasterJames, I think I'll stay as much as possible with the existing JSONLoader format just because there is a lot of content and a lot of existing converters. I want to change the classes that load it though once it is in ThreeJS and how one uses the animation content.

The other format that may be of interest to load into the same structures would be @fabrobinet's glTF's format.

Both the JSONLoader and glTF are designed for the web, so I'd favor them.

FBX is sort of problematic as it is a proprietary format that changes a couple times a year and it is very hard to support -- most 3D content creation tools do not support it properly. But this design should be generally compatible with the internal organization of the FBX file format. :)

But no matter what format you choose to use as the data format, it doesn't really change the main design of the run-time structures.

[MasterJames]

Okay cool. This corispondence helps bring clarity to the task at hand. Thanks.

[tschw]

I think I'll stay as much as possible with the existing JSONLoader format

Yes, please. It's improvable at places - but it's actually capable of development. There's no such freedom with any other format.

Both the JSONLoader and glTF are designed for the web, so I'd favor them.

glTF is an interesting specification with good ideas and a 200% complete data model. Something to learn from and hopefully stop near the hundred. What is a super flexible format without tools that can handle its flexibility? Other than for glTF we have workable content pipelines and complete and efficient loaders for three.json today.

[bhouston]

A quick first pass at the system I described. Still needs a ton of work before it is functional:

dev...bhouston:modernAnimationSystem

/ping @amirebrahimi

[MasterJames]

Looking good!
I only though this repeated code should be put in a temp var/let
data.hierarchy[ h ].keys[ k ].rot

[amirebrahimi]

Great first pass at laying down the broad structure. Can you share more about the Track class and what other types of Tracks you expect to be used? For example, ConstantTrack to me is the same as a KeyframeTrack with one key at 0. If there aren't many other Tracks that you have planned, then I would propose simply having Track being synonymous with KeyframeTrack (i.e. merging the concept).

[bhouston]

One other Track type could be a Procedural / Equation Track type. Basically something like getAt( time ) { return Math.cos( time ); }.

[amirebrahimi]

Okay, so it looks like you're considering usage outside of what most DCC's would export.

[bhouston]

So I have it working with multiple different types of clips.

The latest is here:

dev...bhouston:modernAnimationSystem

It can drive morphTarget animations, visibility, material properties (colors, etc.), position, rotation, scale, and it can drive subcomponents, like position[x], rather than the whole position value. Just have to get it driving bones and the core of the system is complete.

Examples of how to create animations (morph, color, visible, position, rotation, scale) are here:

dev...bhouston:modernAnimationSystem#diff-5b5dbd25ed58e9a9ed8710f62cbbcc76R163

I think this can replace a lot of stuff in the Character animations, but also a lot of the Collada and @fabrobinet's glTF loader -- right now those loaders are sort of very special case in terms of how they handle loading and playing back animations.

[mrdoob]

Looking good! Thanks a lot for cleaning this up!

[getzelone]

That's a must do : different animation clips which we can blend or mix with different layers (walking and say hi with the hand in the same time).

[bhouston]

@getzelone I've added a lot of this already.

• AnimationMixer.fadeIn
• AnimationMixer.fadeOut
• AnimationMixer.warp
• AnimationMixer.crossFade

You can set the weight of all Actions and it will blend between them properly and between the original value on the objects.

I've also started to convert the standard ThreeJS animation examples to use the new system. Here is a skeleton keyframe animation (240 tracks) and a morph target animation (4 morph targets) running together through the mixer at 60fps:

http://exocortex.github.io/three.js/examples/webgl_animation_skinning_morph.html

[crobi]

Nice work! I have been thinking about an animation redesign myself for some time, so here are some random related thoughts. I might have mentioned some of this already in another issue, so apologies if I'm repeating myself.

https://github.com/bhouston/three.js/tree/modernAnimationSystem/src/animation

Track

a set of keys that are composed of times and values

If you define a track as a pure function that maps time values to weights/positions/rotations/scales, you can support non-uniform keyframe tracks (array of time/value pairs), uniform keyframe tracks (array of values), and parametric curves.
Looks like you implemented this already as THREE.KeyframeTrack.getAt(time).

Note: Evaluating a non-uniform keyframe track at a given time involves finding the two neighboring keyframes. As a full search might be too slow, such tracks may want to cache the last found position. This does not work if you want to play back the same track on multiple scene objects - if you want to render an army of marching characters, you want to store the marching animation only once.
That's why, in my opinion, the tracks should be completely stateless and any cache should be part of the object that actually plays back the animation (AnimationAction?).
The interface THREE.KeyframeTrack.getAt(time) does not allow the track to use a cache. Maybe you could add an optional parameter: THREE.KeyframeTrack.getAt(time, cache)?

Note: Looks like your THREE.KeyframeTrack uses an array of objects. Wouldn't be a pair of Float32Arrays - one for the times, one (interleaved) for the values - be more performant? More memory coherency, less impact on the garbage collector?

Note: I have no experience with this whatsoever, but wouldn't it pay off to uniformely resample animations into what I have called uniform keyframe tracks above? You'd have a guaranteed O(1) access and no headaches with finding the right keyframes. When storing sparse keyframes, you have to know the interpolation method used (or you get large deviations), so in practice sparse keyframes are only useful for animations authored with the same interpolation method as the three.js runtime animation system uses.

Action

Would have a blend weight associated with it and can be queued into animation system.

Do actions need to know about their weight? Shouldn't this be controlled by the mixer?

Mixer / Animator

As far as I understand it, the Unity 5.0 Mecanim system has three levels of animation mixing:

• Animation Blend Trees interpolate between similar animations. The leaves of the tree are animation clips. E.g., a blend between a walking and sprinting animation to generate a jogging animation.
• Animation State Machines control the transition between unrelated animations. Nodes of the state machine graph are animation blend trees (or individual animation clips). E.g., if you want to transition from running to jumping, you have to wait until a foot touches the ground, at which point the character can jump off. There might be a significant delay between the time when you request a transition and the time when the transition is executed.
• Animation Layers allow animations to be applied to different parts of a skeleton. Each layer has an animation state machine and a bone mask. Each skeleton bone is affected by the top-most layer that includes the bone.

Animation blend trees

Animation blend trees are very flexible. In additional to blending between similar animations (walk/run), blend tree nodes can be used to transition between two unrelated animations. If the transition is short, if often looks ok. Unreal engine for example has a "blend by int" node where you specify an array of animations and an index of the desired animation, and the node will smoothly blend between the current and the desired animation.

When designing a blend tree, I would consider separating the following concepts:

• The input parameters. Defined by the application state for each animated object, this could be key/value pairs such as {"speed":10, "idle":false}
• The blend tree shape. The actual tree of blend nodes. Each node blend tree node would store which input parameter name(s) it is associated with. This can be shared by multiple objects (again, consider rendering a marching army).
• The blend tree state. Holds all internal state of blend tree nodes, such as the current weights of all blend tree nodes. Each animated object needs its own state.

I have implemented my own proof of concept for animation blend trees in this project. The code completely replaces the three.js animation code in a very hacky way, but is capable of applying a blend tree to a character with 50 bones at several hundred fps. It is a very early prototype, but feel free to use any ideas if you like. The blending code is based on the primitives sampleAnimation (sample an animation clip at a given time, store as a "pose"), blendPose (linear blend between "poses"), and exportPose (flatten a "pose" into a bone matrix texture). In this design, blend tree nodes would ask their children to compute their poses and then call blendPose to interpolate between those poses (such as here).

[bhouston]

I've submitted a PR that implements this -- #6934 Off to bed now.

[bhouston]

@crobi thanks for the input!

Regarding your ideas for tracks, I've implemented most of what you have suggested. Although I am using an array/object representation for the track data. I think that someone can create a BufferKeyframeTrack to optimize that part fairly easily - and it should be a drop in replacement that will still be compatible with the rest of the mixer.

Regarding the mixer/animator. I've only created the first layer, but I believe you need to start somewhere. I also think that the next layers are needed, but hopefully can be done as further improvements to this system.

Although the clips are sparse with regards to bones. Thus you can specify which bones you want to drive using a clip, and it doesn't have to be all of them. Thus there is sort of layer support. We can add further features there without too much difficulty I think.

The blend trees are very interesting-- I didn't know about them before. I think that one could create a blend tree using the Clip/Action and putting it into a tree form. Not too much work I think, but I would prefer to do that as a separate PR rather than further expanding this PR -- it is already huge.

[crobi]

The blend trees are very interesting-- I didn't know about them before.

Many game engines and animation middlewares (including Unity and Unreal Engine) use blend trees. Here's some links:

• Gamasutra post about blend trees in mech warrior
• Unity blend trees
• Non-trivial blend tree example in Unity
• Unreal engine blend trees
• Unreal engine blend nodes

Even though many engines use blend trees, finding good articles about them is not easy. I guess that's because everyone uses a slightly different design. I've heard of projects using many small blend trees and state transitions between them (see below), and of other projects using a single huge blend tree that includes all the different states and transitions.

I also think that the next layers are needed, but hopefully can be done as further improvements to this system.

The animation state machine of Unity is an advanced system that I've seen less often than the blend trees. Consider the following example of a super mario game:

1. The character is running
2. The player presses the jump button
3. The running animation keeps playing, no immediate blending is started
4. Near the time when the character's foot touches the ground, a short crossfade to the jumping animation is initiated.

The reverse transition is even more obvious - you cannot start blending into the running animation until the character has landed (the jumping animation has finished). You can see this in action in this video.

This is a complex system for defining when animations may transition, and may be even used for a two-way interaction with the application state (by firing scripting events on transitions). This is in contrast to blend trees, which simply define how animations blend.

[bhouston]

@crobi So I think we should implement blend trees, but I think it can be done after the acceptance of this PR. One of the things I would like to avoid is implementing everything in one PR, as the larger a PR gets the harder it is to get accepted.

[liammagee]

I'm not currently using animation in three.js, but plan to eventually. Both
the PR and the accompanying discussion are very helpful, thanks.

On 1 August 2015 at 19:47, Ben Houston [email protected] wrote:

@crobi https://github.com/crobi So I think we should implement blend
trees, but I think it can be done after the acceptance of this PR. One of
the things I would like to avoid is implementing everything in one PR, as
the larger a PR gets the harder it is to get accepted.

—
Reply to this email directly or view it on GitHub
#6881 (comment).

[bhouston]

BTW the more people who can have a look at the new PR and ensure that it is ready to merge,it the better.

[liammagee]

Happy to look it over, though I'm not familiar with the existing animation
code.

On 1 August 2015 at 20:21, Ben Houston [email protected] wrote:

BTW the more people who can have a look at the new PR and ensure that it
is ready to merge,it the better.

—
Reply to this email directly or view it on GitHub
#6881 (comment).

[bhouston]

I've completed the new animation system and it includes updates to the Blender exporter to support BlendShapes and node animations: #6934

[EskelCz]

Please can someone confirm if there's a support for the layered animations in the new system?
Like crobi mentioned, walking and waving simultaneously for example. Thank you

[bhouston]

You can blend between animations and run different animations on different tracks -- animations can be sparse and overlapping with existing animations in terms of time.

[majimboo]

Hello, what kind of data does tracks accept? Is it local to hierarchy or model space?