This tutorial was written for anybody working on animations that want to a character to automatically lipsync. Lipsyncing is used to move the mouth of a character to match the dialogue it is speaking. How the mouth should move is calculated in the code using the voice recording of the dialogue as well as a written transcript. This takes the line and converts it into phonemic characters. These are then converted into visemes which are used to move the mouth. The visemes has to be created by hand and that is what this tutorial will detail.

A viseme is a position that the mouth has when making a sound. We use a 3rd party imlementation from Annosoft. They have three different viseme mappings that can be used, each of different quality. The best quality comes from using 17 Visemes.

These pages show which visemes are needed and how to model them

http://www.annosoft.com/docs/Visemes17.html

http://www.annosoft.com/docs/Visemes12.html

http://www.annosoft.com/docs/Visemes9.html

Modeling

This is preferably done in Maya and the latest Collada plug-in shall be used. This has been included in the “plugin” folder.

Make sure that the unit type is set to meters!

The object shall be positioned in the file so that the bottom of the object is at the center of the screen.

When texturing the objects, make sure that the diffuse texture is what is set as texture for all sub meshes.

Material

Each material will require one or more the following textures:

Diffuse
Size: Normal.
File format: dds, dxt5.
Suffix: None.
This is simply the color and is needed for pretty much all materials.

Normal map
Size: Normal.
File format: dds, 3dc.
Suffix: “nrm”.
The normal map for the material and uses the standard format rgb = xyz and is in tangent space. Note that this needs to be converted to a format of XY only though (3dc)!

Specular map
Size: Half of diffuse
File format: dds, dxt5.
Suffix: “spec”.
The specular map handles the specular intensity and power for each pixel. The intensity uses the R,G and B channel and the power in the Alpha channel. Intensity sets the amount of specularity (for each color channel) applied to a pixel, the higher value of the channel, the higher intensity. Power sets the sharpness of the specular highlight, a low value give a dull and diffuse look, while a high value give a wet/shiny look.

Alpha
Size: Normal
File format: tga, single channel (greyscale).
Suffix: “alpha”.
Sets the alpha for each texture and for any lit surface this is a simply 0 or 1 decision per pixel since a lit pixel can never be half transparent, only fully visible or invisible.

Height map
Size: Half of diffuse
File format: tga, single channel (greyscale).
Suffix: “height”.
Height map is more like a depth map actually. White = largest depth, Black = no depth and this is important to remember when creating it.

Illumination map
Size: Half of diffuse
File format: dds, dxt1.
Suffix: “illum”.
This map will be added additively on top of the model when rendering and will not be affected by lighting. It is useful for creating glowing surfaces such as lightbulb, ember, etc.

With size means the size of texture width and height. So if diffuse is 512×512 in size, then a map with half of the size of diffuse will be 256×256. Normal simply means that it has the same size of the diffuse texture. Also note that the size of map might be specified otherwise in the description of an object, and if so the specified size shall be used.

All of the textures in a material (and the material itself!) must be named according the model which uses it. An example of naming is as follows:

The model has the name of “ball01” and its material has a diffuse, normal map and specular map. The textures will then be named like this:

”ball01.dds”, the diffuse.
“ball01_nrm.dds”, the normal map. “ball01_spec.dds”, the specular map.

If the model has uses several materials then add some extra name after the name of the file when settings up the textures. Here is an example:

The model has the name “frontdoor01” and have two submeshes: “frame” and “door”. Each of these submeshes have its own texture and therefore also material. The base name for each submesh material will be: “frontdoor01_frame” and “frontdoor01_door” respectively. This means that the textures will have the names:

For frame:
“frontdoor01_frame.dds”, the diffuse.
“frontdoor01_frame_nrm.dds”, the normal map.
“frontdoor01_frame_spec.dds”, the specular map

For door:
“frontdoor01_door.dds”, the diffuse.
“frontdoor01_door_nrm.dds”, the normal map.
“frontdoor01_door_spec.dds”, the specular map.

Can be used to edit the anno files. More to come…