HOME About Download Subscription FAQ User Guide User Page Promotion Contact 日本語

Table of Contents

Let's Get Started

How to Use:

  1. Make sure that the two windows Webcam Motion Capture and Webcam Motion Receiver are opening.
  2. Check if the webcam on your computer is ON and your upper body is shown in the Webcam Motion Capture window.
    - If your Webcam image is not shown in the Webcam Motion Capture window, please see the FAQ.
  3. Move your hands and fingers, tilt your head, and blink!
    - You will see that your avatar moves according to your move😊
    - Also check the recommended setup!
  4. Adjust Body Filter, Mouth Filter, Blink Filter, and Expression Filter values.
    - Small value ⇨ fast reaction but noisy.
    - Large value ⇨ slow reaction but smooth.
    - (Tip) Set it to as large value as possible while the reaction is still fast enough.
  5. For instructions on how to use Webcam Motion Capture for live streaming or to save it as a video file, please proceed to Set Up OBS Studio.
  6. Please also watch a great tutorial from Cooki Kunai!

Use Video as an Input

If your computer is not fast enough to perform smooth real-time tracking, you can try this approach!
  1. First, shoot a video using your smartphone, etc. and send it to your computer. (You can also use this sample video for a quick test.)
  2. In the Webcam Motion Capture window, select Video.
  3. Select New Video as a source. Then, click Open Video to select your video.
  4. Click Output Folder to specify a folder where processed video information is stored.
  5. Click Process.
    - Once processing video is done, your processed video and control icons will appear.
  6. Hit play icon.
    - You will see that your avatar moves according to your video!
  7. Adjust Body Filter, Mouth Filter, Blink Filter, and Expression Filter values.
    - Small value ⇨ fast reaction but noisy.
    - Large value ⇨ slow reaction but smooth.
    - (Tip) Set it to as large value as possible while the reaction is still fast enough.
  8. For instructions on how to use Webcam Motion Capture for live streaming or to save it as a video file, please proceed to Set Up OBS Studio.

Set Up OBS Studio

To use Webcam Motion Capture for live streaming on platforms like YouTube or to save it as a video file, OBS Studio is commonly used. OBS Studio is free, established software for video recording and live streaming.

The following only describes how to set up Webcam Motion Capture in OBS Studio. For instructions on how to use OBS Studio, please refer to external sources.

Windows

There are 2 options: Use Spout2 (Only for subscribing users) or Game Capture

A. Spout2

  1. Download and install obs-spout2-plugin.
  2. In the Webcam Motion Receiver window, open App Settings at the bottom of the window, and enable USE SPOUT2.
  3. Use Spout2 Capture as a source in OBS.
  4. In the Spout2 Capture settings, select Webcam Motion Receiver for the SpoutSenders and select Default for the Composite mode.
  5. If you want to make the background transparent, select Transparent for the Background in the Webcam Motion Receiver window.

B. Game Capture

  1. Use Game Capture as a Source in OBS
  2. In the Game Capture settings, select Capture specific window for mode and select Webcam Motion Receiver for Window.
  3. If you want to make the background transparent, check the Allow Transparency option in the Game Capture settings. Then, select Transparent for the Background in the Webcam Motion Receiver window.

MacOS

There are 2 options: Use Syphon Client (Only for subscribing users) or MacOS Screen Capture

A. Syphon Client

  1. In the Webcam Motion Receiver window, open App Settings at the bottom of the window, and enable USE SYPHON.
  2. Use Syphon Client as a Source in OBS
  3. If you want to make the background transparent, check Allow Transparency in the Syphon Client settings and select Transparent for the Background in the Webcam Motion Receiver window.

B. MacOS Screen Capture

  1. Use MacOS Screen Capture as a Source in OBS
  2. In the MacOS Screen Capture settings, select Window Capture for the Method and select Webcam Motion Receiver for the Window.
  3. If you want to make the background transparent, select Color for the Background in the Webcam Motion Receiver window, and set the color as a chroma key in OBS.

Facial Expressions Settings

Switching Expressions

Figure 1

You can swtich Facial Expressions using the Keyboard.

By default, Auto Expression is enabled. You can enable each assigned Emotion (Neutral, Fun, Angry, etc.) using the F1, F2, F3, ... keys (Figure 1 (B)), and switch back to Auto Expression by a key (Figure 1 (A)) on your keyboard.

When Auto Expression is enabled, all the Facial Expressions are controlled by the Webcam. Otherwise, the Webcam controls only the Mouth Shapes and Blinks.

Auto Expressions / Mouth Movement Settings

To control the Emotions, Mouth Shapes and Blinks using the Webcam, you need to select one of the following modes: Face Tracking Samples and PerfectSync as shown in Figure 2.
  1. Face Tracking Samples: The avatar's facial expression is set by comparing your facial expression captured by the webcam and the registered Face Tracking Samples in the selected sample set.
  2. PerfectSync: AI-based approach. Each facial feature (eyebrows, eyes, mouth shapes, etc.) is tracked individually, and the corresponding feature on the avatar is set accordingly.

Figure 2

A. Face Tracking Samples

Figure 3

You can configure the Face Tracking Samples as follow.
  1. Select either a Preset or Custom sample set from the dropdown menu (Figure 3 (A)).
    • Each sample set includes:
    • Neutral Expression (Figure 3 (B))
    • 5 Mouth Shapes Expressions (Figure 3 (C))
      • "A", "I", "U", "E", "O"
    • 4 Emotions Expressions (Figure 3 (D))
      • Select 4 preferred expressions ("FUN", "SURPRISED", etc.) from the Facial Expressions (blendshapes) your avatar has from the corresponding dropdown menu (Figure 3 (E)).
    • You can disable a sample itself (Figure 3 (G)) and, for the Emotions Expressions, choose whether to enable blink or add mouth shapes (Figure 3 (H)) when the corresponding emotion expression is chosen.
  2. For the Preset samples:
    - The default registered facial expression is displayed for each corresponding avatar expression (Figure 3 (F)).
    For the Custom samples:
    - You can register your custom facial expressions (Figure 3 (F)) for each avatar's expression as follows.
    1. Make sure that the webcam is enabled and click the corresponding "Update Samples" button (Figure 3 (I)).
    2. Your current expression is instantly reflected on the face model in real-time.
    3. To register the expression, click the "Register" button.
      - You can restore the default facial expression by clicking the "Reset" button (Figure 3 (J)).
    Tips: All the facial expressions are recognized as deviations from the Neutral expression. Therefore, it is advisable to register your typical facial expression as a Neutral expression.

B. PerfectSync

PerfectSync trackes the movement of each facial feature (eyebrows, eyes, mouth shapes, etc.) individually. The tracked features correspond to the BlendShapeLocation defined in ARKit (Apple's AR platform for iOS).

  1. If you only need a subset of the ARKit's 52 features, Webcam-based PerfectSync is an ideal option. In this option, your avatar does not have to be compatible with PerfectSync.
  2. If you would like to use some features that are not supported by the Webcam-based PerfectSync, you will need to use a mobile app. This option is available only if your avatar is compatible with PerfectSync. See Set Up Mobile App for details.

Webcam-based PerfectSync Settings

Figure 4

Use Idle Animation

You can combine your favorite animation with Face and Hand Tracking to control your avatar. The loaded animation is applied to all the bones except the following bones.
When your hand is visible:
Head*1 / Body Rotation Target*1*2 / Upper Arms / Lower Arms / Hands / Fingers
When your hand is Not visible:
Head*1 / Body Rotation Target*1*2
  • *1: You can also apply animation to "Head" and "Body Rotation Target" by disabling the corresponding tracking from "Advanced Tracking Settings" in the Webcam Motion Receiver window.
  • *2: Body Rotation Target Bone can be set to either "Spine", "Chest", or "Upper Chest" from "Advanced Tracking Settings" in the Webcam Motion Receiver window.
  1. Enable animation by checking Use Idle Animation in the top-right corner of the "Webcam Motion Receiver" window.
  2. You can select either Preset animation or Custom Animation.
    For the Preset animation:
    Select animation from the dropdown menu below.
    For the custom animation:
    You can load your own custom animation in the FBX file format from the "Load Animation" button. You can have your own custom animation either by:
    1. Downloading your favorite animation from Mixamo for free. All the 2500+ animation data on Mixamo are royalty free even for commercial use!
    2. Generating your custom animation using a Computer Graphics software such as Blender. Once generated, export the animation in the FBX file format.
      • Please don't change the bone names while editing/exporting your avatar.
      • You need to install VRM Add-on for Blender to edit your VRM avatar in Blender.
In addition to the main animation, you have the option to set up to five sub-animations. You can switch between them by using the assigned keys.

Set Up Mobile App

If you have an iPhone/iPad (iOS 13.0 or later), you can use a mobile app to enhance your avatar's facial expressions using Face Tracking! (For an Android phone, there is an alternative app.)

waidayo

  1. Have a VRM model which is compatible with Perfect Sync.
    • The default avatar is compatible with Perfect Sync so you can use it for a quick test. (You can download the avatar from here.)
  2. Open waidayo on your iPhone/iPad.
  3. Transfer your avatar to your iPhone/iPad by following this.
  4. By following this, set up the connection between waidayo on your iPhone/iPad and waidayo for PC on your Windows/Mac.
    • Once you confirm that the connection is established, you can close "waidayo for PC".
  5. Then, open Webcam Motion Capture and, in the Webcam Motion Receiver window, click Facial Expressions Settings button. Once the Facial Expressions Settings is opened, in Select Tracking Mode, select PerfectSync, and check Mobile App in Tracking Device as shown in this image. (If this option is not available, your avatar is not compatible with Perfect Sync.)
  6. Now, you will see that your avatar's facial expressions animate according to your facial expressions!
    • Try puffing your cheeks, sticking out your tongue, etc.😊

Facemotion3d, iFacialMocap, MeowFace (Android)

  1. Have a VRM model which is compatible with Perfect Sync.
    • The default avatar is compatible with Perfect Sync so you can use it for a quick test. (You can download the avatar from here.)
  2. Make sure that your computer and your phone are connecting to the same WiFi.
  3. Open Facemotion3d/iFacialMocap/MeowFace on your Phone.
  4. In the Webcam Motion Receiver window, click Facial Expressions Settings. Once the Facial Expressions Settings is opened, in Select Tracking Mode, select PerfectSync, and check Mobile App in Tracking Device as shown in this image. Then, select the corresponding app from the drop-down menu below. (If this option is not available, your avatar is not compatible with Perfect Sync.)
    • The mobile app will automatically connect to the Webcam Motion Receiver. If it does not automatically connect, enter your phone's IP address in the IP address next to the drop-down menu and press the Set button*.
  5. Now, you will see that your avatar's facial expressions animate according to your facial expressions!
    • Try puffing your cheeks, sticking out your tongue, etc.😊

* If you specify the IP address directly and it still does not connect, the network traffic may be blocked by a firewall on your PC.

Open "Windows Defender Firewall with Advanced Security" by pressing "Win + r", typing "wf.msc", and pressing Enter. Select "Inbound Rules" and find "webcam motion receiver.exe." By double clicking it, open "webcam motion receiver.exe Properties." Choose "Allow the connection" in the "Action" section (Refer to this image).

Contact

E-mail: contact@webcammotioncapture.info
E-mail: contact@webcammotioncapture.info
KWCL Inc.
Ochiai 1-46-1 Cocolia Tama-Center 7F., Tama, Tokyo
KWCL Inc.
Ochiai 1-46-1 Cocolia Tama-Center 7F., Tama, Tokyo