7 February 2021

Week 7 : Finger tracking and locomotion

Structure

This tutorial is structured as follow :

Requirements

This hands-on is based on the final outcome of the Design of a basic interaction hands-on. Thus, you can directly checkout the last commit and can create a new branch for this hands-on.

Reload the baseline project

If your working directory is not clean (i.e. you have non committed changes), you can stash your work or commit your changes if you want to be able to recover those changes later.

Make sure your work is stashed or committed before the following instruction as it will erase all data

As we won’t take advantage of the previously implemented structure to collect items we can checkout the older integration of the raw OVRPlayerController for this hands-on session. Thus, from the root directory of your project you can run :

rm -rf *

If you did not add a tag on your baseline commit you can retrieve its hash with the following command :

git log

Otherwise you can directly checkout the commit of the basic integration of the oculus framework with the following command :

git checkout integrating_player_rig -f

Once reloaded the scene should looks like :

Reloaded scene

Finger tracking features

Checkout new branches

As we won’t remerge this branch with the master branch we will create a second master_finger_tracking branch and the associated development branch develop_finger_tracking_feature :

git checkout -b master_finger_tracking
git checkout -b develop_finger_tracking_feature

Integrating virtual tracked hands in the scene

Add hands prefabs

The first thing to do is to enable the support for the Hand Tracking Support from the OVRCamerRig :

Enable finger tracking

Then you can remove the previous controllers from the scene as we won’t use them in this section :

Remove controllers

Then we must drag and drop the OVRHandPrefab to both LeftHandAnchor and RightHandAnchor :

Select OVRHandPrefab

Drop OVRHandPrefab

Change virtual hands material

In order to change the default material used for the hands we can select both newly added OVRHandPrefabs :

Select both hands prefabs

And click on the small arrow to extend the material list to change the BasicHandMaterial to another one :

Open material window

And pick a new material :

Select new material

Set the right hand as a right hand

Now we must change the default left hand side in the OVRHandPrefab attached to the RightHandAnchor.

Let’s start with the script side :

Edit OVRHandPrefab script side

Now the skeleton side :

Edit OVRHandPrefab skeleton side

And finally the mesh side :

Edit OVRHandPrefab mesh side

Build

Now we should be able to build the scene (Ctrl + B) and have both of our hands in the virtual scene.

Here is a preview of what you might expect :

Commit

Now that we tested this feature we can commit our changes but we won’t merge it with the master branch but with the master_finger_tracking one :

# Commit
git add .
git commit -m "Add virtual tracked hands in the virtual scene"

# Merge
git checkout master_finger_tracking
git merge develop_finger_tracking_feature --no-ff
git tag virtual_hands_integrated -a -m "Virtual hands integrated in the virtual scene"

# Push
git push --all
git push --tags

Locomotion

The idea now is to build a locomotion mechanism based on hands’ gestures as we no longer hold controllers in our hands.

In this section we will implement a teleportation mechanism where :

N.B. : We want our approach to work with no regard on the hand used to aim.

Checkout a new branch

git checkout -b develop_locomotion

Attach the new locomotion script to the player

The first thing to do is to add a new script attached to OVRPlayerController to handle the locomotion behavior :

Add new component to the OVRPlayerController

Let’s name this script FingerLocomotion.cs :

Name script

This script can be downloaded in its final state here : FingerLocomotion.cs

Expose public instance variables

To interact with the virtual hands we will take advantage of the OVRHand class which will provide us with information about finger pinched.

The full documentation is available here : Oculus - Hand Tracking in Unity

Here we will use it’s method bool GetFingerIsPinching( HandFinger a_hand_finger ) to know whether the player is pinching a given finger or not.

As our FingerLocomotion class need to be linked to both of our hands we can already expose two instance variables storing instances of the left and right OVRHand that we will link later in the Inspector View :

[Header( "Hands" )]
// Bindings with OVR Hands
public OVRHand leftHand;
public OVRHand rightHand;

N.B. : This requires the following using :

using UnityEngine;
using static OVRHand;

As we also want to provide the player a marker to give a visual feedback on the expected position we can add those lines :

[Header( "Marker" )]
// Store the refence to the marker prefab used to highlight the targeted point
public GameObject markerPrefab;

Finally, we don’t want our player to teleport up to the end of the map in a single action thus we will limit the allowed distance for the teleportation :

[Header( "Maximum Distance" )]
[Range( 2f, 30f )]
// Store the maximum distance the player can teleport
public float maximumTeleportationDistance = 15f;

Implement the locomotion behavior

We will take advantage of the object oriented programming to avoid a succession of tests (if) to handle the case where the first hand used it the right or the left one.

Indeed, we will store the first hand involved under the protected instance variable protected OVRHand pointing_hand and the other one as protected OVRHand non_pointing_hand.

Thus, in our Update method we can already place the following lines to detect the hand used to point to the direction :

// If no pointing hand is defined check if one hand is pinching
if ( pointing_hand == null ) {
	if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
	if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
}

We can also already add a part to detect when the pointing_hand is no longer in a pinch state to reset it to null :

// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

And determine the non_pointing_hand with :

// Deduce the non pointing hand
non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

Once we have our hand used to determine the direction we must compute the targeted position and prevent the user to pick a location outside of the allowed range. This can be done with the following method :

protected bool aim_with ( OVRHand a_hand, out Vector3 target_point ) {

        // Default the "output" target point to the null vector
        target_point = new Vector3();

        // Launch the ray cast and leave if it doesn't hit anything
        RaycastHit hit;
        if ( !Physics.Raycast( a_hand.transform.position, a_hand.PointerPose.forward, out hit, Mathf.Infinity ) ) return false;

        // If the aimed point is out of range (i.e. the raycast distance is above the maximum distance) then prevent the teleportation
        if ( hit.distance > maximumTeleportationDistance ) return false;

        // "Output" the target point
        target_point = hit.point;
        return true;
}

The idea in here is to launch a ray starting from the hand’s position with the direction pointed by the hand and look for the first collision point the ray hits.

The direction aimed by the hand is given through the rotation of the transform : a_hand.PointerPose. Thus we only need to retrieve its forward component to define the output direction vector for our raycast.

Then, if the ray hits something we obtain in the RaycastHit hit the position of the collision point we can “output” if the distance is bellow the maximum distance allowed.

N.B. : In this implementation the return state tells if the targeted location is valid and in such a case the targeted position is return through the second variable taken as argument with the keyword out.

Thus we can add, after having determined the pointing_hand, computed and verified that the targeted position is valid the mechanism to :

Teleportation

To perform the teleportation we will use the Move( Vector3 motion ) method from the CharacterController class attached by default to our OVRPlayerController.

Thus we need to retrieve and store this component in a first time with :

// Retreive the character controller used later to move the player in the environment
protected CharacterController character_controller;
void Start () { character_controller = this.GetComponent<CharacterController>(); }

And to teleport the player to Vector3 target_point with :

// Tell the character controller to move to the teleportation point
character_controller.Move( target_point - this.transform.position );

N.B. : Using the method Move( Vector3 motion ) instead of directly editing the OVRPlayerController’s position allows the developer to make sure the player can actually reach the destination without going through walls for instance !

Two hands detection

The idea is that once the pointing_hand set (i.e. one hand is already with the index pinched) to check whether or not the second hand’s index is pinched. If it is the case we just need to continue the execution to reach the call of the teleportation. On the other case then we just need to abort the Update method to prevent the execution to reach the teleportation.

As a first draft this provide the following structure for the Update method :

void Update () {

		// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
		if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

		// If no pointing hand is defined check if one hand is pinching
		if ( pointing_hand == null ) {
			if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
			if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
		}

		// Store the position of the targeted point
		Vector3 target_point;
		if (
			pointing_hand != null                           // If one hand is pinching
			&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
			&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
		) {

			// Deduce the non pointing hand
			non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

			// Check if the other hand is pinching or not
			if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) return;

			// Tell the character controller to move to the teleportation point
			character_controller.Move( target_point - this.transform.position );
		}
	}

Prevent teleportation loop

To prevent this loop of teleportation we will store as an instance variable protected bool teleportation_locked (set as false by default) whether the teleportation can be triggered or not. Then we will add this segment of code just before calling the teleportation from the previous schema of the Update method:

// Prevent continuous teleportation
if ( teleportation_locked ) return;
teleportation_locked = true;

The idea is the following one : When both of indexes are pinched the Update loop will go the first time through the test (as by default teleportation_locked is set to false) an set teleportation_locked to true.

As on the next frame it is very likely that both indexes are still pinched the Update method will re-arrive on the same segment of code with more or less the same conditions but this time with teleportation_locked set to true. Thus, instead of continuing its execution the Update method will be aborted and the teleportation won’t be re-triggered.

The only remaining task it to implement the conditions unlocking the teleportation : i.e. when at least one of the indexes is not pinched which gives the following schema of implementation of the Update method :

// Keep track of the teleportation state to prevent continuous teleportation
protected bool teleportation_locked = false;

void Update () {

	// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
	if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

	// If no pointing hand is defined check if one hand is pinching
	if ( pointing_hand == null ) {
		if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
		if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
	}

	// Store the position of the targeted point
	Vector3 target_point;
	if (
		pointing_hand != null                           // If one hand is pinching
		&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
		&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
	) {
		// Deduce the non pointing hand
		non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

		// Check if the other hand is pinching or not
		if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) {
			// Unlock the teleportation as the second hand is not pinched
			teleportation_locked = false;
			return;
		}

		// Prevent continuous teleportation
		if ( teleportation_locked ) return;
		teleportation_locked = true;

		// Tell the character controller to move to the teleportation point
		character_controller.Move( target_point - this.transform.position );


	} else {
		// Unlock the teleportation as at least the pointing hand is not valid
		teleportation_locked = false;
	}

}

Handling marker

As we previously said we want to make a marker to appear we need to store it’s instance once the marker is instantiated :

protected GameObject marker_prefab_instanciated;

And we can instantiate it once the valid aim condition is validated using the GameObject.Instantiate( GameObject prefab, Transform parent ). However, as we doesn’t want to instantiate one marker per frame and consuming all our memory quickly we can check that the marker is not already instantiated. Thus we can place just after the test on the aimed direction the following lines :

// Instantiate the marker prefab if it doesn't already exists and place it to the targeted position
if ( marker_prefab_instanciated == null ) marker_prefab_instanciated = GameObject.Instantiate( markerPrefab, this.transform );

// Place the marker to the targeted position
marker_prefab_instanciated.transform.position = target_point;

Conversely, if the aim test fails we want our marker to be removed from the scene. Thus on the else case we can add the following lines :

// Destroy marker
if ( marker_prefab_instanciated != null ) Destroy( marker_prefab_instanciated );
marker_prefab_instanciated = null;

As a sum up, our Update method should like :

void Update () {

	// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
	if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

	// If no pointing hand is defined check if one hand is pinching
	if ( pointing_hand == null ) {
		if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
		if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
	}

	// Store the position of the targeted point
	Vector3 target_point;
	if (
		pointing_hand != null                           // If one hand is pinching
		&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
		&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
	) {

		// Instantiate the marker prefab if it doesn't already exists and place it to the targeted position
		if ( marker_prefab_instanciated == null ) marker_prefab_instanciated = GameObject.Instantiate( markerPrefab, this.transform );
		marker_prefab_instanciated.transform.position = target_point;

		// Deduce the non pointing hand
		non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

		// Check if the other hand is pinching or not
		if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) {
			// Reset the teleportation state
			teleportation_locked = false;
			return;
		}

		// Prevent continuous teleportation
		if ( teleportation_locked ) return;
		teleportation_locked = true;

		// Tell the character controller to move to the teleportation point
		character_controller.Move( target_point - this.transform.position );


	} else {
		// Remove the cursor
		if ( marker_prefab_instanciated != null ) Destroy( marker_prefab_instanciated );
		marker_prefab_instanciated = null;

		// Reset the teleportation state
		teleportation_locked = false;
	}

}

Setup the virtual scene

Now that the script edition is complete it should appear as illustrated bellow in the Inpsector View :

Script preview

Create the teleportation marker prefab

As explained above, our script FingerLocomotion will instantiate a marker prefab to show the player the targeted position. Thus we must create this marker prefab in the first place to reference it :

Create marker as a sphere

Then we can convert it into a prefab by dragging the instance into the Assets folder :

Convert marker to a prefab

As we want our target marker to be easily visible we can create a new material to apply on the sphere :

Create new material

And edit it’s color :

Edit material color

Now we can open the prefab to edit its content :

Open prefab

We can start by dragging the new color onto the target :

Drag color onto the marker

Now we must remove the Collider from the sphere (otherwise the raycast will hit the sphere itself) :

Remove sphere collider

We can scale the sphere :

Scale sphere

Leave the prefab edition :

Leave prefab edition

And remove the instance of the prefab from the sphere (as it will be automatically instantiated by our script) :

Remove prefab instance from the scene

Let’s start by selecting the OVRHandPrefab from the left hand :

Select left hand OVRHandPrefab

And drag it to the Left Hand field of our FingerLocomotion script attached to the OVRPlayerController :

Drag left hand OVRHandPrefab

You can do the same for the Right Hand.

Now we select our created teleportation_marker prefab :

Select teleportation_marker

And drop it into the Marker Prefab field of our Locomotion Script :

Drop marker prefab

And finally we can change the layer of the OVRPlayerController to make sure the raycast won’t hit the player’s collider :

Change player layer

And apply the same layer for childrens :

Aplly layer for childrens

Build

Everything should be ready now and you can build your project (Ctrl + B) and you should expect such kind of results where you can point at a direction with one hand and trigger the teleportation with the other one :

Commit

Now that we tested this feature we can commit our changes and merge it with the master_finger_tracking branch :

# Commit
git add .
git commit -m "Add new locomotion based on pinching indexes"

# Merge
git checkout master_finger_tracking
git merge develop_locomotion --no-ff
git tag finger_locomotion_implemented -a -m "Finger locomotion implemented"

# Push
git push --all
git push --tags

Thanks

Thank you for your attention during these sessions. I hope you learned things which might help you to enjoy creating your own game, expressing your creativity, or for other topics ! Please feel free to use the forum if you have questions or want to give your feedback on this hands-on or on this course !