HO4 : Finger tracking and locomotion

Structure

This tutorial is structured as follow :

• Reload the baseline project
• Finger tracking features
• Locomotion

Requirements

This hands-on is based on the final outcome of the Design of a basic interaction hands-on. Thus, you can directly checkout the last commit and can create a new branch for this hands-on.

Reload the baseline project

If your working directory is not clean (i.e. you have non committed changes), you can stash your work or commit your changes if you want to be able to recover those changes later.

Make sure your work is stashed or committed before the following instruction as it will erase all data

As we won’t take advantage of the previously implemented structure to collect items we can checkout the older integration of the raw OVRPlayerController for this hands-on session. Thus, from the root directory of your project you can run :

rm -rf *

If you did not add a tag on your baseline commit you can retrieve its hash with the following command :

git log

Otherwise you can directly checkout the commit of the basic integration of the oculus framework with the following command :

git checkout integrating_player_rig -f

Once reloaded the scene should looks like :

Finger tracking features

Checkout new branches

As we won’t remerge this branch with the master branch we will create a second master_finger_tracking branch and the associated development branch develop_finger_tracking_feature :

git checkout -b master_finger_tracking
git checkout -b develop_finger_tracking_feature

Integrating virtual tracked hands in the scene

Add hands prefabs

The first thing to do is to enable the support for the Hand Tracking Support from the OVRCamerRig :

Then you can remove the previous controllers from the scene as we won’t use them in this section :

Then we must drag and drop the OVRHandPrefab to both LeftHandAnchor and RightHandAnchor :

Change virtual hands material

In order to change the default material used for the hands we can select both newly added OVRHandPrefabs :

And click on the small arrow to extend the material list to change the BasicHandMaterial to another one :

And pick a new material :

Set the right hand as a right hand

Now we must change the default left hand side in the OVRHandPrefab attached to the RightHandAnchor.

Let’s start with the script side :

Now the skeleton side :

And finally the mesh side :

Build

Now we should be able to build the scene (Ctrl + B) and have both of our hands in the virtual scene.

Here is a preview of what you might expect :

Commit

Now that we tested this feature we can commit our changes but we won’t merge it with the master branch but with the master_finger_tracking one :

# Commit
git add .
git commit -m "Add virtual tracked hands in the virtual scene"

# Merge
git checkout master_finger_tracking
git merge develop_finger_tracking_feature --no-ff
git tag virtual_hands_integrated -a -m "Virtual hands integrated in the virtual scene"

# Push
git push --all
git push --tags

Locomotion

The idea now is to build a locomotion mechanism based on hands’ gestures as we no longer hold controllers in our hands.

In this section we will implement a teleportation mechanism where :

• the player pinch the index to aim with the hand involved
• the teleportation is triggered when the second index is pinched

N.B. : We want our approach to work with no regard on the hand used to aim.

Checkout a new branch

git checkout -b develop_locomotion

Attach the new locomotion script to the player

The first thing to do is to add a new script attached to OVRPlayerController to handle the locomotion behavior :

Let’s name this script FingerLocomotion.cs :

This script can be downloaded in its final state here : FingerLocomotion.cs

Expose public instance variables

To interact with the virtual hands we will take advantage of the OVRHand class which will provide us with information about finger pinched.

The full documentation is available here : Oculus - Hand Tracking in Unity

Here we will use it’s method bool GetFingerIsPinching( HandFinger a_hand_finger ) to know whether the player is pinching a given finger or not.

As our FingerLocomotion class need to be linked to both of our hands we can already expose two instance variables storing instances of the left and right OVRHand that we will link later in the Inspector View :

[Header( "Hands" )]
// Bindings with OVR Hands
public OVRHand leftHand;
public OVRHand rightHand;

N.B. : This requires the following using :

using UnityEngine;
using static OVRHand;

As we also want to provide the player a marker to give a visual feedback on the expected position we can add those lines :

[Header( "Marker" )]
// Store the refence to the marker prefab used to highlight the targeted point
public GameObject markerPrefab;

Finally, we don’t want our player to teleport up to the end of the map in a single action thus we will limit the allowed distance for the teleportation :

[Header( "Maximum Distance" )]
[Range( 2f, 30f )]
// Store the maximum distance the player can teleport
public float maximumTeleportationDistance = 15f;

Implement the locomotion behavior

We will take advantage of the object oriented programming to avoid a succession of tests (if) to handle the case where the first hand used it the right or the left one.

Indeed, we will store the first hand involved under the protected instance variable protected OVRHand pointing_hand and the other one as protected OVRHand non_pointing_hand.

Thus, in our Update method we can already place the following lines to detect the hand used to point to the direction :

// If no pointing hand is defined check if one hand is pinching
if ( pointing_hand == null ) {
	if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
	if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
}

We can also already add a part to detect when the pointing_hand is no longer in a pinch state to reset it to null :

// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

And determine the non_pointing_hand with :

// Deduce the non pointing hand
non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

Once we have our hand used to determine the direction we must compute the targeted position and prevent the user to pick a location outside of the allowed range. This can be done with the following method :

protected bool aim_with ( OVRHand a_hand, out Vector3 target_point ) {

        // Default the "output" target point to the null vector
        target_point = new Vector3();

        // Launch the ray cast and leave if it doesn't hit anything
        RaycastHit hit;
        if ( !Physics.Raycast( a_hand.transform.position, a_hand.PointerPose.forward, out hit, Mathf.Infinity ) ) return false;

        // If the aimed point is out of range (i.e. the raycast distance is above the maximum distance) then prevent the teleportation
        if ( hit.distance > maximumTeleportationDistance ) return false;

        // "Output" the target point
        target_point = hit.point;
        return true;
}

The idea in here is to launch a ray starting from the hand’s position with the direction pointed by the hand and look for the first collision point the ray hits.

The direction aimed by the hand is given through the rotation of the transform : a_hand.PointerPose. Thus we only need to retrieve its forward component to define the output direction vector for our raycast.

Then, if the ray hits something we obtain in the RaycastHit hit the position of the collision point we can “output” if the distance is bellow the maximum distance allowed.

N.B. : In this implementation the return state tells if the targeted location is valid and in such a case the targeted position is return through the second variable taken as argument with the keyword out.

Thus we can add, after having determined the pointing_hand, computed and verified that the targeted position is valid the mechanism to :

• wait for the second hand to trigger the teleportation
• prevent a continuous teleportation (i.e. the teleportation to be triggered each frame)

Teleportation

To perform the teleportation we will use the Move( Vector3 motion ) method from the CharacterController class attached by default to our OVRPlayerController.

Thus we need to retrieve and store this component in a first time with :

// Retreive the character controller used later to move the player in the environment
protected CharacterController character_controller;
void Start () { character_controller = this.GetComponent<CharacterController>(); }

And to teleport the player to Vector3 target_point with :

// Tell the character controller to move to the teleportation point
character_controller.Move( target_point - this.transform.position );

N.B. : Using the method Move( Vector3 motion ) instead of directly editing the OVRPlayerController’s position allows the developer to make sure the player can actually reach the destination without going through walls for instance !

Two hands detection

The idea is that once the pointing_hand set (i.e. one hand is already with the index pinched) to check whether or not the second hand’s index is pinched. If it is the case we just need to continue the execution to reach the call of the teleportation. On the other case then we just need to abort the Update method to prevent the execution to reach the teleportation.

As a first draft this provide the following structure for the Update method :

void Update () {

		// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
		if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

		// If no pointing hand is defined check if one hand is pinching
		if ( pointing_hand == null ) {
			if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
			if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
		}

		// Store the position of the targeted point
		Vector3 target_point;
		if (
			pointing_hand != null                           // If one hand is pinching
			&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
			&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
		) {

			// Deduce the non pointing hand
			non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

			// Check if the other hand is pinching or not
			if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) return;

			// Tell the character controller to move to the teleportation point
			character_controller.Move( target_point - this.transform.position );
		}
	}

Prevent teleportation loop

To prevent this loop of teleportation we will store as an instance variable protected bool teleportation_locked (set as false by default) whether the teleportation can be triggered or not. Then we will add this segment of code just before calling the teleportation from the previous schema of the Update method:

// Prevent continuous teleportation
if ( teleportation_locked ) return;
teleportation_locked = true;

The idea is the following one : When both of indexes are pinched the Update loop will go the first time through the test (as by default teleportation_locked is set to false) an set teleportation_locked to true.

As on the next frame it is very likely that both indexes are still pinched the Update method will re-arrive on the same segment of code with more or less the same conditions but this time with teleportation_locked set to true. Thus, instead of continuing its execution the Update method will be aborted and the teleportation won’t be re-triggered.

The only remaining task it to implement the conditions unlocking the teleportation : i.e. when at least one of the indexes is not pinched which gives the following schema of implementation of the Update method :

// Keep track of the teleportation state to prevent continuous teleportation
protected bool teleportation_locked = false;

void Update () {

	// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
	if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

	// If no pointing hand is defined check if one hand is pinching
	if ( pointing_hand == null ) {
		if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
		if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
	}

	// Store the position of the targeted point
	Vector3 target_point;
	if (
		pointing_hand != null                           // If one hand is pinching
		&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
		&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
	) {
		// Deduce the non pointing hand
		non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

		// Check if the other hand is pinching or not
		if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) {
			// Unlock the teleportation as the second hand is not pinched
			teleportation_locked = false;
			return;
		}

		// Prevent continuous teleportation
		if ( teleportation_locked ) return;
		teleportation_locked = true;

		// Tell the character controller to move to the teleportation point
		character_controller.Move( target_point - this.transform.position );


	} else {
		// Unlock the teleportation as at least the pointing hand is not valid
		teleportation_locked = false;
	}

}

Handling marker

As we previously said we want to make a marker to appear we need to store it’s instance once the marker is instantiated :

protected GameObject marker_prefab_instanciated;

And we can instantiate it once the valid aim condition is validated using the GameObject.Instantiate( GameObject prefab, Transform parent ). However, as we doesn’t want to instantiate one marker per frame and consuming all our memory quickly we can check that the marker is not already instantiated. Thus we can place just after the test on the aimed direction the following lines :

// Instantiate the marker prefab if it doesn't already exists and place it to the targeted position
if ( marker_prefab_instanciated == null ) marker_prefab_instanciated = GameObject.Instantiate( markerPrefab, this.transform );

// Place the marker to the targeted position
marker_prefab_instanciated.transform.position = target_point;

Conversely, if the aim test fails we want our marker to be removed from the scene. Thus on the else case we can add the following lines :

// Destroy marker
if ( marker_prefab_instanciated != null ) Destroy( marker_prefab_instanciated );
marker_prefab_instanciated = null;

As a sum up, our Update method should like :

void Update () {

	// Make sure the pointing hand is still pinched otherwise reset the pointing hand to null
	if ( pointing_hand != null && !pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = null;

	// If no pointing hand is defined check if one hand is pinching
	if ( pointing_hand == null ) {
		if ( leftHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = leftHand;
		if ( rightHand.GetFingerIsPinching( HandFinger.Index ) ) pointing_hand = rightHand;
	}

	// Store the position of the targeted point
	Vector3 target_point;
	if (
		pointing_hand != null                           // If one hand is pinching
		&& pointing_hand.IsPointerPoseValid             // Skip invalid pointer pose
		&& aim_with( pointing_hand, out target_point )  // The computation of the target position returned a valid point
	) {

		// Instantiate the marker prefab if it doesn't already exists and place it to the targeted position
		if ( marker_prefab_instanciated == null ) marker_prefab_instanciated = GameObject.Instantiate( markerPrefab, this.transform );
		marker_prefab_instanciated.transform.position = target_point;

		// Deduce the non pointing hand
		non_pointing_hand = ( pointing_hand == leftHand ) ? rightHand : leftHand;

		// Check if the other hand is pinching or not
		if ( !non_pointing_hand.GetFingerIsPinching( HandFinger.Index ) ) {
			// Reset the teleportation state
			teleportation_locked = false;
			return;
		}

		// Prevent continuous teleportation
		if ( teleportation_locked ) return;
		teleportation_locked = true;

		// Tell the character controller to move to the teleportation point
		character_controller.Move( target_point - this.transform.position );


	} else {
		// Remove the cursor
		if ( marker_prefab_instanciated != null ) Destroy( marker_prefab_instanciated );
		marker_prefab_instanciated = null;

		// Reset the teleportation state
		teleportation_locked = false;
	}

}

Setup the virtual scene

Now that the script edition is complete it should appear as illustrated bellow in the Inpsector View :

Create the teleportation marker prefab

As explained above, our script FingerLocomotion will instantiate a marker prefab to show the player the targeted position. Thus we must create this marker prefab in the first place to reference it :

Then we can convert it into a prefab by dragging the instance into the Assets folder :

As we want our target marker to be easily visible we can create a new material to apply on the sphere :

And edit it’s color :

Now we can open the prefab to edit its content :

We can start by dragging the new color onto the target :

Now we must remove the Collider from the sphere (otherwise the raycast will hit the sphere itself) :

We can scale the sphere :

Leave the prefab edition :

And remove the instance of the prefab from the sphere (as it will be automatically instantiated by our script) :

Link items

Let’s start by selecting the OVRHandPrefab from the left hand :

And drag it to the Left Hand field of our FingerLocomotion script attached to the OVRPlayerController :

You can do the same for the Right Hand.

Now we select our created teleportation_marker prefab :

And drop it into the Marker Prefab field of our Locomotion Script :

And finally we can change the layer of the OVRPlayerController to make sure the raycast won’t hit the player’s collider :

And apply the same layer for childrens :

Build

Everything should be ready now and you can build your project (Ctrl + B) and you should expect such kind of results where you can point at a direction with one hand and trigger the teleportation with the other one :

Commit

Now that we tested this feature we can commit our changes and merge it with the master_finger_tracking branch :

# Commit
git add .
git commit -m "Add new locomotion based on pinching indexes"

# Merge
git checkout master_finger_tracking
git merge develop_locomotion --no-ff
git tag finger_locomotion_implemented -a -m "Finger locomotion implemented"

# Push
git push --all
git push --tags

Thanks

Thank you for your attention during these sessions. I hope you learned things which might help you to enjoy creating your own game, expressing your creativity, or for other topics ! Please feel free to use the forum if you have questions or want to give your feedback on this hands-on or on this course !