Professional Documents
Culture Documents
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Legal Notice
Copyright 2012 by Omek Interactive. All rights reserved. All information included in this document, such as text, graphics, photos, logos and images, is
the exclusive property of Omek Interactive. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or
translated into any language or computer language, in any form or by any means, electronic, mechanical or otherwise without prior written permission
of Omek Interactive.
This document is provided strictly on an as is basis without warranty of any kind, either express or implied, including, but not limited to, the implied
warranties of merchantability or fitness for a particular purpose Omek Interactive assumes no responsibility for any errors that may have been included in
this document, and reserves the right to make changes to the document without notice. Omek Interactive disclaims any responsibility for incidental or
consequential damages in connection with the furnishing, performance, or use of this document.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Table of Contents
Preface ................................................................................................................................................................................ 5
Gesture Authoring Tool (GAT) Overview ............................................................................................................................ 5
What is a Gesture? .......................................................................................................................................................... 5
Training GAT to Recognize Gestures ............................................................................................................................... 5
Machine Learning ............................................................................................................................................................ 6
Basic Entities and Terms .................................................................................................................................................. 7
Frame .......................................................................................................................................................................... 7
Gesture........................................................................................................................................................................ 7
Gesture Pack ............................................................................................................................................................... 7
Sequence ..................................................................................................................................................................... 8
Clip .............................................................................................................................................................................. 8
Building a Pack................................................................................................................................................................. 8
Designing a Gesture Pack ................................................................................................................................................ 9
Guidelines for Designing a Gesture ............................................................................................................................. 9
Gesture Training Overview ............................................................................................................................................10
Stage One: Creating a Pack and Adding Gestures ............................................................................................................12
Creating a Gesture Pack ................................................................................................................................................12
Pack Dashboard......................................................................................................................................................... 14
Adding and Modifying Gestures ....................................................................................................................................15
Adding a Basic Gesture ............................................................................................................................................. 15
Adding a Preview Image to a Gesture ....................................................................................................................... 17
Creating Composite Gestures ................................................................................................................................... 18
Importing Gestures ................................................................................................................................................... 20
Adding Rules ..................................................................................................................................................................21
Stage Two: Preparing Sequences......................................................................................................................................23
Adding Sequences .........................................................................................................................................................23
Recording Sequences Guidelines and Tips ............................................................................................................. 23
Importing Sequences ................................................................................................................................................ 25
Marking Sequences .......................................................................................................................................................27
Marking Sequences Guidelines and Tips ................................................................................................................ 28
Stage Three: Building the Pack .........................................................................................................................................32
Choosing a Build Mode..................................................................................................................................................32
Training Gestures Separately ........................................................................................................................................33
Initiating the Build .........................................................................................................................................................34
Understanding and Improving Results ..........................................................................................................................35
Build History .............................................................................................................................................................. 37
Tips for Dealing with Misses ..................................................................................................................................... 38
2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Preface
This document describes the process of training Gestures using the Omek Gesture Authoring Toolkit (GAT). It
provides an overview of the basic training process and workflow, explains the different stages, and introduces
the interactive application for training basic Gestures.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Machine Learning
Machine Learning is a type of computer algorithm, which learns to identify certain patterns by analyzing
positive and negative examples of the pattern. This method has many applications. For example, using machine
learning, you can train software to identify subject categories of a given text, musical genres of recorded music
clips, or a specific persons face from among photographs of different people. In our case, we train software to
recognize certain Gestures from among many possible gestures.
In order to learn by example, a machine learning algorithm must receive as input both instances of the pattern it
is trying to identify, and instances that do not conform to the pattern. These are often referred to as positive
and negative instances. In the case of gesture recognition, a positive instance is a video clip that contains the
gesture, while a negative instance is a video clip that does not contain the gesture.
When you are working with GAT, and you mark Gestures within their sequences, what you are doing is defining
positive instances for the machine learning algorithm to work with. Negative instances are taken from randomly
selected locations throughout the unmarked portions of the sequences.
The machine learning algorithm then analyzes all of the examples according to features that a programmer
defines. In the case of Gestures, the features might include the relative joint positions of the tracked person, the
speed of their motions, and so on. By comparing these features between positive and negative instances, the
algorithm learns to identify positive instances of the defined Gestures, not only those it encounters in the
training set, but also new instances that it encounters at runtime.
Since everything the algorithm learns is derived from the instances it receives during training, its important to
represent the real world as far as possible in the set of training instances. This means that you should record
the same gesture performed by several people, from various positions in relation to the sensor. You should
record gestures performed by men, women and children, depending on who will use the application.
When the algorithm is trained well, it can achieve highly accurate gesture identification. This may take several
iterations of training, involving adding more instances, marking instances more accurately, and preventing GAT
from trying to learn from noisy data or ambiguous instances.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Frame
A video recording is not a continuous stream, but is in fact composed of several discrete images or frames. The
number of frames recorded per second depends on the specific sensor used. A typical frame rate for the types of
sensors used for motion tracking is 30 frames per second. The longer a video sequence is, the more frames it
contains.
Gesture
A Gesture is a unique, distinctive pose or motion of one or more body joints. This may be a static (single-frame)
pose, such as the right hand stretched forward, or a dynamic (multi-frame) motion, such as the right hand
drawing a circle.
GAT supports the following types of gestures:
Basic Gestures, which are the fundamental building blocks of GAT. These Gestures are created using GAT
training process. GAT distinguishes between three types of Basic Gestures:
Single Motion Gesture: a gesture that is completed in a short period of time, such as a salute.
Pose: a single, static body pose, such as standing with arms stretched upwards.
Continuous Gesture: an ongoing motion, such as running or spinning.
When creating a Basic Gesture, you can create its Mirror Gesture with no additional effort. A Mirror
Gesture is the mirror image of a Basic Gesture. For example you can define a jumpRight Gesture and at
the same time define a jumpLeft Gesture as its mirror image. You dont have to train a Mirror Gesture
separately GAT trains it automatically.
Composite Gestures, which are a combination of two Basic Gestures performed one after the other, in a
certain order and within a specified time range.
Gesture Pack
A Gesture Pack is a collection of Gestures that are trained and packaged together. Typically you create a single
Gesture Pack that contains all the custom gestures that your application uses. When you add a Basic Gesture to a
Gesture Pack, you must retrain all of the gestures in the pack. This is because GATs machine learning algorithm must
take into account all of the types of gestures it can encounter. A positive instance of a new Basic Gesture also serves
as a negative instance of other Basic Gestures, thereby affecting their training process.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Sequence
In order for GAT to be able to recognize a Basic Gesture, you need to teach it what the Gesture looks like. You do
this by recording short video segments of people performing the Gesture. A video segment, together with its tracking
data produced by the Beckon SDK, is called a Sequence. A sequence is a recording of a person performing the
Gesture, with a length of up to 60 seconds.
Once you have recorded sequences, you need to indicate which frame or frames contain the required Gesture. This is
called Marking.
Clip
You can mark three types of segments or Clips:
Gestures - Each performance of a given Gesture in a sequence is called an Instance. Typically, there
are several instances of a Gesture in a sequence. You must mark all of the Gesture instances in the
sequence to ensure correct learning.
Bad Data - You can assist the training process by marking faulty tracking. This refers to places in the
sequence where you see that identified joint locations are incorrect in relation to the true joint
positions. When you mark the faulty frame or frames as Bad Data, GAT will ignore these portions
when learning Gestures.
XNegative - Sometimes you may encounter segments which are confusing to the learning process, as
they contain a movement or pose that resembles one of the defined Gestures but should not be
identified as a Gesture. In this case, you can mark the segment as XNegative, which ensures that this
specific segment will be taken as a negative instance for all Gestures.
The marked segment of a sequence which represents a specific event (Gesture, Bad Data or XNegative) is called
a Clip.
Building a Pack
Once all Gestures are created in the Pack, and sequences are marked for each Gesture, you can build the Pack. During
the build process, GAT learns from the examples you have provided and creates Classifiers for the Gestures. A
Classifier is a software component that identifies a given segment as the target Gesture, or rejects it.
The build process not only creates Classifiers, but also tests their quality. To do this, GAT takes a random subset
of the marked sequences, and uses them to train the Classifiers. Then it tests them on the rest of the marked
sequences. This means that GAT runs the Classifiers on the marked sequences that were not used for training,
and produces a Build Report, which shows what Gestures were identified in this stage.
The test results include:
The number of Hits (true positives in machine learning terms), which are Gestures that were detected
correctly.
The number of False Hits (false positives in machine learning terms), which are cases where a Gesture
was identified although it was not performed.
The number of Misses (false negatives in machine learning terms), which are cases where a Gesture was
performed but was not identified.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Name: The Gesture name should be unique and comprehensive, for example, rightFrontPunch".
Note:
A Gesture name must be unique within a single Pack. The same Gesture name may appear in
more than one Pack. However, if several Packs are used in the same application, there may not
be duplicate Gesture names among these Packs.
Granularity: Each Basic Gesture should be short and simple. Avoid defining combinations of more than one
basic movement as a Single Motion Gesture. Optionally, two simple Gestures can be combined into a
Composite Gesture. For example, instead of training a leftRightFrontPunch Gesture (which includes a left
front punch and then a right front punch), you can train a rightFrontPunch and a leftFrontPunch, and
then you can easily define any combination of these punches.
Participants: Determine whether the target participants include men, women and children. Make sure that
the video clips show several different people from among the target groups, performing the relevant
Gestures. The Gestures should be performed several times, with various people in various locations.
Description: When defining a Gesture, make sure its description is clear and specific. For example: The
right hand starts from the right side of the body, at the height of the head or the torso, and moves fast and
straight towards an opponent, who stands in front of the player. A clear, detailed description will help
when recording video segments.
Conditions: Detail the conditions and situations under which the Gesture will be performed, regarding start
and end positions, speed of motion, and any other factors of the gesture. For example: A punch can be
executed from a neutral pose (while the right hand is at the side of the body, in front of the torso, or close
to the right shoulder), or from a defense pose (where the hands defend the face). A punch can be executed
from a standing position, while dodging an opponents hit, or while stepping forward. Also, if there are
conditions in which the Gesture will not be performed, describe these as well.
Avoid confusion: If there are similar motions or poses that should not be detected as the Gesture, describe
them as well. For example: A slow movement of the hand should not be detected as a front-punch. A
hook-punch should not be detected. If there such confusing motions or poses, make sure to record
them as part of the input to the Build process. Mark them as XNegative clips, to ensure that GAT does not
learn them as Gesture instances.
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Build
Refine
Analyze/Live Test
Publish
Note:
Before building the Pack with all Gestures enabled, you can also choose to build one Gesture at
a time. See Training Gestures Separately for more details.
10
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
In each sequence, mark Gesture instances, Bad Data and XNegative instances.
Stage Three: Building the Pack: Build the Pack. GAT creates a Classifier for each gesture.
Stage Four: Testing the Pack: Test and improve the Pack:
If you want to improve performance, return to stage 2, add sequences and improve marking, then build and
test.
Stage Five: Publishing the Pack: Create the software module that you will integrate into your application.
11
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
12
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
1.
In the Name text box, type a name for the new Pack.
Note:
Gesture Pack names can only contain lower-case letters, upper-case letters, and underscores.
2.
Click Create.
The Pack Dashboard window appears. For a new Pack with no Gestures defined, the Pack Dashboard looks
like this:
Note:
To rename a Pack after it is created, on the File menu, click Rename Pack. Do not rename Pack
files manually as this could cause the Pack to malfunction.
13
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Pack Dashboard
The Pack Dashboard displays all of the Gestures defined in the Pack. From this window you can add and modify
Gestures in the Pack.
4.
14
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Click Open Pack to display the Open window. Select the .gpp file you want to open.
The Pack Dashboard appears with all the Gestures defined in the pack.
2.
In the Name text box, type a name for the new Gesture.
Note:
Gesture names can only contain lower-case letters, upper-case letters, and underscores.
3.
From the Type drop-down list, select the type of Basic Gesture you want to define: Single Motion, Pose or
Continuous.
4.
Optionally, you can choose to create a Mirror Gesture for this Basic Gesture. (See Gesture for an
explanation of Mirror Gestures). When you check the Create Mirror Gesture box, the Mirror Name area is
displayed. Type a name for the Mirror Gesture.
15
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
5.
Click OK.
The Pack Dashboard displays an icon for the new Gesture.
Note:
If you added a Mirror Gesture, the Mirror Gesture icon
Mirror Gesture.
Note:
Defining a new Mirror Gesture requires a rebuild of the Gesture Pack, but deleting an existing
Mirror Gesture does not.
16
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
You can change this image to a different image that illustrates the actual Gesture you are defining. The following
image files types are supported: JPEG, TIFF, PNG, GIFF.
To add a preview image:
1. Right-click the icon of the Gesture to which you want to add the image, and choose Add Preview Image.
The Open window appears.
2.
Select the image file type from the list, and browse to the location of the file you want to use as a preview
image.
3.
Click Open.
The Gesture icon now displays the image you chose.
17
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
3.
4.
From the first Gesture dropdown list, select the first Gesture of the Composite Gesture.
5.
In the Minimum Gap textbox, type the minimum gap between the two Gestures in frames.
6.
In the Maximum Gap textbox, type the maximum gap between the two Gestures in frames.
7.
From the second Gesture dropdown list, select the second Gesture of the Composite Gesture.
18
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
8.
Click OK.
The Pack Dashboard displays the new Composite Gesture icon.
Note:
When you create a Composite Gesture from Gestures that are already trained, you do not need to
rebuild the Gesture Pack. Deleting a Composite Gesture also does not require a rebuild.
19
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Importing Gestures
You can import a Gesture from one Pack to another. When you import the Gesture, you import all its definitions as
well as sequences associated with the Gesture and their markings.
To import a Gesture:
1.
2.
3.
Check the Gesture or Gestures you want to add and click Add.
Note:
If the Create a copy of sequences when I add them to the pack option is checked (see Appendix B:
GAT Configuration), the imported sequence files will be copied to your Pack folder.
20
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Adding Rules
A Rule is a condition or constraint that must be met in order for a Gesture to be recognized, in addition to its
motion properties. The following Rule is supported:
Refractory Period Rule - This Rule dictates that a Gesture will only be recognized once within a specific number
of frames. A new instance of the Gesture is detected only if the specified number of frames has passed since the
previous occurrence of the Gesture.
To define a Rule:
1.
In the Pack Dashboard, from the Pack menu, choose Pack Rules or click the Rules button
the Gesture Pack Rules window.
, to display
Note:
If you define a rule for a Gesture that has a Mirror Gesture, the rule will automatically be applied to
the Mirror Gesture as well. You can override this behavior by explicitly adding a different rule for the
Mirror Gesture.
Note:
Creating a new rule or updating an existing rule does not require rebuilding the Gesture Pack.
Deleting a rule does require a rebuild.
21
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
2.
Click the
3.
From the drop-down list, select the Gesture for which you want to define the Rule.
4.
5.
Click OK.
Once the Pack is built, this Rule will affect the way the Gesture is recognized.
22
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
2.
Adding Sequences
You can record new video sequences or import pre-recorded sequences. A GAT sequence is composed of two
files: a file that contains the feed from the sensor, and an additional file that contains the Beckon motion
tracking information (the .seq file). Both files should be located in the same folder and have the same name with
different suffixes.
By default, the sequences recorded by GAT are saved in the Pack Folder. You can change the target folder for the
recorded sequences (see Gesture Pack Locations).
Short sequences: record short sequences (typically, sequences should be no longer than 30 seconds, with
no more than 4 Gesture instances).
Single person: Only one person should be in the sensors field of view when recording sequences.
Person variation: record several different people (one at a time) of different sizes.
Location variation: record people in different locations and angles, relative to the sensor.
Clarity: Try to record typical, unambiguous instances of the Gesture. In particular, it should be obvious
when the Gesture begins, and when it ends.
Demonstrate XNegatives: In addition, if there are movements that look similar to one of the Packs
Gestures, but should not be identified as a Gesture, record these too and mark them as XNegative.
23
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Note:
In order to record sequences, you need to connect a sensor to your computer and make sure it is
operating properly.
To record a sequence for a Gesture:
1.
Double-click the Gesture to which you want to add the sequence. The Gesture Dashboard is displayed.
2.
Click Add Sequence, choose Record New and wait for the sensor to initialize.
The Viewing Pane displays the sensor input.
3.
In the Recording dialog, you can click the Options arrow to display the recording options.
24
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
4.
Note:
When you record a new sequence, GAT will name it automatically as seq followed by a number.
Sequences are numbered in order of creation, starting from 0.
After a sequence is created, you can right-click on it and choose Rename to rename it.
Sequence names can only contain lower-case letters, upper-case letters, underscores and digits.
Importing Sequences
You can also import pre-recorded sequences for training Gestures. You can import a single file or a folder with
several files.
By default, importing sequences does not copy the sequence files into the Pack folder. If you want to copy the
files, activate this option in the Options window (see Appendix B: GAT Configuration). A sequence file that
already exists in the Pack will not be recopied when importing sequences.
If the tracking file (.seq) file is missing (for example if the sequence was not recorded using GAT, or if the .seq file
was created using an older version of the Beckon SDK) you have the option of retracking the sequence. This
means that GAT processes the sensor sequence and creates tracking information in a new .seq file.
2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com
25
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
IMPORTANT!
Importing a folder containing a large amount of sequences can take a long time, especially if the
sequences need to be retracked.
If a sequence needs retracking after it has been added, GAT will indicate this on the specific sequence in the
Sequences pane on the left of the Gesture Dashboard.
To retrack a sequence, right-click on the sequence and click Retrack.
2.
3.
4.
Browse to the sequence file you want to import and click Open.
The Sequence is added to the Gesture Dashboard.
You can also import several sequences at once, if they reside in the same folder.
To import a folder:
1.
2.
3.
4.
26
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Marking Sequences
Once sufficient sequences have been added, you must mark the Gesture instances in the sequences. While you
play back the sequence in GAT, you can mark a clip the frames in which the Gesture appears.
The number of clips or instances required to train and test a Gesture can vary, depending on the complexity and
the uniqueness of the Gesture. Single Motion gestures require more instances than Pose or Continuous
Gestures.
Note:
As a rule of thumb, before building a Pack, make sure that you have:
1. 30 marked clips for each Single Motion Gesture.
2. 10 marked clips for each Pose and Continuous Gesture.
You may have to add more clips after the first build, to improve results.
Here are some numbers taken from the real training scenario of a RightClick gesture:
15 sequences
40 marked Gesture clips
15 XNegative clips
In addition to marking Gesture clips, you can also mark segments as Bad Data and XNegative.
Bad Data refers to places in the sequence where you see that identified joint locations are incorrect in relation
to the true joint positions. When you mark the faulty frames as Bad Data, GAT will ignore these portions when
learning Gestures.
XNegative refers to motions that are similar to one of the Gestures in your Pack, and could be confused with it,
but should not be detected as a Gesture. For instance, if you see that hopping on one foot is sometimes
confused with jumping on both feet, mark the hopping-on-one-foot sequences as XNegative clips (assuming
hopping-on-one-foot is not a Gesture in the Pack).
IMPORTANT!
XNegative clips are learned as negative instances of all the Gestures in a Pack (they do not relate to
one specific Gesture). Therefore, take care not to mark an XNegative clip on a motion that should be
recognized as one of the Gestures in the Pack, because then GAT will try to learn it as both a positive
and a negative instance.
Note:
If a single sequence contains several different types of Gestures, add it to each Gesture in turn. Then
mark the clips of each Gesture within the Gesture Dashboard.
Note:
If a sequence is not manually marked with any type of clip (Gesture, XNegative or Bad Data) it will
not be used when the Pack is built. A sequence which has automatically-marked Bad Data clips will
still be excluded unless it has user-defined clips.
2012 Omek Interactive, Ltd. All rights reserved. | www.omekinteractive.com
27
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Completeness: Each and every instance of the Gesture must be marked. If you do not mark a correct
instance of a Gesture, its frames could potentially be used as a negative instance, causing GAT to learn from
incorrect input and harming its accuracy.
Consistency: When marking instances of a specific Gesture, be consistent about marking the beginnings and
ends of the Gesture. Each instance should contain the same motion (and not only part of the motion).
Partial Gestures: All unmarked frames can potentially be used as negative instances. If only part of a
Gesture appears at the beginning or end of a sequence, it should not be marked as a positive instance, since
only part of the Gesture is available. On the other hand, it should not be considered a negative instance
either. In this case, mark the partial Gesture instance as Bad Data, so that it will be ignored by the learning
process.
Accurate Marking: make sure to mark the start and end of a Gesture accurately, and not to include frames
in the clip that dont belong to the Gesture. When in doubt, its better to make a clip shorter rather than
too long.
No Overlap: make sure that there is no overlap between clips of any type (Gesture, XNegative or Bad Data).
If the same sequence was added to more than one Gesture, take special care not to mark Gestures that
overlap with other Gestures clips in the sequence.
3D View: We recommend working in 3D View when you are marking clips. In this view, you can adjust the
angle in which you view the tracked skeleton, so that you can optimally detect the Gestures start and end.
For instance, if a person stretches their arm forward and back, its easier to see the boundaries of this
motion from a side view rather than a frontal view.
XNegatives: You should use XNegative clips to mark specific motions that could be confused with one of the
Gestures you are training. Usually you will detect these segments when they are identified as false hits
during the build. As with marking Gesture clips, its important to mark XNegative clip ranges accurately, and
its better to make a clip shorter rather than too long.
28
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Bad Data: Gestures should only be marked on areas in the sequence where the tracking is accurate.
Inaccurate joint data reduces the quality of the Pack. Therefore, you should pay attention to the Low
Confidence indicator, which appears as a red underline in the sequence playback and automatically marks
low-quality segments as Bad Data, as in the example below:
Do not mark Gestures on segments which GAT has automatically marked as Bad Data. Additionally, if there
are segments where you can see that the joint tracking is inaccurate, even if GAT has not marked them as
such, you can explicitly mark them as Bad Data.
29
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
2.
Tip:
You can decrease the playback speed by clicking the Speed arrow and selecting or . It is easier to
mark clips in slow motion.
3.
4.
Use the Mark In and Mark Out buttons to mark the beginning and end frames of a clip.
to control the
30
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Click the Mark arrow to display the clip options and click the type you want to mark. Choose:
5.
XNegative: to mark a clip as a segment that should not to be identified as a Gesture (although it is
similar enough to one of the Gestures to be confusing).
Make sure to mark all the occurrences of the Gesture. You should also take care to mark as XNegative all events
that might be mistakenly detected as a Gesture.
31
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Fast
Normal
Slow
Although the names of the build modes appear to indicate their speed, they actually differ in the number of
learning iterations that each one performs. The Fast mode performs one learning iteration, the Normal mode
performs three iterations, and the Slow mode can perform a varying number of iterations, up to half the
number of sequences in the pack. The modes also differ in the number of sequences that they use to train the
Classifiers. Therefore, the quality of results that each mode produces may differ. There is no absolute rule as to
which mode is always fastest in practice, or which mode produces the best results.
Note:
As a rule of thumb, select the Fast build mode to start with. Use this mode to examine the initial
results and improve your sequences and their marking. Then you can try the Normal and Slow
modes, to see which mode produces the best results.
32
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Note:
Building Gesture Classifiers separately is optional. If this method is used, it is only a preliminary stage
before building the entire Pack. Ultimately, GAT must train on all Gestures together, so that it can
learn to differentiate among them.
33
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
2.
In the Pack menu, click Build Settings, then click the build mode you want.
3.
4. The build process can take from a few minutes up to hours depending on the number of sequences in the
Pack. Building the Pack creates a build sub-directory in the Pack directory.
When the build is finished, GAT displays the test scores for the Basic Gestures.
Note:
By default, GAT will try to build several classifiers in parallel, to shorten build time. If your Gestures
contain large numbers of sequences, this mode may fail due to lack of memory. If this happens, you
will see this message:
In this case, click Build Serial to build in serial mode. This mode takes longer but requires less
memory.
34
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
35
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
The following table summarizes the Build Reports information. The numbers in the first column refer to the areas
marked in Figure 29: Build Report above.
Table 1: Build Report Items
Name
Description
Score
The total score of the Gesture, on a scale of 0-100. (See note below
regarding the calculation of the score value).
Hits
False Hits
Misses
The number of events that were marked as Gestures but were not detected.
Sequence
False Hits/
Misses
For each of the sequences, the number of False Hits and Misses and specific
frames where they were detected.
Note:
The score value is calculated according to this formula:
Score = (# Hits) / (# Hits + # Misses + # False Hits)
Note:
To analyze an error in detection, double-click on the error frame number in the False Hits or Misses
column. The Gesture Dashboard opens, showing the sequence and frame of the error.
36
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Build History
In addition to the results of the current build, GAT saves the build results of the previous 10 versions.
To see the results of previous builds:
1. Click the Show Build Reports button
The Build Reports window appears.
2. Click the Build Reports History tab. A list of the last 10 build date-times is displayed.
3. Click on the arrow to the right of each builds date-time to see its results.
Note:
If you want to retrieve the Gesture Packs classifiers from older builds, you can find them in subfolders under the Gesture Packs Build folder. The sub-folders are named according to the date and
time of the build, as they are displayed in the Build Reports History window.
37
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Note:
The Build stage uses some of the marked clips for training the Classifiers, and the rest for testing
them. The Build Report refers only to the test set results. The division between train and test sets is
constant, so that while youre improving results, you can examine the changes on the same set.
If there are Gestures in the Pack that were not included in the build, they are also listed.
Examine each of the falsely identified events, review their sequences, and see whether their clip marking should
be adjusted. Below are some tips on how to do this.
When you are satisfied with the build results, you can go on to test the Pack on live input from a sensor or on a
recorded video sequence.
Add and mark more sequences with Gesture instances, so that GAT has more examples to learn from.
If you notice a tracking problem, you can use the Bad Data option to exclude the specific frame or
frames from the build.
If the beginning or the end of a marked event is not accurately marked, you can drag either of the
borders of the clip to fine-tune the marking.
If a particular movement is identified incorrectly as a Gesture, you can use the XNegative option to mark
the specific segments as explicit negative instances. (Remember: XNegative clips will be used as negative
instances of all Gestures, so make sure you dont mark a valid Gesture as an XNegative clip). You can
also record new instances of this confusing movement, and mark them as XNegative clips as well.
If you see a False Hit within the range of a marked Gesture instance, this is due to the fact that GAT
refines the marked Gesture ranges during the build. In this case, we recommend correcting the start and
end positions of the clip manually, so that only the Gesture is included.
Note:
Bad Data clips are ignored in the Classifier training stage, but False Hits and Misses may occur in
faulty data segments during a Live Test. This represents a real-life scenario and can occur at
application runtime as well.
38
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Note:
When testing a Pack on a pre-recorded sequence, avoid frequent presses on the Play/Pause button,
and do not move the playback thumb
. These actions disrupt the timing of the sequences
input and can cause incorrect Gesture recognition.
You can now refine the Pack by adding marked training sequences according to the problems you detected and
rebuilding the Pack.
Figure 32 describes the fine-tuning feedback cycle.
39
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
The following table summarizes the Test window parts and controls:
Table 2: Test Window Items
Option
Use to
Display detected Gestures and Alerts, including name and frame number
Filters pane
Video panel
Playback Timeline
Displays the playback progress with any marked clips, and the Confidence
indicator.
Input source
Play
40
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
5.
In the Filters pane, click Gestures to display the Gestures tab and select the gestures you want to detect.
You can click the arrow next to the color and select a different color for the display of the detected Gesture.
The Gestures you selected will appear during the test on the timeline and in the Gestures and Alerts pane.
Note:
Gestures whose classifiers have not been built cannot be viewed in Live Test.
6.
In the Filters pane, click Alerts to display the Alerts tab, and select the Alerts you want to display.
The Alerts you selected are displayed in the Gestures and Alerts pane.
7.
Test live sensor input. Make sure you have a sensor connected, then click Live Camera. After the
sensor initializes, the video panel displays the input from the sensor. Have a person in view of the
sensor perform the Gesture, and observe the detection.
41
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Test from a video sequence: Click Sequence, the click the Browse
location of the sequence. Click
42
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Each time a Gesture is detected, the event is marked on the timeline in the color selected for the specific
Gesture.
The Gestures and Alerts pane displays occurrences of the detected Gestures and Alerts, according to the
filters selected in the Filters tabs.
You now need to analyze the results and if necessary, add or modify training sequences according to the
problems you detected.
43
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
44
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Appendix A: Glossary
Bad Data
A marked segment of a sequence that will not be used when building the Pack. This option is used to mark faulty
tracking data or partial Gestures.
Classifier
A software component that identifies a given motion as the target Gesture, or rejects it.
Clip
A marked segment of a sequence which represents a specific event (Gesture, Bad Data, XNegative).
False Hit
A unique, distinctive motion or pose of one or more body joints. This may be a static pose, such as the right
hand stretched forward, or a motion, such as the right hand drawing a circle.
Gesture, Basic
Pose Gesture
Continuous Gesture
Gesture, Composite
A Gesture that is defined as a combination of two Basic Gestures, performed in a specific order within a
specified time range.
Gesture, Mirror
A Gesture that is a mirror image of a Basic Gesture. For example, the Mirror Gesture of a jumpRight Gesture
would be a jumpLeft Gesture.
Hit
One occurrence of a given Gesture in a sequence. There can be several instances of a Gesture in a single
sequence.
45
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Miss
A pose or motion that is similar to one of the defined Gestures, but should not be identified as a Gesture.
XNegative examples are used to train the Classifier to reject such instances.
Sequence
A short video clip, which includes the tracking data as computed by the Beckon SDK.
46
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Build Folder: The default location for the classifier files that GAT produces. Older build files are also stored
here, under sub-folders that are named according to the date and time of the build.
Sequence Handling
Create a copy of sequences when I add them to the pack: When checked, sequences imported into the
Pack are copied into the Pack folder. This option is turned off by default.
Clear all previous marking when I create a copy of the sequences: When checked, clip markings on the
imported sequences are removed from the copied versions. This option is turned off by default.
Startup
Show Step-by-Step Guide: When checked, the Step-by-Step Guide is displayed by default.
Open Last Pack: When checked, when GAT is opened, it automatically opens the last Pack that was edited.
This option is turned off by default.
47
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
48
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
The following table summarizes the parts and controls of the Gesture Dashboard.
Description
Use to
Add Sequence
Sequences Pane
Display all sequences added to the Gesture. The right-click menu enables
you to rename a sequence, delete a specific sequence or all sequences,
retrack a specific sequence or all sequences, clear all markings or display
statistics.
Set up the viewing panes 2D, 3D and overlays views (see Setting Up the
Viewing Pane).
Video Pane
Playback Timeline
Marker
Navigate Playback
Timeline
Playback Toolbar
Control playback.
Speed
Go
49
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
The Viewing pane can display additional optional information. The following table lists the overlays available in
the 2D and 3D Views.
Table 4: 2D View/3D View Overlays
Overlay
Description
Available in
RGB
2D
Depth
2D
Show Joints
Displays joints as colored dots. You can turn joint display on and off, and also
display only a subset of joints that you select in the Select Joints window.
2D/3D
Show Bones
Displays bones as colored lines. You can turn bone display on and off.
2D/3D
Joint Info
2D/3D
Select Joints
Opens a window that allows you to select the joints you want to display in the
overlay.
2D/3D
50
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
2.
Click each joint to select/de-select it. You can also click Select All to select all joints or Clear to de-select all
joints.
3.
You can also click Joints List to display a list of the joints. Click a joint name in the list to select/de-select the
joint.
Note:
Selecting joints in the 2D/3D view only affects their display. It does not affect the set of tracked
joints.
51
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Options
Use to
Sequence
Video Panel
Playback Timeline
Displays the playback progress with any marked clips, and the Confidence
indicator.
Navigate Playback
Timeline
Playback Toolbar
Speed
Frame
Enter a frame number and click Go, to skip to that frame in the playback.
52
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Playback Toolbar
You can use the Playback toolbar icons
playback of a sequence.
to control the
Table 6: Playback Options
Button
Click to
Stop playback and go to the beginning of the sequence.
53
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Appendix E: Statistics
The Statistics feature provides information about the Pack, the Gestures and the sequences in the Pack.
To display Pack statistics, do one of the following:
Total Number of Sequences: the number of sequences in all the Gestures in the Pack.
Average Score: the average build score for the Gestures in the Pack.
Last Build Total Runtime: the time that the latest build took to complete.
Last Build Type: The build type (Normal, Fast, or Slow) chosen for the last build.
54
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
To view a Gestures Statistics, on the left pane, click the Gesture you want to view.
The statistics for the Gesture you selected appear in the right pane.
The Gesture Statistics include the following information for each of the Gestures in the Pack:
Mirror Gesture: the name of the corresponding mirror gesture, if there is one.
Excluded from Build: whether this Gesture will be ignored during the next build.
Number of Marked Sequences: the number of marked sequences for the Gesture.
Number of Unmarked Sequences: the number of unmarked sequences for the Gesture.
55
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
In the left pane, click the Gesture arrow to view its sequences.
2.
The sequence statistics include the following information for each of the sequences in the Pack:
Clips Marked as Gesture: the number of clips in the sequence that are marked as Gesture instances.
Clips marked as Bad Data: the number of clips in the sequence that were marked as Bad Data.
Clips Marked as XNegative: the number of clips in the sequence that were marked as XNegative.
56
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Appendix F: FAQs
Question:
How many sequences should I record to train a Classifier?
Answer:
The number of required sequences varies for different Gestures, as some are easy to detect and some are more
difficult. In general, use the following approach. Begin by recording at least 10 training sequences, of which the
majority will show the target Gesture. A few of the sequences should exhibit other motions that are similar to
the Gesture, but should not be detected as the Gesture.
Since the examples that the Build learns from are the marked clips of the gesture (and not the entire sequence),
you need to make sure that the set of sequences includes:
Question:
I trained a new Gesture Classifier, but it gets very few hits. What should I do to improve it?
Answer:
Add more Gesture instances to the training sequences.
Question:
I need to train the same Gesture for the left and right hands. Do I have to record sequences for each side?
Answer:
You do not need to mark and record sequences for each side separately. Instead, you can use the Mirror Gesture
feature. While defining a Gesture, check the Create Mirror Gesture option. Name the original and mirror
Gestures informatively. For instance, if the original Gesture is jumpRight, name the mirror Gesture jumpLeft.
You only need to add sequences to the original Gesture. GAT will automatically train the mirror Gesture on a
flipped version of the original sequences.
Question:
I trained a new Gesture and it works well, with no misses at all. But I am getting quite a few false hits, what
should I do?
Answer:
Record sequences of a person performing the motion that causes the false hits, and add a few such sequences
to the training set.
57
O M E K G E ST U R E AU T H O R I N G TO O L FO R B E C KO N 3 . 0 - U SE R M A N UAL
Question:
The trained Gesture is not reliable, although I have many sequences.
Answer:
Make sure the input data is accurate. Pay particular attention to the Gestures clips and the tracking in the
training sequences. Make sure all the Gestures are marked. If the tracking is not accurate for a particular
sequence, mark the problematic frames as Bad Data.
Question:
What are the typical numbers of sequences and clips in a real-life Gesture training scenario?
Answer:
Here are some numbers taken from the real training scenario of a RightClick gesture:
15 sequences
40 marked Gesture clips
15 XNegative clips
This Gesture took 4 cycles of building and refinement, which were completed in a few hours.
58