How to save/recall a gesture for later use

[bridgewaterrobbie]

Hey sorry to bother you again, we have been trying to get up and running but running into a few issues that we think you might be able to solve quickly. We tried looking through signadventure, but havent had much luck there since it seems you use a different process, and we were unable to find the methods you used to initially create the gestures.

We are trying to create a gesture, then save it for later use with .toJSON, restoring it with .fromJSON. In sign adventure, you seem to use a .byte file to recall old gestures, but we can't figure out the path you took to load them from the excercise prefab into LeapClassifier, nor how to create a new byte file.

Also we have one very strange thing happen as we call toJson. Despite checking that the Gestures dictionary does contain the key with "this.gestures.ContainsKey(gestureName)", the code crashes immediately after with the line "var gestures = this.gestures[gestureName];" with a

KeyNotFoundException: The given key was not present in the dictionary.
System.Collections.Generic.Dictionary`2[System.String,System.Collections.Generic.List`1[System.Collections.Generic.List`1[Point]]].get_Item (System.String key) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
LeapTrainer.toJSON (System.String gestureName) (at Assets/LeapClasifier/LeapTrainer.cs:893)
ThomasLeapTrainer.FixedUpdate () (at Assets/Scripts/ThomasLeapTrainer.cs:30)

We also have not managed to get a gesture to be recognized after creating it. Our code looks like this:

    void Update () {
        if (Input.GetButtonDown("Fire3") && trainer != null)
        {
            startTraining();
        }


        trainer.OnGestureRecognized += (name, value, allHits) => {
            Debug.Log("Gesture recognized");
        };



    }

    void startTraining()
    {
        if (trainer == null || isTraining) return;

        isTraining = true;
        Debug.Log("Button Pressed");

        trainer.Create("TestGest", false);
        Debug.Log("Gesture Created");
    }
}

Again, sorry for the bother and we appreciate any feedback/solutions you have time to give.

[Victorma]

They way I used it was simply but is not avaliable atm (discussed later).

I just used the LeapMotion Scene to record as binary assets the gestures I needed, then during the loading proccess I just loaded all the frames using the LeapRecorder class and finally I added the gesture using this method (I think you knew this already):
https://github.com/Victorma/LeapTrainer.CSharp/blob/master/Assets/LeapClasifier/LeapTrainer.cs#L555

As you are trying to use the JSON feature you should know that the recorder implemented inside the LeapTrainer is untested since I didn't needed to use it. You're getting the error because the gesture is NOT being recorded at all.
So you have two options depending on the way your project is going to work:

• If you need to create gestures runtime, and suposing that the gesture training is working you should:
  • First: know that this is disabled and you'll need to reenable it and see if it is working inside: https://github.com/Victorma/LeapTrainer.CSharp/blob/master/Assets/LeapClasifier/LeapTrainer.cs#L244 (If you make it work and/or solve problems, a PR would be awesome :D)
  • Second: your code should look like this:

    LeapTrainer trainer;
    bool isTraining = false;

    void Start()
    {
        if (trainer)
        {
            // These are DELEGATES so one adition is enought to receive callbacks
            trainer.OnGestureCreated += Trainer_OnGestureCreated;
            trainer.OnGestureRecognized += Trainer_OnGestureRecognized;
            trainer.OnGestureUnknown += Trainer_OnGestureUnknown;
            trainer.OnTrainingCountdown += Trainer_OnTrainingCountdown;
            trainer.OnTrainingStarted += Trainer_OnTrainingStarted;
            trainer.OnTrainingComplete += Trainer_OnTrainingComplete;

            // This will only be needed in case you change the parameter inside LeapTrainer
            // because youll be able to track when a training gesture is recorded and notify the
            // repetitions left. 
            //  int trainingGestures = 1;   // The number of gestures samples that collected during training
            trainer.OnTrainingGestureSaved += Trainer_OnTrainingGestureSaved;
        }
    }

    private void Trainer_OnGestureCreated(string name, bool trainingSkipped)
    {
        Debug.Log("Gesture " + name + " has been created");
    }

    private void Trainer_OnTrainingCountdown(int countdown)
    {
        Debug.Log("Training will start in " + countdown + " seconds");
    }

    private void Trainer_OnTrainingStarted(string name)
    {
        Debug.Log("Started training gesture " + name);
    }

    private void Trainer_OnTrainingComplete(string name, List<List<Point>> gestures, bool isPose)
    {
        Debug.Log("Finished training gesture " + name);
    }

    private void Trainer_OnGestureRecognized(string name, float value, Dictionary<string, float> allHits)
    {
        Debug.Log("Gesture " + name + " recognized");
    }

    private void Trainer_OnGestureUnknown(Dictionary<string, float> allHits)
    {
        Debug.Log("No gesture recognized...");

        // However you can still work with the hits in case I detect some recording are underperforming...
        // For instance: if you have a recording that the best match is a value of 0.5 but the threshold is 0.7 you'll 
        //   never be notified, but you'll still receive the value here.
    }

    private void Trainer_OnTrainingGestureSaved(string name, List<List<Point>> gestures)
    {
        Debug.Log("Training gesture recorded... X more to go.");
    }

    void Update()
    {
        if (Input.GetButtonDown("Fire3") && trainer != null)
        {
            startTraining();
        }
    }

    void startTraining()
    {
        if (trainer == null || isTraining) return;

        isTraining = true;
        Debug.Log("Requesting new training...");

        trainer.Create("TestGest", false);
    }

• If you want to have all the possible gestures and use them runtime (as in my project) I recommend you to use the same process as I did. However this is not working right now for orion and I'll try to create a new Scene for recording this week as @mugabi0978 is requesting as well. The way it should be done should be like:

For the new recording scene:

• Create a new scene just like the old record an playback one.
• Using the LeapRecorder, add all the frames of the recording.
• Finally, call LeapRecorder SaveToNewFile.

For the runtime load:

• Using the recorded asset, call to LeapRecorder Load to restore all the frames.
• Use GetFrames to obtain all the frames from the asset.
• Use LeapTrainer.loadFromFrames to train the trainer.

I'll try to implement something for this as soon as I have some time but anyway if you're going for the first, let me know if you succeed since I don't have a LeapMotion at all.

Bests!

[bridgewaterrobbie]

@Victorma
Haven't finished implementing it, but I think I got it now. Thank you for the information, I think I should have our preliminary project up and running in the next few days now!

Thank you for your help again here. I feel like this could be seriously useful for AR implementations like ours.

[bridgewaterrobbie]

@Victorma
So I learned something "Fun" with Orion. It does not have a Serialize method for frames. So LeapRecorder does not work natively...
See https://developer.leapmotion.com/documentation/csharp/devguide/Leap_Serialization.html for offical note of that.
I'm going to try and make my own like it says to. I have never done something like this before though... it will be interesting.

I wanted to let you know since you said you are trying to make a new recording scene, this will be an issue to overcome with any attempt. If I succeed, I will share it for sure.

EDIT

that wasnt so bad, and I learned new things. I thought I would have to make it another byte file from scratch which is why I worried. JsonUtility took care of it just fine, now I have a working Orion LeapRecorder.

Ill post it tomorrow with a recorder scene once the code is cleaned up. In the future, it probably wouldnt be too hard to just merge the LeapRecorder method into the main LeapClassifier's create method, rather than using the exact method roboleary used. .

[Victorma]

@bridgewaterrobbie that is great! Even if its not byte json solution is better for most of the cases because it has better portabillity. Whenever you have the updated version of LeapRecorder I'll be very pleased if you share it with me in a PR here 😄

Thanks for your effort!

[bridgewaterrobbie]

@Victorma
Ok just did the PR. I included two scenes, based off of Unity's included scenes: VR and desktop usage.

I intend to integrate leaprecorder into LeapTrainer later, to make it a bit more cohesive. That might take a while though. Is there any reason that you know of that the solution from roboleary (when working) is superior to the method LeapRecorder uses of just capturing all the frames for later analysis? Otherwise I think Ill strip his unimplemented stuff out, and replace it with LeapRecorder's functionality.

[mugabi0978]

if successful please let me know because, since am new to unity 3D-- am
finding it a bit difficult to cop up;

I appreciate all your works;

On Thu, Oct 20, 2016 at 4:52 AM, bridgewaterrobbie <notifications@github.com

wrote:

@Victorma https://github.com/Victorma
Ok just did the PR. I included two scenes, based off of Unity's included
scenes: VR and desktop usage.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKaawcwONnDsWkZdaZPjWZF6DtrenyKtks5q1sllgaJpZM4KYGeE
.

[bridgewaterrobbie]

@mugabi0978
Here is a more complete tutorial on how to get the current version up and running. I am mostly posting what Victorma posted here, just a bit more verbose.

First download LeapClassifier with the new LeapRecorder from github (My link is here: https://github.com/bridgewaterrobbie/LeapTrainer.CSharp , Victorma will probably pull it into the main one later)

Use the scenes I made to record a series of actions to a file for later. You can use the playback mode (Z by default) to check your recording if you want. Note that the playback here may be a bit faster, but it wont matter here.

Next open your own scene up. You will need to instantiate an instance of LeapTrainer as well as LeapRecorder. Use LeapRecorder to load in the file you made earlier with Load. This will load all of the frames to your instance of LeapRecorder.

Now call LeapTrainer.loadFromFrames, passing in LeapRecorder.GetFrames(Returns a list of the Frame objects). This will create the gesture for the LeapTrainer to recognize.

Finally, use delegate methods like Victorma showed above to act on a gesture being recognized. I haven't done that last step yet, but I imagine switch statements would be your friend here.

[mugabi0978]

@bridgewaterrobbie

Whenever i run the scene i get this error below:-

The referenced script on this Behaviour (Game Object 'LeapHandController')
is missing!

If possible, please can you make a video to illustrate your statements
please; which i can later view on youtube with the provided link; please do.

i think that could be a easier way to learn what you are saying. Thank You.

On Thu, Oct 20, 2016 at 10:44 AM, bridgewaterrobbie <
notifications@github.com> wrote:

@mugabi0978 https://github.com/mugabi0978
Here is a more complete tutorial on how to get the current version up and
running. I am mostly posting what Victorma posted here, just a bit more
verbose.

First download LeapClassifier with the new LeapRecorder from github (My
link is here: https://github.com/bridgewaterrobbie/LeapTrainer.CSharp ,
Victorma will probably pull it into the main one later)

Use the scenes I made to record a series of actions to a file for later.
You can use the playback mode (Z by default) to check your recording if you
want. Note that the playback here may be a bit faster, but it wont matter
here.

Next open your own scene up. You will need to instantiate an instance of
LeapTrainer as well as LeapRecorder. Use LeapRecorder to load in the file
you made earlier with Load. This will load all of the frames to your
instance of LeapRecorder.

Now call LeapTrainer.loadFromFrames, passing in
LeapRecorder.GetFrames(Returns a list of the Frame objects). This will
create the gesture for the LeapTrainer to recognize.

Finally, use delegate methods like Victorma showed above to act on a
gesture being recognized. I haven't done that last step yet, but I imagine
switch statements would be your friend here.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKaawbTNBWo0quiU4aSVNIFiiAf43u0Pks5q1xvPgaJpZM4KYGeE
.

[bridgewaterrobbie]

@mugabi0978
Not sure how that happened, but the fix is very easy. Drag the "LeapHandRecorder" script from Assets/ onto the missing script, and you should be good to go.

EDIT Im going ahead and pushing the small fix needed too.

[mugabi0978]

Ok i downloaded last update and no errors...thanks for the update....

On Thu, Oct 20, 2016 at 11:48 AM, powell lorenz powellorenz@gmail.com
wrote:

i dont get you well;- I drag it onto the missing script???

What do you mean by this?

I tried dragging it to LeapMotion/Scripts/ but error persists. what am i
doing wrong please?

On Thu, Oct 20, 2016 at 11:30 AM, bridgewaterrobbie <
notifications@github.com> wrote:

@mugabi0978 https://github.com/mugabi0978
Not sure how that happened, but the fix is very easy. Drag the
"LeapHandRecorder" script from Assets/ onto the missing script, and you
should be good to go.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKaawZSPaLZJaDFdVRgKOeA6gtiBwT7eks5q1yadgaJpZM4KYGeE
.

[Victorma]

@bridgewaterrobbie related to: #2 (comment)
As far as I navigated into the code, the @roboleary solution is intended to finish the recording as soon as the user stopped generating recordable flames, which happens mostly when the user stops moving. Also, @roboleary solution has the special case of "poses" that handle the case of not moving gestures.
Nevertheless, as I tried, if you have all the frames you can perform more rich movements that can include little pauses and you'll have no problems with poses (as you can see in my project I had no problems detecting the letters).
To be honest, my guesses are into the idea that @roboleary solution is trying to implement a simple system to stop recording gestures and avoid recording the last frames that are generated when you try to press a key to stop recording. (In fact, in my game I remove like 100 frames from the records to remove that part)
If @roboleary wants to share his experience with us he might be able to solve the clue 😄

Now I'm heading to review the PR and merge it ^^

Also, thanks for your cooperation with @mugabi0978

@mugabi0978, I'll try to update SignAdventure with the new updates here and I'll let you know.

[mugabi0978]

Wow, it would be my pressure if you do @Victorma
https://github.com/Victorma

Am really awaiting for that anytime from now;-

And also please update the readme...to match people like us. (unity 3d
beggineers)

@bridgewaterrobbie https://github.com/bridgewaterrobbie and @Victorma
https://github.com/Victorma , i really appreciate all your efforts on
this; i wish one day we can have a google hangout....

On Thu, Oct 20, 2016 at 12:25 PM, Victorma Perez Colado <
notifications@github.com> wrote:

@bridgewaterrobbie https://github.com/bridgewaterrobbie related to: #2
(comment)
#2 (comment)
As far as I navigated into the code, the @roboleary
https://github.com/roboleary solution is intended to finish the
recording as soon as the user stopped generating recordable flames, which
happens mostly when the user stops moving. Also, @roboleary
https://github.com/roboleary solution has the special case of "poses"
that handle the case of not moving gestures.
Nevertheless, as I tried, if you have all the frames you can perform more
rich movements that can include little pauses and you'll have no problems
with poses (as you can see in my project I had no problems detecting the
letters).
To be honest, my guesses are into the idea that @roboleary
https://github.com/roboleary solution is trying to implement a simple
system to stop recording gestures and avoid recording the last frames that
are generated when you try to press a key to stop recording. (In fact, in
my game I remove like 100 frames from the records to remove that part)
If @roboleary https://github.com/roboleary wants to share his
experience with us he might be able to solve the clue 😄

Now I'm heading to review the PR and merge it ^^

Also, thanks for your cooperation with @mugabi0978
https://github.com/mugabi0978

@mugabi0978 https://github.com/mugabi0978, I'll try to update
SignAdventure with the new updates here and I'll let you know.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKaawURti3S50sI1LG_mJK8twkbP4aEbks5q1zNsgaJpZM4KYGeE
.

[bridgewaterrobbie]

@Victorma

Ok, I think I am one line of code from being done, but this is not quite working yet. I am now getting an error in GeometricalMatcher, at line ~80 (Ive added some debug statements)

The line is
if (!matched[j]) {
The full error log is:

Argument is out of range.
Parameter name: index : at System.Collections.Generic.List1[System.Boolean].get_Item (Int32 index) [0x0000c] in /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633 at GeometricalMatcher.gestureDistance (System.Collections.Generic.List1 gesture1, System.Collections.Generic.List1 gesture2, Int32 start) [0x00033] in C:\Users\Robbie\Desktop\LeapTrainer.CSharp\Assets\LeapClasifier\GeometricalMatcher.cs:81 at GeometricalMatcher.match (System.Collections.Generic.List1 gesture, System.Collections.Generic.List1 trainingGesture) [0x0002a] in C:\Users\Robbie\Desktop\LeapTrainer.CSharp\Assets\LeapClasifier\GeometricalMatcher.cs:51 at LeapTrainer.correlate (System.String gestureName, System.Collections.Generic.List1 trainingGestures, System.Collections.Generic.List1 gesture) [0x0002b] in C:\Users\Robbie\Desktop\LeapTrainer.CSharp\Assets\LeapClasifier\LeapTrainer.cs:857 at LeapTrainer.recognize (System.Collections.Generic.List1 gesture, Int32 frameCount) [0x00064] in C:\Users\Robbie\Desktop\LeapTrainer.CSharp\Assets\LeapClasifier\LeapTrainer.cs:790
at LeapTrainer.onFrame (Leap.Frame frame) [0x00123] in C:\Users\Robbie\Desktop\LeapTrainer.CSharp\Assets\LeapClasifier\LeapTrainer.cs:256
UnityEngine.Debug:Log(Object)
LeapTrainer:onFrame(Frame) (at Assets/LeapClasifier/LeapTrainer.cs:260)
TrainerListener:OnFrame(Object, FrameEventArgs) (at Assets/LeapClasifier/LeapTrainer.cs:151)
Leap.CSharpExtensions:DispatchOnContext(EventHandler`1, Object, SynchronizationContext, FrameEventArgs)
LeapInternal.Connection:handleTrackingMessage(LEAP_TRACKING_EVENT&)
LeapInternal.Connection:processMessages()

My best guess: Something about how Orion works is formatting the data differently enough to anger GeoMatcher. This difference is not signifigant enough to make stored V2 frames complain (As evidenced by your SignAdventure still recognizing frames from Orion and matching them to your V2 Leap Recorder)

I am trying to figure this out now, but I figured I would post here now too.

It seems that the Recognize function always either returns 0, or has this error.

Thank you again.

[Victorma]

@bridgewaterrobbie I don't know what could it be... I wasn't able to make Orion SDK work in Sign Adventure because of the deserialization issue. Nevertheless, this wednesday I'm having the Leap Motion so I'll be able to test this and fix it.
I am sorry for the delays you're probably having...

[bridgewaterrobbie]

@Victorma no worries, not like you are permenently responsible for your code on github. Any help is appreciated, but we will take care e of anything you don't want to (though we wouldn't have realized the create method was not working with your help admittedly)

I'll make a pull request for anything we discover. The issue is either in how the point array is getting passed to match, or something about how Orion handles frames is exposing a bug in geometric matched. I'm going to reference the original $p to see if that can fix things