In my AWE preview article, I told you Prehension AI is a startup creating an SDK to help developers implement hand gestures in their MR/VR applications. At AWE US, I was able to get to it quickly, and these are my impressions.
AI presence
Prehension is a startup that provides the Unity SDK to easily detect hand gestures. The gestures the company focuses on are animated gestures, so not simple hands (like a fist) but gestures made with one hand, like this (very Italian) one.
The problems with animated gestures are mainly two:
- You need to find a gesture that is not a simple hand gesture, but that Includes movement of hand and fingers over time
- Each person will perform the gesture in a different wayAnd finding a gesture that works for every user is incredibly difficult.
Purpose of Prehension To solve these two problemss provides a ready-to-use plugin that detects any gesture you want.
Hands-on: recording gesture
Alex from Prehension showed me the process of registering a new gesture in AWE. Basically you Go to a specific unity view provided by the plugin, Put on your Quest+ Link and then press play. At runtime, select which gesture you’re recording (gestures are associated with name), then You hit record and make a gesture with your hand (Note that you are using Quest+Link, so the hands will be detected by the Quest runtime). To provide more samples, you can perform the gesture several times: Perfectly, You need to ask more people to do the gesture You will have more variation in the training data about how the gesture is executed.
After you’ve recorded all the samples for all the gestures you want, you stop Play mode To start training the machine learning model, press the specific editor button of the plugin. Training happens in the cloud, then The model is downloaded back to your project, So it can work even without internet connection. Training time is very short: in AWE, three detected gestures and 5-6 samples for each gesture, It was like 1 minute.
Note that You can update the model later. In AWE, Prehension was able to add a specific gesture sample on top of all the samples it had already recorded. But for each new model, You need to refresh the entire model again; Since new models may change all classification weights, you cannot update only a part of it.
Hands-on: Gesture Detection
After the ML model is ready, you can create a Unity view there A specific prehension script will tell you when a gesture is detected. Note that the computer only detects a gesture if your movement truly resembles a gesture. What I mean is that if you record three gestures and you make a movement that doesn’t match any of the three, the classifier will return that no gesture was detected; It does not mandate recognition for any of the three classes.
I was able to test a simple scenario there A cube performed animations based on my hand movements (I think this is the perfect demo for me: I love cubes, and as an Italian, I love hand gestures). It worked well: Once I understood how to interact with the demo, By moving my hands I was able to trigger the animation of the cube I wanted.
It’s interesting to note that the system doesn’t work properly for the only animation I recorded my own sample, but For other hand gestures. That means The classifier was already “general” enough Also detect strange movements of a person that are not present in the training data. This says a lot about the quality of this SDK.
Price and release date
Prehension AI’s SDK is currently in private beta: You can contact the company on this website to test the SDK and start cooperation with them. When it’s publicly available, you can use it Payment of monthly subscription fee.
Final considerations
I had a few minutes to test prehension, but I came away with good impressions of it. SDK was there Very easy to useand detecting gestures worked well, Even in chaotic situations like a small booth at AWE. Some things still feel “beta”.The recording scene is graphically very rough, but who cares if the product works?
I think the product is interesting because This saves a lot of time for developers who want to integrate hand gestures in their applications. Doing the same by hand would require a lot of time and effort (if you have ML knowledge), while Prehension offers it out of the box.
The biggest downside to its potential is the price. If it’s free I’ll use it myself tomorrow. But if the price is too high, no indie developers can afford it. Another big question Sustainability of the company: In a negative moment for VR, the company needs to understand whether there are enough VR developers who specifically require animated hand gestures for their applications to sustain their business.
I guess we’ll see. For now, we can say that this is an interesting XR SDK that solves a specific problem in our field. For others, time will tell.
Disclaimer: This blog contains advertisement and affiliate links. If you click on an affiliate link, I will be very happy because I will receive a small commission on your purchase. My boring full disclosure can be found here.
related to
#Prehension #easy #implement #gestures #games