Carunda24 Parrot DroneIn 2016, we at Carunda24 decided to start building a wrist-mounted gesture recognition system based wearable sensors and research in soft body dynamics (SBD). With the emerging trends in augmented and virtual reality, one of the biggest challenges we’ll face in society and tech in the coming years will be how to interact with immersive technologies. In the past we’ve followed a pattern of establishing some input methods to a computer (historically with a keyboard and mouse), and then modifying our behavior to the machine interface. There has also been a desire for Natural User Interaction (NUI), and the development of technologies which allow us to interact with computers in the ways we naturally interact with people and the world.

NUI Drone Development

While researching soft body dynamics and gesture recognition for various applications, it seemed for us a natural and interesting challenge to start with the NUI control of a drone. As part of the development journey we thought of which connected devices we would like to interact with, and flying a drone was one of our first inspirations. We were looking for something we could play with, and a small Parrot drone already almost feels like a pet bird or dog rather than a flying computer. So we set out to connect our wearable sensor to the Parrot Mini Cargo drone.

We chose the Parrot Mini drone since it is relatively small (can be flown indoors) and also includes an SDK and a Bluetooth Low Energy receiver, which made it ideal to fly in combination with our Arduino hardware prototype via a JavaScript program.

Carunda24 Wearable SensorWearable Sensor Technology

At the core of the smart strap is a Soft Condensed Matter Sensor (SCMS), essentially a conductive polymer composite material developed at Empa in Switzerland. The SCMS allows us to measure the strain on the skin continuously, so we can actually detect the moment of tendons as the hand creates different gestures. It’s based on a high tech combination of nano-textured conductive carbon particles mixed with a highly elastic thermoplastic elastomer. Research has shown the SCMS can measure strain related to hand gestures and is even sensitive enough to measure the pulse-wave of blood (Piezoresistive Soft Condensed Matter Sensor for Body-Mounted Vital Function Applications).

Carunda24 smartstrap 15 2Hardware/Software Prototyping

For our first flight we wanted to be flexible and agile in development, so we chose to prototype our hardware solution with a Adafruit Feather HUZZAH and a software program written in JavaScript. This enabled us to prototype the drone control as well as form a development platform to future applications such as NUI VR/AR interaction, remote robotics, animation interaction, etc. Our JavaScript program included a hand calibration and gesture recognition approach, and then we mapped those gestures to the necessary drone commands. These were then sent to the Parrot via a Bluetooth connection on a host computer.

Next Steps

For us the Parrot drone was just the beginning, in the future we will be further developing our gesture recognition smart strap technology and also integrating with different robotic and AR/VR systems.