Screen Shot 2017 01 14 at 19.28.20

Last time we introduced our smart strap prototype and wearable strain sensor (Flying a Parrot Drone With a Wearable Sensor), where we sent flight commands to a Parrot minidrone through natural hand motions. This was done by integrating a wearable strain sensor into a watchband and then connecting it to a hardware prototyping platform. In this update we’d like to share more information about the hardware and software prototyping approach we took and how we're building towards an API for wearable sensor development using body-mounted sensors.

 

Hardware

For the prototype board, we decided to go with easily available and flexible components, to ensure that anyone could repeat our findings and improve on them without costing too much. The microcontroller of choice was the ESP2866 from Espressif, which is compatible with the Arduino IDE and has Wifi integrated. The Huzzah Feather Board from Adafruit integrates it into a nice package, adding a LiPo Charger and Port and and USB-Bridge for easy programming. It’s a powerful board but also small enough to be mounted on the arm for wearables prototyping. Since the Feather Board only has analog input channel to the ADC (Analog-to-Digital), we decided to add an external ADC. We went with the ADS1015, a 12-bit ADC with adjustable gain that can be accessed via I2C, making it easy to integrate with various microcontrollers. This enabled us to add a second wearable sensor (integrated in the watchband) and to increase the sensing resolution from the 10-bit ADC included on the Feather.

Having an adjustable gain means to keep the measured voltage between 0 and 1 Volt. A voltage divider (https://learn.sparkfun.com/tutorials/voltage-dividers) circuit was used to bring the input voltage of 3.3v down to around 0.5v. The two main components and two voltage dividers (one for each sensor) were then placed on a breadboard.

Drone Demo Prototype 001 bb

 Software

For software, we decided to go with a split model: have the reading of the ADC values done on the ESP2866 (coded via the Arduino IDE) and the rest on the host-pc, using JavaScript (node.js). This gave us flexibility in development and allows future expansion in the node.js ecosystem. JavaScript is already one of the most used programming languages and node.js has over 350000 installable packages. This made it very easy to add in the Parrot minidrone control using one of the available libraires (such as parrot-rolling-spider). 

 

Microcontroller Program

The microcontroller program was developed in the Arduino IDE. It runs in a loop and does the following:   

- gets a timestamp

- reads the ADC value of channel 0

- gets another timestamp

- reads the ADC values of channel 1

It then combines timestamps and adc values to a string and outputs that to the serial port via serial.writeln

 

#include <Wire.h>
#include <Adafruit_ADS1015.h>
// Adafruit_ADS1115 ads; /* Use this for the 16-bit version */
Adafruit_ADS1015 ads; /* Use thi for the 12-bit version */
 
void setup(void)
{
Serial.begin(250000);
ads.setGain(GAIN_TWO); // 2x gain +/- 2.048V 1 bit = 1mV 0.0625mV
ads.begin();
}
 
void loop(void)
{
int16_t adc0, adc1, adc2, adc3;
String o0,o1,o3;
int16_t ts;
 
// get timestamp
String out2 = String(millis());
// read adc value0
adc0 = ads.readADC_SingleEnded(0);
// get timestamp
String out3 = String(millis());
// read adc value1
adc1 = ads.readADC_SingleEnded(1);
 
// convert adc
String out0 = String(adc0);
String out1 = String(adc1);
 
// make sure that values below 1000 have 4 digits by adding a leading 0
if (out0.length()==3) {
out0="0"+out0;
};
 
// make sure that values below 1000 have 4 digits by adding a leading 0
if (out1.length()==3) {
out1="0"+out1;
};
 
// construct output, consisting of timestamp & adc value
o3=out2+";"+out0+";"+out3+";"+out1;
// send output to serial
Serial.println(o3);
 
delay(90);
}

Node.js Program

The serialport npm module allows for access of the serialport from node.js. That way, data written to the serialport on the microcontroller can be easily be accessed by a javascript program.

var parsers = SerialPort.parsers; 

var serialport = new SerialPort("/dev/tty.SLAB_USBtoUART",{
baudrate: 250000,
 parser: parsers.readline("\n")
});
serialport.on('open',function(){
serialport.on('data', function(data){
var a=data.split(";");

 

The serialport module works with parsers to recognize the pattern of the incoming data. In our case, we're using writeln on the sending side, which just adds a return character after each data "packet.". On the JavaScript side, the serialport ondata handler i called each time a new packet arrives. This packet is simply split at the ; character to access the sensor values and the timestamps for reference.

 

Sensor Data Filtering and Scaling

First, the sensor values are filtered with a Kalman filter. Since the sensor values are inherently very dynamic,the use of a Kalman filter makes them more stable. After filtering, the sensor values are scaled between 0 and 100, based on the minimum and maximum values measured during calibration. Lastly, the sensor values are compared to pre-existing thresholds, which were determined while wearing the smartstrap and noting down the scaled values for the various gestures, and the recognized gesture is stored. 

 

 // implement Kalman filtering of raw ADC values
var fval0 = vals0.map(function(v) {
return kalmanFilter0.filter(v);
});
 
var fval1 = vals1.map(function(v) {
return kalmanFilter1.filter(v);
});
 
//console.log("raw data",this.value, this.raw);
console.log("raw0",val0);
console.log("filtered raw0",fval0[0]);
console.log("max0",max0,"min0",min0);
console.log("raw1",val1);
console.log("filtered raw1",fval1[0]);
console.log("max1",max1,"min1",min1);
 
//scaled0=scaleBetween(val0,0,100,min0,max0);
scaled0=scaleBetween(fval0,0,100,min0,max0);
console.log("scaled0",scaled0);
//scaled1=scaleBetween(val1,0,100,min1,max1);
scaled1=scaleBetween(fval1,0,100,min1,max1);
console.log("scaled1",scaled1);

Next Steps

The Parrot mindrone demo showed us that we could easily implement gesture recognition using the wearable sensor. With the flexibility of node.js and the Arduino hardware, we’re now expanding into different use cases and thinking up new ways to integrate natural user interface (NUI) and soft body dynamics (SBD) into applications. As we develop the platform further we’ll expand our gesture recognition API so that it can flexibly integrate with other programs and hardware platforms including Drone Control, Robot Arm Control or User Interface Control.

Do you have an idea for wearable sensors and gesture recognition? Please get in touch.