Tactile Sensor Could Give Robots Human Touch

Imagine opening a door using only your sense of vision to guide you, without any sense of touch. Vision would help you guide your hand to the doorknob, but how do you know when your hand has made contact with the knob? How will you know which way to twist the doorknob, and whether to push or pull the door open? Trial and error is an inefficient way to handle this, especially considering how many doors one might encounter in a day. A sense of touch is needed to perform a greater range of tasks more effectively and efficiently. “At some point in time robots will need this kind of capability,” notes Youssef Benmokhtar, CEO of tactile sensor company Gelsight.

To spin off of his work at MIT, vision science professor Edward Adelson co-founded Gelsight in 2011 together with his postdoctorial student Kimo Johnson, and local technologist and entrepreneur Janos Rohaly. Benmokhtar says Adelson was inspired to investigate tactile sensing by watching his infant daughter use her sense of touch to explore the world around her. “That led Adelson to ask the question, ‘Is there a sensor that mimics human touch and feel?’ And the answer was no, there is no such thing,” Benmokhtar explains. “He’s one the very few people in the world that can take that answer and say, ‘Well, okay, if it doesn't exist, I'm just going to invent it.’”

The result, potentially, will be a means for a robot to locate parts that is intuitive to us in one sense, yet also counterintuitive relative to ways we are accustomed to seeing robots move. Rather than moving to a part based on coordinate programming or based on vision measurement, the robot will be able to “feel” its way to grasping the part accurately.

Except, as Benmokhtar describes, this artificial sense of “touch” also relies on vision.

Making Touch Visible

Gelsight’s solution features a tactile sensor made of an elastomeric material that’s designed to conform on contact with an object like human skin. A camera located behind the sensor captures multiple images of what the sensor is detecting. An algorithm stitches them together to create a real-time 3D map of the object’s surface.  “It’s basically like having a camera inside your finger,” Benmokhtar explains. “We make touch visible, so you can actually see details of textures.” Gelsight’s sensor takes human touch a step further into submicron precision, thanks to its use of photometric stereo, a technique where multiple photos of the same object under different lighting conditions are stitched together. This can provide pixel-level information on the depth of the object’s surface.

The company’s initial mission was to give robots and other machines the same tactile senses humans have. But as the founders started publishing papers about their technology, they started to garner interest from the industrial world, including interest related to quality rather than part handling. An engineer from an aerospace company brought a bag of parts to Gelsight’s offices and asked if the company’s sensor could detect scratches, dents, knicks and other defects on its parts. Gelsight’s sensor was able to pick up defects the engineer hadn’t picked up with any other inspection technology. “That's how the company started its journey into surface metrology,” Benmokhtar says.

The company has released three products for the surface metrology market: the Gelsight Mobile, a portable tactile sensor, the Gelsight Max, a higher-resolution tactile sensor, and the Gelsight Mobile Shock Protected Edition, which has a more ruggedized design. These products are designed to be used in metrology labs, on shop floors or in the field to find and measure part characteristics such as defects, surface roughness, and part features in a range of materials, including metal, glass and carbon fiber.

Applications for Robot Touch

Gelsight’s technology finds use in metrology, the company is also still pursuing its initial vision of giving robots and other machines a human-like sense of touch. Much of this work is advancing through collaborations with other companies. It recently worked with Nvidia to launch support for Gelsight’s tactile sensor in Nvidia’s simulation tool. Gelsight also partnered with Meta to distribute and sell a version of its sensor called the Digit, which is a more basic version of its tactile sensor that’s designed to be mounted on robotic hands and other grippers. The company also offers the Gelsight Mini, which is a smaller version of its sensor with higher resolution software than the Digit sensor. Benmokhtar says this sensor hasn’t yet been deployed in an industrial application, but it’s being used in corporate and academic research.

A sense of touch could enable robots to handle more tasks. “It's actually quite easy for a robot to hold a pallet and move it around,” Benmokhtar notes, but robots often struggle with complex assembly tasks involving small parts. “And this is where I think, just like humans, you're going to need a combination of senses to be able to do those kinds of complex tasks.” Tactile sense could even unlock general-purpose robots that handle a variety of tasks in homes, healthcare applications and other environments with a lot of variation. “Let’s say you have an unpredictable, unconstrained environment,” he notes. “If you think about humans or any animal, when we interact with the world, it's a multi-sensory experience. We think tactile sensing will be absolutely critical in enabling these applications.”