Humans are very good at using tools in a roundabout way. Don’t have a spoon? You will use a pen to stir your coffee. Missing a bolt to hang a lamp? A rubber band will do the trick temporarily. Need a tray? A book or digital tablet will suffice. This capacity for improvisation is not born “ex nihilo”, it results from cognitive capacities which allow us to make links between objects, the tools at our disposal, their uses, etc.

Tool use in robots has so far been seen as an exploration and learning problem: a robot needs to discover how a tool can be used, either by trying various strategies , or by observing and imitating other humans or robots. In our study published in Nature Machine Intelligence, we showed that we could teach robots to think more creative way, “outside the box”, as the English speakers would say.

To enable robots to use tools intuitively like humans, we first looked at how we humans are able to do so.

Towards “tool cognition” in robots

A robot needs to pick up a bucket, but the path is blocked by some obstacles. This robot cannot move obstacles or jump over them. But he was able to search for — and find — a cleaning stick and use it as a tool to scoop up the bucket.

This simple use of a tool may seem obvious to a human, but it is a complex challenge for a robot.

Indeed, the use of tools first requires the robot to understand that it cannot perform a task without tools (1). He must then find an object in his environment that he can use as a tool to carry out the requested task (2). To do this, he must find out how to use this tool, that is to say, determine the actions to perform (3). And finally, of course, execute these actions… and therefore complete the task (4).

The second and third challenges are fundamental cognitive challenges, at which humans seem to be very good… and robots, much less so.

Our new algorithm for “tool cognition” in robots takes a big step forward in meeting these challenges.

How do humans recognize that an object is sufficient to serve as a tool for a given task?

In our work, we have categorized tools based on how we humans use them. This categorization shows that humans can in fact intuitively recognize only a very specific category of tools..

These “Category 1” tools are tools that allow you to perform tasks that are already in the human repertoire. For example, pliers help us grasp objects, but we can already do that with our fingers; a hammer helps us hit objects, which we can already do with our fist.

Using these “Category 1” tools requires the same action we would do without a tool: pliers need you to pinch, which is similar to what you would do with your fingers; a hammer requires you to make swinging striking motions, and you would do the same motion without a hammer.

Category 1 tools are probably the most common in human and animal life. The first tools used by humans, stones for hitting and breaking fruits and cutting them, were category 1 tools, because humans could perform these tasks, with similar actions, with their fist and their nails, respectively. And in fact, thetools used by animals can be classified in category 1.

We call “category 2” tools tools that allow us to carry out tasks that humans can carry out without tools but with very different gestures (like a car jack to lift a car).

Category 3 tools allow you to carry out tasks that are outside the human repertoire (such as a vacuum cleaner or a chainsaw).

These category 2 and 3 tools cannot be used intuitively by humans: you must read instructions, imitate another user, or explore the tool to discover how to use it.

The tools humans use intuitively

It is believed in psychology that one of the reasons humans are good at using tools is that using of tools leads to an “embodiment” of the tools, that is to say that the brain ends up considering the tools as an “extension” of our body.

In our work on the “embodiment” of the tool and the member a> by humans, we observed that the similarity of the characteristics of an object with the functional characteristics of the limb is important for its incarnation.

This observation, along with our characterization of tools, suggests that to identify Category 1 tools, humans may be using their limbs as a reference: a hammer is recognized as a tool for hitting objects because it closely resembles a fist and an arm, pliers are recognized as a tool for pinching because they resemble pure fingers, and similarly, a plate resembles a palm and a bowl resembles cupped hands. By identifying and comparing the posture of our limbs when we perform a task, we can identify tools that can be used for the task performed with the same posture.

Using similarity with its limbs is the key idea that is used to program our robot.

Teaching robots to recognize tools that resemble what they know

Our algorithm uses this idea to allow robots to recognize objects (even those they see for the first time) as tools for each task they have the ability to perform “without tools” (i.e. -say with their members, without additional object).

Once a robot has learned to perform a task with its limbs, our algorithm allows it to use its limbs as a reference to recognize tools that help it accomplish the same task. Additionally, by definition, because Category 1 tools require the same action as without tools, the robot can use the same skill (“controller”) to use the tool by simply increasing its body kinematics based on the new tool.

The robot does not need to have ever used a single tool or even observed the use of the tools by others.

In our proposal, we also provide a grip planning algorithm to grasp and use the identified tools. Unlike thealgorithms previously described in the literature scientific, ours is the first to allow robots to use tools without having to first learn with the same tools or with other tools.

However, at the moment our algorithm only allows robots to use category 1 tools, that is, tools for tasks that the robot is capable of performing without tools. For category 2 and 3 tools, learning or observation (followed by imitation) is necessary, like the previous tool use algorithms.

In addition, we can extend the algorithm to the use of tools similar to those already known to the robot: basically, we replace “similarity with a member” by “similarity with a known tool”, as suggested by thealgorithms previously described.

But as we have seen, Category 1 tools appear to be the most basic tools in humans and animals and so we believe our algorithm is a significant step toward tool cognition in robots.

Author Bio: Ganesh Gowrishankar is a Researcher at the Montpellier Computer Science, Robotics and Microelectronics Laboratory