Listen to this article

Fruit spoilage is a major source of lost revenue for grocers. | Image credit: Clem Onojeghuo from Pexels

Simbe Robotics was recently granted US Patent 11,200,537 covering the application of autonomous mobile robot and vision for tracking the freshness/ripeness of fruits and vegetables in a grocery display.

According to the USDA Economic Research Service (ERS), in 2010 a total of 133 billion pounds (31%) of the 430 billion pounds of available food supply at retail and consumer levels went uneaten, with an estimated retail value of $162 billion. Estimates of the average supermarket loss rate was 11.6% for 31 fresh vegetable types.

Grocers operate on thin margins, so every little bit of savings counts. This is potentially a huge application opportunity in a market where Simbe Robotics already has a presence. Simbe recently announced it is expanding the use of inventory robots at 100% of the Schnucks Grocery stores in the U.S.

The difficult part of this application is the machine vision and creating an algorithm that can determine when food is starting to go bad, and subsequently when it’s time to raise an alert to a fresh food clerk to remediate the situation. In the patent application, Simbe outlines the use of hyper-spectral imaging to determine the freshness of fruits and vegetables and the characterization of the quality.

The Simbe Tally robot is one of the market leaders in grocery store inventory management. Up until now, Tally could track the inventory of all of the stable food items and hard goods within a store. But fresh food inventory and spoilage has long been a difficult application to solve, so this hasn’t been a capability of the Tally robot. Until now.

The patent outlines the workflow to identify spoiled fruit with a camera on a mobile robot and alert a clerk. | Image credit: USPTO

With the patent in hand, Simbe can now begin the process of commercializing this idea. It is likely the company already has it working in a lab environment, but the real proof will come from validating the imaging and analysis with real-world environments of a grocery store with changing lighting conditions throughout the day.

We reached out to the company for comment, but at this time Simbe is remaining quiet on its commercialization plans.

This application has been a “holy grail” opportunity since the early days of in-store inventory management with mobile robots. The biggest hurdle has been solving the lighting, imaging and analysis of fresh food products in display cases. Imaging hard goods on standard grocery shelves was solved by a number of competitors in this space. However, all these competitors noted that imaging fresh fruits and vegetables remained a difficult problem to solve.

Simbe’s Tally robot moves autonomously through the store to check inventory levels. | Image credit: Simbe Robotics

First of all, imaging fruits and vegetables on display requires a different lens and depth-of-field setup from shelf imaging. It requires that the camera look down from a different vantage point than a camera looking at items on shelves. This is because fruits and vegetables are typically arranged at different angles as the items are stacked and presented to the buyer. Other problems include reflections from overhead lighting, used to illuminate the items for purchase, as well as inconsistent reflections from moisture on an item’s surface from frequent water spraying.

The patent is long and intricate at 32 pages. It covers the design of an application that assesses the state of the fruit, identifies fruit that is spoiling, and subsequently alerting a clerk to check and/or replace the spoiling fruit. The outcome of the new capabilities will be the ability to alert the fresh food clerk that items on display need to be removed from sale and replaced with fresher offerings.

The end game for Simbe is better utilization of the Simbe mobile robot in-store along with the capabilities to cover more SKUs in the store, but especially the support for one of the highest-labor intensive processes (managing the fresh food displays) in the store.