BACKGROUND

I. Technical Field

The present disclosure relates generally to systems, methods, and devices for identifying products in retail stores, and more specifically to systems, methods, and devices for capturing, collecting, and automatically analyzing images of products displayed in retail stores for purposes of providing one or more functions associated with the identified products.

II. Background Information

Shopping in stores is a prevalent part of modern daily life. Store owners (also known as “retailers”) stock a wide variety of products on store shelves and add associated labels and promotions to the store shelves. Typically, retailers have a set of processes and instructions for organizing products on the store shelves. The source of some of these instructions may include contractual obligations and other preferences related to the retailer methodology for placement of products on the store shelves. Nowadays, many retailers and suppliers send people to stores to personally monitor compliance with the desired product placement. Such a monitoring technique, however, may be inefficient and may result in nonuniform compliance among retailers relative to various product-related guidelines. This technique may also result in significant gaps in compliance, as it does not allow for continuous monitoring of dynamically changing product displays. To increase productivity, among other potential benefits, there is a technological need to provide a dynamic solution that will automatically monitor retail spaces. Such a solution, for example and among other features, may automatically determine whether a disparity exists between a desired product placement and an actual product placement.

The disclosed devices and methods are directed to providing new ways for monitoring retail establishments using image processing and supporting sensors.

SUMMARY

Embodiments consistent with the present disclosure provide systems, methods, and devices for capturing, collecting, and analyzing images of products displayed in retail stores. For example, consistent with the disclosed embodiments, an example system may receive an image depicting a store shelf having products displayed thereon, identify the products on the store shelf, and trigger an alert when disparity exists between the desired product placement and the actual product placement.

Another aspect of the present disclosure is directed to a computer program product for identifying products and monitoring planogram compliance using analysis of image data embodied in a non-transitory computer-readable medium and executable by at least one processor, the computer program product including instructions for causing the at least one processor to execute the method described above.

In an embodiment, a non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for planning deployment of image sensors. The method may include determining a location of a store shelf within a retail store and obtaining a first coverage parameter corresponding to a first product type and a second coverage parameter corresponding to a second product type. The method may also include accessing a database to determine a first height of products of the first product type and a second height of products of the second product type. The method may further include determining a position for placing a camera configured to capture images of at least a portion of the store shelf by analyzing the location of the store shelf, the first coverage parameter, the second coverage parameter, the first height, and the second height. The method may also include providing, to a user interface of a user device, information relating to the determined position of the camera.

In an embodiment, a system for planning deployment of image sensors may include at least one processor programmed to determine a location of a store shelf within a retail store. The at least one processor may also programmed to obtain a first coverage parameter corresponding to a first product type and a second coverage parameter corresponding to a second product type. The at least one processor may further programmed to access a database to determine a first height of products of the first product type and a second height of products of the second product type. The at least one processor may also programmed to determine a position for placing a camera configured to capture images of at least a portion of the store shelf by analyzing the location of the store shelf, the first coverage parameter, the second coverage parameter, the first height, and the second height. The at least one processor may further programmed to provide, to a user interface of a user device, information relating to the determined position of the camera.

In an embodiment, a method for planning deployment of image sensors may include determining a location of a store shelf within a retail store. The method may also include obtaining a first coverage parameter corresponding to a first product type and a second coverage parameter corresponding to a second product type. The method may further include accessing a database to determine a first height of products of the first product type and a second height of products of the second product type. The method may also include determining a position for placing a camera configured to capture images of at least a portion of the store shelf by analyzing the location of the store shelf, the first coverage parameter, the second coverage parameter, the first height, and the second height. The method may further include providing, to a user interface of a user device, information relating to the determined position of the camera.

In an embodiment, a non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for routing a cleaning robot. The method may include receiving an indication of at least one store shelf of a retail store. The method may also include causing a first adjustment to a cleaning robot route of the cleaning robot within the retail store based on at least one location within the retail store corresponding to the at least one store shelf. The first adjustment may be configured to enable an image sensor associated with the cleaning robot to capture one or more images of at least one portion of the at least one store shelf. The method may further include receiving a first image acquired by the image sensor associated with the cleaning robot. The first image may include a representation of the at least one portion of the at least one store shelf. The method may also include analyzing the first image to determine a need for a second image of the at least one portion of the at least one store shelf. The method may further include causing a second adjustment to the cleaning robot route within the retail store, in response to the determined need. The second adjustment may be configured to enable the image sensor to capture the second image of the at least one portion of the at least one store shelf.

In an embodiment, a system for planning deployment of image sensors may include at least one processor programmed to receive an indication of at least one store shelf of a retail. The at least one processor may also programmed to cause a first adjustment to a cleaning robot route of the cleaning robot within the retail store based on at least one location within the retail store corresponding to the at least one store shelf. The first adjustment may be configured to enable an image sensor associated with the cleaning robot to capture one or more images of at least one portion of the at least one store shelf. The at least one processor may further programmed to receive a first image acquired by the image sensor associated with the cleaning robot. The first image may include a representation of the at least one portion of the at least one store shelf. The at least one processor may also programmed to analyze the first image to determine a need for a second image of the at least one portion of the at least one store shelf. The at least one processor may further programmed to cause a second adjustment to the cleaning robot route within the retail store, in response to the determined need. The second adjustment may be configured to enable the image sensor to capture the second image of the at least one portion of the at least one store shelf.

In an embodiment, a method for routing a cleaning robot may include receiving an indication of at least one store shelf of a retail store. The method may also include causing a first adjustment to a cleaning robot route of the cleaning robot within the retail store based on at least one location within the retail store corresponding to the at least one store shelf. The first adjustment may be configured to enable an image sensor associated with the cleaning robot to capture one or more images of at least one portion of the at least one store shelf. The method may further include receiving a first image acquired by the image sensor associated with the cleaning robot. The first image may include a representation of the at least one portion of the at least one store shelf. The method may also include analyzing the first image to determine a need for a second image of the at least one portion of the at least one store shelf. The method may further include causing a second adjustment to the cleaning robot route within the retail store, in response to the determined need. The second adjustment may be configured to enable the image sensor to capture the second image of the at least one portion of the at least one store shelf.

In an embodiment, a non-transitory computer-readable medium includes instructions that when executed by a processor cause the processor to perform a method for reacting to changes of items hanging on peg-hooks connected to pegboards. The method may include receiving a first indication indicative of a change of items hanging on a first peg-hook connected to a pegboard; receiving a second indication indicative of a change of items hanging on a second peg-hook connected to the pegboard; in response to the first indication, causing an action related to a first product type; and in response to the second indication, causing an action related to a second product type, the second product type differing from the first product type.

In an embodiment, a method for reacting to changes of items hanging on peg-hooks connected to pegboards may include receiving a first indication indicative of a change of items hanging on a first peg-hook connected to a pegboard; receiving a second indication indicative of a change of items hanging on a second peg-hook connected to the pegboard; in response to the first indication, causing an action related to a first product type; and in response to the second indication, causing an action related to a second product type, the second product type differs from the first product type.

In an embodiment, a system for reacting to changes of items hanging on peg-hooks connected to pegboards may include at least one processor programmed to receive a first indication indicative of a change of items hanging on a first peg-hook connected to a pegboard; receive a second indication indicative of a change of items hanging on a second peg-hook connected to the pegboard; in response to the first indication, cause an action related to a first product type; and in response to the second indication, cause an action related to a second product type, the second product type differs from the first product type.

In an embodiment, a non-transitory computer-readable medium includes instructions that when executed by a processor cause the processor to perform a method for identifying products from on-shelf sensor data and image data. The method may include receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf; receiving an image of the at least part of the retail shelf and at least one of the one or more products; and analyzing the captured data and the image to determine a product type of the one or more products.

In an embodiment, a method for identifying products from on-shelf sensor data and image data may include receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf; receiving an image of the at least pan of the retail shelf and at least one of the one or more products; and analyzing the captured data and the image to determine a product type of the one or more products.

In an embodiment, a system for identifying products from on-shelf sensor data and image data may include at least one processor programmed to receive data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf, receive an image of the at least part of the retail shelf and at least one of the one or more products; and analyze the captured data and the image to determine a product type of the one or more products.

In an embodiment, a method for providing visual navigation assistance in retail stores may comprise receiving a first indoor location of a user within a retail store; receiving a target destination within the retail store; providing first navigation data to the user through a first visual interface; after providing the first navigation data, receiving a second indoor location of the user within the retail store; determining that the second indoor location is within a selected area around the target destination, the selected area not including the first indoor location; and in response to the determination that the second indoor location is within the selected area around the target destination, providing second navigation data to the user through a second visual interface, the second visual interface differing from the first visual interface.

In an embodiment, a computer program product may identify products and monitoring planogram compliance using analysis of image data embodied in a non-transitory computer-readable medium and executable by at least one processor. The computer program product may include instructions for causing the at least one processor to execute the method described above.

In an embodiment, a method for making gradual adjustments to planograms is disclosed. The method may comprise receiving a first image of at least part of a shelf; analyzing the first image to determine a first placement of products on the at least part of the shelf; based on the determined first placement of products, determining a planned first adjustment to the determined first placement of products on the at least part of the shelf; and based on the planned first adjustment to the determined first placement of products, providing first information, the first information being configured to cause the planned first adjustment to the determined first placement of products. The method may further comprise receiving a second image of the at least part of the shelf captured after the first information was provided; analyzing the second image to determine a second placement of products on the at least part of the shelf; based on the determined second placement of products, determining a planned second adjustment to the determined second placement of products on the at least part of the shelf; and based on the planned second adjustment to the determined second placement of products, providing second information, the second information being configured to cause the planned second adjustment to the determined second placement of products.

In an embodiment, a system for making gradual adjustments to planograms comprise at least one processor. The at least one processor may be programmed to receive a first image of at least part of a shelf; analyze the first image to determine a first placement of products on the at least part of the shelf; based on the determined first placement of products, determine a planned first adjustment to a determined first placement of products on the at least part of the shelf; and based on the planned first adjustment to the determined first placement of products, provide first information, the first information being configured to cause the planned first adjustment to the determined first placement of products. The processor may further be programmed to receive a second image of the at least part of the shelf captured after the first information was provided; analyze the second image to determine a second placement of products on the at least part of the shelf; based on the determined second placement of products, determine a planned second adjustment to the determined second placement of products on the at least part of the shelf; and based on the planned second adjustment to the determined second placement of products, provide second information, the second information being configured to cause the planned second adjustment to the determined second placement of products.

In an embodiment, a method for testing of planograms is disclosed. The method may comprise receiving a first image of at least part of a shelf; analyzing the first image to determine a first placement of products on the at least part of the shelf; based on the first placement of products, determining a planned adjustment to the first placement of products on the at least part of the shelf; and generating first instructions to implement the planned first adjustment. The method may further comprise receiving a second image of the at least part of the shelf captured after the first instructions were generated; analyzing the second image to determine a second placement of products on the at least part of the shelf, the second placement of products resulting from the planned first adjustment; receiving an indication of an impact of the second placement of products; determining a planned second adjustment to the second placement of products on the at least part of the shelf, the planned second adjustment being determined based on the impact; and generating second instructions to implement the planned second adjustment.

In an embodiment, a system for testing of planograms may comprise at least one processor. The at least one processor may be programmed to receive a first image of at least part of a shelf; analyze the first image to determine a first placement of products on the at least part of the shelf; based on the first placement of products, determine a planned first adjustment to the first placement of products on the at least part of the shelf; and generate first instructions to implement the planned first adjustment to the first placement of products. The at least one processor may further be programmed to receive a second image of the at least part of the shelf captured after the first instructions were generated; analyze the second image to determine a second placement of products on the at least part of the shelf, the second placement of products resulting from the planned first adjustment; receive an indication of an impact of the second placement of products; determine a planned second adjustment to the second placement of products on the at least part of the shelf, the planned second adjustment being determined based on the impact; and generate second instructions to implement the planned second adjustment.

In an embodiment, a method for assessing quality of tasks performed by persons in retail stores may comprise receiving an indication that a person completed a task corresponding to at least one shelf in a retail store; receiving at least one image of the at least one shelf captured using an image sensor after the completion of the task; analyzing the at least one image to determine at least one property associated with performing the task; and using the at least one property to determine a reward for performing the task.

In an embodiment, a system for assessing quality of tasks performed by persons in retail stores include at least one processor; and a non-transitory computer-readable medium containing instructions that, when executed by the at least one processor, cause the at least one processor to: receive an indication that a person completed a task corresponding to at least one shelf in a retail store; receive at least one image of the at least one shelf captured using an image sensor after the completion of the task; analyze the at least one image to determine at least one property of associated with performing the task; and use the at least one property to determine a reward for performing the task.

In an embodiment, a computer program product may assess quality of tasks performed by persons in retail stores embodied in a non-transitory computer-readable medium and executable by at least one processor. The computer program product may include instructions for causing the at least one processor to execute the method described above.

In some embodiments, an indication that a person completed a task corresponding to at least one shelf in a retail store may be received, and at least one image of the at least one shelf may be received, the at least one image being captured using an image sensor after the completion of the task. The at least one image may be analyzed to determine at least one property associated with performing the task, and the at least one property may be used to determine a reward for performing the task. For example, the indication that the person completed the task may be based on an input from the person, may be based on an analysis of the at least one image, and so forth. In one example, the at least one property associated with performing the task may include a quality indication of at least one aspect of performing the task, and the determination of the reward may be based on the quality indication. In one example, the at least one image may be at least one image captured by at least one of the person or an image sensor mounted to a shelf. In one example, the at least one image may be analyzed to determine a property of the person, and the determination of the reward may be based on the property of the person. In one example, the task may include at least one of a restocking of a product associated with the at least one shelf, a correction of product facings at the at least one shelf, removing a product from the at least one shelf, changing a price of at least one product, and changing an orientation of at least one product on the at least one shelf. In one example, the at least one image may be analyzed to determine at least one additional available task corresponding to the at least one shelf in the retail store, and an indication of the additional available task may be provided to the person. In one example, the at least one image may be analyzed to determine that the person performed a positive action corresponding to the at least one shelf in the retail store (the positive action may not be included in the task), and the determination of the reward may be based on the positive action. In one example, an impact of the performed task may be determined, and the determination of the reward may be based on the impact. In one example, the task may include at least one of positioning or removing a promotional sign at the at least one shelf. In one example, the at least one image may be analyzed to determine at least one aspect lacking in the performance of the task, an indication of the at least one aspect may be provided to the person, at least one additional image of the at least one shelf may be received (the at least one image may be at least one image being captured using the image sensor after the indication of the at least one aspect is provided), and the at least one additional image may be analyzed to determine the reward for performing the task. In one example, the reward may include a monetary reward, and the determination of the reward may include a determination of an amount associated with the monetary reward. In one example, input from at least one pressure sensor positioned on the at least one shelf may be received, and the determination of the at least one property may be based on an analysis of the received input. In one example, input from at least one touch sensor positioned on the at least one shelf may be received, and the determination of the at least one property may be based on an analysis of the received input. In one example, input from at least one weight sensor positioned on the at least one shelf may be received, and the determination of the at least one property may be based on an analysis of the received input. In one example, input from at least one light sensor positioned on the at least one shelf may be received, and the determination of the at least one property may be based on an analysis the received input. In one example, analyzing the at least one image to determine the at least one property may comprise identifying at least one visual indicator in the at least one image (the visual indicator may comprise at least one of: a color, a brightness, a character, and a shape), comparing the at least one visual indicator to a reference visual indicator, and determining a degree of similarity between the identified at least one visual indicator and the reference visual indicator.

In an embodiment, a non-transitory computer-readable medium may include instructions that when executed by a processor cause the processor to perform a method for selecting available assignments for users based on mobile computing devices of the users, the method comprising: receiving an indication of at least one parameter of a mobile computing device associated with a user; accessing a plurality of available assignments, each assignment of the plurality of available assignments involving use of at least one mobile computing device; based on the at least one parameter of the mobile computing device associated with the user, selecting a subset of the plurality of available assignments; and offering the selected subset of the plurality of available assignments to the user.

In an embodiment, a method for selecting available assignments for users based on mobile computing devices of the users may include receiving an indication of at least one parameter of a mobile computing device associated with a user; accessing a plurality of available assignments, each assignment of the plurality of available assignments involving use of at least one mobile computing device; based on the at least one parameter of the mobile computing device associated with the user, selecting a subset of the plurality of available assignments; and offering the selected subset of the plurality of available assignments to the user.

In an embodiment, a system for selecting available assignments for users based on mobile computing devices of the users may include at least one processor programmed to: receive an indication of at least one parameter of a mobile computing device associated with a user; access a plurality of available assignments, each assignment of the plurality of available assignments involving use of at least one mobile computing device; based on the at least one parameter of the mobile computing device associated with the user, select a subset of the plurality of available assignments; and offer the selected subset of the plurality of available assignments to the user.

In an embodiment, a non-transitory computer-readable medium may include instructions that when executed by a processor cause the processor to perform a method for selecting available assignments, the method comprising: receiving an indication of an external assignment, wherein the external assignment is associated with one or more locations external to a plurality of retail stores; based on the indication of the external assignment, selecting a retail store among the plurality of retail stores; based on the selected retail store, selecting at least one available assignment in the selected retail store; and offering the selected at least one available assignment in the selected retail store to a user.

In an embodiment, a method for selecting available assignments may include receiving an indication of an external assignment, wherein the external assignment is associated with one or more locations external to a plurality of retail stores; based on the indication of the external assignment, selecting a retail store among the plurality of retail stores; based on the selected retail store, selecting at least one available assignment in the selected retail store; and offering the selected at least one available assignment in the selected retail store to a user.

In an embodiment, a system for selecting available assignments in retail stores for users based on external assignments may include: at least one processor programmed to: receive an indication of an external assignment, wherein the external assignment is associated with one or more locations external to a plurality of retail stores; based on the indication of the external assignment, select a retail store among the plurality of retail stores; based on the selected retail store, select at least one available assignment in the selected retail store; and offering the selected at least one available assignment in the selected retail store to a user.

Consistent with other disclosed embodiments, non-transitory computer-readable storage media may store program instructions, which are executed by at least one processing device and perform any of the methods described herein.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.

