BACKGROUND

Technical Field

The present disclosure relates to imager based systems, for example image-based systems that automatically detect, monitor and track objects in an environment, for instance in a retail store.

Description of the Related Art

Many applications would benefit by automated detection, monitoring or tracking of objects in the respective environment. For example, retail, warehouse, baggage sorting, parcel sorting, or gaming casino environments may benefit by automated detection, monitoring or tracking of objects using image-based techniques.

In many applications or environments, objects to be monitored or tracked are moving or in transit. Camera systems designed to capture images of moving objects typically employ a global shutter imager with short exposure time, and often employing pulsed illumination.

For example, a retail environment (e.g., grocery or convenience store) may employ ceiling mounted cameras used for detection of items in shopping carts. Such cameras must be capable of capturing relatively high-resolution images in order to discern the specific objects or items. In particular, machine vision algorithms require a certain number of pixels in order to properly identify objects or items. Thus, when a large area is to be covered, a camera or imager mounted in a ceiling would have to have a very high resolution. As an estimate, in order to recognize items as small as 5 inches by 5 inches in size (e.g., a box of Tide® laundry detergent), an image would require approximately 200 pixels by 200 pixels. At approximately 40 pixels per inch, a 5 Megapixel imager could only cover an area of about 5 feet by 4 feet while still providing sufficient resolution to identify objects or items. For example, if the application is monitoring or tracking objects or items at a checkout station or counter of, for instance a grocery store (e.g., on a conveyor belt at the checkout station or counter, and/or in the shopping cart), this area of coverage is not sufficient. In particular, a position of a shopping cart at a checkout station or counter can vary substantially from one checkout transaction to another, or even during a single checkout transaction, making selection of an appropriate 5 foot by 4 foot area for imaging virtually impossible.

Further, sharp image capture is needed for successfully reading linear or one-dimensional machine-readable symbols (e.g., barcode symbols) and/or two-dimensional machine-readable symbols (e.g., area or matrix code symbols). For moving objects, sharp image capture typically requires very fast shutter speeds, which can be impossible using ambient light, leading to the need for expensive and large active illumination systems. Active illumination can interfere with other opto-electronic components of a system or device, and may present an annoyance to human operators.

BRIEF SUMMARY

Global shutter imagers typically have lager pixel size, and are more expensive and noisier than rolling shutter imagers.

Applicants have realized that a smaller, less expensive camera system that requires less light to operate can be implemented, for example, if: 1) an area to be imaged is kept small, and/or 2) relative object motion can be minimized. Such may advantageously allow for lower cost, smaller size, faster tracking, and lower levels of required illumination than existing global shutter imager-based approaches.

In at least one implementation, a field-of-view of an imager or camera, for example a rolling shutter camera, is directed or caused to track one or more objects via a steering mirror, for instance a fast steering mirror. A tracking subsystem may include a tracking imager or camera, or some other sensor(s) or transducer(s) that capture data (e.g., 3-dimensional data) that characterizes objects in an environment, for instance a retail environment. In other implementations, images captured by a first imager may be used to determine characteristics of objects, which characteristics are used to cause a respective field-of-view of the first imager to track one or more objects. Characteristics may, for example, include appearance, presence, location, position, speed, and/or direction of travel of the object. Characteristics may also, for example, include physical characteristics of the object and/or packaging, which characteristics allow classifying the object as a certain type of object (e.g., stock keeping unit or SKU, restricted sale type item). The steering mirror directs the field-of-view of a relatively higher resolution imager with a relatively narrow field-of-view to track an object, for example an object spotted in a wider field-of-view of another imager, and/or an object that is in motion. For moving objects, relative motion between the object and the field-of-view of the higher resolution imager is reduced or eliminated, allowing use of a rolling shutter imager or camera.

Image sensors of rolling shutter imagers typically have smaller pixels than image sensors of global shutter imagers. Rolling shutter imagers typically have lower noise due to the ability to cancel kTC noise, which may otherwise be a dominant noise source. Use of object tracking allows the exposure time of a rolling shutter imager to be longer than would otherwise be possible. Such may advantageously allow use of ambient light or lower levels of active lighting than might otherwise be employed with a global shutter imager.

While some implementations may advantageously employ rolling shutter imagers, for example as the relatively narrow field-of-view imager, other implementations may employ a global shutter image for the relatively narrow field-of-view imager and/or for the relatively wide field-of-view imager. For example, a global shutter imager may be employed as the relatively narrow field-of-view imager in the InCart™ and ShelfHawk™ applications, discussed below.

The relatively smaller pixel size typical of image sensors of rolling shutter imagers may allow a smaller imager package size, with a shorter focal length lens than might otherwise be required. Typically, for an equivalent f-number, the aperture diameter of a rolling shutter imager is smaller than that of a global shutter imager. This can advantageously reduce cost while increasing tracking speed.

The imaging system may be employed in loss prevention, for example in retail environments. For example, two imagers may be employed to cover an area (e.g., checkout station or counter, aisle, game pit), one imager having a relatively wider field-of-view and the other having a relatively narrower field-of-view which is steerable. For instance the relatively narrower field-of-view may be steerable via a steering mirror. The type of steering mirror, and in particular the speed at which the steering mirror can operate, may be a function of the specific application. For example, some applications may have relatively slow moving objects to track or objects that move at a fixed speed (e.g., on a conveyor belt) or a known speed (e.g., speed sensed with one or more encoders), with trigger signals received via one or more sensors or transducers (e.g., photocells, inductive sensors, ultrasonic sensors). In such applications, the steering mirror may take the form of a mechanical motorized platform that allows the imager (e.g., image sensor) to move with respect to an optic assembly. For instance, devices and techniques employed for stabilization in digital single lens reflect (DSLR) cameras may be suitable. Other applications may employ a fast steering mirror, for example the Eagle Eye™ fast steering mirror, developed by DataLogic.

This approach can be employed, for example, as a security camera in retail (e.g., grocery stores, convenience stores) or other locations, advantageously avoiding the need for additional security cameras. This approach can be employed, for example, to advantageously detect objects or items in a shopping cart, shopping basket, or elsewhere at a checkout station or counter. For instance, objects or items left in a shopping cart or shopping basket may be intentionally or unintentionally removed from the retail environment generating an alert to retail personnel to prevent such from occurring. This approach can additionally or alternatively be employed, for example, to advantageously detect, prevent and otherwise deter what is called “sweet-hearting” by retail personnel such as checkout cashiers or clerks. In “sweet-hearting,” the cashier or clerk intentionally fails to scan or “ring up” an item, for example pretending to scan or “ring the item up,” for customers who are known by or colluding with the cashier or clerk. This allows the customer to steal items in a way that is typically very difficult to detect. Monitoring items at the checkout station or counter, for instance on a conveyer belt, may prevent such. Further, this approach can be employed, for example, to recognize individuals (e.g., customers), including ones that may have previously been banned from the premises, or to provide prompts to personnel allowing customers to be greeted by name or otherwise be recognized. This approach can be employed, for example, to advantageously monitor shopping at locations other than the checkout station or counter of a retail environment. For example, this approach can be employed, for example, to monitor or track the selection of items or objects from shelves and placement into a shopping cart or shopping basket in aisles of a retail environment, allowing better assessment of shopping patterns or purchasing decisions or, conversely, detection of shop lifting. Such could likewise be employed to monitor pick-and-place operations in a warehouse environment, or gaming wagering, card dealing or play in a gaming establishment such as a casino.

An imaging system to image objects in an environment may be summarized as including a first imager having a first imager field-of-view to capture images of the environment; a steering mirror interposed along a first optical path between the first imager and the environment, the steering mirror selectively operable to steer at least a portion of the first imager field-of-view relative to one or more objects in the environment; and an object tracking subsystem that includes at least one hardware processor, the object tracking subsystem communicatively coupled to cause the steering mirror to steer the first imager field-of-view based at least in part on information indicative of at least one of a presence, a position, a speed or a direction of at least one object in the environment to at least partially track objects in the environment. The first imager may be a rolling shutter imager. The first imager may be a rolling shutter imager having a complementary metal oxide semiconductor (CMOS) image sensor and at least one of a mechanical or electronic rolling shutter. The steering mirror may be a fast scanning mirror.

The object tracking subsystem may include a second imager having a second imager field-of-view to capture images of the environment, the first imager field-of-view relatively more narrow than the second imager field-of-view; and a control subsystem, the control subsystem communicatively coupled the second imager to receive information directly or indirectly therefrom, and communicatively coupled to cause the steering mirror to steer the first imager field-of-view based at least in part on information received via the second imager. The object tracking subsystem may detect an entrance of at least a first object into the second field-of-view. The object tracking subsystem may identify an object entering the second field-of-view as corresponding to a defined type of object. The object tracking subsystem may detect an entrance of at least a first object into the first field-of-view. The object tracking subsystem may identify an object entering the first field-of-view as corresponding to a defined type of object. The object tracking subsystem may determine a position of the object. The object tracking subsystem may determine at least an estimate of a speed of the object. The object tracking subsystem may determine a direction of the object. The object tracking subsystem may determine a direction of the object.

The imaging system may further include a steering mirror actuator drivingly coupled to the steering mirror and responsive to signals from the control subsystem of the object tracking subsystem to steer the first imager field-of-view.

The first imager may be a rolling shutter imager having an image sensor and at least one of a mechanical or electronic rolling shutter, the first imager having a higher resolution than the second imager, and may further include a steering mirror actuator drivingly coupled to the steering mirror and responsive to signals from the control subsystem of the object tracking subsystem to concurrently steer an entire field-of-view of the image sensor of the first imager. The control subsystem of the object tracking system may cause the steering mirror actuator to move the steering mirror to an initial scan position at a first speed, then to immediately follow a rolling shutter exposure by panning the sensor at a second speed, the second speed slower than the first speed. The environment may be a retail environment. The retail environment may be a checkout stand with a point-of-sale terminal, and the first and the second fields-of-view may encompass at least a portion of the checkout stand. The retail environment may be a checkout stand with a conveyor belt, and the first and the second fields-of-view may encompass at least a portion of the conveyor belt. The retail environment may be an aisle spaced remotely from a checkout stand, and the first and the second fields-of-view may encompass at least a portion of the aisle. The object tracking subsystem may identify a first object in the first field-of-view as corresponding to a defined stock keeping unit (SKU) of a retail object. The object tracking subsystem may identify a second object in at least one of the first or the second fields-of-view as corresponding to at least one of a shopping cart or a shopping basket in which one or more retail objects reside. The object tracking subsystem may determine whether one or more retail objects are retained in a shopping cart or a shopping basket at a defined point in a retail transaction.

The imaging system may further include a variable focus lens in the first optical path between the first imager and the environment; an illumination source positioned and oriented to illuminate at least a portion of the environment in the first field-of-view; and a polarizer in the first optical path between the first imager and the environment. The system may compensate to recover blur and motion induced artifacts in rolling shutter imagers. The system may additionally or alternatively coordinate operation of wide field-of-view FOV imagers and narrow field-of-view imagers, for example generating coordinates of object of interest in the wide field-of-view that allows tracking in the narrow field-of-view, tracking multiple objects concurrently, and selecting objects including prioritizing objects.

A method of operation in an imaging system to image objects in an environment, the imaging system including a first imager having a first imager field-of-view, a steering mirror interposed along a first optical path between the first imager and the environment, and an object tracking subsystem that includes at least one hardware processor, may be summarized as including detecting at least one of a presence, a position, a speed or a direction of at least one object in the environment by the object tracking subsystem; steering the first imager field-of-view relative to one or more objects in the environment based at least in part on the detecting at least one of a presence, a position, a speed or a direction of the at least one object in the environment to at least partially track objects in the environment; and capturing images of the at least one object by the first imager as the first imager field-of-view is steered.

The method may further include operating the first imager as a rolling shutter imager. The imaging system may include a fast scanning mirror, and steering the first imager field-of-view may include moving the fast scanning mirror.

The object tracking subsystem may include a control subsystem that includes at least one hardware processor and a second imager, the second imager having a second imager field-of-view of the environment, and the method may further include receiving information by the control subsystem directly or indirectly from the second imager; and analyzing the received information by the control subsystem to detect at least one of the presence, the position, the speed or the direction of the at least a first one of the objects in the environment. The detecting by the object tracking subsystem may include detecting an entrance of at least a first one of the at least one object into the second field-of-view. The detecting by the object tracking subsystem may include identifying at least a first one of the at least one object entering the second field-of-view as corresponding to a defined type of object. The detecting by the object tracking subsystem may include detecting an entrance of at least a first one of the at least one object into the first field-of-view. The detecting by the object tracking subsystem may include identifying at least a first one of the at least one object entering the first field-of-view as corresponding to a defined type of object. The detecting by the object tracking subsystem may include determining a position of the first one of the objects. The detecting by the object tracking subsystem may include determining at least an estimate of a speed of the first one of the objects. The object tracking subsystem may include determining a direction of the first one of the objects. The detecting by the object tracking subsystem may include determining a direction of the first one of the objects.

The method may further include providing a control signal by a control system, directly or indirectly, to a steering mirror actuator which is drivingly coupled to the steering mirror to steer the first imager field-of-view. Providing a control signal to a steering mirror actuator may include providing control signals to the steering mirror actuator to move to the steering mirror to an initial scan position at a first speed, then to follow a rolling shutter exposure by panning a sensor at a second speed, the second speed slower than the first speed. The environment may be a retail environment and capturing images of the at least one object by the first imager may include capturing images of a location in the retail environment. Capturing images of a location in the retail environment may include capturing images that encompass at least a portion of a checkout stand in the retail environment. Capturing images of a location in the retail environment may include capturing images that encompass at least a portion of a conveyor belt in the retail environment. Capturing images of a location in the retail environment may include capturing images that encompass at least a portion of an aisle spaced remotely from a checkout stand in the retail environment.

The method may further include identifying a first object in the first field-of-view as corresponding to a defined stock keeping unit (SKU) of a retail object.

The method may further include identifying a second object in at least one of the first or the second fields-of-view as corresponding to at least one of a shopping cart or a shopping basket in which one or more retail objects reside.

The method may further include determining whether one or more retail objects are retained in a shopping cart or a shopping basket at a defined point in a retail transaction.

The method may further compensate to recover blur and motion induced artifacts in rolling shutter imagers. The method may additionally or alternatively coordinate operation of wide field-of-view FOV imagers and narrow field-of-view imagers, for example generating coordinates of an object of interest in the wide field-of-view that allows tracking in the narrow field-of-view, tracking multiple objects concurrently, or selecting objects including prioritizing objects.

