CROSS-REFERENCES

Some references, which may include patents, patent applications and various publications, are cited and discussed in the description of this disclosure. The citation and/or discussion of such references is provided merely to clarify the description of the present disclosure and is not an admission that any such reference is “prior art” to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entireties and to the same extent as if each reference was individually incorporated by reference.

FIELD

The present disclosure relates generally to self-checkout techniques, and more particularly to systems and methods of self-checkout that uses visual information to prevent shoplifting.

BACKGROUND

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

Self-checkout (also known as self-service checkout or semi-attended customer-activated terminal (SACAT)) machines provide service for customers to process their own purchases from a retailer. It is an alternative to the traditional cashier-staffed checkout. That is, the customer performs the job of the cashier themselves, by scanning and applying payment for the items. Self-checkout is vulnerable to some shoplifting, causing a lot of losses to the retailer. For example, a person may directly take the item without scanning. To that end, some stores decide to install a surveillance system and hire some staffs to monitor shoplifters. However, it is difficult for the staffs to concentrate on monitoring multiple self-checkout machines at all times.

Therefore, an unaddressed need exists in the art to address the aforementioned deficiencies and inadequacies.

SUMMARY

In certain aspects, the present disclosure relates to a system for self-checkout. The system includes a scanner, an imaging device, and a computing device. The computing device includes a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to:

initiate a self-checkout event;

instruct the imaging device to capture video frames of a region of interest (ROI), where the ROI includes a table region for placing a product, and the table region includes a scanning region for scanning the product;

track the product in the video frames;

record scanning status and location status of each of the product, wherein the scanning status includes scanned and unscanned, the default scanning status is unscanned, and the location status includes in the scanning region, in the table region (including “on the table” where the product is placed on top surface of the table and is in contact with the top surface of the table, and “above the table” where the product is in the air above the top surface of the table and is not in contact with the table surface), and out of the table region but within the ROI;

in response to receiving a scanning signal from the scanner when the product is placed in the scanning region, record scanning status of the tracked product from unscanned to scanned;

calculate a shoplifting risk score based on a number of the product having the scanning status of unscanned and disappear from the table region or the ROI; and

provide a shoplifting warning when the shoplifting score is greater than a predetermined value.

In certain embodiments, the computer executable code is configured to track all the products that have been observed in the video frames. In certain embodiments, the computer executable code may be configured to only track the products that have been scanned.

In certain embodiments, the computer executable code is configured to initiate a self-checkout event when the customer touches “Start” on a touchscreen of the computing device, or when the customer starts scanning a product, or the customer scans his membership card.

In certain embodiments, the step of calculating a shoplifting risk score is performed after the computing device receives an instruction from a payment device indicating that the self-checkout event has been completed, or after the computing device receives a signal because the customer touched “End” on the touchscreen of the computing device, or after both the actions. In certain embodiments, the step of calculating a shoplifting risk score may also be delayed for a short period of time after the checkout event has been completed, or until the customer leaves the ROI. In certain embodiments, the computer executable code may be configured to process recorded video and the corresponding scanning information after an extended time period.

In certain embodiments, the computer executable code is configured to, before track the product: segment the video frames such that each pixel of the video frames is labeled with hand of a customer, product in hand, product on table, and background; and detect the product in hand and the product on table based on the labels of the pixels. In certain embodiments, the object detection methods, e.g. Mask R-CNN, Faster R-CNN, and RefineDet, can be used to detect the product in hand or on the table.

In certain embodiments, the computer executable code is configured to segment and detect using a deep convolutional neural network (CNN).

In certain embodiments, the computer executable code is configured to track the product using tracking-by-detection and greedy search when the product is the product in hand, and track the product using appearance based tracker when the product is the product on table.

In certain embodiments, the shoplifting risk score R is calculated by: R=r1×N1+r2×N2. N1is a number of the product that has the scanning status of unscanned and disappears from the table region, and N2is a number of the product that has the scanning status of unscanned and disappears from the ROI, r1is a medium risk factor, and r2is a high risk factor.

In certain embodiments, the medium risk factor is in a range of about 0.5-1.0, and the high risk factor is in a range of about 1.0-2.0. In certain embodiments, the high risk factor is set at about two times of the medium risk factor.

In certain embodiments, the computer executable code is further configured to: provide the shoplifting warning that shoplifting happened when R≥β1×(N3)α; and provide the shoplifting warning that shoplifting might happened when β1×(N3)α>R≥β2×(N3)α. N3is a number of the product that has the scanning status of scanned, α is a predetermined number in a range of about 0.2-1.0, β1is a predetermined number in a range of about 1.0-2.0, and β2is a predetermined number in a range of about 0.5-1.0. When R<β2×(N3)α, the computer executable code is configured to keep silent or send a notice that no shoplifting happened.

In certain embodiments, the computer executable code is further configured to: provide a scan difficulty message when the product is placed in the scanning region and has the scanning status of unscanned, and the computing device has not received a scanning signal of the product from the scanner for an accumulated time greater than a predetermined time.

In certain embodiments, the predetermined time is in a range of about 2-15 seconds. In certain embodiments, the predetermined time is 3-5 seconds. In certain embodiments, the predetermined time is 3 or 5 seconds. In certain embodiments, the predetermined time may be configured according to a retailer's need.

In certain embodiments, the imaging device is a red green blue-depth (RGBD) camera.

In certain embodiments, the computing device is an embedded device. In certain embodiments, the computing device is a cloud device.

In certain aspects, the present disclosure relates to a method for self-checkout. In certain embodiments, the method includes:

initiating, by a computing device, a self-checkout event;

instructing, by the computing device, an imaging device to capture video frames of a region of interest (ROI), where the ROI includes a table region for placing a product, and the table region includes a scanning region for scanning the product;

tracking, by the computing device, the product in the video frames;

recording, by the computing device, scanning status and location status of each of the product, where the scanning status includes scanned and unscanned, the default scanning status is unscanned, and the location status includes in the scanning region, in the table region, and out of the table region but within the ROI;

in response to receiving a scanning signal from a scanner when the product is placed in the scanning region, recording, by the computing device, scanning status of the tracked product from unscanned to scanned;

calculating a shoplifting risk score based on a number of the product having the scanning status of unscanned and disappears from the table region or the ROI; and

providing a shoplifting warning when the shoplifting risk score is greater than a predetermined value.

In certain embodiments, the method further includes, before the step of tracking the product: segmenting, by the computing device, the video frames such that each pixel of the video frames is labeled with hand of a customer, product in hand, product on table, and background; and detecting, by the computing device, the product in hand and the product on table based on the labels of the pixels.

In certain embodiments, at least one of the steps of segmenting and detecting is performed using a deep convolutional neural network (CNN).

In certain embodiments, the step of tracking the product is performed using tracking-by-detection and greedy search when the product is the product in hand, and performed using appearance based tracker when the product is the product on table.

In certain embodiments, the shoplifting risk score R is calculated by: R=r1×N1+r2×N2, where N1is a number of the product that has the scanning status of unscanned and disappears from the table region, N2is a number of the product that has the scanning status of unscanned and disappears from the ROI, r1is a medium risk factor in a range of about 0.5-1.0; and r2is a high risk factor in a range of about 1.0-2.0.

In certain embodiments, the method further includes: providing the shoplifting warning that shoplifting happened when R≥β1×(N3)α; and providing the shoplifting warning that shoplifting might happened when β1×(N3)α>R≥β2×(N3)α. N3is a number of the product that has the scanning status of scanned, a is a predetermined number in a range of about 0.2-1.0, β1is a predetermined number in a range of about 1.0-2.0, and β2is a predetermined number in a range of about 0.5-1.0. When R<β2×(N3)α, the computer executable code is configured to keep silent or send a notice that no shoplifting happened.

In certain embodiments, the method further includes: providing a scan difficulty message when the product is placed in the scanning region and has the scanning status of unscanned, and the computing device has not received a scanning signal of the product from the scanner for an accumulated time greater than a predetermined time.

In certain embodiments, the imaging device is a red green blue-depth (RGBD) camera. In certain embodiments, the computing device is an embedded device or a cloud device.

In certain aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. The computer executable code, when executed at a processor of a computing device, is configured to perform the method described above.

These and other aspects of the present disclosure will become apparent from following description of the preferred embodiment taken in conjunction with the following drawings and their captions, although variations and modifications therein may be affected without departing from the spirit and scope of the novel concepts of the disclosure.

