BACKGROUND

Heavy equipment, such as excavators, dump trucks, loaders, bulldozers, mixers, and the like typically operate at earthmoving sites. Some examples of earthmoving sites include, but are not limited to, building/bridge/road construction sites, quarries, and mines. In many cases, the heavy equipment are provided by many different contractors and subcontractors, and are often equipped with provider-specific monitoring devices to track the activity and productivity of each heavy equipment during the construction process. Thus, it may be difficult for a single entity, e.g., a construction manager or owner, to track the productivity of all of the heavy equipment through the construction process.

SUMMARY

Embodiments provide image capturing methods and systems for capturing productivity at an earthmoving site using computer vision to track productivity of heavy equipment without provider-specific monitoring devices. Image capturing devices can be positioned at an earthmoving site to capture images of heavy equipment and surface contours of the earthmoving site. The system can analyze the captured images to identify the heavy equipment's type, movement, and position at the earthmoving site. The system can also analyze surface contours of the earthmoving site and selectively send images of the earthmoving site to a server for further image processing. Such methods and systems improve the simplicity, versatility, and efficiency at which productivity is captured at an earthmoving site.

In some embodiments, a method for capturing objects in action at an earthmoving site includes: capturing, by an image capturing device positioned at the earthmoving site, an image of a region of the earthmoving site including an object, the image being recorded as captured digital data; identifying a classification of the object using a trained algorithm existing in memory of the image capturing device; sending, by the image capturing device, the classification of the object to a remote server through a network; determining a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image; sending, by the image capturing device, a set of images including the image to the remote server, the set of images representing movement of the object at the earthmoving site; determining an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site; and outputting a report to a user based on the activity performed by the object at the earthmoving site.

The object can be a first object, the pixel location can be a first pixel location, the boundary can be a first boundary, and the image can include the first object and a second object. The method can further include determining a second pixel location and a second boundary of the second object within the field of view of the image capturing device based on positions of pixels of the second object in the image; determining a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of a second image capturing device; determining a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device; determining a distance between the first 3D location and the 3D location; comparing the distance to a proximity threshold indicating a minimum distance between the first and second objects; and in response to a determination that the distance is below the proximity threshold, performing the sending of the portion of the image to the remote server. The portion of the image can include the first object and the second object when the distance between the first 3D location and the second 3D location are below the proximity threshold. The method can further include generating, by the image capturing device, a warning signal in response to the determination that the distance is below the proximity threshold. The method can further include determining that the classification of the object is one of a set of predefined objects; and in response to the determination that the object is one of a set of predefined objects, performing the sending of the portion of the image to the remote server. The image capturing device can be a first image capturing device positioned at a first location within the earthmoving site, the image can be a first image, the region is a first region of the earthmoving site, the captured digital data can be a first captured digital data, and where the method further includes: capturing, by a second image capturing device positioned at a second location within the earthmoving site different from the first location, a second image of a second region of the earthmoving site including the object from a different perspective than the first image capturing device, the second image being recorded as second captured digital data; and using a first known position of the first image capturing device and second known position of the second image capturing device, along with a known 3D position of a common point captured within fields of view of the first and second image capturing devices, to determine a position of the object at the earthmoving site in 3D space. Determining the position and the boundary of the object can be performed by the image capturing device at the earthmoving site. The method can further include sending, by the image capturing device, the position and the boundary of the object into the remote server through the network. The boundary can be a boundary box that forms a rectangle around the object and that dynamically moves in location and dimension corresponding to a movement of the object through the set of images. The object can be at least one of a group consisting of heavy equipment and personnel.

In certain embodiments, an image capturing device for capturing objects in action at an earthmoving site includes: a camera for capturing images of a region of the earthmoving site; communication circuitry operable to send data to a remote server; memory for storing at least a portion of the captured images of the earthmoving site, the memory comprising an image classifier module or an object detection module; and a processor coupled to the memory, camera, and communication circuitry, the processor capable of executing instructions in the memory to: capture an image of a region of the earthmoving site including an object, the image being recorded as captured digital data; identify a classification of the object using a trained algorithm of the image classifier module or the object detection module existing in memory of the image capturing device; send the classification of the object to a remote server through a network; determine, by the object detection module, a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image; send a set of images including the image to the remote server, the set of images representing movement of the object at the earthmoving site; determine an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site; and output a report to a user based on the activity performed by the object at the earthmoving site.

The object can be a first object, the pixel location can be a first pixel location, the boundary is a first boundary, and the image can include the first object and a second object. The processor can be further configured to: determine a second pixel location and a second boundary of the second object within the field of view of the image capturing device based on positions of pixels of the second object in the image; determine a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of a second image capturing device; determine a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device; determine a distance between the first 3D location and the 3D location; compare the distance to a proximity threshold indicating a minimum distance between the first and second objects; and in response to a determination that the distance is below the proximity threshold, perform the sending of the portion of the image to the remote server. The set of images can include the first object and the second object when the distance between the first 3D location and the second 3D location are below the proximity threshold. The processor can be further configured to: generate a warning signal in response to the determination that the distance is below the proximity threshold.

In certain additional embodiments, a system for capturing an object in action at an earthmoving site includes: a first image capturing device positioned within the earthmoving site and operable to capture first images of a first region of the earthmoving site; and a second image capturing device positioned within the earthmoving site and configured to capture second images of a second region of the earthmoving site, wherein the first region overlaps with at least a portion of the second region, and wherein the first image capturing device and the second image capturing device are communicatively coupled to a remote server over a network, wherein the first image capturing device comprises: a camera for capturing the first images of the first region; communication circuitry operable to send data to the remote server; memory for storing at least a portion of the captured images of the earthmoving site, the memory comprising an image classifier module and an object detection module; and a processor coupled to the memory, camera, and communication circuitry, the processor capable of executing instructions in the memory to: capture an image of a region of the earthmoving site including an object, the image being recorded as captured digital data; identify a classification of the object using a trained algorithm existing in memory of the image capturing device; send the classification of the object to a remote server through a network; determine a pixel location and a boundary of the object within a field of view of the image capturing device based on positions of pixels of the object in the image; send a set of images including the image to the remote server, the set of images representing movement of the object at the earthmoving site; and determine an activity being performed by the object based on an analysis of digital data associated with the classification of the object and the movement of the object at the earthmoving site, wherein a report based on the activity performed by the object at the earthmoving site is outputted to a user.

The first image capturing device can be communicatively coupled to the second image capturing device so that data can be transferred between the first and second image capturing devices. The object can be a first object, the pixel location can be a first pixel location, the boundary can be a first boundary, and the image includes the first object and a second object. The processor can be further configured to: determine a second pixel location and a second boundary of the second object within the field of view of the image capturing device based on positions of pixels of the second object in the image; determine a first three-dimensional (3D) position of the first object in the earthmoving site based on the first pixel location in the field of view of the first image capturing device and a third pixel location in a field of view of a second image capturing device; determine a second 3D position of the second object in the earthmoving site based on the second pixel location in the field of view of the first image capturing device and a fourth pixel location in the field of view of the second image capturing device; determine a distance between the first 3D location and the 3D location; compare the distance to a proximity threshold indicating a minimum distance between the first and second objects; and in response to a determination that the distance is below the proximity threshold, perform the sending of the portion of the image to the remote server.

A better understanding of the nature and advantages of embodiments of the present invention may be gained with reference to the following detailed description and the accompanying drawings.

