CROSS-REFERENCE TO RELATED APPLICATIONS

This application represents the U.S. National Phase of International Application number PCT/GB2019/052575 entitled “Manufacturing Assistance System” filed 13 Sep. 2019, which claims priority to Great Britain Application number 1817061.3 filed 19 Oct. 2018, all of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

This invention relates to a system for assisting a worker in performing manufacturing tasks using an augmented-reality display system.

Manufacturing tasks, such as may be performed by one or more workers in a factory, often require high levels of concentration and precision in order to avoid mistakes. They may also require a worker to have a high degree of knowledge about how the tasks should be performed. These requirements can place significant physical and/or cognitive stress on the worker. This can, over time, lead to reduced productivity or to a higher risk of mistakes being made.

The present invention seeks to provide a technological approach that may help to reduce these problems.

SUMMARY OF THE INVENTION

From a first aspect, the invention provides a system for assisting a worker in performing manufacturing tasks involving one or more objects, the system comprising:an augmented-reality display system;a task-state sensing system for determining a state of a manufacturing task involving one or more objects;a physiological-state-sensing system for detecting a physiological state of the worker; andan electronic processing system;
wherein:the electronic processing system is configured to determine a guidance setting for the worker from a detected physiological state of the worker; andthe electronic processing system is configured to determine assistance information to display to the worker in dependence on the state of the manufacturing task and also in dependence on the determined guidance setting for the worker; andthe augmented-reality display system is configured to display the determined assistance information to the worker in a controlled spatial relationship to the one or more objects.

From a second aspect, the invention provides a method of assisting a worker in performing a manufacturing task involving one or more objects, the method comprising:determining a state of the manufacturing task;detecting a physiological state of the worker;determining, on an electronic processing system, a guidance setting for the worker from the detected physiological state of the worker;the electronic processing system determining assistance information to display to the worker in dependence on the state of the manufacturing task and also in dependence on the determined guidance setting for the worker; anddisplaying the determined assistance information to the worker in a controlled spatial relationship to the one or more objects through an augmented-reality display system.

Thus it will be seen that, in accordance with the invention, relevant assistance information can be provided to a worker in a controlled spatial relationship to the objects that are being worked on, and at a level that is responsive to the physiological state of the worker. In this way, the worker may automatically be guided at a level that takes account of the state of mind of the worker (e.g., how tired or stressed the worker is); this can help to reduce errors and improve the worker's productivity at the manufacturing task.

The system may use any appropriate type of augmented-reality display system. The augmented-reality display system may comprise an augmented-reality display device that is wearable by the worker, although this is not essential. In some embodiments, the augmented-reality display system may comprise a component that is worn on the worker's head, such as a headset or glasses. In other embodiments, it may comprise a heads-up display, which may be spaced away from the worker.

Augmented reality (AR) display systems can fuse computer-generated information with a real-world object or scene. This may be done purely visually, with computer-generated information, such as text or graphics, being overlaid on an actual or synthesised image of a real-world scene (typically the worker's immediate surroundings), or the augmented-reality display system may additionally provide computer-generated sounds, haptic feedback, olfactory messages, or other information to the worker in any sensory modality.

The augmented-reality display system may be configured to overlay the assistance information on the worker's direct view of the one or more objects—e.g., by projecting the assistance information onto a transparent screen through which the worker can view of the objects.

Alternatively, it may be configured to combine the assistance information with an electronically-generated representation of the one or more objects—e.g., by incorporating the assistance information into image data obtained from one or more cameras. The combined image may then be displayed to the worker using an immersive display device—e.g., a virtual-reality headset.

In either case, the augmented-reality display system preferably allows the worker to view the one or more objects in stereo vision (i.e., with different perspectives for the worker's left and right eyes). The display system may provide adjustable accommodation; for example, it may control a focal distance to the assistance information in dependence on the position of the one or more objects. Such features may be desirable as they can mean that the provision of the assistance information need not impair the worker's ability to perceive depth when performing the manufacturing task.

The augmented-reality display system may comprise a display device that may be head worn, or hand held, or world fixed, and that may be optical see through, or video see through, or purely virtual.

The augmented-reality display system may comprise a sensor for determining a respective position and/or orientation of each of the one or more objects in space—e.g., one or more cameras. It may comprise processing logic for displaying the guidance information in a controlled spatial relationship to one or more objects based on data from the sensor. The guidance information may be displayed to the worker such that it appears to remain in a fixed alignment with one or more fixed points in space (e.g., relative to the Earth), or in a fixed alignment with one or more points on one or more of the objects or on a manufacturing machine such as a conveyor belt, even as the worker shifts his eyes or as the worker moves his head and/or his body (e.g. so that the information appears to be attached to a work-piece or to a machine).

For example, as an engine moves along a conveyor belt, information relating to the assembly of the engine may appear anchored to the engine. Movements of the worker's eyes or head may be tracked in multiple degrees of freedom—for example, in three degrees of freedom (3DoF), e.g., three-dimensional orientation only, or in six degrees of freedom (6DoF), e.g. three-dimensional position and orientation. Known methods of controlling the spatial relationship of displayed information to real-world objects may be employed.

The augmented-reality display system may further be configured to display the determined assistance information to the worker in a controlled semantic relationship to the one or more objects.

The task-state sensing system may determine the state of the manufacturing task in any appropriate way. It may receive outputs from one or more manufacturing machines or other equipment. It may comprise one or more cameras, motion sensors, switches, human interfaces, etc. It may comprise processing logic, such as software executing on a processor, for determining the state of the task. The state may be at least partly determined by a position in a set of steps. The set may be predefined. It may be or comprise a linear sequence of steps, which may be carried out in the same order over a plurality of performances of the same manufacturing task, or it may comprise a branching set of steps (i.e., having multiple possible paths which may be followed differently from one performance of the task to another). The set of steps may be known to the task-state sensing system (e.g., being pre-programmed in advance of the manufacturing task starting). The task-state sensing system may comprise a context-aware sensor which may be configured to track a state of the context surrounding the manufacturing task. It may be configured to track the state of the context surrounding the manufacturing task by producing a stream of cues, which may be used to update a set of context models.

The manufacturing task may comprise any mechanical or chemical assembly, disassembly or other processing. The task may comprise a plurality of intended or actual steps. The one or more objects may include solid or liquid objects. In one set of preferred embodiments, the manufacturing task may comprise an assembly process, such as assembly of a complex product from a plurality of component input objects. The state of the manufacturing task may represent progress of an assembly process. The same manufacturing task may be performed multiple times by the same worker, or by different workers.

The system may be configured to assist a plurality of workers, simultaneously or at different times. In particular, where a manufacturing task involving one or more objects is performed by a plurality of workers (e.g., a team), the system may comprise one or a plurality of augmented-reality display systems (e.g., one for each worker) and may be configured to display respective assistance information to each worker in a controlled spatial relationship to the one or more objects. The assistance information for each worker may be determined in dependence on a respective guidance setting for the worker. The assistance information for a worker may be determined independently for that worker, or it may depend on the guidance setting of at least one other worker of the plurality of workers. The displayed assistance information may be unique to each worker, or may differ between at least two of the workers, or the same assistance information may be displayed to all of the workers.

The physiological-state-sensing system may comprise one or more sensors for sensing a physiological state of the worker. It may be configured to detect a state of the worker's autonomic nervous system—i.e. a state that is not typically under conscious control of the worker, such as detecting an elevated breathing rate. It may be configured to detect a mental or cognitive state of the worker, such as an increase in mental stress or mental fatigue. While the physiological-state-sensing system could comprise a sensor or imaging system, such as an electroencephalogram (EEG) or functional near-infrared spectroscopy (fNIRS) system, for direct monitoring of the worker's brain (e.g., for measuring neural activity), preferred embodiments do not directly monitor brain activity but instead comprise a processing system configured to infer a physiological state, such as a mental state, from the output of one or more sensors. The physiological-state-sensing system may comprise one or more sensors, which may sense respective properties of the worker's body. The system may detect or measure one or more properties of the worker's muscular, circulatory, respiratory and/or integumentary systems. In some embodiments, the system may measure or detect any one or more of: breathing rate; heart rate; blood volume pulse; heart-rate variability; sweat; skin conductance; eye movement; eyelid droop; pupil dilation; body temperature; facial temperature; head position; head movement; head orientation. The physiological-state-sensing system may determine quantitative or qualitative values or changes.

A physiological-state sensor of the physiological-state-sensing system may be incorporated into a component of the augmented-related display system, such as a headset, or it may be located on a body part of the worker, or it may be located in an environment external to the worker. In some embodiments it may comprise a camera, such as an infrared or thermographic camera. It may comprise processing logic for analysing a camera output, such as for determining a breathing rate from a sequence of thermographic images of the worker's face.

The detected physiological state may represent a binary state, such as conscious or unconscious, but preferably represents a position on a qualitative or quantitative scale. The physiological state may be determined by detecting a change in a property of the worker's body, such as an increase in heart rate above a threshold gradient, or from an absolute value of a measurand, such as a heart rate above a threshold level.

The physiological-state-sensing system or the electronic processing system may determine a value representing the physiological state of the worker, such as an alertness score. It may determine a value representing a likelihood that the worker will make a mistake in the manufacturing task; or a concentration level of the worker; or a distraction level of the worker; or a stress level of the worker; or a mental loading of the worker.

The guidance setting may be a determined from a set or range of possible guidance settings, which may comprise two, three, ten, a hundred or more settings. The guidance setting may represent a level of guidance, or a type of guidance, or a combination of a type (e.g., category) of guidance and a guidance level. The setting may be selected from an ordered set or range. In some preferred embodiments, the guidance setting has at least three possible values; in this way, the guidance may be finer-grained than merely turning a guidance mechanism on or off. The guidance setting may be or comprise a value from a discrete or continuous scale.

The guidance setting may be determined from one or more values output by the physiological-state-sensing system, representative of one or more physiological states of the worker—e.g., by applying one or more thresholds to such a value, or by using a formula or look-up table. Any suitable process could be used to translate an output of the physiological-state-sensing system to a guidance setting, including a machine-learning algorithm. For example, the processing system may be configured to translate an alertness score having one of a finite set of possible values to a guidance setting having one of a finite set of possible values.

The assistance information may comprise any textual, graphical, photographic, video or other information. It may comprise verbal or pictorial instructions. It may refer to an object of the one or more objects, or to an operation to be performed on an object of the one or more objects. It may comprise a diagram of part of all of such an object or operation. It may represent a sequence of two or more steps displayed sequentially over time.

In some embodiments, the augmented-reality display system may be configured to determine where to display assistance information in a field of view of the worker in dependence on a priority level of the assistance information—e.g., with higher priority information being displayed closer to the worker or closer to an object of the one or more objects.

The guidance setting may determine the quantity, type, content, order and/or position of the assistance information that is displayed to the worker. In some embodiments, the possible guidance settings are ordered, with a higher guidance setting or level causing the processing system to display more assistance information over the course of the manufacturing task compared with the amount of assistance information that would be displayed at a lower guidance level. The displayed assistance information may be selected from a set or range of possible assistance information, which set or range may have at least two, three or more members. In some embodiments, a lower guidance level causes the processing system to omit one or more steps from a sequence of information relating to a sequence of manufacturing steps that is available for display to the worker. In some embodiments, a higher guidance level causes the processing system to display the assistance information more prominently to the worker. In some embodiments, a higher guidance level causes the processing system to offer or present additional information to the worker, such as a video clip demonstrating one or more steps. In some embodiments, a higher guidance level causes the processing system to offer or present additional information to the worker, such as a video clip demonstrating one or more steps.

In some embodiments, a higher guidance level causes the processing system to obscure one or more areas in a field of vision of the worker. This can help to focus the worker's attention on a non-obscured area, which may include one of the manufacturing objects. The obscuring may comprise displaying a geometrical object, such as a solid black rectangle, to the worker. The obscuring may be partial (e.g., a partial darkening of an area) or may be total (e.g., complete masking). Total obscuring may be particularly suited to immersive display systems in which the manufacturing objects are viewed in a computer-processed image rather than being viewed directly.

In some embodiments, a higher guidance level causes the processing system to emphasise one or more areas in a field of vision of the worker. This can help to focus the worker's attention on the emphasised area, which may include one of the manufacturing objects. The emphasising may comprise displaying a geometrical object to the worker, such as a line adjacent a boundary of the emphasised area, or a ring around the emphasised area, or a highlighting overlay on the emphasised area, or an animation in the vicinity of the emphasised area.

The system may determine respective guidance settings relating to a plurality of different physiological states, each of which may be used by the processing system to determine what assistance information to present to the worker. For example, a guidance setting in respect of distractedness may control the masking of one or more zones of the worker's field of view, while a guidance setting in respect of fatigue may control a level of detail level or a level of stimulus (e.g. to startle the worker into a state of higher alertness) in the assistance information.

The state of a manufacturing task may determine the displayed assistance information in any appropriate way. The position of one or more of the objects involved in the manufacturing task may determine what assistance information is displayed. The processing system may use a current position in a sequence of manufacturing steps, output by the task-state sensing system, to determine what assistance information is displayed. For example, a sequence of instructions may be displayed to the worker, over time, synchronised with the actual progress of the manufacturing task. This can ensure that relevant information is always displayed at any given moment, in contrast to a sequence of instructions being presented at a fixed temporal rate.

The electronic processing system may, of course, also take account of other inputs when determining what assistance information to display to the worker. In particular, in some embodiments, the electronic processing system is configured to determine what assistance information to display to the worker in further dependence on a proficiency setting for the worker. The proficiency setting may be represented by one or more values stored in a memory of the electronic processing system.

The proficiency setting may represent a level of skill and/or experience of the worker. Respective proficiency levels or settings may be stored for a plurality of workers; the settings may differ between the workers. The proficiency setting may set by a human operator, such as a manager or human-resources officer, or it may be determined by the system—e.g., based on sensor feedback relating to the worker's performance at the manufacturing task or at other manufacturing tasks, such as speed of completing the task and/or an error rate—or it may be set based on human and computer-based inputs. The proficiency setting may be represented by a value on a continuous or discrete scale of possible values; the scale may comprise at least two, three or more values. In some embodiments, a higher proficiency level may cause the electronic processing system to implement one or more of the behaviours disclosed above in respect to lower guidance levels (and vice versa for lower proficiency level). The processing system may set the guidance setting in dependence on the proficiency setting.

The proficiency setting may be stored persistently in memory—e.g., persisting over a plurality of performances of the manufacturing tasks, or over a plurality of hours, day, weeks or months. This does not, however, mean that the proficiency setting must always remain constant; on the contrary, it may be adjusted at times. However, it is preferably not re-initialised to a default value between instances of the manufacturing tasks, or each day or week.

The applicant has recognised that the use of a proficiency setting in this way may be beneficial even in systems that do not necessarily determine a guidance setting for the worker from a detected physiological state of the worker.

Thus, from a further aspect, the invention provides a system for assisting a worker in performing manufacturing tasks, the system comprising:an augmented-reality display system;a task-state sensing system for determining a state of a manufacturing task involving one or more objects;a memory for storing information representing a proficiency setting for the worker; andan electronic processing system;
wherein:the electronic processing system is configured to determine assistance information to display to the worker in dependence on the state of the manufacturing task and also in dependence on a proficiency setting for the worker stored in the memory; andthe augmented-reality display system is configured to display the determined assistance information to the worker in a controlled spatial relationship to the one or more objects.

From another aspect, the invention provides a method of assisting a worker in performing a manufacturing task involving one or more objects, the method comprising:determining a state of the manufacturing task;retrieving stored information representing a proficiency setting for the worker;determining, on an electronic processing system, assistance information to display to the worker in dependence on the state of the manufacturing task and also in dependence on the retrieved proficiency setting for the worker; anddisplaying the determined assistance information to the worker in a controlled spatial relationship to the one or more objects through an augmented-reality display system.

Any of the features disclosed with reference to the earlier aspects may be features of embodiments of these aspects also, in any appropriate combinations. The system may determine a guidance setting, as disclosed previously, or may determine what assistance information to display based on one or more factors already disclosed herein, but without necessarily calculating an explicit guidance setting.

The task-state sensing system, physiological-state-sensing system and electronic processing system may comprise any number of individual processing units. Each may comprise one or more of: processors, DSPs, FPGAs, ASICs, volatile memory, non-volatile memory, inputs, outputs, wired network connections, wireless network connections, etc. Any of the processing steps disclosed herein may be performed by hardware, software or a combination of both. They may be distributed across two or more devices or locations. In some embodiments, the electronic processing system comprises a central server. In some embodiments, the electronic processing system may be implemented, at least in part, on a device carried or worn by the worker. The systems disclosed herein may comprise a plurality of task-state sensing systems, physiological-state-sensing systems, augmented-reality display devices, and/or electronic processing systems. The electronic processing system may be configured to display respective assistance information to a plurality of workers, potentially simultaneously.

Features of any aspect or embodiment described herein may, wherever appropriate, be applied to any other aspect or embodiment described herein. Where reference is made to different embodiments or sets of embodiments, it should be understood that these are not necessarily distinct but may overlap.

