BACKGROUND

Technical Field

The present disclosure relates to fields of robotics and artificial intelligence (AI). More particularly the present disclosure relates to computerized robotic systems employing a coupling device for coupling one or more objects to a robotic system, and a locking mechanism for locking the one or more objects with the robotic system. Further, the present disclosure also relates to integration of marker system with the robotic system, for grasping and interacting with the one or more objects. Furthermore, the present disclosure also relates to integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.

Background Art

Research and development in robotics have been undertaken for decades, but the progress has been mostly in the field of heavy industrial applications such as automobile manufacturing automation or military applications. Simple robotics systems have been designed for the consumer markets, but they have not seen a wide application in the home-consumer robotics space, thus far. With advances in technology, combined with a population with higher incomes, the market may be ripe to create opportunities for technological advances to improve people's lives. Robotics has continued to improve automation technology with enhanced artificial intelligence and emulation of human skills and tasks in many forms in operating a robotic apparatus or a humanoid.

The notion of robots replacing humans in certain areas and executing tasks that humans would typically perform is an ideology in continuous evolution since robots were first developed in the 1970s. Manufacturing sectors have long used robots in teach-playback mode, where the robot is taught, via pendant or offline fixed-trajectory generation and download, which motions to copy continuously and without alteration or deviation. Companies have taken the pre-programmed trajectory-execution of computer-taught trajectories and robot motion-playback into such application domains as mixing drinks, welding or painting cars, and others. However, all of these conventional applications use a 1:1 computer-to-robot or tech-playback principle that is intended to have only the robot faithfully execute the motion-commands, which is usually following a taught/pre-computed trajectory without deviation.

Additionally, in conventional robotic systems, one or more objects or manipulators or end-effectors are coupled directly to the robotic systems. In the conventional systems, the coupling devices are often characterized with stability issues, which may render inefficient or inaccurate operation of the robotic systems. Though, there has been improvements in the coupling devices to improve stability and accuracy of the coupling, these systems tend to be cumbersome and complex. Also, the configuration of coupling devices in the conventional systems may require the entire coupling device to be replaced or altered as per the configuration of the one or more objects or manipulators or end-effectors to be coupled to the robotic system, which is undesirable. The present disclosure is directed to overcome one or more limitations stated above.

The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgment or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

SUMMARY

Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a multi-level robotic system for high speed and high fidelity manipulation operations segmented into, in one embodiment, two physical and logical subsystems made up of instrumented, articulated and controller-actuated subsystems, including a larger and coarser-motion macro-manipulation system operating responsible for operations in larger unconstrained environment workspaces at a reduced endpoint accuracy, and a smaller and finer-motion micro-manipulation system responsible for operations in a smaller workspace and while interacting with tooling and the environment at a higher endpoint motion accuracy, carrying out mini-manipulation trajectory-following tasks based on mini-manipulation commands provided through a dual-level database specific to the macro- and micro-manipulation subsystems, supported by a dedicated and separate distributed processor and sensor architecture operating under an overall real-time operating system communicating with all subsystems over multiple bus interfaces specific to sensor, command and database-elements.

Systems and methods are provided for operating universal robotic assistant systems. In some embodiments, a method for operating robotic assistant system comprises: receiving, by one or more processors configured in a robotic assistant system, environment data corresponding to a current environment, from one or more sensors configured in the robotic assistant system; determining, by the one or more processors, a type of the current environment based on the collected environment data; detecting by the one or more processors, one or more objects in the current environment, wherein the one or more objects are associated with the type of the current environment; identifying, by the one or more processors, for each of the one or more objects, one or more interactions based on type of the one or more objects and the type of the current environment; retrieving, by the one or more processors, interaction data corresponding to the one or more objects from a remote storage associated with the robotic assistant system; and executing, by the one or more processors, the one or more interactions on the corresponding one or more objects, based on the interaction data.

In some embodiments, determining the type of the current environment includes transmitting, by the one or more processors, the environment data to a remote storage associated with the universal robotic assistant systems, wherein the remote storage comprises a library of environment candidates; and receiving, by the one or more processors, in response to the transmitted environment data, the type of the current environment determined based on the environment data, from among the library of environment candidates.

In some embodiments, each of the one or more processors is communicatively connected to a central processor associated with the robotic assistant system.

In some embodiments, the environment data includes position data and image data of the current environment.

In some embodiments, the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices.

In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.

In some embodiments, the one or more objects are detected from a plurality of objects associated with the type of the current environment, wherein the plurality of objects are retrieved from a remote storage.

In some embodiments, the object data is collected by the one or more sensors comprising one or more cameras.

In some embodiments, detecting the one or more objects and the type of the one or more objects further comprises analysing features of the one or more objects, wherein the features comprises at least one of shape, size, texture, color, state, material and pose of the one or more objects.

In some embodiments, analyzing the features of the one or more objects includes detecting one or more markers disposed on each of the one or more objects.

In some embodiments, the one or more interactions identified for each of the one or more objects based on the type of objects and the type of the current environment indicates the one or more interactions to be performed by the respective object or on the respective object within the current environment.

In some embodiments, the interaction data of each of the one or more interactions comprises a sequence of motions to be performed by or on the one or more objects and one or more optimal standard positions of one or more manipulation devices, configured to interact with the one or more objects, relative to the corresponding one or more objects.

In some embodiments, executing at least one of the one or more interactions on the corresponding one or more objects includes, for each of the one or more interactions: positioning, by the one or more processors, one or more manipulation devices within a proximity of the corresponding one or more objects; identifying, by the one or more processors, an optimal standard position of the one or more manipulation devices relative to the corresponding one or more objects, wherein the optimal standard position is selected from one or more standard positions of the one or more manipulation devices; positioning, by the one or more processors, the one or more manipulation devices at the identified optimal standard position using one or more positioning techniques; and executing, by the one or more processors, using the one or more manipulation devices, the one or more interactions on the corresponding one or more objects.

In some embodiments, the one or more positioning techniques includes at least one of object template matching technique and marker-based technique, wherein the object template matching technique is used for standard objects and the marker-based technique is used for standard and non-standard objects.

In some embodiments, positioning one or more manipulation devices at an optimal standard position using the object template matching technique includes: retrieving, by the one or more processors, an object template of a target object from a remote storage associated with the universal robotic assistant system, wherein the target object is an object currently being subjected to one or more interactions, wherein the object template comprises at least one of shape, color, surface and material characteristics of the target object; positioning, by the one or more processors, the one or more manipulation devices to a first position proximal to the target object; receiving, by the one or more processors, one or more images, in real-time, of the target object from at least one image capturing device associated with the one or more manipulation devices, wherein the one or more images are captured by at least one image capturing device when the one or more manipulation devices are at the first position; comparing, by the one or more processors, the object template of the target object with the one or more images of the target object; and performing, by the one or more processors, at least one of: adjusting position of the one or more manipulation devices towards the optimal standard position based on position of the one or more manipulation devices in previous iteration and reiterating the steps of receiving and comparing, when the comparison results in mismatch; or inferring that the one or more manipulation devices reached the optimal standard position when the comparison results in a match.

In some embodiments, positioning one or more manipulation devices at optimal standard position using the marker-based technique includes: detecting one or more markers associated with a target object; and adjusting position of the one or more manipulation devices towards the optimal standard position based on the detected one or more markers associated with the target object, wherein the position is adjusted using a real-time image of the target objected received from at least one image capturing device associated with the one or more manipulation devices.

In some embodiments, the one or more markers includes at least one of: a physical marker disposed on the target object; and a virtual marker corresponding to one or more points on the target object, wherein the one or more markers enable computation of position parameters comprising distance, orientation, angle, and slope, of the one or more manipulation devices with respect to the target object.

In some embodiments, the one or more markers associated with the target object are physical markers when the target object is a standard object and the one or more markers associated with the target object are virtual markers when the target object is a non-standard object.

In some embodiments, the one or more markers include the physical marker disposed on the target object, wherein the physical marker is a triangle-shaped marker, and wherein adjusting position of the one or more manipulation devices includes: moving, by the one or more processors, the one or more manipulation devices towards the triangle-shaped marker until at least one side of the triangle-shaped marker has a preferred length; rotating, by the one or more processors, the one or more manipulation devices until a bottom vertex of the triangle-shaped marker is disposed in a bottom position of the real-time image of the target object; shifting, by the one or more processors, the one or more manipulation devices along an X and/or Y axis of the real-time image of the target object until a center of the triangle-shaped marker is in a center position of the real-time image of the target object; and adjusting, by the one or more processors, a slope of the one or more manipulation devices until each angle of the triangle-shaped marker are at least one of equal to approximately 60 degrees or equal to a predetermined maximum difference between the angles that is smaller than their difference prior to initiating the adjustment of the position of the one or more manipulation devices, wherein achieving at least one of the two conditions mentioned above, indicates that the one or more manipulation devices reached the optimal standard position.

In some embodiments, the one or more markers include the physical marker disposed on the target object, wherein the physical marker is a chessboard-shaped marker, and wherein adjusting position of the one or more manipulation devices includes: calibrating, by the one or more processors, each image capturing device associated with the one or more manipulation devices using the chessboard-shaped marker, wherein the calibration comprises estimating at least one of focus length, principal point and distortion coefficients of each image capturing device with respect to the chessboard-shaped marker; identifying, by the one or more processors, in real-time, images of the target object and image co-ordinates of corners of square slots in the chessboard-shaped marker; assigning, by the one or more processors, real-world coordinates to each internal corner among the corners of the square slots in the real-time image based on the image co-ordinates; and determining, by the one or more processors, position of the one or more manipulation devices based on the calibration, image co-ordinates and the real-time co-ordinates with respect to the chessboard-shaped marker, wherein the steps of calibrating, identifying, assigning and determining are repeated until the position of the one or more manipulation devices is equal to the optimal standard position.

In some embodiments, the virtual markers are placed on the target object using at least one of shape analysis technique, particle filtering technique and Convolutional Neural Network (CNN) technique.

In some embodiments, placing the virtual markers using shape analysis technique includes: receiving, by the one or more processors, real-time images of a target object from at least one image capturing device associated with one or more manipulating devices; determining, by the one or more processors, shape of the target object and longest and shortest sides of the target object. The sides of the target object are determined as longest and shortest with reference to length of each side of the target object; determining, by the one or more processors, geometric centre of the target object based on the shape of the target object and, the longest and the shortest sides of the target object; and projecting, by the one or more processors, an equilateral triangle on the target object, wherein each side of the equilateral triangle is equal to half of the shortest side of the target object; the equilateral triangle is oriented along the longest side of the target object; and geometric centre of the equilateral triangle is coinciding with the geometric centre of the target object; and placing, by the one or more processors, the virtual markers at each vertex of the equilateral triangle.

In some embodiments, placing the virtual markers using particle filtering technique includes: retrieving, by the one or more processors, one or more ideal values corresponding to ideal positions of a target object from a remote storage associated with the universal robotic assistant systems; receiving, by the one or more processors, real-time images of the target object from at least one image capturing device associated with one or more manipulating devices; generating, by the one or more processors, special points within boundaries of the target object using the real-time images; determining, by the one or more processors, an estimated value for combination of visual features in neighborhood of each special point, wherein the visual features comprises at least one of histograms of gradients, spatial color distributions and texture features; comparing, by the one or more processors, each estimated value with each of the one or more ideal values to identify respective proximal match; and placing, by the one or more processors, the virtual markers at each position on the target object corresponding to each proximal match.

In some embodiments, placing the virtual markers using the CNN technique includes: downloading, by the one or more processors, a CNN model corresponding to a target object from libraries stored in a remote storage associated with the universal robotic assistant systems; and detecting positions on the target object for placing the virtual markers based on the CNN model.

Hereafter, various embodiments of the present disclosure are explained in terms of a kitchen environment. However, this should not be construed as a limitation of the present disclosure, as the present disclosure may be applicable to any environment other than a kitchen environment.

Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food dish with substantially the same result as if a chef had prepared the food dish. In a first embodiment, the robotic assistant system in a standardized robotic kitchen comprises two robotic arms and hands that replicate the precise movements of the chef in same sequence (or substantially the same sequence). The two robotic arms and hands replicate the movements in the same timing (or substantially the same timing) to prepare the food dish based on a previously recorded document (a recipe-script) of the chef's precise movements in preparing the same food dish. In a second embodiment, a computer-controlled cooking apparatus prepares a food dish based on a sensory-curve, such as temperature over time, which was previously recorded in a software file where the chef prepared the same food dish with the cooking apparatus with sensors for which a computer recorded the sensor values over time when the chef previously prepared the food dish on the cooking apparatus fitted with the sensors. In a third embodiment, the kitchen apparatus comprises the robotic arms in the first embodiment and the cooking apparatus with sensors in the second embodiment to prepare a dish that combines both the robotic arms and one or more sensory curves, where the robotic arms are capable of quality-checking a food dish during the cooking process, for such characteristics as taste, smell, and appearance, allowing for any cooking adjustments to the preparation steps of the food dish. In a fourth embodiment, the kitchen apparatus comprises a food storage system with computer-controlled containers and container identifiers for storing and supplying ingredients for a user to prepare the food dish by following the chef's cooking instructions. In a fifth embodiment, a robotic kitchen comprises a robotic assistant system with arms and a kitchen apparatus in which the robotic assistant system moves around the kitchen apparatus to prepare a food dish by emulating a chef's precise cooking movements, including possible real-time modifications/adaptations to the preparation process defined in the recipe-script.

A robotic cooking engine comprises detection, recording, and chef emulation cooking movements, controlling significant parameters, such as temperature and time, and processing the execution with designated appliances, equipment, and tools, thereby reproducing a gourmet dish that tastes identical to the same dish prepared by a chef and served at a specific and convenient time. In one embodiment, a robotic cooking engine provides robotic arms for replicating a chef's identical movements with the same ingredients and techniques to produce an identical tasting dish.

The underlying motivation of the present disclosure centers around humans being monitored with sensors during their natural execution of an activity, and then, being able to use monitoring-sensors, capturing-sensors, computers, and software to generate information and commands to replicate the human activity using one or more robotic and/or automated systems. While one can conceive of multiple such activities (e.g. cooking, painting, playing an instrument, etc.), one aspect of the present disclosure is directed to the cooking of a meal: in essence, a robotic meal preparation application. Monitoring a human chef is carried out in an instrumented application-specific setting (a standardized kitchen in this case), and involves using sensors and computers to watch, monitor, record, and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in an environment that is capable of allowing a robotic or automated system in a robotic kitchen prepare the same dish to the standards and quality as the dish prepared by the human chef.

The use of multimodal sensing systems is the means by which the necessary raw data is collected. Sensors capable of collecting and providing such data include environment and geometrical sensors, such as two-dimensional (cameras, etc.) and three-dimensional (lasers, sonar, etc.) sensors, as well as human motion-capture systems (human-worn camera-targets, instrumented suits/exoskeletons, instrumented gloves, etc.), as well as instrumented (sensors) and powered (actuators) equipment used during recipe creation and execution (instrumented appliances, cooking-equipment, tools, ingredient dispensers, etc.). All this data is collected by one or more distributed/central computers and processed by various processes. The processors of the distributed/central computers will process and abstract the data to the point that a human and a computer-controlled robotic kitchen can understand the activities, tasks, actions, equipment, ingredients and methods, and processes used by the human, including replication of key skills of a particular chef. The raw data is processed by one or more software abstraction engines to create a recipe-script that is both human-readable and, through further processing, machine-understandable and machine-executable, spelling out all actions and motions for all steps of a particular recipe that a robotic kitchen would have to execute. These commands range in complexity from controlling individual joints, to a particular joint-motion profile over time, to abstraction levels of commands, with lower-level motion-execution commands embedded therein, associated with specific steps in a recipe. Abstraction motion-commands (e.g. “crack an egg into the pan”, “sear to a golden color on both sides”, etc.) can be generated from the raw data, refined, and optimized through a multitude of iterative learning processes, carried out live and/or off-line, allowing the robotic kitchen systems to successfully deal with measurement-uncertainties, ingredient variations, etc., enabling complex (adaptive) minimanipulation motions using fingered-hands mounted to robot-arms and wrists, based on fairly abstraction/high-level commands (e.g. “grab the pot by the handle”, “pour out the contents”, “grab the spoon off the countertop and stir the soup”, etc.).

The ability to create machine-executable command sequences, now contained within digital files capable of being shared/transmitted, allowing any robotic kitchen to execute them, opens up the option to execute the dish-preparation steps anywhere at any time. Hence, it allows the option to buy/sell recipes online, allowing users to access and distribute recipes on a per-use or subscription basis.

The replication of a dish prepared by a human is performed by a robotic kitchen, which is in essence a standardized replica of the instrumented kitchen used by the human chef during the creation of the dish, except that the human's actions are now carried out by a set of robotic arms and hands, computer-monitored and computer-controllable appliances, equipment, tools, dispensers, etc. The degree of dish-replication fidelity will thus be closely tied to the degree to which the robotic kitchen is a replica of the kitchen (and all its elements and ingredients), in which the human chef was observed while preparing the dish.

Broadly stated, a humanoid having a robot computer controller operated by robot operating system (ROS) with robotic instructions comprises a database having a plurality of electronic minimanipulation libraries, each electronic minimanipulation library including a plurality of minimanipulation elements. The plurality of electronic minimanipulation libraries can be combined to create one or more machine executable application-specific instruction sets, and the plurality of minimanipulation elements within an electronic minimanipulation library can be combined to create one or more machine executable application-specific instruction sets; a robotic structure having an upper body and a lower body connected to a head through an articulated neck, the upper body including torso, shoulder, arms, and hands; and a control system, communicatively coupled to the database, a sensory system, a sensor data interpretation system, a motion planner, and actuators and associated controllers, the control system executing application-specific instruction sets to operate the robotic structure.

In addition, embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus for executing robotic instructions from one or more libraries of minimanipulations. Two types of parameters, elemental parameters and application parameters, affect the operations of minimanipulations. During the creation phase of a minimanipulation, the elemental parameters provide the variables that test the various combinations, permutations, and the degrees of freedom to produce successful minimanipulations. During the execution phase of minimanipulations, application parameters are programmable or can be customized to tailor one or more libraries of minimanipulations to a particular application, such as food preparation, making sushi, playing piano, painting, picking up a book, and other types of applications.

Minimanipulations comprise a new way of creating a general programmable-by-example platform for humanoid robots. The state of the art largely requires explicit development of control software by expert programmers for each and every step of a robotic action or action sequence. The exception to the above are for very repetitive low-level tasks, such as factory assembly, where the rudiments of learning-by-imitation are present. A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks, such as cooking, taking care of the infirm, or other tasks performed by the next generation of humanoid robots. More specifically, unlike the previous art, the present disclosure provides the following distinctive features. First, a potentially very large library of pre-defined/pre-learned sensing-and-action sequences are called minimanipulations. Second, each mini-manipulation encodes preconditions required for the sensing-and-action sequences to produce successfully the desired functional results (i.e. the post conditions) with a well-defined probability of success (e.g. 100% or 97% depending on the complexity and difficulty of the minimanipulation). Third, each minimanipulation references a set of variables whose values may be set a-priori or via sensing operations, before executing the minimanipulation actions. Fourth, each minimanipulation changes the value of a set of variables to represent the functional result (the post conditions) of executing the action sequence in the minimanipulation. Fifth, minimanipulations may be acquired by repeated observation of a human tutor (e.g. an expert chef) to determine the sensing-and-action sequence, and to determine the range of acceptable values for the variables. Sixth, minimanipulations may be composed into larger units to perform end-to-end tasks, such as preparing a meal, or cleaning up a room. These larger units are multi-stage applications of minimanipulations either in a strict sequence, in parallel, or respecting a partial order wherein some steps must occur before others, but not in a total ordered sequence (e.g. to prepare a given dish, three ingredients need to be combined in exact amounts into a mixing bowl, and then mixed; the order of putting each ingredient into the bowl is not constrained, but all must be placed before mixing). Seventh, the assembly of minimanipulations into end-to-end-tasks is performed by robotic planning, taking into account the preconditions and post conditions of the component minimanipulations. Eighth, case-based reasoning wherein observation of humans performing end-to-end tasks, or other robots doing so, or the same robot's past experience can be used to acquire a library of reusable robotic plans form cases (specific instances of performing an end-to-end task), both successful ones to replicate, and unsuccessful ones to learn what to avoid.

In a first aspect of the present disclosure, the robotic apparatus performs a task by replicating a human-skill operation, such as food preparation, playing piano, or painting, by accessing one or more libraries of minimanipulations. The replication process of the robotic apparatus emulates the transfer of a human's intelligence or skill set through a pair of hands, such as how a chef uses a pair of hands to prepare a particular dish; or a piano maestro playing a master piano piece through his or her pair of hands (and perhaps through the feet and body motions, as well). In a second aspect of the present disclosure, the robotic apparatus comprises a humanoid for home applications where the humanoid is designed to provide a programmable or customizable psychological, emotional, and/or functional comfortable robot, and thereby providing pleasure to the user. In a third aspect of the present disclosure, one or more minimanipulation libraries are created and executed as, first, one or more general minimanipulation libraries, and second, as one or more application specific minimanipulation libraries. One or more general minimanipulation libraries are created based on the elemental parameters and the degrees of freedom of a humanoid or a robotic apparatus. The humanoid or the robotic apparatus are programmable, so that the one or more general minimanipulation libraries can be programmed or customized to become one or more application specific minimanipulation libraries specific tailored to the user's request in the operational capabilities of the humanoid or the robotic apparatus.

Some embodiments of the present disclosure are directed to the technical features relating to the ability of being able to create complex robotic humanoid movements, actions and interactions with tools and the environment by automatically building movements for the humanoid, actions, and behaviors of the humanoid based on a set of computer-encoded robotic movement and action primitives. The primitives are defined by motion/actions of articulated degrees of freedom that range in complexity from simple to complex, and which can be combined in any form in serial/parallel fashion. These motion-primitives are termed to be Minimanipulations (MMs) and each MM has a clear time-indexed command input-structure, and output behavior-/performance-profile that are intended to achieve a certain function. MMs can range from the simple (‘index a single finger joint by 1 degree’) to the more involved (such as ‘grab the utensil’) to the even more complex (‘fetch the knife and cut the bread’) to the fairly abstract (‘play the 1stbar of Schubert's piano concerto #1’).

Thus, MMs are software-based and represented by input and output data sets and inherent processing algorithms and performance descriptors, akin to individual programs with input/output data files and subroutines, contained within individual run-time source-code, which when compiled generates object-code that can be compiled and collected within various different software libraries, termed as a collection of various Minimanipulation-Libraries (MMLs). MMLs can be grouped in to multiple groupings, whether these be associated to (i) particular hardware elements (finger/hand, wrist, arm, torso, foot, legs, etc.), (ii) behavioral elements (contacting, grasping, handling, etc.), or even (iii) application-domains (cooking, painting, playing a musical instrument, etc.). Furthermore, within each of these groupings, MMLs can be arranged based on multiple levels (simple to complex) relating to the complexity of behavior desired.

It should thus be understood that the concept of Minimanipulation (MM) (definitions and associations, measurement and control variables and their combinations and value-usage and—modification, etc.) and its implementation through usage of multiple MMLs in a near infinite combination, relates to the definition and control of basic behaviors (movements and interactions) of one or more degrees of freedom (movable joints under actuator control) at levels ranging from a single joint (knuckle, etc.) to combinations of joints (fingers and hand, arm, etc.) to ever higher degree of freedom systems (torso, upper-body, etc.) in a sequence and combination that achieves a desirable and successful movement sequence in free space and achieves a desirable degree of interaction with the real world so as to be able to enact a desirable function or output by the robot system, on and with, the surrounding world via tools, utensils, and other items.

Examples for the above definition can range from (i) a simple command sequence for a digit to flick a marble along a table, through (ii) stirring a liquid in a pot using a utensil, to (iii) playing a piece of music on an instrument (violin, piano, harp, etc.). The basic notion is that MMs are represented at multiple levels by a set of MM commands executed in sequence and in parallel at successive points in time, and together create a movement and action/interaction with the outside world to arrive at a desirable function (stirring the liquid, striking the bow on the violin, etc.) to achieve a desirable outcome (cooking pasta sauce, playing a piece of Bach concerto, etc.).

The basic elements of any low-to-high MM sequence comprise movements for each subsystem, and combinations thereof are described as a set of commanded positions/velocities and forces/torques executed by one or more articulating joints under actuator power, in such a sequence as required. Fidelity of execution is guaranteed through a closed-loop behavior described within each MM sequence and enforced by local and global control algorithms inherent to each articulated joint controller and higher-level behavioral controllers.

Implementation of the above movements (described by articulating joint positions and velocities) and environment interactions (described by joint/interface torques and forces) is achieved by having computer playback desirable values for all required variables (positions/velocities and forces/torques) and feeding these to a controller system that faithfully implements them on each joint as a function of time at each time step. These variables and their sequence and feedback loops (hence not just data files, but also control programs), to ascertain the fidelity of the commanded movement/interactions, are all described in data-files that are combined into multi-level MMLs, which can be accessed and combined in multiple ways to allow a humanoid robot to execute multiple actions, such as cooking a meal, playing a piece of classical music on a piano, lifting an infirm person into/out of a bed, etc. There are MMLs that describe simple rudimentary movement/interactions, which are then used as building-blocks for ever higher-level MMLs that describe ever-higher levels of manipulation, such as ‘grasp’, ‘lift’, ‘cut’ to higher level primitives, such as ‘stir liquid in pot’/‘pluck harp-string to g-flat’ or even high-level actions, such as ‘make a vinaigrette dressing’/‘paint a rural Brittany summer landscape’/‘play Bach's Piano-concerto #1’, etc. Higher level commands are simply a combination towards a sequence of serial/parallel lower- and mid-level MM primitives that are executed along a common timed stepped sequence, which is overseen by a combination of a set of planners running sequence/path/interaction profiles with feedback controllers to ensure the required execution fidelity (as defined in the output data contained within each MM sequence).

The values for the desirable positions/velocities and forces/torques and their execution playback sequence(s) can be achieved in multiple ways. One possible way is through watching and distilling the actions and movements of a human executing the same task, and distilling from the observation data (video, sensors, modeling software, etc.) the necessary variables and their values as a function of time and associating them with different minimanipulations at various levels by using specialized software algorithms to distill the required MM data (variables, sequences, etc.) into various types of low-to-high MMLs. This approach would allow a computer program to automatically generate the MMLs and define all sequences and associations automatically without any human involvement.

Another way would be (again by way of an automated computer-controlled process employing specialized algorithms) to learn from online data (videos, pictures, sound logs, etc.) how to build a required sequence of actionable sequences using existing low-level MMLs to build the proper sequence and combinations to generate a task-specific MML.

Yet another way, although most certainly more (time-) inefficient and less cost-effective, might be for a human programmer to assemble a set of low-level MM primitives to create an ever-higher level set of actions/sequences in a higher-level MML to achieve a more complex task-sequence, again composed of pre-existing lower-level MMLs.

Modification and improvements to individual variables (meaning joint position/velocities and torques/forces at each incremental time-interval and their associated gains and combination algorithms) and the motion/interaction sequences are also possible and can be effected in many different ways. It is possible to have learning algorithms monitor each and every motion/interaction sequence and perform simple variable-perturbations to ascertain outcome to decide on if/how/when/what variable(s) and sequence(s) to modify in order to achieve a higher level of execution fidelity at levels ranging from low- to high-levels of various MMLs. Such a process would be fully automatic and allow for updated data sets to be exchanged across multiple platforms that are interconnected, thereby allowing for massively parallel and cloud-based learning via cloud computing.

Advantageously, the robotic apparatus in a standardized robotic kitchen has the capabilities to prepare a wide array of cuisines from around the world through a global network and database access, as compared to a chef who may specialize in one type of cuisine. The standardized robotic kitchen also is able to capture and record favorite food dishes for replication by the robotic apparatus whenever desired to enjoy the food dish without the repetitive process of laboring to prepare the same dish repeatedly.

The structures and methods of the present disclosure are disclosed in detail in the description below. This summary does not purport to define the disclosure. The disclosure is defined by the claims. These and other embodiments, features, aspects, and advantages of the disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings.

In some embodiments, an electronic inventory system comprises a storage unit configured to store one or more objects; one or more image capturing devices configured in the storage unit to: capture one or more images of each of the one or more objects, in real-time; and transmit each of the one or more images to a display screen configured on the storage unit and one or more embedded processors configured in the storage unit; one or more sensors configured in the storage unit to provide corresponding sensor data to at least one of the one or more embedded processors associated with position and orientation of each of the one or more objects; one or more light sources configured in the storage unit to facilitate the one or more image capturing devices in capturing one or more images of each of the one or more objects in the storage unit, by providing uniform illumination in the storage unit; one or more embedded processors configured in the storage unit, wherein the one or more embedded processors interact with a central processor of the robotic assistant system through a communication network, configured to: detect each of the one or more objects stored in the storage unit based on the one or more images and the sensor data; and transmit the one or more images and the sensor data to the central processor in real-time or periodically.

In some embodiments, the one or more sensors comprises at least one of a temperature sensor, a humidity sensor, an ultrasound sensor, a laser measurement sensors and SONAR.

In some embodiments, the one or more embedded processors detect each of the one or more objects by detecting presence/absence of the one or more objects, estimating content stored in the one or more objects, detecting position and orientation of each of the one or more objects, reading at least one of visual markers and radio type markers attached to each of the one or more objects and reading object identifiers.

In some embodiments, the one or more embedded processors detect the one or more objects based on Convolutional Neural Network (CNN) techniques.

In some embodiments, the storage unit comprises a display screen fixed on external surface of the storage, configured to display images and videos of the one or more objects and one or more interactions performed on each of the one or more objects, in real-time.

In some embodiments, the display screen enables a user to visualize and locate each of the one or more objects stored in the storage unit, without opening doors of the storage unit.

In some embodiments, the storage unit is further configured with motor devices to enable performing one or more actions on doors of the storage unit, automatically, wherein the one or more actions comprise at least one of opening, closing, locking and unlocking the doors of the storage unit.

In some embodiments, each of the one or more sensors, each of the one or more light sources and each of the one or more image capturing devices of the storage unit are electrically connected to an extension board configured in the storage unit, wherein extension board of each storage unit is connected to a Power over Ethernet (PoE) switch.

In some embodiments, the storage unit is further configured with a fan block for providing air circulation inside the storage unit and a thermoelectric cooler element to cool electric components in the storage unit.

In one non-limiting embodiment of the present disclosure, a coupling device for coupling one or more objects to a robotic system is provided. The coupling device comprising a first coupling member defined onto the robotic system and a second coupling member defined onto the one or more objects, and connectable with the first coupling member. A locking mechanism is defined at an interface of each of the first coupling member and the second coupling member, for coupling the one or more objects with the robotic system.

In an embodiment, the first coupling member is defined by a first connection surface connectable to the robotic system and a first mating surface defined with a plurality of first projections along its periphery.

In an embodiment, the second coupling member is defined by a second connection surface connectable to the one or more objects and a second mating surface defined with a plurality of second projections along its periphery.

In an embodiment, the plurality of first projections and the plurality of second projections are complementary to each other to facilitate coupling of the first coupling member with the second coupling member.

In an embodiment, the first connection surface is connectable with the robotic system by at least one of a mechanical means, an electromechanical means, a vacuum means and a magnetic means.

In an embodiment, the second connection surface is connectable to the one or more objects by at least one of the mechanical means, the electromechanical means, the vacuum means and the magnetic means.

In an embodiment, the material of the first coupling member and the second coupling member are selected to facilitate joining between the first mating surface and the second mating surface.

In an embodiment, the first coupling member is made of either of an electromagnetic material or a ferromagnetic material.

In an embodiment, the second coupling member is made either of the ferromagnetic material or the electromagnetic material.

In an embodiment, an interface port is defined on the first coupling member and interfaced to the robotic system, for peripheral connection between the robotic and the second coupling member, to facilitate manipulation of the one or more objects by the robotic system.

In an embodiment, each of the one or more objects is at least one of a kitchen appliance and a kitchen tool.

In an embodiment, at least one sensor unit is defined in the robotic system, wherein the at least one sensor unit is configured to detect orientation of the plurality of first projections with the plurality of second projections, during coupling of the first coupling member with the second coupling member.

In an embodiment, the locking mechanism comprises at least one notch defined on the first mating surface and at least one protrusion defined on the second mating surface. The at least one protrusion is adapted to engage with the at least one notch for coupling the first mating surface with the second mating surface.

In an embodiment, the at least one protrusion is shaped corresponding to the configuration of the at least one notch.

In an embodiment, the locking mechanism comprises at least one notch defined on the second mating surface and at least one protrusion defined on the first mating surface. The at least one protrusion is adapted to engage with the at least one notch for coupling the first mating surface with the second mating surface.

In an embodiment, the at least one notch is shaped in at least one of a triangular shape, a circular shape, and a polygonal shape.

In another non-limiting embodiment of the present disclosure, a coupling device for coupling one or more objects to a robotic system is provided. The coupling device comprising a first coupling member defined onto the robotic system and a second coupling member defined onto the one or more objects, and connectable with the first coupling member. A locking mechanism is defined at an interface of each of the first coupling member and the second coupling member, for coupling the one or more objects with the robotic system. The locking mechanism comprises at least one triangular notch defined on either of the first coupling member and the second coupling member and at least one triangular protrusion defined on the corresponding first coupling member and the second coupling member. The at least one triangular protrusion is adapted to engage with the at least one triangular notch for coupling the first coupling member with the second coupling member.

In another non-limiting embodiment of the present disclosure, a coupling device for coupling one or more objects to a robotic system is provided. The coupling device comprises a first coupling member defined onto the robotic system and a second coupling member defined onto the one or more objects, and connectable with the first coupling member. A locking mechanism is defined at an interface of each of the first coupling member and the second coupling member, for coupling the one or more objects with the robotic system. The locking mechanism comprises at least one circular notch defined on either of the first coupling member and the second coupling member and at least one circular protrusion defined on the corresponding first coupling member and the second coupling member. The locking mechanism is adapted to engage with the at least one circular notch for coupling the first coupling member with the second coupling member.

In another non-limiting embodiment of the present disclosure, a coupling device for coupling one or more objects to a robotic system is provided. The coupling device comprises a first coupling member defined onto the robotic system and a second coupling member defined onto the one or more objects, and connectable with the first coupling member. A locking mechanism defined at an interface of each of the first coupling member and the second coupling member, for coupling the one or more objects with the robotic system. The locking mechanism comprises at least one notch defined on either of the first coupling member and the second coupling member, wherein each of the at least one notch is configured to receive an electromagnet. Also, at least one protrusion is defined on the corresponding first coupling member and the second coupling member and adapted to engage with the electromagnet in the at least one notch for coupling the first coupling member with the second coupling member.

In an embodiment, the at least one protrusion is made of ferromagnetic material for joining with the electromagnet in the at least one notch.

In an embodiment, the at least one notch includes a groove defined along its periphery.

In an embodiment, the at least one protrusion includes a pin, shaped corresponding to the configuration of the groove in the at least one notch and adapted to engage with the groove for improving stability of the coupling between the first coupling member and the second coupling member.

In an embodiment, a locking mechanism for securing one or more objects to a robotic system is provided. The locking mechanism comprising at least one first locking member fixed on a manipulator of the robotic system and at least one second locking member is mounted on the manipulator and adapted to be operable between a first position and a second position. At least one actuator assembly is associated with the at least one second locking member and adapted to operate the at least one second locking member between the first position and the second position. The at least one actuator operates the at least one second locking member from the first position to the second position, to engage each of the one or more objects between the at least one first locking member and the at least one second locking member, thereby securing the one or more objects with the robotic system.

In an embodiment, the at least one first locking member and the at least one second locking member located in a same plane of the manipulator.

In an embodiment, the at least one actuator assembly is configured on a rear surface of the manipulator.

In an embodiment, the at least one first locking member and the at least one second locking member are located on a front surface of the manipulator.

In an embodiment, each of the one or more objects includes a holding portion, defined with a plurality of slots along its periphery for engaging with the at least one first locking member and the at least one second locking member.

In an embodiment, shape of the plurality of slots corresponds to the configuration of the at least one first locking member and the at least one second locking member.

In an embodiment, the at least one actuator assembly is actuated by the robotic system, to slide the at least one second holding member from the first position to the second position, when the manipulator approaches vicinity of each of the one or more objects.

In an embodiment, the manipulator includes a guideway for guiding each of the at least one second holding member between the first position and the second position.

In an embodiment, the at least one first holding member and the at least one second holding member is a hook member.

In an embodiment, the at least one actuator assembly is selected from at least one of a linear actuator and a rotary actuator.

In an embodiment, the at least one actuator assembly comprises a lead screw mounted onto the manipulator, a motor interfaced with the robotic system and coupled to the lead screw, to axially rotate the lead screw and a nut mounted onto the lead screw and engaged with the at least one second holding means. The nut is configured to traverse along the lead screw during its axial rotation, thereby operating the at least one second holding means between the first position and the second position.

In an embodiment, a lead screw holder is provided for mounting the lead screw on the manipulator, such that the lead screw is aligned along a horizontal axis of the manipulator.

In an embodiment, the lead screw includes a plurality of threads with a lead angle ranging from about 6 degrees to about 12 degrees, to restrict movement of the nut, when the motor ceases to operate.

In an embodiment, the nut is engaged with the at least one second holding means via at least one bracket member.

In an embodiment, the nut is configured to slide the at least one second holding means from the first position to the second position, during clockwise rotation of the lead screw.

In an embodiment, the nut is configured to slide the at least one second holding means from the second position to the first position, during anti-clockwise rotation of the lead screw.

In an embodiment, the nut is configured to slide the at least one second holding means from the first position to the second position, during anti-clockwise rotation of the lead screw.

In an embodiment, the nut is configured to slide the at least one second holding means from the second position to the first position, during clockwise rotation of the lead screw.

In an embodiment, the motor is supported onto the manipulator via a clamp.

In an embodiment, the at least one second holding means, extends from the rear surface of the manipulator and protrudes over the front surface of the manipulator to position itself in the same plane as that of the at least one first holding means.

In an embodiment, the at least one actuator assembly comprises a housing mounted onto the manipulator, the housing includes a solenoid coil configured to be energized by a power source. A plunger is accommodated within the housing and suspended concentrically to the solenoid coil, wherein the plunger is adapted to be actuated by the solenoid coil in an energized condition. A frame member is mounted to the plunger and connected to the at least one second holding means. The frame member is configured to transfer actuation of the plunger to the at least one second holding means during the energized condition of the solenoid coil, thereby operating the at least one second holding means between the first position and the second position.

In an embodiment, the power source for energizing the solenoid coil is selected from at least one of an alternating current and a direct current.

In an embodiment, a damper member is provided such that, one end is fixed to the housing and another end connected to the frame member, to control movement of the frame member.

In an embodiment, the frame member includes one or more link members connected to each of the at least one second holding means.

It is to be understood that the aspects and embodiments of the disclosure described above may be used in any combination with each other. Several of the aspects and embodiments may be combined to form a further embodiment of the disclosure.

The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

The structures and methods of the present disclosure are disclosed in detail in the description below. This summary does not purport to define the disclosure. The disclosure is defined by the claims. These and other embodiments, features, aspects, and advantages of the disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings.

