This application is the National Stage Application of International Patent Application No. PCT/CN2018/082105, filed on Apr. 8, 2018, which claims priority to Chinese Patent Application No. 201710329549.9, filed on May 11, 2017, which is incorporated by reference for all purposes as if fully set forth herein.

FIELD OF THE INVENTION

The present invention relates to the field of large area surveillance, and more particularly to a large area surveillance method based on weighted double deep Q-learning and a surveillance robot.

DESCRIPTION OF THE RELATED ART

Surveillance systems such as traffic light surveillance at road intersections and security surveillance in residential areas are practically omnipresent in our daily lives. Surveillance systems integrate the knowledge about multimedia technologies, computer networks, industrial control, artificial intelligence, among other things, and may be used for security protection, information acquisition, scheduling and commanding, and the like. Surveillance systems may further provide a variety of services for production procedures and distance education. However, for some large area environments in which specific tasks need to be completed, for example, an unidentified object needs to be found, tracked, and monitored, comprehensive surveillance still cannot be conducted by using current surveillance systems. The reasons mainly lie in two aspects. In the first aspect, a surveillance camera has inherent disadvantages such as the problems of a limited surveillance area and limited information capacity. In the second aspect, the surveillance costs need to be considered. Synchronous monitoring of a plurality of locations using multiple cameras is required for large area surveillance. As a result, the surveillance is difficult, and the costs are high. In conclusion, how to monitor a large-area environment already becomes a problem to be urgently resolved.

Existing surveillance systems have the following disadvantages: (1). The transmission distance of an analog video signal over a cable is limited. When the transmission distance is greater than 1 km, problems such as signal attenuation, distortion, and group delay occur easily, and the picture quality is severely degraded. (2). Information is generally stored in the form of video records in most surveillance systems. However, the capacity is limited by using such a method, and information needs to be regularly uploaded or storage media need to be regularly replaced. As a result, automated surveillance cannot be implemented. (3). Surveillance using a plurality of cameras usually means excessively high costs. Moreover, the synchronization of a plurality of devices has not been considered for existing surveillance cameras.

SUMMARY OF THE INVENTION

The object of the present invention is to provide a large area surveillance method based on weighted double deep Q-learning, so that not only the problem of failing to carry out comprehensive surveillance using a surveillance system because of an excessively large surveillance area is resolved, but also the problem of limited camera capacity is resolved. Moreover, a deep estimation network is used to directly perform feature extraction on image information and determine whether an unidentified object appears in a surveillance area, where features are directly correlated to a Q-value table, so that the synchronization of a plurality of cameras is avoided, and thus the costs are reduced.

To achieve the above purpose, the present invention provides the following technical solutions.

In one aspect, the invention provides a large area surveillance method based on weighted double deep Q-learning, and the method includes the following steps:

S1. providing a large space and a robot in the large space, wherein the robot in a working state reaches a target state from a current state by using a double Q-learning method, a Q-value table of the robot includes a QA-value table and a QB-value table, and a Q-value is calculated by using a deep estimation network parameter θ, where

an update formula of a QA-value is as follows:

⁢βA=|QB⁡(s′,a*;θ)-QB⁡(s′,aL;θ)|c+|QB⁡(s′,a*;θ)-QB⁡(s′,aL;θ)|,δ=R⁡(s,a)+γ⁡[βA⁢QA⁡(s′,a*;θ)+(1-βA)⁢QA⁡(s′,a*;θ)]-QA⁡(s,a;θ),and⁢⁢QB←QB⁡(s,a;θ)+α⁡(s,a)⁢δ,

an update formula of a QB-value is as follows:

⁢βB=|QA⁡(s′,a*;θ)-QA⁡(s′,aL;θ)|c+|QA⁡(s′,a*;θ)-QA⁡(s′,aL;θ)|,⁢δ=R⁡(s,a)+γ⁡[βB⁢QB⁡(s′,a*;θ)+(1-βB)⁢QA⁡(s′,a*;θ)]-QB⁡(s,a;θ),⁢⁢and⁢QB←QB⁡(s,a;θ)+α⁡(s,a)⁢δ,

where βAand βB represent weights, s′ represents a next state, a represents a best action of a next state, aLrepresents a worst action of a next state, c is a free parameter and c≥0, δ represents a time differential, R represents a reward value, γ represents a target discount and 0≤γ≤1, s represents the current state, a represents a current action, α represents a learning rate, α is within a range (0, 1), and θ represents the deep estimation network parameter,

in the working state, an unidentified object is present in the large space, and

in the target state, the unidentified object is in a surveillance area of the robot;

S2. setting an initial state of the robot as the current state s by the robot;

S3. detecting and determining whether the current state s is the working state by the robot, where if not, the process turns to S4, and if yes, the process turns to S5;

S4. switching to standby mode to reach a next state s′ by the robot, where the process turns to S11;

S5. detecting and determining whether the current state s is the target state by the robot using a deep estimation network, where if not, the process turns to S6, and if yes, the process turns to S7;

S6. selecting and executing the current action a to reach a next state s′ by the robot, where the process turns to S8;

S7. selecting and executing the current action a to reach a next state s′ and monitoring the unidentified object by the robot, where the process turns to S8;

S8. obtaining the reward value R according to the next state s′ by the robot, where the process turns to S9;

S9. selectively updating the QA-value or the QB-value with equal probability by the robot, where the process turns to S10;

S10. determining whether the Q-value table of the robot converges by the robot, where if not, the process turns to S11, and if yes, the process turns to S12;

S11. resetting the next state s′ as the current state s by the robot, where the process returns to S3;

S12. formulating an optimal surveillance strategy by the robot, where the process turns to S13;

S13. resetting the next state s′ as the current state s by the robot, where the process turns to S14;

S14. detecting and determining whether the current state s is the working state by the robot, where if not, the process turns to S15, and if yes, the process turns to S16;

S15. switching to standby mode to reach a next state s′ by the robot, where the process returns to S13;

S16. detecting and determining whether the current state s is the target state by the robot, where if not, the process turns to S17, and if yes, the process turns to S18;

S17. reaching a next state s′ according to the optimal surveillance strategy by the robot, where the process returns to S13; and

S18. selecting and executing the current action a to reach a next state s′ and monitoring the unidentified object by the robot, where the process returns to S13,

where in a same large space, the Q-value, the learning rate α, and the target discount γ of the robot, the structure and parameter θ of the deep estimation network, an action selection manner, and the weight β are only initialized before the robot selects and executes the current action a for the first time.

In a weighted double deep Q-learning method, state information such as video images and feedback signals (such as a reward) provided by an environment are input in a deep estimation network. A convolutional layer neural network that may extract spatial structure information is first used to extract feature information of an important target in an image. Nonlinear transformation of a full connected layer is then used to perform classification or regression. Eventually, a Q-value of each action is generated at an output layer. In the network, a reward value and an error item are reduced to a limited range by using a network parameter θ, to ensure that the Q-value is in an appropriate range and improve the stability of the method. In the network, various improvements have been made to a conventional Q-learning algorithm, including: (1). An experience replay mechanism is used during training, and the obtained state transition samples are stored in a memory unit. During training, a small batch of samples is randomly selected from a sample pool to update the network parameter θ, so that the size of the available samples is increased, association between samples is reduced, and the algorithm stability is improved. (2). In addition to the use of a deep network to represent a current Q-value, a target network is further additionally used to generate a target Q-value. A linear combination of the current Q-value and the target Q-value is used to select an action, and a mean squared error between the current Q-value and the target Q-value is minimized to update the network parameter θ. After the target network is introduced, the target Q-value is kept unchanged with a period of time, so that the impact of value fluctuation on a training process is reduced, and the algorithm stability is improved.

In a preferable embodiment, the large space is divided into several subspaces, after selecting and executing the current action a, the robot remains still in a current subspace or moves to a subspace adjacent to the current subspace, and each subspace is not larger than the surveillance area of the robot.

In another preferable embodiment, when the robot detects the current state s, a sensor of the robot is used to acquire an approximate location lociof the unidentified object and a precise location locaof the robot, denoted as s=loci,loca.

In a further preferable embodiment, the robot acquires image information by using a camera of the robot, performs feature extraction and classification by using the deep estimation network, and determines by itself whether an unidentified object is present in the surveillance area, if yes, an alarm is raised by an alarm device of the robot.

In a specific embodiment, the robot selects the current action a in an Ú-greedy manner.

In the foregoing technical solution, when the robot selects the current action a, there is a larger probability of selecting an action represented by a maximum Q-value, and there is a smaller probability of selecting any another action.

In the foregoing technical solution, the reward value R is set as follows:

R⁡(s,a)={10loci=loca-0.1loci≠loca,
where locais a precise location of the robot, and lociis an approximate location of an unidentified object, that is, when an unidentified object is in the surveillance area of the robot, a positive reward is provided, and when the robot observes no unidentified object, a negative reward is provided.

In a preferable embodiment, the robot keeps updating the learning rate α of the robot,

α⁡(s,a)=1n⁡(s,a)0.8,
where when the robot executes the current action a, an unidentified object also moves to form a double-movement state, and n is the number of executing the action a in the double-movement state.

In another aspect, the invention further provides a large area surveillance robot based on weighted double deep Q-learning, where a Q-value table of the robot includes a QA-value table and a QB-value table, and a Q-value is calculated by using a deep estimation network parameter θ, where

an update formula of a QA-value is as follows:

⁢βA=|QB⁡(s′,a*;θ)-QB⁡(s′,aL;θ)|c+|QB⁡(s′,a*;θ)-QB⁡(s′,aL;θ)|,⁢δ=R⁡(s,a)+γ⁡[βA⁢QA⁡(s′,a*;θ)+(1-βA)⁢QB⁡(s′,a*;θ)]-QA⁡(s,a;θ),⁢⁢and⁢QA←QA⁡(s,a;θ)+α⁡(s,a)⁢δ,
an update formula of a QB-value is as follows:

⁢βB=|QA⁡(s′,a*;θ)-QA⁡(s′,aL;θ)|c+|QA⁡(s′,a*;θ)-QA⁡(s′,aL;θ)|,⁢δ=R⁡(s,a)+γ⁡[βB⁢QB⁡(s′,a*;θ)+(1-βB)⁢QA⁡(s′,a*;θ)]-QB⁡(s,a;θ),⁢⁢and⁢QB←QB⁡(s,a;θ)+α⁡(s,a)⁢δ,

where βAand βB represent weights, S′ represents a next state, a represents an optimal action of a next state, aLrepresents a worst action of a next state, c is a free parameter and c≥0, δ represents a time differential, R represents a reward value, γ represents a target discount and 0≤γ≤1, S represents the current state, a represents a current action, α represents a learning rate, α is within a range (0, 1), and θ represents the deep estimation network parameter; and

In a preferable embodiment, the robot is also provided with a sensor for detecting a precise location of the robot and an approximate location of an unidentified object in real time, and a camera for monitoring the unidentified object. The sensor and the camera are respectively electrically connected to a main control chip of the robot.

In another preferable embodiment, the robot is further provided with an alarm device which is electrically connected to the main control chip of the robot.

By means of the above technical solutions, as compared with the prior art, the present invention has the following advantages:

(1) The large area surveillance method based on weighted double deep Q-learning disclosed in the invention can desirably handle the problem of movement tracking by a robot of a target (an unidentified object) whose movement manner is uncontrolled in a large space. In a weighted double deep Q-learning method, a perceived state is determined, and a risk Q-value in the state is calculated. Subsequently, the robot is guided according to the risk Q-value to select an action to perform movement, to monitor a large area when the intention of a target (an unidentified object) is unknown and it is difficult to determine the behavior of the target (the unidentified object). The weighted double deep Q-learning method is a compromise method between a Q-learning method and a double Q-learning method and used to guide a next action of an Agent. An original Q-learning method may be used for a large-space but cannot deal with a case in which the movement manner of a target is uncontrolled. A double Q-learning method effectively resolves the problem of monitoring a target whose movement manner is uncontrolled, but cannot be applied to a large-area scenario. The weighted double deep Q-learning method can desirably deal with the problem of real-time surveillance in a large-area environment. When the weighted double deep Q-learning method is used to carry out real-time surveillance, a Q-value table needs to be updated. The Q-value table includes a QA-value table and a QB-value table. After each action, there is a 50% probability of updating a value in one of the tables. If a value in the QA-value table needs to be updated, an action having a maximum Q-value is selected from all possible next states in QAtable, and the action is then used to update values in the QA-value table and the QB-value table. Utilization ratio of the QA-value and the QB-value table depends on a weight value β. The state of an unidentified object, and the state and a current action of the robot are considered during the calculation of β. A compromise is made between a conventional Q-learning method and a double Q-learning method. In the method of the invention, a difference δ between a target estimate and a current estimate is continuously reduced to eventually reach convergence to obtain an optimal surveillance strategy that can monitor and track an unidentified object.

(2) The robot perceives the location of the robot and an approximate location of an unidentified object in a large space, and reaches a target state by using double Q-learning to carry out surveillance, to acquire image information of the unidentified object and raise an alarm according to the image information.

(3) The robot can perform navigation to track an unidentified object. Therefore, only one robot is needed to complete the surveillance of an entire dangerous area. There is no need to consider the synchronization of a plurality of monitors, and the costs are reduced.

(4) A state perceived by the robot is a state having a Markov property. A future state of a state having a Markov property is only related to a current state and is not related to a previous state. Therefore, a camera does not need to store past information and only needs to focus on the current locations of the robot and an unidentified object. Therefore, the problem of limited information capacity of a camera is resolved, and the costs are reduced.

(5) When the robot performs navigation to track an unidentified object, a reward value is received, and a good case and a bad case can be effectively distinguished according to the reward value, so that the learning speed of the robot is improved, and the tracking of an unidentified object by the robot is facilitated to a certain degree.

(6) The robot chooses whether it is needed to return to a charging point for automatic charging, according to the locations of the robot and a target (an unidentified object).

