the present invention is of classification systems , methods , and computer software to identify driving situations from labeled examples of previous occurrences . the purpose of the classifier is to provide physical context to a separate system that mitigates unnecessary distractions , allowing the driver to maintain focus during periods of high difficulty . in one embodiment , while watching videos of driving , different users were asked to indicate their perceptions of the current situation . a classifier was trained to emulate the human recognition of driving situations . in unstructured conditions , such as driving in urban areas and the german autobahn , the classifier was able to correctly predict human perceptions of driving situations over 95 % of the time . the second embodiment advances the first in the following ways : it eliminates the need to determine a priori the useful contexts . the second embodiment automatically identifies the salient patterns , or “ contexts ” in the data . since the contexts are automatically identified from the data itself , the system can be redeployed quickly to new operating environments ( e . g ., transitioning from on - road to off - road conditions ). as a corollary , the second embodiment eliminates the concern about which contexts to identify . for example , are developers not identifying a context that could be potentially useful ? this question is eliminated as the second embodiment automatically finds the statistical regularities in the data . using the first embodiment , difficulty estimates were conducted in an ad hoc manner . the new system allows one to construct optimal difficulty estimates using the contexts as inputs and human - generated estimates of perceived difficulty as the ground truth . the difficulty estimator then minimizes the error between its predictions and those of the human . the first embodiment also did not allow adaptation to new users . the classifier was created from a set of training data , and is thereafter a static system . the second embodiment allows automatic adaptation to new users . the first embodiment of the invention is next described in detail . the test vehicle for experiments employed with the first embodiment of the invention was a mercedes - benz s - class sedan , equipped with specialized sensors for the present research . the vehicle supplies a wide range of physical data such as speed , turn signals , etc . the posture of the driver is measured by a pressure - sensitive chair and ultrasonic six - degree - of - freedom head - tracking system , both developed by the university of pittsburgh . several hours of data were collected in unstructured driving conditions in both urban areas and on the german autobahn . humans were also asked humans to label videos of these driving runs according to a list of potential situations . consequently , the high - level goal of the work was to predict the time - series of human - recognized situations using the various sensors as input . to this end , the sandia cognitive framework ( scf ) as described in c . forsythe and p . g . xavier , “ human emulation : progress toward realistic synthetic human agents ,” in proceedings of the 11 th conference on computer - generated forces and behavior representation , 2002 , was employed to integrate the information of the driver posture and vehicle state to estimate the current driving situations . the pattern - recognition component of scf is a type of nonlinear dynamical system ( nds ). the present invention comprises the learning algorithms used to tune the parameters of the nds to recognize driving situations . a series of experiments was conducted to ascertain the most useful situations to provide the mitigation strategy with the appropriate driving context . as the goals of the mitigation strategy evolved , its information requirements changed accordingly . for example , during what driving situations would a driver least desire to receive a mobile phone call ? conversely , during what driving situations does a mobile phone call least impact safety ? the questions were repeated for each extraneous task that the driver may encounter . in this manner , a list of potentially useful situations were identified and , in later stages , superfluous situations were removed from the list . after several iterations , the remaining situations are assumed to be the minimum set needed to perform the desired classifications and demonstrate an operational performance gain by mitigating extraneous tasks during difficult driving conditions . when the classifier of the invention indicates that the driving context is no longer difficult , mitigation ends and the vehicle operates normally . typically , the driver is unaware that the system proactively intervened on her behalf . five human subjects were instructed to drive on a predefined circuit of german roads measuring about 200 km , ranging from urban streets to the autobahn , and each subject made three runs of the circuit . no modifications were made to the roadways or the ambient driving conditions , such as traffic or road construction . data from the vehicle and driver posture were sampled at a rate of four hertz . from these data - collection experiments , a total of almost 24 hours of data ( 343 , 946 samples ) was obtained . in addition to the sensor streams , a wide - angle video camera was also used to capture a driver - like perspective out the front of the vehicle . in order to use supervised learning to classify the data according to driving situations , it is first necessary to obtain ground - truth labels . as mentioned earlier , candidate situations were vetted by the information required by the mitigation strategy . after several iterations , the following situations were decided to be the most useful : ( 1 ) approaching or waiting at intersection ; ( 2 ) leaving intersection ; ( 3 ) entering on - ramp or high - speed roadway ; ( 4 ) being overtaken ; ( 5 ) high acceleration or dynamic state of vehicle ; ( 6 ) approaching slow - moving vehicle ; ( 7 ) preparing to change lanes ; and ( 8 ) changing lanes . to generate the labels , a tool was created that displayed to the human labeler the frontal video as well as a set of check - boxes , one for each of the eight candidate driving situations . the human labeler indicated their perception of the current driving situation by checking and unchecking the appropriate boxes . the tool also allowed the user to rewind , pause , save , load , and correct previously labeled time periods . after the user completed the labeling of a video segment , a zero - order hold was performed to associate a label with each input sample . this resulted in a sequence of binary vectors , { x * 0 , . . . , x * n } where └ x * n ┘ i = 1 if the ith situation was active at time n , and └ x * n ┘ i = 0 otherwise . two users generated labels for each of the five driver subjects for each of their three circuits , for a total of 15 labeled data sets . formulated as a supervised - learning problem , a goal of the first embodiment of the present invention was to find an “ optimal ” mapping from a time - series of sensory inputs to a time - series of driving - situation labels . since a goal of the invention is human centric , deriving a classifier that emulates the human recognition of driving situations , the sandia cognitive framework ( scf ) was employed . from an engineering perspective , the situation - recognition component of scf is a type of nonlinear dynamical system ( nds ). the inputs to the nds are the processed sensory inputs , described above . the outputs of the nds are estimated activation levels of the various situations as a function of time . the estimated situation activations can be considered a trajectory through the state space of the nds . it is preferred to use the ordered sequence of labels , described above , as the groundtruth targets for a learning algorithm . from this formulation , the goal of a supervised - learning algorithm is to tune the parameters of the nds to minimize the error between the estimated situations and the ground - truth situations generated by the human labelers . to derive the preferred learning algorithm , one needs a few elementary functions and notation . one writes the ( i , j ) th entry of a matrix as [ a ] i , j and the ith entry of a vector as [ x ] i . { tilde over ( τ )} α ( x )= x ( h α ( x )+ ψ h − α ( − x )), ( 1 ) where ψ ∈ ( 0 , 1 ) is some small , positive constant to ensure that its derivative is nonzero . in the experiments , the setting was typically 0 . 01 . for convenience , write the piecewise - continuous compression function used in the scf is defined as note that equation 2 has a continuous derivative that is nonzero everywhere , except as x →∞. in the experiments , one typically set l = 3 . for convenience , write equation 1 and equation 2 are both more easily demonstrated in graphical form and are shown in fig1 ( a ) and ( b ). where α & lt ; β . in the experiments , one typically set α = 0 . 4 and β = 0 . 6 . let the vector of inputs at time n be r n ∈ d , where d is the number of sensors . let the vector of estimated activation levels of the situations at time n be { circumflex over ( x )} n ∈ c , where c is the number of situations to classify . the dynamical equations are y n = a { circumflex over ( x )} n − 1 — ƒ l ( bu n ), { circumflex over ( x )} n = ƒ l ({ tilde over ( τ )} β ( y n )) ( 3 ) where a is the ( c × c ) matrix of feedback weights , and b is the ( c × d ) matrix of feedforward weights . from a high - level perspective , the general form of equation 3 implies that the current driving situations are a function of previous driving situations and the current sensor inputs . except in trivial cases , there does not exist a closed - form optimal solution for the parameters of equation 3 , given a sequence of known sensor inputs and desired output trajectory . therefore , any learning algorithm will rely on iterative procedures to compute locally optimal estimates of a and b . to minimize the error between the ground - truth labels and the estimated labels from equation 3 , one may pursue two different approaches . the first uses a gradient - descent approach and the second uses a genetic algorithm ( ga ) formulation . according to the human labelers , the majority of the time ( 52 %) during the experiments , none of the target situations occurred . the most common driving situation , “ begin overtaken ,” occurs 28 % of the time , while the rarest situation , “ entering on - ramp or high - speed roadway ,” occurs less than 1 % of the time . given this imbalanced data set , a typical least - squares estimation procedure , such as regression , will tend to generate only false negatives because it can achieve 99 % accuracy by simply classifying “ not entering on - ramp .” by raising the “ punishment ” for misclassifying rare situations , a system will be forced to learn the causes of those rare , but important , events . one can do this by weighting the samples inversely proportional to how frequently they occur , one of several well - established approaches to the “ rare - event problem ”. with such a weighting scheme , incorrectly classifying a time sample as “ entering on - ramp ” ( false positive ) results in the error being weighted by 0 . 01 , whereas missing a classification of “ entering on - ramp ” ( false negative ) results in an error weight of 0 . 99 . this has the effect of minimizing the number of false negatives , which is important in designing a system that mitigates against potentially difficult , though infrequent , driving situations . let n be the total number of labeled samples gathered during the experiments . let n i be the number of samples where the ith situation was “ active ,” according to the ground - truth labels . define the weighting function as in the experiments , it was found that a classifier performs much better when incorporating this weighting scheme . this is because it is impossible for a system classifying situations based solely on their relative frequency to achieve better than 50 % correct when weighting classifications by equation 4 . one preferably chooses error similar to that of large - margin classification , where learning focuses on finding the most constraining vectors to shatter a training set . consider that an estimated situation is correct if it is “ near ” the target label , i . e ., if the tristate value of the estimated situation equals the target label , α , β ({ circumflex over ( x )} i )= x * i . define the error between the label for the ith situation as the supervised - learning formulation is finding the parameter matrices a and b from equation 3 that minimize the sum - squared error both the gradient descent and the genetic algorithm seek locally optimal solutions to equation 6 . in the gradient - descent formulation , one seeks to minimize the error measure between the target and estimated labels by tuning the parameters according to the gradient of the error . to do this , one needs the gradients of equation 5 with respect to the parameters of equation 3 , namely the feedback matrix a and the feedforward matrix b . at time n , let the estimated situation activation for the ith situation be [{ circumflex over ( x )} n ] i from equation 3 and let the corresponding ground - truth label be └{ circumflex over ( x )}* n ┘ i . the gradient of the error measure with respect to the feedforward weights at time n for the ith situation label is then ∂ ∂ [ a ] i , j ⁢ 1 2 ⁢   ɛ ⁡ ( [ x n * ] i , [ x ^ n ] i )   2 = { 0 , ⁢ ϑ α , β ⁡ ( [ x ^ n ] i ) = [ x n * ] i - ω ⁡ ( [ x n * ] i ) ⁢ ( [ x n * ] i - [ x ^ n ] i ) ⁢ ∂ ∂ [ a ] i , j i ⁡ [ x ^ n ] i , one must now apply the chain rule several times and , skipping some lengthy steps , one receives the following result : to compute the full gradient , it is necessary to vary the indices ( i and j ) over all possibilities . the derivation of the feedforward weights is similar to equation 7 , the result being : note that both equation 7 and equation 8 use the gradient from the previous time step , which could lead to unstable updates as the error accumulates . various researchers in the field of adaptive control have identified sufficient conditions to ensure that gradient - descent update rules yield stable estimates . the pseudo - code for the preferred gradient - descent algorithm is shown below , where { x * 0 , . . . , x * n } is the sequence of target labels , { r * 0 , . . . , r * n } is the sequence of sensor inputs , η ∈ ( 0 , 1 ) is the step size , and σ & gt ; 0 is the stopping criterion . ν := 0 for all r n ∈{ r 0 , . . . , r n } δ a := ∂ ∂ a ⁢ 1 2 ⁢   ɛ ⁡ ( [ x n * ] i , [ x ^ n ] )   2 δ b := ∂ ∂ b ⁢ 1 2 ⁢   ɛ ⁡ ( [ x n * ] i , [ x ^ n ] )   2 a := a − ηδ a b := b − ηδ b ν := ν +∥ δ a ∥ f +∥ δ b ∥ f end for all it can be shown that the algorithm and update rules are stable , and consequently guaranteed to converge and terminate at a local minimum , if the step size η is decayed in the standard manner . for the genetic algorithm embodiment , the dakota optimization package was employed to create a genetic algorithm ( ga ) to find locally optimal solutions for the parameters of the dynamical system in equation 3 . the genome is preferably simply a column - stacked vector of the feedforward and feedback matrices , y = vec ([ a ; b ]). the fitness criterion for a given parameterization was the error measure between all ground - truth and estimated labels , as in equation 6 , which is the same as the gradient - descent formulation . between generations , keep the n best genomes and the stochasticity is preferably handled solely by genome mutations , as crossover was not allowed for . the genome mutations were selected by sampling from a gaussian distribution centered about the keeper genome from the previous generation , y ( i + 1 ) ˜ n ( y ( i ) , σ ). the covariance matrix , σ , was determined by hand a priori to contain reasonable values . the driving data was divided into a training set ( 18 . 3 hours ) and a test set ( 5 . 6 hours ). each of the five subjects drove the roadway circuit three times . each driver had two circuits randomly assigned to the training set and one to the test set . both the gradient - descent and genetic - algorithm formulations were run on these data sets . in every performance statistic that was measured , the gradient - descent algorithm outperformed the genetic algorithm . fig2 shows the performance in absolute percentage correct . the gradient - descent algorithm predicted the human recognition of driving situations over 95 % of the time on the test set , while the ga managed about 85 % correct . as mentioned earlier , a most important statistic in measuring the success of the algorithms is the weighted percentage correct ( equation 4 ). fig3 shows the performance in weighted percentage correct . when weighting the classification score by equation 4 , the gradient - descent algorithm predicted human recognition of driving situations about 88 % of the time on the test set , while the ga managed about 84 % correct . the gradient - descent algorithm is also much more efficient at finding a solution than the ga . on the 18 . 3 hours driving data in the training set , the gradient descent algorithm typically converged in about one hour of computation time and 101 parameter evaluations . the ga , on the other hand , averaged 1611 hours ( about 67 days ) of computation time and 50 , 013 evaluations . the relative efficiency of the gradient - descent algorithm is due to the massive amount of problem - specific knowledge incorporated into the algorithm — the gradient of the error . since this information is not incorporated into the ga , it is not surprising that it used substantially more parameter evaluations to arrive at a locally optimal solution . however , even when the ga was given the solution of the gradient - descent algorithm as its initial genome , the ga was unable to improve performance after several machine - days of computation . the identification and extraction of patterns from observed data goes back to ancient times , and pattern recognition is now a necessary capability in many fields . researchers are still developing novel methods to identify and extract patterns from large amounts of data . other researchers are interested in identifying contexts , or situations , in response to human behavior to improve system performance , with applications in computing and mobile phones . incorporating pattern recognition to assist humans in vehicle driving has a relatively long history in robotics , most notably the navlab project at carnegie - mellon university . in order to simplify the process of writing control software , the navlab project quickly started using supervised - learning algorithms based on observations gained while humans drove the target vehicle . in the present invention , learning algorithms were employed to estimate the optimal parameters of a nonlinear dynamical system . broadly speaking , estimating the parameters of a dynamical system that minimize some cost function is known as optimal control . there has been a substantial amount of work in nonlinear systems , though finding optimal solutions to most nonlinear systems is generally intractable . while one is able to write down the gradient of the system with respect to the tunable parameters and derive stable update equations , this is usually quite cumbersome and time consuming . this is the reason that general stochastic optimization techniques , such as genetic algorithms , are often used to solve for the unknown optimal parameters . while the experiments herein compared the performance of a designed gradient - descent algorithm to a standard genetic - algorithm formulation , there are a large number of other techniques to classify time - series data that could be employed with the present invention , including recurrent neural networks and hidden markov models . the manner by which the contexts are generated in the second embodiment of the invention uses an approach called “ unsupervised learning .” the form of unsupervised learning that is preferred attempts to find regularities in the data . this is done by first transforming temporal sensory signals into a high - dimensional vector . it is preferred to use linear regression to fit a straight line to a temporal signal over a predefined time window , shown in fig5 , resulting in regression coefficients shown in fig6 . the slope ( m ) and offset ( b ) values are two coordinates in a vector space . for n sensors , this regression approach yields a 2 * n dimensional vector . the preferred unsupervised learning algorithm searches for “ clusters ” in this high - dimensional vector space , as shown in fig7 . one of the assumptions of this work is that regularities in this vector space are caused by underlying physical contexts . for example , entering an onramp will result in the driver accelerating , putting on a turn signal , and looking over her shoulder . it is this type of regularity that the second embodiment &# 39 ; s algorithm captures . once the classifier is constructed , and new sensory data are being input into the system , one estimates the vector - space parameters from the regression mentioned above , using a buffer to keep the data on hand . the classifier estimates the probability that the vector - space parameters were generated by each of the k clusters . this probability distribution is the output of the classifier . to perform difficulty classification , as shown in fig4 , one obtains a time series of difficulty estimates from a human , and these human - scored difficulty estimates are analog ( e . g ., 0 - 100 ) and are diagnostic of the context classifier . to estimate the “ optimal ” difficulty classifier , one takes the time series of context probabilities from the unsupervised context classifier as input and the human - scored difficulty estimates as ground truth . one then estimates a cognitive model ( using the approach described in the first embodiment ) that optimally predicts the human perception of difficulty , also on an analog scale of , e . g ., 0 - 100 , using the context probabilities as input . to adapt a system to a new user , suppose one has a context classifier constructed from the approach mentioned above that is constructed from the data of a group of people for which we have human - scored difficulty estimates . then a new user , for whom we do not have difficulty estimates , begins driving the vehicle . in the second embodiment one can adapt the previously constructed context classifier to the unique driving style of this new user . as data from the new user becomes available , simply add these to the training set mentioned above and run the unsupervised - learning algorithms to update the locations of the contexts , resulting in an adapted context classifier . one then uses the probability distributions from the adapted classifier to estimate a new optimal difficulty classifier using only the inputs and outputs for which one has human - scored difficulty estimates . while such an approach may initially sound costly , it can be computed in real time in parallel with the regular operation of the context classification . the present invention , in either embodiment , may be conjoined with a motor vehicle and concomitant sensor inputs in a variety of hardware implementations . for example , the data processing can be performed by an appropriately programmed microprocessor , application specific integrated circuit ( asic ), field programmable gate array ( fpga ), or the like , in conjunction with appropriate memory and bus elements . the methods of the invention can be implemented by appropriate software coded in c ++, java , microcode , etc ., as understood by one of ordinary skill in the art . although the invention has been described in detail with particular reference to these preferred embodiments , other embodiments can achieve the same results . variations and modifications of the present invention will be obvious to those skilled in the art and it is intended to cover in the appended claims all such modifications and equivalents . the entire disclosures of all references , applications , patents , and publications cited above are hereby incorporated by reference .
computer software for and a method of enhancing safety for an operator of a motor vehicle comprising employing a plurality of sensors of vehicle and operator conditions , matching collective output from the sensors against a plurality of known dangerous conditions , and preventing certain activity of the operator if a known dangerous condition is detected .