embodiments of the disclosure relate generally to the field of disease recognition in the heart . for example , embodiments of the disclosure relate to systems and methods for exploiting spatio - temporal information for view recognition and disease recognition in cardiac echo videos . the inventors of the present invention have published an article entitled “ exploiting spatio - temporal information for view recognition in cardiac echo videos ,” and published by the ieee computer society , 2001 l street , nw . suite 700 , washington , d . c . 20036 - 4910 usa , on jun . 27 , 2008 , which is incorporated by reference into this application as if set forth fully herein . two of the inventors also published an article entitled “ cardiac disease recognition in echocardiograms using spatio - temporal statistical models ,” published by ieee engineering in medicine and biology society , 445 hoes lane , piscataway , n . j . 08854 - 4141 usa , on aug . 23 , 2008 , which is incorporated by reference into this application as if fully set forth herein . throughout the description , for the purposes of explanation , numerous specific details are set forth in order to provide a thorough understanding of the present disclosure . it will be apparent , however , to one skilled in the art that the present disclosure may be practiced without some of these specific details . fig1 is illustrates different cardiac echo view points used in an echo cardiographic examination . an echographic exam involves scanning a heart from four standard transducer positions , namely , parasternal , apical , subcostal and suprasternal . for each transducer position , the transducer is rotated and oriented to obtain multiple tomographic images of the heart along its long and short axes . of the viewpoints generated by standard transducer positions , the most common are the apical 4 - chamber 110 , the parasternal long axis 120 , the parasternal short axis 130 and the apical 2 - chamber 140 . tomographic imaging from the four standard transducer positions shows different combinations of anatomical regions of the heart , as illustrated in fig1 . because different regions of the heart may be diseased without potentially affecting other regions , it is desirable to image the different regions of the heart . for example , apical 4 - chamber view 110 illustrates the left ventricle , left atrium , right ventricle , right atrium , septal wall , the mitral and tricuspid valves and can also be used to study left ventricle hypertrophy . as seen in fig2 a and 2b , there can be significant variation in the appearance of the heart for a given transducer position . this may be caused by anatomical differences between diseased and normal hearts as well as possibly due to a variety of other factors such as the variability between patients , instruments , operator experience or imaging modalities . images 210 and 220 show a four - chambered view of two different hearts . image 210 illustrates a patient with hypertrophy , while image 220 illustrates excessive enlargement of the left ventricle . images 230 , 240 and 250 show three different echocardiograph frames where 230 and 240 are 2 - chamber viewpoints and the 250 is a short - axis viewpoint of the same heart . thus , the echocardiograph of the same heart can generate very different images depending on the transducer position and image . the effect of zoom and pan can also make the determination of viewpoint difficult . images 260 and 270 illustrate this , where 270 is a zoomed in version of the region depicted in 260 . the determination of viewpoint becomes even more challenging for real - time recordings , as it must distinguish standard viewpoints from transitional ones when the transducer is being moved before settling in a new position . image 290 shows a transitional view that is from a continuous recording with before and after views shown in 280 and 295 , repsectively . image 290 must be classified as a spurious view . fig2 c shows a preferred embodiment of how visual appearance is represented by our modeling approach . each image of a heart is represented by a shape vector s obtained by concatenating the ( x , y ) locations of n feature points f 1 , f 2 , . . . f n as seen in 210 c and in equation ( 1 ). the shape vectors s are geometrically normalized by a similarity transform γ sim that factors out variation in rotation , scale , and position . the final shape vectors are represented in a normalized space 220 c . for instance , procrustes analysis using two anchor points can be used for this normalization step . thus , the full description of shape includes the vector s as well as a similarity transform fs that positions the shape in the image through a rotation , scale , and a translation . s =[ x 1 , y 1 , . . . , x n , y n ] t [ equation 1 ] given an image with n features , the texture vector t concatenates the pixels from a set of patches centered on the feature points into a long vector , where patch size is matched to the pixel spacing between features . just as the shape vectors are normalized for geometry , the texture vector t raw is similarly normalized for echo gain and offset by subtracting the mean and dividing by standard deviation as seen in equation ( 2 ). since cardiac motion is a useful feature for cardiac analysis , we would also like to create a canonical representation of motion . considering a cardiac cycle with m frames will produce m sets of feature points { s 1 , s 2 , . . . , s m }, where each s is a column vector of stacked shape ( x , y ) feature coordinates , as shown in 230 c . to create a canonical representation , the motion of the cardiac cycle is vectorized by normalizing for image plane geometry and standardizing the time axis to a target length n . align the first frame s 1 to a canonical position using a similarity transform γ 1 sim . to standardize the remaining frames i , 2 ≦ i ≦ m , apply the same similarity transform to standardize the frames , as shown in equation 3 to standardize the sequence length from input length m to a target length n , interpolate the s i &# 39 ; s in time using piecewise linear interpolation . next , to decouple shape from our motion representation , we factor out shape by subtracting out frame 1 , creating our final motion vector m shown in equation 4 . in what follows , the shape , texture , and motion vectors are used to represent images both in model training and model fitting steps . in the model training step , a user trains the model representation by hand annotating the shape in all the training images . in contrast , the model fitting stage automatically fits a model of shape and texture to an input sequence to analyze . fig3 a is a block diagram illustrating an example method for training a spatiotemporal model of a heart 300 . this may also be referred to as training stage block diagram 300 . in one embodiment beginning at 310 , a user first prepares a set of training data by annotating the ( x , y ) positions of feature points in the images , usually taken to sample along the contours of the heart walls and valves . in 320 , the user builds an active shape model ( asm ), a spatial and textural model of heart appearance , for a known transducer viewpoint from the annotated training data . to construct the asm model , the dimensional of the spatial and textural vector is reduced using pca to find a small set of eignshapes and eigentextures . the shape s and the texture t can be linearly modeled to form the active shape model as seen in equation ( 5 ). the p - dimensional vector a and the q - dimensional vector b are the low dimensional vector representations of shape and texture . proceeding to 330 - 350 , a motion model is generated for a transducer viewpoint class by tracking the viewpoint &# 39 ; s asm within the training set of annotated echo sequences and modeling the resulting tracks with a linear generative motion model . in block 330 , the asm model built in block 320 is used to densely track the asm feature points throughout the entire training data , which is necessary since the manual annotation step in block 310 only requires a fraction of the training set to be hand - annotated . in block 340 the asm tracks are vectorized into motion vectors by normalizing for heart rate period and the positioning of the heart in the image . block 350 applies a dimensionality reduction procedure to the training motion vectors , modeling them with a low - dimensional linear model , as shown in equation 6 . the r - dimensional vector c is the low dimensional vector representations of motion , and the columns of matrix m are a set of r “ eigenmotions ” that form a low - dimensional basis for the cardiac motions seen in the training set . the final output of the training procedure 300 is a viewpoint - specific asm model and motion model . fig3 b shows an algorithm for the method for training spatio - temporal models of the heart from training sequences seen in the block diagram shown in fig3 a . fig4 a shows how the asm and motion models from block diagram 300 can be fit to a new image sequence to extract heart shape and motion . this may be referred to as the fitting stage block diagram 400 in block 410 , the asm model is fit to the first frame in the image sequence , image i 1 . since there is no prior information of the overall heart location in the image , this step is preceded by a segmentation step that locates the heart chambers . given the initial asm fit from i 1 , block 420 tracks the feature locations frame by frame through the remainder of the sequence using the asm model . fitting an asm to a new sequence involves finding a similarity transform γ sim to position the model appropriately and recovering the shape and text vectors a and b . this is iteratively estimated by alternating between shape and texture update steps . to evaluate an asm fit at a given position , error of fit is measured in shape and texture space using mahalanobis distance and the normalized reconstruction error . for image i , this can be seen in equation ( 7 ). fit ( a , b , γphu sim )= a t σ shp − 1 a + b t σ tex − 1 b + 2 r 2 / λ tex q + 1 [ equation 7 ] where r =∥ t − tt t t ∥, t = l ( γ sim ( x , y )). λ tex q + 1 is the ( q + 1 ) th texture eigenvalue , and σ shp and σ tex are diagonal matrices with pca eigenvalues . the overall motion of the asm tracks from block 420 is analyzed in blocks 430 - 440 . in block 430 , the asm feature tracks are vectorized into a motion vector m , as in stage 340 , normalizing for heart rate period and a global positioning transform . proceeding to 440 , the motion vector is projected into an eigen - motion feature space of a transducer viewpoint class , as shown in equation ( 8 ). proceeding to 450 , the matching algorithm estimates a measure of a sequence fit for each cardiac view , wherein the sequence fit is an average appearance fit over the cardiac cycle plus a mahalanobis distance and reconstruction error , as shown in equation ( 9 ). where the function “ fit ” is defined in equation ( 7 ), r mot =∥ m − mm t m ∥, λ mot r + 1 is the ( r + 1 ) th texture eigenvalue , and σ mpt is a diagonal ( r × r ) matrix of eigenvalues corresponding to e 1 m , e 2 m ,. . . . , e r m . the mahalanobis term is a weighted distance from the pca projection of m to the mean motion m ( but within the pca model space ). the reconstruction term measures the distance from m to the pca projection of m , and it tells how well the model explains the data . fig4 b shows an algorithm for fitting steps of the spatio - temporal models to a new echo sequence seen in the block diagram shown in fig4 a . fig5 a is a block diagram 500 of a method for recognizing cardiac viewpoint in echo sequences , divided into a model building stage 500 a and a runtime recognition stage 500 b . the model building stage 500 a iterates over all the viewpoints and builds a viewpoint - specific asm and motion model for each viewpoint . the inputs for a 500 a are a set of viewpoints v , l ≦ v ≦ v , and a set of training sequences . in block 510 , the iteration variable v is initialized to 1 since this is the initial viewpoint . proceeding to block 520 , asm and motion models are built for viewpoint v from the training data specific for that viewpoint . this is implemented as a procedure call to the training block 300 in fig3 . after the asm and motion models for view v = l are completed , we move to the next viewpoint in block 530 by increasing v by 1 . in decision block 540 , the algorithm determines if the all available views v have been considered . if v is less than or equal to v , blocks 520 - 530 are repeated . if v is greater than v , model building stage 500 a is completed . view recognition stage 500 b recognizes cardiac viewpoint v in input sequence i using the viewpoint - specific asm and motion models generated by stage 500 a . as in stage 500 a , the view recognition algorithm iterates over the viewpoints v , and it returns the viewpoint with the best model fit . in block 550 , the iteration variable v is initialized to 1 since this is the initial viewpoint . proceeding to block 560 , the viewpoint fit , fit ( v ) ( see equation 9 ), is computed by fitting the asm and motion models for viewpoint v to input sequence i . block 560 is implemented as a procedure call to the fitting block 400 in fig4 . after the model fitting is completed , we move to the next viewpoint in block 570 by increasing v by 1 . in decision block 580 , the algorithm determines if the all available views v have been considered . if v is less than or equal to v , blocks 560 - 570 are repeated . if v is greater than v , we proceed to block 590 , where the view v that minimizes fit ( v ) is returned as the recognized view . fig5 b shows the algorithm for recognizing cardiac viewpoint shown in the block diagram of fig5 a . fig6 a and 6b show a block diagram of disease recognition 600 , which includes a model building portion 600 a in fig6 a and a disease recognition portion 600 b in fig6 b . the model building portion 600 a in fig6 a builds models for a set of diseases , where the disease models are , in turn , broken down into a collection of viewpoint - specific asm and motion models . the inputs for 600 a include a set of training sequences , viewpoints v , l ≦ v ≦ v , and diseases d , l ≦ d ≦ d . block diagram 600 a mostly consists of flow control over two nested loops , the outer loop over diseases d and an inner loop over viewpoints v . block 605 begins with the first disease , d = l , and block 610 begins with the first viewpoint , v = l . block 615 creates an asm and motion model for disease d and viewpoint v , using the training data for d and v . after model building and proceeding to block 620 , v is increased by 1 . in decision block 625 , the algorithm determines if the all available views v have been modeled . if v is less than or equal to v , blocks 615 - 620 are repeated to model an additional viewpoint . when v is greater than v , the disease model for a particular disease , d , is the union of asm ( d , v ) and motionmodel ( d , v ) over all the viewpoints v , as shown in block 630 . for example , a disease model for an enlarged heart would contain appearance - based asm models that all generate enlarged chambers in all the viewpoints . it should be noted that the disease model also creates a “ normal ” model such that a healthy heart is part of the models included in this process . thus , the process can recognize a healthy heart , rather than attempting to associate a heart disease with a healthy heart . block 635 shows the algorithm proceeding to the next disease and block 640 shows a decision block that tests for additional diseases to model by comparing d with d . if d is less than or equal to d , the process repeats blocks 610 - 635 to model an additional disease . if d is greater than d , the model building stage 600 a completes . fig6 b is the disease recognition portion 600 b of block diagram 600 and shows blocks 645 , 650 , 655 , 660 , 665 , 670 , 675 , 680 and 685 . the input to disease recognition is an input sequence ito classify , the disease models generated in 600 a , viewpoints v , l ≦ v ≦ v , and diseases d , l ≦ d ≦ d . similar to the model building portion 600 a , disease recognition 600 b consists of two nested loops , the outer loop over diseases d and an inner loop over viewpoints v . block 645 begins with the first disease , d = l , and block 650 begins with the first viewpoint , v = l . block 655 fits the input sequence i to asm and motion models for disease d and viewpoint v , which may be described by the algorithm in fig4 and equation 9 , described above . block 660 increments v by 1 , moving to the next viewpoint . decision block 665 shows that if v is less than or equal to v , the process is returned to block 655 . if v is greater than v , the algorithm proceeds to block 670 , where the overall disease fit , fit ( d ), is taken as the average fit ( d , v ) over all views v . block 675 moves to analyze the input i with the next disease , incrementing d by one . decision block 680 processes the next disease by returning to block 650 if d is less than or equal to d . if d is greater than d , then we proceed to block 685 . block 685 shows that the disease recognized by the algorithm is the disease that minimizes fit ( d ). fig6 c shows the model building and disease recognition algorithms that correspond to block diagrams 600 a and 600 b . thus , at block 685 , the algorithm has recognized the heart disease most present in the patient in input sequence i . as discussed above , the null hypothesis is that the patient does not have a disease , wherein a normal class is also modeled . the diagnostic viewpoints of the normal class are taken as the union of viewpoints from all of the disease classes . fig7 illustrates an example computer architecture for implementing the methods and flows as described in fig3 - 6 . the exemplary computing system of fig7 includes : 1 ) one or more processors 701 ; 2 ) a memory control hub ( mch ) 702 ; 3 ) a system memory 703 ( of which different types exist such as ddr ram , edo ram , etc ,); 4 ) a cache 704 ; 5 ) an i / o control hub ( ich ) 705 ; 6 ) a graphics processor 706 ; 7 ) a display / screen 707 ( of which different types exist such as cathode ray tube ( crt ), thin film transistor ( tft ), liquid crystal display ( lcd ), dpl , etc . ); and / or 8 ) one or more i / o devices 708 . the one or more processors 701 execute instructions in order to perform whatever software routines the computing system implements . the instructions frequently involve some sort of operation performed upon data . both data and instructions are stored in system memory 703 and cache 704 . cache 704 is typically designed to have shorter latency times than system memory 703 . for example , cache 704 might be integrated onto the same silicon chip ( s ) as the processor ( s ) and / or constructed with faster sram cells whilst system memory 703 might be constructed with slower dram cells . by tending to store more frequently used instructions and data in the cache 704 as opposed to the system memory 703 , the overall performance efficiency of the computing system improves . system memory 703 is deliberately made available to other components within the computing system . for example , the data received from various interfaces to the computing system ( e . g ., keyboard and mouse , printer port , lan port , modem port , etc .) or retrieved from an internal storage element of the computing system ( e . g ., hard disk drive ) are often temporarily queued into system memory 703 prior to their being operated upon by the one or more processor ( s ) 701 in the implementation of a software program . similarly , data that a software program determines should be sent from the computing system to an outside entity through one of the computing system interfaces , or stored into an internal storage element , is often temporarily queued in system memory 703 prior to its being transmitted or stored . the ich 705 is responsible for ensuring that such data is properly passed between the system memory 703 and its appropriate corresponding computing system interface ( and internal storage device if the computing system is so designed ). the mch 702 is responsible for managing the various contending requests for system memory 703 access amongst the processor ( s ) 701 , interfaces and internal storage elements that may proximately arise in time with respect to one another . one or more i / o devices 708 are also implemented in a typical computing system . i / o devices generally are responsible for transferring data to and / or from the computing system ( e . g ., a networking adapter ); or , for large - scale non - volatile storage within the computing system ( e . g ., hard disk drive ). ich 705 has bi - directional point - to - point links between itself and the observed i / o devices 708 . referring back to fig3 - 6 , portions of the different embodiments of the described methods and flows may be embodied in software , hardware , firmware , or any combination thereof . any software may be software programs available to the public or special or general - purpose processors running proprietary or public software . the software may also be a specialized program written specifically for signature creation and organization and recompilation management . for the exemplary methods illustrated in fig3 - 6 , embodiments of the invention may include the various processes as set forth above . the processes may be embodied in machine - executable instructions that cause a general - purpose or special - purpose processor to perform certain steps . alternatively , these processes may be performed by specific hardware components that contain hardwired logic for performing the processes , or by any combination of programmed computer components and custom hardware components . embodiments of the invention do not require all of the various processes presented , and it may be conceived by one skilled in the art as to how to practice the embodiments of the invention without specific processes presented or with extra processes not presented . for example , while it is described that a user may perform portions of the methods , those portions alternatively or in conjunction may be performed by an automated or computer process . in another embodiment , the models may be provided as previously generated . the foregoing description of the embodiments of the invention has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the invention to the precise forms disclosed . numerous modifications and adaptations are apparent to those skilled in the art without departing from the spirit and scope of the invention .
a method for recognizing heart diseases in a cardiac echo video of a heart with an unknown disease using a spatio - temporal disease model derived from a training echo video , comprising the steps of : generating a plurality of training models for heart diseases , wherein the cardiac echo videos are each derived from a known viewpoint and the disease of the heart is known ; analyzing the video of the heart with the unknown disease by fitting a model of shape and motion for each frame and combining the results across the frames ; and , reporting the disease using a classification method for choosing among the diseases of interest .