Method and apparatus for selectively processing paginated output
This application is a continuation of application Ser. No. 07/257,218, filed Oct. 13, 1988, now abandoned. 
    
    
     A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. 
     TECHNICAL FIELD 
     This invention relates to digital data processing and, more particularly, to the selective post-processing of paginated output from digital data processing systems. 
     BACKGROUND OF THE INVENTION 
     Many digital data processing application programs in use today produce output in the form of a sequence of two-dimensional images or planes, exemplified by the printed page or the display on a cathode ray tube. Typical examples of such application programs include word processing programs, spread-sheet programs and data base manager programs. The contents of these images include all of the information obtained from the operation of the application program, arranged or formatted in a manner believed to be useful by the application programmer. Some limited ability to control or edit these output images are sometimes included in the application package. Unfortunately, however, such alternate output formats must be provided either by laborious, line-by-line manual editing of the original output, by hard coding of the new format by the application programmer, or else a large and complex report generator must be included in the application program to allow the user to format his or her own output. 
     The major disadvantage of the inflexible output format is that it sometimes serves to conceal rather than reveal the desired data, due either to the sparsity of the desired data among the total data output, or due to the indirect manner in which it is displayed. The disadvantages of report generators associated with application programs are the time and difficulty in learning how to use the report generator, only to find that it is then necessary to learn a new report generator for a new and different application program. The disadvantages of manual editing are obvious. 
     SUMMARY OF THE INVENTION 
     In accordance with the illustrative embodiment of the present invention, these and other problems are solved by providing a general purpose application program output post-processor system which permits the user to selectively format, process and display portions of the data contained in a series of two-dimensional images for display or for printing. The post-processing system in accordance with the present invention (hereinafter called &#34;phaser&#34;) is independent of the application program with which it is used and hence can be used with a wide variety of different application programs. 
     More specifically, the post-processing system of the present invention looks at the two-dimensional plane-oriented outputs from any application program and selectively identifies one or two-dimensional portions of those outputs either by pattern content, or by location on the page. That is, information is located either by the content of the information, or by line and character offset (column) location on the page. It should be noted that the patterns and/or the physical locations need not be contiguous. In order to permit common processing of data identified by these different locators, they are both converted to a single descriptor form, for example, lists of line segments. Such a &#34;segment list&#34; can then be used to control the processing of the data so identified to permit the selective display, replacement or omission of the described data. In particular, data located by pattern matching, for example, is converted to line segment location and placed on the segment list in line segment format. 
     The plane-oriented post-processing system of the present invention therefore can process the data identified in the segment list by deriving other data from the data identified on the segment list, or merely displaying the identified data in any desired format. 
     A major advantage of the output-oriented post-processing formatter of the present invention over the data-oriented, pre-processing formatters of the prior art is the ability to use the same formatter for many different application programs, from word processing programs to data base managing programs, and from spread-sheet programs to information retrieval programs. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     A complete understanding of the present invention may be gained by considering the following detailed description in conjunction with the accompanying drawing, in which: 
     FIG. 1 shows a general block diagram of a page-oriented computer application program system showing the post-processing phaser formatter in accordance with the present invention; 
     FIG. 2 shows a graphical example of the type of post-processing that can be done on paginated output using the phaser system of the present invention; 
     FIG. 3 shows a more detailed block diagram of the phaser post-processing page-oriented processing system in accordance with the present invention; 
     FIGS. 4A and 4B show graphical representations of the syntax diagram for a compilation procedure that can be utilized in the phaser system of the present invention; and 
     FIG. 5 shows a flowchart of an illustrative phaser procedure showing the ordering of working set routines. 
     To facilitate reader understanding, identical reference numerals are used to designate elements common to the figures. 
    
    
     DETAILED DESCRIPTION 
     In FIG. 1 there is shown a general block diagram of a computer system in which the present invention can be used. The system of FIG. 1 includes a data source 10 which supplies the data to be processed by the computer system of FIG. 1. Typically, such data sources are located on magnetic disks, usually after being assembled from large numbers of data generating stations such as point-of-sale terminals, banking terminals, business inventory control stations, and so forth. Data from source 10 is supplied to a standard computer processing system 11 which performs well-known types of computer processing steps on the data from source 10. A typical result of such processing is the generation of a plurality of &#34;pages&#34; of output. In this context, the term &#34;pages&#34; includes successive cathode ray tube screen images, or any other two-dimensional, planar output, as well as printed pages. Such output can be categorized as standard planar output 12 from computer processing system 11. The major problem with such output, as anyone who has had to deal with it fully understands, is that the specific data the user requires is most often buried somewhere in a very large volume of data, spread over a large number of possibly non-successive pages of data, the majority of which is not of interest to that user. It is to the solution of this problem that the present invention is directed. 
     In accordance with the present invention, the standard planar output 12 from the computer processing system 11 is applied to a special post-processing system 13, hereinafter called the &#34;phaser&#34; processing system. The function of the phaser processing system 13 is to analyze the planar output from block 12, to select the portions of that output desired by the present user, to further process such selected portions, if required, and to display the results of such further processing, along with whatever portions of the original output are desired. The result of the operation of phaser processing system 13 is customized planar output 14 which includes all of the data wanted by the user, and only the data wanted by the user, reformatted and further processed in exactly the manner wanted by that user. Other users of the same standard planar output 12 can use the same phaser processing system 13 to provide another, different customized output 14 suitable for the other user&#39;s needs. 
     In the prior art, it was always possible for the supplier of the particular computer processing system 11 to customize the output for the imagined needs of future users, or to customize the output for a particular user, given adequate time and resources. In either case, however, the customization was &#34;hard coded&#34; into the application program, not thereafter readily susceptible to changes by the user. Different users, of course, required different customizations, all at significant costs in time and other resources. It is a significant advantage of the phaser processing system 13 of the present invention that the format and content of the customized planar output 14 is entirely under the control of the user. 
     It was also possible in the prior art for the supplier of the particular computer processing system 11 to include in the application software a more or less general purpose report generator. Such report generators had the capability of allowing the user to select the data and the format for the planar output 12, thus supplying one of the needs met by the present invention. Unfortunately, however, such report generators are at least as complex to implement as the application itself, and hence expensive to provide. Furthermore, the degree of variation permitted in formatting or data content was limited, dependent in part on the software supplier&#39;s view of the needs of the ultimate user. For example, many report generator programs simply permitted the user to display or not display the contents of specific fields in the data records, with no more sophisticated selection criteria and no further processing possible. More importantly, such report generators were specific to the applications software in which they resided. Hence, when a new application processing system 11 is used, an entirely new report generator must be provided or devised. In addition to the cost and delay time in providing such new report generators, it is typical for the new report software to require retraining of the entire user community, since the methodology and the conventions for the new report generator are normally different from other prior report generators. It is another significant advantage of the present invention that the phaser processing system 13 can be applied to the paginated, planar output from any computer processing system 11, provided only that the output is divided into pages or images on a terminal screen, that is, as long as the output is a succession of two-dimensional data arrays which can be divided into rows and columns, into lines and characters, or into any other two-dimensional coordinate identifiers. 
     Although general purpose report generators are available in the prior art, in order to use these report generators, the data base itself must be reformatted to the specifications of the report generator and the data base manager supplied with the report generator used to access information from the data base. The difficulty with this arrangement is the obverse of the difficulties pointed out above. That is, the data base universe has to be tailored to the particular report generator whereas, above, the report generator universe had to be tailored to the particular data base. Neither arrangement deals efficiently with the actual data base arrangements present in the real world. The major advantage of the present invention is the provision of tailored output from any data base by tailoring the user-readable output of the data base rather than the data base itself. Indeed, it is possible for the system of the present invention to convert data which is not paginated into paginated data, and hence it can be used to convert a stream of data, originally intended for electronic consumption, into a series of pages useful for human readers. 
     It is to be noted that the purpose of customizing the output of a particular computer processing system 11 may be to transfer the customized data on to another computer processing system rather that to present the data to the user. The phaser system of the present invention is perfectly suitable for this arrangement and, in doing so, can reduce the delay in performing such further processing to virtually zero. It should also be noted that one very important use of decimated computer processing output is in regression testing. Regression testing generally involves the comparison of the outputs from new and old versions of the same program. If they are identical, the new version is considered to be correct. In regression testing, it is undesirable to compare all of the output since the time required would be prohibitive and some portions of the output tend to be date or time dependent and hence not comparable. Regression testing is therefore the art of selecting test cases which are typical, repeatable, and cover all of the major capabilities of the system. The decimation of the output data from the new and old versions of the same program for regression testing comparisons is a major use of the phaser processing system of the present invention. 
     In FIG. 2 there is shown a typical example of the type of post-processing of paginated output permitted by the phaser processing system 13 of the present invention. In FIG. 2, there are shown three different views 15, 16 and 17 of the same paginated portion of the output from a computer processing system such as system 11 of FIG. 1. Page 15 shows a window 18 containing the data characters &#34;origin&#34; in the lower, central portion of page 15. As suggested by the text between the pages 15-7, the appearance of the characters &#34;o-r-i-g-i-n&#34; in window 15 causes the contents &#34;France&#34; of window 19 in the upper right hand corner of page 16 to be changed to &#34;Italy&#34; in window 19 of page 17. This type of pattern recognition and conditional pattern changing is only typical of the phaser processing system 13 of the present invention. Virtually any other type of two-dimensional processing is possible. 
     The general types of analysis actions available in the phaser system are replace, choose, remove and replace everything but. Combinations of these four basic actions provide virtually universal flexibility in decomposing paginated output. Similarly, the general categories of objects upon which these actions can be taken are virtually universal. Thus, the replace, choose, remove and replace everything but actions can be applied to pages, lines, columns, strings, boxes (like windows 19 in FIG. 2), blocks (lists of strings, not necessarily contiguous), and &#34;clouds&#34; (entire pages treated as one continuous string for SPITBOL® pattern searches). SPITBOL patterns are described, for example, in SPITBOL-68K Program Reference Manual, by R. B. K. Dewar, Catspaw, Inc., Salida, Colo., 1987. SPITBOL is a dialect of the SNOBOL language, described in The SNOBOL4 Programming Language, R. E. Griswold et al., Prentice-Hall, Englewood Cliffs, N.J., 2nd Ed., 1971. The pages of information thus decomposed by the replace, choose, remove and replace everything but actions can be recomposed however the user wishes by the merge action which recombines the decomposed elements into new pages with new formats. 
     In operation, the user of the phaser system of the present invention describes the post-processing desired by a sequence of statements involving the actions and objects described above. These statements make up a phaser program and are used by the phaser system as described in connection with FIG. 3. Turning then to FIG. 3, there is shown a flowchart of the phaser processing system 13 of FIG. 1. The process of FIG. 3 starts in start box 20. In box 21, most of the variables used in the phaser system of FIG. 3 are initialized. Similarly, in box 22, most of the patterns used in the phaser system are initialized. The phaser program 24 consists of a number of statements involving actions and objects as described above. In box 23, the phaser program is compiled and a phaser listing 25 produced for assistance in trouble shooting. The phaser program compiled in box 23 is executed in box 26 to translate the standard planes of data 27 into the customized planes of data 28, all in accordance with the user&#39;s wishes as expressed in phaser program 24. 
     The phaser program 24 of FIG. 3 is typically divided up into a number of working sets, each working set defining the processing steps to be carried out on one set of input data to produce one particular data output. A phaser program can include any number of such working sets, each working set working on a different data input set, or providing different processing steps for the same input data. The basic phaser statement types are defined below, where &lt;argument&gt; is a variable name, where capital letters identify the predefined actions, where &#34;objects&#34; are as defined above, and where square brackets around an argument indicate that the argument is optional: 
     1. define &lt;working set name&gt; using &lt;using group&gt;. This statement is used to mark the beginning of a working set. The working set name is the unique name of this working set. The using group in the using clause is a list of the inputs to this working set, including filenames (in parentheses) or set names of other working sets. 
     2. end &lt;working set name&gt;. This statement is used to mark the end a working set. 
     3. choose &lt;object1&gt; [where &lt;object2&gt; &lt;contain phrase&gt; &lt;pattern&gt;]. The current text set will be modified to include only the &lt;object1&gt; portion of the input text if the &lt;object2&gt; portion of the input text meets the specifications of the where clause. 
     4. remove &lt;object1&gt; [where &lt;object2&gt; &lt;contain phrase&gt; &lt;pattern&gt;]. The current text set will be modified so that it will not include the &lt;object1&gt; portion of the input text if the &lt;object2&gt; portion of the input text meets the specifications of the where clause. 
     5. replace &lt;object1&gt; with &lt;string&gt; [where &lt;object2&gt; &lt;contain phrase&gt; &lt;pattern&gt;]. The current text set will have the &lt;object1&gt; portion of the input text replaced with repetitions of the with string only if, and only if, the &lt;object2&gt; portion of the input text meets the specifications of the where clause. 
     6. replace everything but &lt;object1&gt; with &lt;string&gt; [where &lt;object2&gt; &lt;contain phrase&gt; &lt;pattern&gt;]. The current text set will have everything in the input text set but the &lt;object1&gt; portion of the input text replaced with the with string if, and only if, the &lt;object2&gt; portion of the input text meets the specifications of the where clause. 
     7. merge &lt;mergetype&gt; &lt;mergeobject1&gt; &lt;mergeobject2&gt;. Merge successive pairs of plane objects (pages from files or from working sets) side-by-side, one-after-another or line-by-line (mergetypes). Plane objects may be the current working set plane contents, a plane read from a file, or a remembered plane. 
     8. pad on the &lt;direction&gt; with &lt;number&gt; &lt;string&gt;. Add &lt;number&gt; copies of &lt;string&gt; on the &lt;direction&gt; side of the current working set plane. The direction is right or left. 
     9. remember as &lt;name&gt;. Save a copy of the contents of the current working set plane for later retrieval under the key &lt;name&gt;. 
     10. translate from &lt;string1&gt; to &lt;string2&gt;. Replace single character in &lt;string1&gt; with single characters in &lt;string2&gt;. This could be done with replace, but not as efficiently. 
     11. write &lt;string&gt; to &lt;filename&gt;. Store a literal string in a file. 
     12. print to &lt;filename&gt;. Print the contents of the current working set to &lt;filename&gt;. 
     The &#34;objects&#34; of the above action statements can be any one or two-dimensional regions on an input plane, not necessarily contiguous. Such regions can be defined in terms of page numbers for groups of pages, line numbers, ranges or patterns for groups of lines, column numbers, ranges or patterns for groups of columns, coordinates or patterns for strings, and coordinates or patterns for a rectangular box. Other definitions of regions include &#34;clouds&#34; defined by patterns and &#34;blocks&#34; defined by lists of line segments. These relationships between the actions and the regions will be discussed in connection with a discussion of the syntax diagrams of FIGS. 4A and 4B. 
     Before turning to FIG. 4A, it should be noted that a phaser statement is a section of source code beginning with one of the action words identified above (define, end, replace, choose, remove, etc.) and ending with a semicolon. A phaser working set is a plurality of phaser statements introduced by a define statement and terminated by an end statement. A phaser program is one or more working sets. Note that the input to a working set can be the output of one or more other working sets as well as the contents of one or more data files. By processing working sets in a user-defined order, the phaser system is able to perform logical unions and intersections of the working set parameters. That is, sequential working sets are ANDed and parallel working sets are ORed. 
     With the above as a background, the syntax diagram of FIGS. 4A and 4B can now be considered. The syntax diagrams of FIGS. 4A-4B are so-called &#34;railroad&#34; diagrams. In these diagrams, movement takes place through the diagram from left to right and various branches can be entered as if railroad switches were operated to permit access to that branch. The direction of the branch curves suggests the unidirectional nature of the branching. 
     In FIG. 4A, the action words of the working set statements are found at reference numeral 40 while the objects affected by these actions and their respective position specifications are found at reference numeral 41. Note that all possible combinations of actions, objects and position specifications are accounted for in the syntax diagram. At reference numeral 42, the replacement value is available for the replace statements. In FIG. 4B, the conditional where clauses are taken care of at reference numeral 43 with the objects and position specification named in the WHERE clause appearing at reference numeral 44. Finally, at reference numeral 45, the text comparison patterns are accounted for. Using the syntax diagram of FIGS. 4A and 4B, the generation of a compiler for phaser statements is straightforward and obvious to persons skilled in the compiler art. 
     It will be noted that the phaser system allows two-dimensional objects in the planar output of a computer processing system to be described in a wide variety of ways: page numbers, column and line numbers, patterns in strings, boxes, blocks and clouds, and so forth. In order to process the contents of such objects, it is essential to provide a common representation of the information location in the planar output, regardless of how such information was described by the user. The common representation used in the phaser system is called the &#34;segment list.&#34; A segment list is a list composed of all of the line segments making up the data identification of an object and its descriptors. Such segment lists are composed of items which take the form: 
     
         (a, b-c) 
    
     where a is the line number in which the line segment is located, b is the starting column number of the line segment and c is the ending column number of the line segment. The asterisk (&#34;*&#34;) is used as a wild card value in segment list items to simplify representations. Thus, &#34;(*, b-c)&#34; represents the segments between columns b and c for all lines, &#34;(a, b-*)&#34; represents the segment on line a from column b to the end of the line, and &#34;(*, 1--*)&#34; represents an entire page. Data in the planar output of a computer processing system is located in terms of all of the locators described above, including pattern matching. Once located, however, all information specifications internal to the phaser system are kept in the form of segment lists and passed between processes in this form. It is therefore unnecessary to provide special processing routines for all of the different forms of data identifiers available in the system (FIGS. 4A-4B). All internal processing is done on segment lists, and all data representations are converted to and from the segment list representation whenever any processing is necessary. Some regions translate directly into segment list notation (lines and columns), but some can be translated to segment list notation only after the data is identified, e.g., by pattern matching. 
     Since segment list items can arise from a variety of phaser statements, it is possible for various items to include portions of overlapping segments. While the system would operate properly with such overlapping items, the processing would be less than optimum since certain segments would be treated more than once. In order to eliminate such multiple processing, each segment list, just prior to processing, is sorted, duplicate segments eliminated and overlapping segments combined into a single segment. A simple comparison of line numbers and column numbers readily permits such segment list optimization. 
     The phaser system represented in block form in FIG. 3 can also be represented by the following pseudo-code in Table I: 
     
                       TABLE I                                                     
______________________________________                                    
main ( )                                                                  
define variables                                                          
define patterns                                                           
parsegen ( )                                                              
       while input                                                        
       getstat ( )                                                        
           read statement                                                 
           if defset                                                      
              then                                                        
                 start workingset code                                    
              end then                                                    
           else if endset                                                 
              then                                                        
                 terminate workingset code                                
              end then                                                    
           else if intersperse merge                                      
              then                                                        
                 buildintersperse                                         
              end then                                                    
           else if action statement                                       
              then                                                        
                 buildcode                                                
              end then                                                    
           else if merge statement                                        
              then                                                        
                 buildmerge                                               
              end then                                                    
           end else                                                       
       end getstat                                                        
end parsegen                                                              
process ( )                                                               
       if no endset                                                       
           then                                                           
              add endset                                                  
           end then                                                       
       end if                                                             
       fileroutine ( )                                                    
           while no NULL                                                  
              readpage ( )                                                
              rout (workingsets)                                          
           end while                                                      
       end fileroutine                                                    
end process                                                               
end main                                                                  
______________________________________                                    
 
    
     The define routines create all of the fixed variables and fixed patterns to be used in the balance of the program. The parsegen routine is the compiler, using the syntax of FIGS. 4A and 4B, to create executable code from the phaser statements. For example, getstat gets the next phaser statement (terminated by a semicolon). The defset routine sets up the data structures specified as input and the working set-data structure control structures necessary to identify and eventually execute each working set against the appropriate data structures. The buildcode, buildmerge and buildintersperse routines create the actual executable code for the action statements, merge statements and intersperse statements, respectively. The endset routine terminates the working set and triggers the actual compilation of the working set code. 
     In TABLE I, the process routine actually executes the compiled phaser executable code. First, the endset routine is called in case the final END statement is missing. The fileroutine routine loops through the input files identified in the define statements (up to an end-of-file, EOF), passing the input planes of data, one by one, on to the rout routine. The rout routine finds each working set which specifies this file in its using clause, and executes the compiled working set against that plane. The rout routine also finds the working sets which must follow other working sets (in accordance with the using clause of the define statements) and insures proper ordering of the execution of the working sets. Execution of the working set code continues until the input has been exhausted and all planes have been processed. SPITBOL code descriptions of each of these routines can be found in the Appendix to this application. 
     An illustrative example of a phaser user program will be given to aid in the understanding of the present invention. It is to be understood that this example is illustrative only, and should not be treated as limiting in any sense. Indeed, the illustrative example was chosen to be relatively simple to aid in the understanding of the invention. The following phaser program is illustrative: 
     
                       TABLE II                                                    
______________________________________                                    
define a using (file1);                                                   
remove pages where they contain `xyz`;                                    
define b using a;                                                         
remove pages where they contain `abc`;                                    
end b;                                                                    
define c using a;                                                         
remove pages where they contain `111`;                                    
end c;                                                                    
define d using b, c;                                                      
replace strings with `xyz` where they contain `kag`;                      
print to (outfile);                                                       
end d;                                                                    
______________________________________                                    
 
    
     The result of compiling and executing the phaser program of TABLE II is shown graphically in FIG. 5. Each box in FIG. 5 represents a working set. Thus the contents of &#34;file1&#34; are operated on (by way of line 50) by working set 51, identified as working set &#34;a.&#34; The output of working set 51 is routed to both working set 52 (&#34;b&#34;) and working set 53 (&#34;c&#34;). The outputs of working sets 52 and 53 are both routed to the input to working set 54 (&#34;d&#34;). Thus, the output of working set 54 on leads 55 is the ORed combination of working sets 52 and 53, subject to (ANDed with) the post-ORing operations of working set d. It is clear that any other Boolean combination of operations can be performed on the input data simply by appropriately specifying the working set inputs. In this sense, phaser applies a very large grain data flow (one plane at a time) to the editing of collections of plane-oriented output. Moreover, the phaser commands exploit the two-dimensional aspects of the planar output to transform two-dimensional arrays or patterns of alphanumeric data by filtering each page-sized plane through a virtually arbitrarily prescribed network of plane-oriented filters. 
     It should be clear to those skilled in the art that further embodiments of the present invention may be made by those skilled in the art without departing from the teachings of the present invention. ##SPC1##