This task has been flagged for clarification. Code on this page in its current state may be flagged incorrect once this task has been clarified. See this page's Talk page for discussion.
Often data is produced by one program, in the wrong format for later use by another program or person. In these situations another program can be written to parse and transform the original data into a format useful to the other. The term "Data Munging" is often used in programming circles for this task.

A request on the comp.lang.awk newsgroup led to a typical data munging task:

I have to analyse data files that have the following format:
Each row corresponds to 1 day and the field logic is: $1 is the date,
followed by 24 value/flag pairs, representing measurements at 01:00,
02:00 ... 24:00 of the respective day. In short:

<date> <val1> <flag1> <val2> <flag2> ...  <val24> <flag24>

Some test data is available at: 
... (nolonger available at original location)

I have to sum up the values (per day and only valid data, i.e. with
flag>0) in order to calculate the mean. That's not too difficult.
However, I also need to know what the "maximum data gap" is, i.e. the
longest period with successive invalid measurements (i.e values with
flag<=0)
The data is free to download and use and is of this format:

Data is no longer available at that link. Zipped mirror available here (offsite mirror).

1991-03-30	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1
1991-03-31	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	20.000	1	20.000	1	20.000	1	35.000	1	50.000	1	60.000	1	40.000	1	30.000	1	30.000	1	30.000	1	25.000	1	20.000	1	20.000	1	20.000	1	20.000	1	20.000	1	35.000	1
1991-03-31	40.000	1	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2
1991-04-01	0.000	-2	13.000	1	16.000	1	21.000	1	24.000	1	22.000	1	20.000	1	18.000	1	29.000	1	44.000	1	50.000	1	43.000	1	38.000	1	27.000	1	27.000	1	24.000	1	23.000	1	18.000	1	12.000	1	13.000	1	14.000	1	15.000	1	13.000	1	10.000	1
1991-04-02	8.000	1	9.000	1	11.000	1	12.000	1	12.000	1	12.000	1	27.000	1	26.000	1	27.000	1	33.000	1	32.000	1	31.000	1	29.000	1	31.000	1	25.000	1	25.000	1	24.000	1	21.000	1	17.000	1	14.000	1	15.000	1	12.000	1	12.000	1	10.000	1
1991-04-03	10.000	1	9.000	1	10.000	1	10.000	1	9.000	1	10.000	1	15.000	1	24.000	1	28.000	1	24.000	1	18.000	1	14.000	1	12.000	1	13.000	1	14.000	1	15.000	1	14.000	1	15.000	1	13.000	1	13.000	1	13.000	1	12.000	1	10.000	1	10.000	1

Only a sample of the data showing its format is given above. The full example file may be downloaded here.

Structure your program to show statistics for each line of the file, (similar to the original Python, Perl, and AWK examples below), followed by summary statistics for the file. When showing example output just show a few line statistics and the full end summary.

INT no data := 0;               # Current run of consecutive flags<0 in lines of file #
INT no data max := -1;          # Max consecutive flags<0 in lines of file #
FLEX[0]STRING no data max line; # ... and line number(s) where it occurs #
 
REAL tot file := 0;             # Sum of file data #
INT num file := 0;              # Number of file data items with flag>0 #
 
# CHAR fs = " "; #
INT nf = 24;
 
INT upb list := nf;
FORMAT list repr = $n(upb list-1)(g", ")g$;
 
PROC exception = ([]STRING args)VOID:(
  putf(stand error, ($"Exception"$, $", "g$, args, $l$));
  stop
);
 
PROC raise io error = (STRING message)VOID:exception(("io error", message));
 
OP +:= = (REF FLEX []STRING rhs, STRING append)REF FLEX[]STRING: (
  HEAP [UPB rhs+1]STRING out rhs;
  out rhs[:UPB rhs] := rhs;
  out rhs[UPB rhs+1] := append;
  rhs := out rhs;
  out rhs
);
 
INT upb opts = 3; # these are "a68g" "./Data_Munging.a68" & "-" #
[argc - upb opts]STRING in files;
FOR arg TO UPB in files DO in files[arg] := argv(upb opts + arg) OD;
 
MODE FIELD = STRUCT(REAL data, INT flag);
FORMAT field repr = $2(g)$;
 
FOR index file TO UPB in files DO
  STRING file name = in files[index file], FILE file;
  IF open(file, file name, stand in channel) NE 0 THEN
    raise io error("Cannot open """+file name+"""") FI;
  on logical file end(file, (REF FILE f)BOOL: logical file end done);
  REAL tot line, INT num line;
  # make term(file, ", ") for CSV data #
  STRING date;
  DO
    tot line := 0;             # sum of line data #
    num line := 0;             # number of line data items with flag>0 #
    # extract field info #
    [nf]FIELD data;
    getf(file, ($10a$, date, field repr, data, $l$));
 
    FOR key TO UPB data DO
      FIELD field = data[key];
      IF flag OF field<1 THEN
        no data +:= 1
      ELSE
        # check run of data-absent data #
        IF no data max = no data AND no data>0 THEN
          no data max line +:= date FI;
        IF no data max<no data AND no data>0 THEN
          no data max := no data;
          no data max line := date FI;
        # re-initialise run of no data counter #
        no data := 0;
        # gather values for averaging #
        tot line +:= data OF field;
        num line +:= 1
      FI
    OD;
 
    # totals for the file so far #
    tot file +:= tot line;
    num file +:= num line;
 
    printf(($"Line: "g"  Reject: "g(-2)"  Accept: "g(-2)"  Line tot: "g(-14, 3)"  Line avg: "g(-14, 3)l$,
          date,
          UPB(data) -num line,
          num line, tot line,
          IF num line>0 THEN tot line/num line ELSE 0 FI))
  OD;
  logical file end done:
    close(file)
OD;
 
FORMAT plural = $b(" ", "s")$,
       p = $b("", "s")$;
 
upb list := UPB in files;
printf(($l"File"f(plural)"    = "$, upb list = 1, list repr, in files, $l$,
        $"Total    = "g(-0, 3)l$, tot file,
        $"Readings = "g(-0)l$, num file,
        $"Average  = "g(-0, 3)l$, tot file / num file));
 
upb list := UPB no data max line;
printf(($l"Maximum run"f(p)" of "g(-0)" consecutive false reading"f(p)" ends at line starting with date"f(p)": "$,
    upb list = 1, no data max, no data max = 0, upb list = 1, list repr, no data max line, $l$))