This task has been flagged for clarification. Code on this page in its current state may be flagged incorrect once this task has been clarified. See this page's Talk page for discussion.
Often data is produced by one program, in the wrong format for later use by another program or person. In these situations another program can be written to parse and transform the original data into a format useful to the other. The term "Data Munging" is often used in programming circles for this task.

A request on the comp.lang.awk newsgroup led to a typical data munging task:

I have to analyse data files that have the following format:
Each row corresponds to 1 day and the field logic is: $1 is the date,
followed by 24 value/flag pairs, representing measurements at 01:00,
02:00 ... 24:00 of the respective day. In short:

<date> <val1> <flag1> <val2> <flag2> ...  <val24> <flag24>

Some test data is available at: 
... (nolonger available at original location)

I have to sum up the values (per day and only valid data, i.e. with
flag>0) in order to calculate the mean. That's not too difficult.
However, I also need to know what the "maximum data gap" is, i.e. the
longest period with successive invalid measurements (i.e values with
flag<=0)
The data is free to download and use and is of this format:

Data is no longer available at that link. Zipped mirror available here (offsite mirror).

1991-03-30	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1
1991-03-31	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	20.000	1	20.000	1	20.000	1	35.000	1	50.000	1	60.000	1	40.000	1	30.000	1	30.000	1	30.000	1	25.000	1	20.000	1	20.000	1	20.000	1	20.000	1	20.000	1	35.000	1
1991-03-31	40.000	1	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2
1991-04-01	0.000	-2	13.000	1	16.000	1	21.000	1	24.000	1	22.000	1	20.000	1	18.000	1	29.000	1	44.000	1	50.000	1	43.000	1	38.000	1	27.000	1	27.000	1	24.000	1	23.000	1	18.000	1	12.000	1	13.000	1	14.000	1	15.000	1	13.000	1	10.000	1
1991-04-02	8.000	1	9.000	1	11.000	1	12.000	1	12.000	1	12.000	1	27.000	1	26.000	1	27.000	1	33.000	1	32.000	1	31.000	1	29.000	1	31.000	1	25.000	1	25.000	1	24.000	1	21.000	1	17.000	1	14.000	1	15.000	1	12.000	1	12.000	1	10.000	1
1991-04-03	10.000	1	9.000	1	10.000	1	10.000	1	9.000	1	10.000	1	15.000	1	24.000	1	28.000	1	24.000	1	18.000	1	14.000	1	12.000	1	13.000	1	14.000	1	15.000	1	14.000	1	15.000	1	13.000	1	13.000	1	13.000	1	12.000	1	10.000	1	10.000	1

Only a sample of the data showing its format is given above. The full example file may be downloaded here.

Structure your program to show statistics for each line of the file, (similar to the original Python, Perl, and AWK examples below), followed by summary statistics for the file. When showing example output just show a few line statistics and the full end summary.

-- demo\rosetta\TextProcessing1.exw
with javascript_semantics -- (include version/first of next three lines only)
include readings.e -- global constant lines, or:
--assert(write_lines("readings.txt",lines)!=-1) -- first run, then:
--constant lines = read_lines("readings.txt")
 
include builtins\timedate.e
 
integer count = 0,
        max_count = 0,
        ntot = 0
atom readtot = 0
timedate run_start, max_start
 
procedure end_bad_run()
    if count then 
        if count>max_count then
            max_count = count
            max_start = run_start
        end if
        count = 0
    end if
end procedure
 
for i=1 to length(lines) do
    sequence oneline = split(lines[i],'\t'), r
    if length(oneline)!=49 then
        ?"bad line (length!=49)"
    else
        r = parse_date_string(oneline[1],{"YYYY-MM-DD"})
        if not timedate(r) then
            ?{"bad date",oneline[1]}
        else
            timedate td = r
            integer rejects=0, accepts=0
            atom readsum = 0
            for j=2 to 48 by 2 do
                r = scanf(oneline[j],"%f")
                if length(r)!=1 then
                    ?{"error scanning",oneline[j]}
                    rejects += 1
                else
                    atom reading = r[1][1]
                    r = scanf(oneline[j+1],"%d")
                    if length(r)!=1 then
                        ?{"error scanning",oneline[j+1]}
                        rejects += 1
                    else
                        integer flag = r[1][1]
                        if flag<=0 then
                            if count=0 then
                                run_start = td
                            end if
                            count += 1
                            rejects += 1
                        else
                            end_bad_run()
                            accepts += 1
                            readsum += reading
                        end if
                    end if
                end if
            end for
            readtot += readsum
            ntot += accepts
            if i>=length(lines)-2 then
                string average = iff(accepts=0?"N/A":sprintf("%6.3f",readsum/accepts))
                printf(1,"Date: %s, Rejects: %2d, Accepts: %2d, Line total: %7.3f, Average %s\n",
                       {format_timedate(td,"DD/MM/YYYY"),rejects, accepts, readsum, average})
            end if
        end if
    end if
end for
 
printf(1,"Average: %.3f (of %d readings)\n",{readtot/ntot,ntot})
end_bad_run()
if max_count then
    printf(1,"Maximum run of %d consecutive false readings starting: %s\n",
            {max_count,format_timedate(max_start,"DD/MM/YYYY")})
end if

?"done"
{} = wait_key()
