The following task concerns data that came from a pollution monitoring station with twenty-four instruments monitoring twenty-four aspects of pollution in the air. Periodically a record is added to the file, each record being a line of 49 fields separated by white-space, which can be one or more space or tab characters.

The fields (from the left) are:

 DATESTAMP [ VALUEn FLAGn ] * 24

i.e. a datestamp followed by twenty-four repetitions of a floating-point instrument value and that instrument's associated integer flag. Flag values are >= 1 if the instrument is working and < 1 if there is some problem with it, in which case that instrument's value should be ignored.

A sample from the full data file readings.txt, which is also used in the Text processing/1 task, follows:

Data is no longer available at that link. Zipped mirror available here

1991-03-30	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1
1991-03-31	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	20.000	1	20.000	1	20.000	1	35.000	1	50.000	1	60.000	1	40.000	1	30.000	1	30.000	1	30.000	1	25.000	1	20.000	1	20.000	1	20.000	1	20.000	1	20.000	1	35.000	1
1991-03-31	40.000	1	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2
1991-04-01	0.000	-2	13.000	1	16.000	1	21.000	1	24.000	1	22.000	1	20.000	1	18.000	1	29.000	1	44.000	1	50.000	1	43.000	1	38.000	1	27.000	1	27.000	1	24.000	1	23.000	1	18.000	1	12.000	1	13.000	1	14.000	1	15.000	1	13.000	1	10.000	1
1991-04-02	8.000	1	9.000	1	11.000	1	12.000	1	12.000	1	12.000	1	27.000	1	26.000	1	27.000	1	33.000	1	32.000	1	31.000	1	29.000	1	31.000	1	25.000	1	25.000	1	24.000	1	21.000	1	17.000	1	14.000	1	15.000	1	12.000	1	12.000	1	10.000	1
1991-04-03	10.000	1	9.000	1	10.000	1	10.000	1	9.000	1	10.000	1	15.000	1	24.000	1	28.000	1	24.000	1	18.000	1	14.000	1	12.000	1	13.000	1	14.000	1	15.000	1	14.000	1	15.000	1	13.000	1	13.000	1	13.000	1	12.000	1	10.000	1	10.000	1

Task

 Confirm the general field format of the file.
 Identify any DATESTAMPs that are duplicated.
 Report the number of records that have good readings for all instruments.


* Read text/2
 
	v = array(24)
	f = array(24)
	tos = char(9) " " ;* break characters are both tab and space
	pat1 = break(tos) . dstamp
	pat2 = span(tos) break(tos) . *v[i] span(tos) (break(tos) | (len(1) rem)) . *f[i]
	rowcount = 0
	hold_dstamp = ""
	num_bad_rows = 0
	num_invalid_rows = 0
 
in0
	row = input :f(endinput)
	rowcount = rowcount + 1
	row ? pat1 = :f(invalid_row)
 
* duplicated datestamp?
* if dstamp = hold_dstamp then duplicated
	hold_dstamp = differ(hold_dstamp,dstamp) dstamp :s(nodup)
	output = dstamp ": datestamp at row " rowcount " duplicates datestamp at " rowcount - 1
nodup
 
	i = 1
in1
	row ? pat2 = :f(invalid_row)
	i = lt(i,24) i + 1 :s(in1)
 
* Is this a goodrow?
* if any flag is < 1 then row has bad data
	c = 0
goodrow
	c = lt(c,24) c + 1 :f(goodrow2)
	num_bad_rows = lt(f[c],1) num_bad_rows + 1 :s(goodrow2)f(goodrow)
goodrow2
 
	:(in0)
 
invalid_row
	num_invalid_rows = num_invalid_rows + 1
	:(in0)
 
endinput
	output =
	output = "Total number of rows                    : " rowcount
	output = "Total number of rows with invalid format: " num_invalid_rows
	output = "Total number of rows with bad data      : " num_bad_rows
	output = "Total number of good rows               : " rowcount - num_invalid_rows - num_bad_rows 
 
end
 
 