The following task concerns data that came from a pollution monitoring station with twenty-four instruments monitoring twenty-four aspects of pollution in the air. Periodically a record is added to the file, each record being a line of 49 fields separated by white-space, which can be one or more space or tab characters.

The fields (from the left) are:

 DATESTAMP [ VALUEn FLAGn ] * 24

i.e. a datestamp followed by twenty-four repetitions of a floating-point instrument value and that instrument's associated integer flag. Flag values are >= 1 if the instrument is working and < 1 if there is some problem with it, in which case that instrument's value should be ignored.

A sample from the full data file readings.txt, which is also used in the Text processing/1 task, follows:

Data is no longer available at that link. Zipped mirror available here

1991-03-30	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1
1991-03-31	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	20.000	1	20.000	1	20.000	1	35.000	1	50.000	1	60.000	1	40.000	1	30.000	1	30.000	1	30.000	1	25.000	1	20.000	1	20.000	1	20.000	1	20.000	1	20.000	1	35.000	1
1991-03-31	40.000	1	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2
1991-04-01	0.000	-2	13.000	1	16.000	1	21.000	1	24.000	1	22.000	1	20.000	1	18.000	1	29.000	1	44.000	1	50.000	1	43.000	1	38.000	1	27.000	1	27.000	1	24.000	1	23.000	1	18.000	1	12.000	1	13.000	1	14.000	1	15.000	1	13.000	1	10.000	1
1991-04-02	8.000	1	9.000	1	11.000	1	12.000	1	12.000	1	12.000	1	27.000	1	26.000	1	27.000	1	33.000	1	32.000	1	31.000	1	29.000	1	31.000	1	25.000	1	25.000	1	24.000	1	21.000	1	17.000	1	14.000	1	15.000	1	12.000	1	12.000	1	10.000	1
1991-04-03	10.000	1	9.000	1	10.000	1	10.000	1	9.000	1	10.000	1	15.000	1	24.000	1	28.000	1	24.000	1	18.000	1	14.000	1	12.000	1	13.000	1	14.000	1	15.000	1	14.000	1	15.000	1	13.000	1	13.000	1	13.000	1	12.000	1	10.000	1	10.000	1

Task

 Confirm the general field format of the file.
 Identify any DATESTAMPs that are duplicated.
 Report the number of records that have good readings for all instruments.


#lang racket
(read-decimal-as-inexact #f)
;; files to read is a sequence, so it could be either a list or vector of files
(define (text-processing/2 files-to-read)
  (define seen-datestamps (make-hash))
  (define (datestamp-seen? ds) (hash-ref seen-datestamps ds #f))
  (define (datestamp-seen! ds pos) (hash-set! seen-datestamps ds pos))
 
  (define (fold-into-pairs l (acc null))
    (match l ['() (reverse acc)]
      [(list _) (reverse (cons l acc))]
      [(list-rest a b tl) (fold-into-pairs tl (cons (list a b) acc))]))
 
  (define (match-valid-field line pos)
    (match (string-split line)
      ;; if we don't hit an error, then the file is valid
      ((list-rest (not (pregexp #px"[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}")) _)
       (error 'match-valid-field "invalid format non-datestamp at head: ~a~%" line))
 
      ;; check for duplicates
      ((list-rest (? datestamp-seen? ds) _)
       (printf "duplicate datestamp: ~a at line: ~a (first seen at: ~a)~%"
               ds pos (datestamp-seen? ds))
       #f)
 
      ;; register the datestamp as seen, then move on to rest of match
      ((list-rest ds _) (=> next-match-rule) (datestamp-seen! ds pos) (next-match-rule))
 
      ((list-rest
        _
        (app fold-into-pairs
             (list (list (app string->number (and (? number?) vs))
                         (app string->number (and (? integer?) statuss)))
                   ...)))
       (=> next-match-rule)
       (unless (= (length vs) 24) (next-match-rule))
       (not (for/first ((s statuss) #:unless (positive? s)) #t)))
 
      ;; if we don't hit an error, then the file is valid
      (else (error 'match-valid-field "bad field format: ~a~%" line))))
 
  (define (sub-t-p/1)
    (for/sum ((line (in-lines))
              (line-number (in-naturals 1)))
      (if (match-valid-field line line-number) 1 0)))  
  (for/sum ((file-name files-to-read))
    (with-input-from-file file-name sub-t-p/1)))
 
(printf "~a records have good readings for all instruments~%"
        (text-processing/2 (current-command-line-arguments)))