The following task concerns data that came from a pollution monitoring station with twenty-four instruments monitoring twenty-four aspects of pollution in the air. Periodically a record is added to the file, each record being a line of 49 fields separated by white-space, which can be one or more space or tab characters.

The fields (from the left) are:

 DATESTAMP [ VALUEn FLAGn ] * 24

i.e. a datestamp followed by twenty-four repetitions of a floating-point instrument value and that instrument's associated integer flag. Flag values are >= 1 if the instrument is working and < 1 if there is some problem with it, in which case that instrument's value should be ignored.

A sample from the full data file readings.txt, which is also used in the Text processing/1 task, follows:

Data is no longer available at that link. Zipped mirror available here

1991-03-30	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1
1991-03-31	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	10.000	1	20.000	1	20.000	1	20.000	1	35.000	1	50.000	1	60.000	1	40.000	1	30.000	1	30.000	1	30.000	1	25.000	1	20.000	1	20.000	1	20.000	1	20.000	1	20.000	1	35.000	1
1991-03-31	40.000	1	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2	0.000	-2
1991-04-01	0.000	-2	13.000	1	16.000	1	21.000	1	24.000	1	22.000	1	20.000	1	18.000	1	29.000	1	44.000	1	50.000	1	43.000	1	38.000	1	27.000	1	27.000	1	24.000	1	23.000	1	18.000	1	12.000	1	13.000	1	14.000	1	15.000	1	13.000	1	10.000	1
1991-04-02	8.000	1	9.000	1	11.000	1	12.000	1	12.000	1	12.000	1	27.000	1	26.000	1	27.000	1	33.000	1	32.000	1	31.000	1	29.000	1	31.000	1	25.000	1	25.000	1	24.000	1	21.000	1	17.000	1	14.000	1	15.000	1	12.000	1	12.000	1	10.000	1
1991-04-03	10.000	1	9.000	1	10.000	1	10.000	1	9.000	1	10.000	1	15.000	1	24.000	1	28.000	1	24.000	1	18.000	1	14.000	1	12.000	1	13.000	1	14.000	1	15.000	1	14.000	1	15.000	1	13.000	1	13.000	1	13.000	1	12.000	1	10.000	1	10.000	1

Task

 Confirm the general field format of the file.
 Identify any DATESTAMPs that are duplicated.
 Report the number of records that have good readings for all instruments.


 
(defn parse-line [s]
  (let [[date & data-toks] (str/split s #"\s+")
        data-fields (map read-string data-toks)
        valid-date? (fn [s] (re-find #"\d{4}-\d{2}-\d{2}" s))
        valid-line? (and (valid-date? date)
                         (= 48 (count data-toks))
                         (every? number? data-fields))
        readings    (for [[v flag] (partition 2 data-fields)]
                      {:val v :flag flag})]
    (when (not valid-line?)
      (println "Malformed Line: " s))
    {:date date
     :no-missing-readings? (and (= 48 (count data-toks))
                                (every? pos? (map :flag readings)))}))
 
(defn analyze-file [path]
  (reduce (fn [m line]
            (let [{:keys [all-dates dupl-dates n-full-recs invalid-lines]} m
                  this-date (:date line)
                  dupl? (contains? all-dates this-date)
                  full? (:no-missing-readings? line)]
              (cond-> m
                dupl? (update-in [:dupl-dates]  conj this-date)
                full? (update-in [:n-full-recs] inc)
                true  (update-in [:all-dates]   conj this-date))))
          {:dupl-dates #{} :all-dates #{} :n-full-recs 0}
          (->> (slurp path)
               clojure.string/split-lines
               (map parse-line))))
 
(defn report-summary [path]
  (let [m (analyze-file path)]
    (println (format "%d unique dates" (count (:all-dates m))))
    (println (format "%d duplicated dates [%s]"
                     (count (:dupl-dates m))
                     (clojure.string/join " " (sort (:dupl-dates m)))))
    (println (format "%d lines with no missing data" (:n-full-recs m)))))
 