<jupyter_start><jupyter_text>Exploratory Analysis of San Diego County TMDL MeasurementsThis dataset, which contains multiple TDML measurement series, was extracted from CEDEN, and is stored in the [Data Library data repository](https://data.sandiegodata.org/dataset/water-quality-project-example-data/resource/4d8d1b40-a70f-450b-b1a2-2f7571643cbb). More information about the dataset is avialble at the [program overview website.](https://www.waterboards.ca.gov/sandiego/water_issues/programs/tmdls/).The largest number of records are for Rainbow Creek, which [has had a long term monitoring and remediation project.](http://missionrcd.org/residential/rainbow-creek-watershed/)<jupyter_code>import pandas as pd
import missingno as msno
df = pd.read_csv('http://ds.civicknowledge.org.s3.amazonaws.com/ceden.waterboards.ca.gov/CEDEN%20TDML.csv', 
                 skiprows=2, low_memory=False)
df['SampleDate'] = pd.to_datetime(df.SampleDate) # Dates are often not automatically convered
df['Result'] = pd.to_numeric(df['Result'],errors='coerce') # Column has some non-numerics, so is recognized as a string
df.head().T
df.Program.value_counts()
# Names of all of the parent projects
df.ParentProject.value_counts()
df.Project.value_counts()
# Identifiers for the various measurements
df.Analyte.value_counts()<jupyter_output><empty_output><jupyter_text>Records Per StationsThe largest number of records are for Rainbow Creek, which [has had a long term monitoring and remediation project.](http://missionrcd.org/residential/rainbow-creek-watershed/)<jupyter_code>df.StationName.value_counts().head(20)
len(df[df.StationName.str.contains('Rainbow Creek')])<jupyter_output><empty_output><jupyter_text>Record CoverageIn these stacked event charts, it is easy to see the Rainbow Creek records, which cover a much longer time range than other locations, but which are not measured as frequently.<jupyter_code>def event_time_plot(df,yaxis='StationName'):
    from datetime import datetime
    import matplotlib.pyplot as plt
    import matplotlib.dates as dt

    _ = df[(df.SampleDate > pd.datetime(1980,1,1))].copy()

    _['secs_since_epoch'] = (_.SampleDate - pd.datetime(1970,1,1))

    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(111)
    ax = plt.plot(_.SampleDate, _[yaxis], marker='|', markersize=2, linestyle='None')<jupyter_output><empty_output><jupyter_text>Here is the time coverage per project<jupyter_code>event_time_plot(df, 'Project')<jupyter_output><empty_output><jupyter_text>And a more detailed coverage per station.<jupyter_code>event_time_plot(df)<jupyter_output><empty_output><jupyter_text>Here is what just the Beaches and Creeks Bacteria TDML records look like.<jupyter_code>bac = df[df.ParentProject == 'Bacteria TMDL 20 Beaches and Creeks'].copy()
event_time_plot(bac)

msno.matrix(bac.sample(1000)[bac.columns[:40]])
msno.matrix(bac.sample(1000)[bac.columns[40:]])
bac.SampleDate.min(), bac.SampleDate.max()
bac.Analyte.value_counts()
bac['SampleDays'] = (bac.SampleDate - bac.SampleDate.min()).dt.days

import seaborn as sns
import matplotlib.pyplot as plt 

g = sns.lmplot('SampleDays', 'Result',  data=bac[bac.Analyte == 'Enterococcus'], 
           hue='StationName', fit_reg=False, size=10)

g.set( yscale="log")

ax = plt.gca()
ax.set_title("Enterococcus measurements by time for all stations in Bacteria TMDL project")

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11989421099030267, 'fraction_numerical': 0.01821921833676168, 'mean_word_length': 6.121338912133891, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '144687', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1190, 'n_tokens_neox': 1123, 'n_words': 260, 'path': 'TMDL.ipynb', 'repo_name': 'san-diego-water-quality/example-analysis', 'size': '3403'}