<jupyter_start><jupyter_text>Our dependencies:<jupyter_code>%pip install pandas<jupyter_output>Collecting pandas
  Downloading pandas-1.0.1-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)
[K     |████████████████████████████████| 10.1 MB 8.0 MB/s eta 0:00:01
[?25hRequirement already satisfied: python-dateutil>=2.6.1 in /home/fracpete/development/projects/fracpete/pynb2docker/venv/lib/python3.7/site-packages (from pandas) (2.8.1)
Collecting pytz>=2017.2
  Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Requirement already satisfied: numpy>=1.13.3 in /home/fracpete/development/projects/fracpete/pynb2docker/venv/lib/python3.7/site-packages (from pandas) (1.18.1)
Requirement already satisfied: six>=1.5 in /home/fracpete/development/projects/fracpete/pynb2docker/venv/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas) (1.14.0)
Installing collected packages: pytz, pandas
Successfully installed pandas-1.0.1 pytz-2019.3
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: scipy in /home/fracpete/development/projects/fracpete/[...]<jupyter_text>The following environment variables are used for determining input and output:* `INPUT` - the input file* `OUTPUT` - the output file* `VERBOSE` - whether to output some debugging information (boolean: true|false)For testing purposes, we set them to manual values if not present:<jupyter_code>import os

input_file = None
output_file = None
verbose = True

if 'INPUT' in os.environ:
    input_file = os.environ['INPUT']
else:
    input_file = "/home/fracpete/development/datasets/uci/numeric/bolts.csv"
if 'OUTPUT' in os.environ:
    output_file = os.environ['OUTPUT']
else:
    output_file = "/home/fracpete/temp/bolts_clean.csv"
if 'VERBOSE' in os.environ:
    verbose = bool(os.environ['VERBOSE'])
if verbose:
    print("INPUT=%s" % input_file)
    print("OUTPUT=%s" % output_file)<jupyter_output>INPUT=/home/fracpete/development/datasets/uci/numeric/bolts.csv
OUTPUT=/home/fracpete/temp/bolts_clean.csv<jupyter_text>Load data:<jupyter_code>import pandas as pd

data = pd.read_csv(input_file)
if verbose:
    print("Input data shape:", data.shape)<jupyter_output>Input data shape: (40, 8)<jupyter_text>Select rows where SPEED1 (2nd column) is greater than 2:<jupyter_code>filtered = data[data.SPEED1 > 2.0]
if verbose:
    print("Filtered shape:", filtered.shape)<jupyter_output>Filtered shape: (24, 8)<jupyter_text>Write output file:<jupyter_code>filtered.to_csv(output_file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1371927042030135, 'fraction_numerical': 0.03727200634417129, 'mean_word_length': 5.622047244094488, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '466973', 'license': 'Apache-2.0', 'n_tokens_mistral': 989, 'n_tokens_neox': 912, 'n_words': 190, 'path': 'jupyter/pandas_filter_pipeline.ipynb', 'repo_name': 'fracpete/pynb2docker', 'size': '2522'}