<jupyter_start><jupyter_text>Copy Training and Prediction Data<jupyter_code>%%bash
mkdir -p /content/mldata
wget https://storage.googleapis.com/cloud-datalab/sampledata/ml/census/census_train.csv -P /content/mldata -q
wget https://storage.googleapis.com/cloud-datalab/sampledata/ml/census/census_test.csv -P /content/mldata -q
wget https://storage.googleapis.com/cloud-datalab/sampledata/ml/census/census_predict.csv -P /content/mldata -q<jupyter_output><empty_output><jupyter_text>Browse and Explore Your CSV Data View several lines from head. "columns" in cell is optional. Without it, the names will be col0, col1...coln.<jupyter_code>%%csv view -i /content/mldata/census_train.csv
columns: label, age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capital-gain, capital-loss, hours-per-week, native-country<jupyter_output><empty_output><jupyter_text>Get stats of columns (--profile). -n is number of lines to read, and is optional (default to 5).<jupyter_code>%%csv view -i /content/mldata/census_train.csv --profile -n 200
columns: label, age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capital-gain, capital-loss, hours-per-week, native-country<jupyter_output><empty_output><jupyter_text>Infer Schema and Generate Feature Class by RunningRun the following command and it will replace cell input with feature class definition in next cell.Note that --target (and --key) can be either a column name, or an index into the columns (0 based, -1 means last).<jupyter_code>%%ml features --csv /content/mldata/census_train.csv --target label
columns: label, age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capital-gain, capital-loss, hours-per-week, native-country<jupyter_output><empty_output><jupyter_text>Define feature classNow the feature class is generated. Modify it as appropriate, such as converting a column from text to categorical. Then execute the cell.<jupyter_code>%%tensorflow feature

import google.cloud.ml.features as features


class CsvFeatures(features.CsvFeatureSet):
  """ This class is generated from command line:
         %%ml csv-schema ...
         Please modify it as appropriate!!!
  """

  def __init__(self):
    columns = 'label','age','workclass','fnlwgt','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country'
    super(CsvFeatures, self).__init__(columns)

  target = features.target('label').classification()
  attrs = [
      features.numeric('age').min_max_scale(-1.0, 1.0),
      features.numeric('capital-gain').min_max_scale(-1.0, 1.0),
      features.numeric('capital-loss').min_max_scale(-1.0, 1.0),
      features.numeric('education-num').min_max_scale(-1.0, 1.0),
      features.numeric('fnlwgt').min_max_scale(-1.0, 1.0),
      features.numeric('hours-per-week').min_max_scale(-1.0, 1.0),
      features.categorical('education').one_of_k(),
      features.categorical('marital-status').one_of_k(),
      features.categorical('occupation').one_of_k(),
      features.categorical('race').one_of_k(),
      features.categorical('relationship').one_of_k(),
      features.categorical('sex').one_of_k(),
      features.categorical('workclass').one_of_k(),
  ]
  native_country = features.text('native-country').bag_of_words(vocab_size=10000)<jupyter_output><empty_output><jupyter_text>Preprocess Training and Testing DataOutput: preprocessed train data, test data, and metadata generated from train data<jupyter_code>%%ml preprocess -o /content/mldata/
train: /content/mldata/census_train.csv
test: /content/mldata/census_test.csv<jupyter_output><empty_output><jupyter_text>Take a peek at metadata generated<jupyter_code>%%bash
cat /content/mldata/metadata.yaml<jupyter_output>columns:
  age:
    max: 90.0
    mean: 38.58164675532078
    min: 17.0
    name: age
    scale:
      max: 1.0
      min: -1.0
    transform: scale
    type: numeric
  capital-gain:
    max: 99999.0
    mean: 1077.6488437087312
    min: 0.0
    name: capital-gain
    scale:
      max: 1.0
      min: -1.0
    transform: scale
    type: numeric
  capital-loss:
    max: 4356.0
    mean: 87.303829734959
    min: 0.0
    name: capital-loss
    scale:
      max: 1.0
      min: -1.0
    transform: scale
    type: numeric
  education:
    items:
      10th: 0
      11th: 1
      12th: 2
      1st-4th: 3
      5th-6th: 4
      7th-8th: 5
      9th: 6
      Assoc-acdm: 7
      Assoc-voc: 8
      Bachelors: 9
      Doctorate: 10
      HS-grad: 11
      Masters: 12
      Preschool: 13
      Prof-school: 14
      Some-college: 15
    name: education
    transform: one_of_k
    type: categorical
  education-num:
    max: 16.0
    mean: 10.0806793403151
    min: 1.0
    name: education-num
    scale[...]<jupyter_text>Preprocessing Prediction DataUsing the metadata we generated to preprocess more data.<jupyter_code>%%ml preprocess -o /content/mldata/
predict: /content/mldata/census_predict.csv
metadata: /content/mldata/metadata.yaml<jupyter_output><empty_output><jupyter_text>Now take a look at all data we preprocessed<jupyter_code>%%bash
ls /content/mldata<jupyter_output>census_predict.csv
census_test.csv
census_train.csv
metadata.yaml
preprocessed_predict
preprocessed_test
preprocessed_train
{'fraction_non_alphanumeric': 0.12200111586386461, 'fraction_numerical': 0.03198809745211084, 'mean_word_length': 4.078375826251181, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '348264', 'license': 'Apache-2.0', 'n_tokens_mistral': 1935, 'n_tokens_neox': 1791, 'n_words': 410, 'path': 'docs/ml/census/Preprocess-Local.ipynb', 'repo_name': 'yukoga/datalab-archives', 'size': '5377'}