<jupyter_start><jupyter_text>Initial datasetHousing price data from single family homes in Denver CO<jupyter_code>prices = pd.read_csv('data/single_family_home_values.csv')
prices['last_sale_date'] = pd.to_datetime(prices['last_sale_date'])
display(prices.shape)
prices.head()<jupyter_output><empty_output><jupyter_text>AugementationWith a larger dataet a model could learn to relate the sale date and sale amount, but if we had an existing measure for this it'd be helpful. Fortunately consumer price index is readily available in csvs! If this wasn't a indepedent measureand we needed to control for leakage we could embed the merging in donatello's ModelDAG (a lightweight execution graph) but since it's not pernicious we can do it up front for simplicity.<jupyter_code>cpi = pd.read_csv('data/denver_cpi.csv')
cpi['DATE'] = pd.to_datetime(cpi['DATE'])
cpi.head()
df = pd.merge_asof(prices.sort_values(by='last_sale_date'), cpi, 
                   left_on='last_sale_date', right_on='DATE', direction='nearest')
df.head()
def load_sculpture(df, 
                   model=LinearRegression(), 
                   paramGrid=None,
                   searchKwargs=None):

    """
    Helper to load sculpture
    """
    d = data.Dataset(raw=df, target='estimated_value')
    
    e = estimator.Estimator(model=model, paramGrid=paramGrid, searchKwargs=searchKwargs)
    
    # sklearn metrics
    m = [measure.Metric(metric) for metric in (r2_score, explained_variance_score, median_absolute_error)]
    # donatello custom metric
    m.append(measure.FeatureWeights())
    
    sculpture = core.Sculpture(dataset=d, estimator=e, metrics=m,
                               validation=True, holdout=True)

    return sculpture

df.select_dtypes(pd.np.number).drop('zipcode', axis=1).head()
sculpture = load_sculpture(df.select_dtypes(pd.np.number).drop('zipcode', axis=1))
sculpture.fit()
from donatello.utils.helpers import view_sk_metric

for name, scores in sculpture.measurements.crossValidation.items():
    if name != 'feature_weights':
        display(view_sk_metric(scores).rename(columns={'score': name}))
    else:
        display(pd.concat([scores.mean.rename(columns={i: '{}_mean'.format(i) for i in scores.mean}), 
                           scores.std.rename(columns={i: '{}_std'.format(i) for i in scores.std})],
                           axis=1).sort_values('coefficients_mean')
               )<jupyter_output><empty_output><jupyter_text>target transforms on an execution graphScikit-learn currently doesn't support y transforms or record number modify transform (imblearn does, but placesrestrictions on what types of transformers can be embedded in Pipelines)Donatello has you covered.The Apply and AccessTransformer allows you to inject a transform function that modifes the design, the target, or both and can be applied only during fit or<jupyter_code>def build_mask(dataset, columns, threshold):
    X = scale(dataset.designData[columns])
    y = dataset.targetData.values
    model = OLS(y, X)
    mask = pd.np.abs((model.fit().outlier_test()[:, 0]) < threshold)
    return mask

def transform(dataset, columns=('square_footage', 'num_bedrooms', 'last_sale_amount'), threshold=2):
    columns = list(columns) if columns else list(dataset.designData)
    mask = build_mask(dataset, columns, threshold)
    X = dataset.designData.loc[mask]
    y = dataset.targetData.loc[mask]
    return X, y<jupyter_output><empty_output><jupyter_text>Here we use the fitOnly option to apply this transformation on fitting pipeline but not the scoring pipeline.<jupyter_code>outlier_removal = transformers.Apply(func=transform, fitOnly=True)
d = sculpture.dataset
d_t = outlier_removal.fit_transform(dataset=d.subset('train'))<jupyter_output>/anaconda/lib/python2.7/site-packages/ipykernel_launcher.py:2: DataConversionWarning: Data with input dtype int64 were all converted to float64 by the scale function.<jupyter_text>QA dataset sizes for a given response (also note the subet method if you're not familiar with it). The last member demonstrates the already fit transformer not altering the dataset<jupyter_code>len(d), len(d.subset('train')), len(d_t), len(d_t.designTrain), len(outlier_removal.transform(dataset=d_t.subset('train')))
def plot(columns=['square_footage', 'num_bedrooms']):
    mask = build_mask(d.subset('train'), columns, 2)
    
    for column in columns:
        plt.figure(figsize=(18,8))
        plt.xlim(0, d.designTrain[column].quantile(.98) * 1.05) 
        plt.ylim(0, d.targetTrain.max() * 1.01) 
        plt.plot(d.designTrain.loc[mask, column].values, d.targetData.loc[mask].values, 'b.', alpha=.1)
        plt.plot(d.designData.loc[~mask, column].values, d.targetData.loc[~mask].values, 'r*', alpha=.75)
        plt.legend(['inliers', 'outliers'])
        plt.xlabel(column)
        plt.ylabel('expected price in current USD')
        plt.show()
plot()<jupyter_output>/anaconda/lib/python2.7/site-packages/ipykernel_launcher.py:2: DataConversionWarning: Data with input dtype int64 were all converted to float64 by the scale function.<jupyter_text>Donatello's execution graph is built on top of networkx and scikit-learn. To ensure parameters are passed through, and estimator clones are performed properly, the ModelDAG requires a set of nodes and a dict of edges. The graph can be intialized with these but  it's typically easier to use the helper methods to build the graph.Scikit-learn transformers do not natively enable passing the target through,<jupyter_code>model = transformers.ModelDAG(set([]), {})
n1 = transformers.Node('scale', transformers.StandardScaler(), enforceTarget=True)
n2 = transformers.Node('rm_outliers', transformers.Apply(func=transform, fitOnly=True))
n3 = transformers.Node('ml', LinearRegression())
model.add_edge_flow(n1, n2)
model.add_edge_flow(n2, n3)
sculpture = load_sculpture(df.select_dtypes(pd.np.number).drop('zipcode', axis=1), model=model)
sculpture.fit()
for name, scores in sculpture.measurements.crossValidation.items():
    if name != 'feature_weights':
        display(view_sk_metric(scores).rename(columns={'score': name}))
    else:
        display(pd.concat([scores.mean.rename(columns={i: '{}_mean'.format(i) for i in scores.mean}), 
                           scores.std.rename(columns={i: '{}_std'.format(i) for i in scores.std})],
                           axis=1).sort_values('coefficients_mean')
               )<jupyter_output><empty_output><jupyter_text>BranchingScikit-learn enables branching feature engineering by nesting FeatureUnions and Pipelines. While servicable, this interface can be hard to manage and makes it more difficult to share transformers and components between problems.To support this feature, the edges of donatello's ModelDAG are actually transformers themselves with the sole task of passing (conducting) data through them. Conduction transformers follow a pattern. By default all columns/keys found during fitting will be passed along but the ModelDAG's add_edge_conductor method makes it easy to modify these options but selecting or inverting (selecting all but) the columns seen during fit by name, a regex match, or dtype.<jupyter_code>model = transformers.ModelDAG(set([]), {})
# intitate branching by selecting all
n0 = transformers.Node('select', transformers.DatasetFlow(invert=True)) 

# first branch
n11 = transformers.Node('ohe', transformers.OneHotEncoder(dropOne=True))

# second branch
n21 = transformers.Node('scale', transformers.StandardScaler(), enforceTarget=True)
n22 = transformers.Node('rm_outliers', transformers.Apply(func=transform, fitOnly=True))

# terminal node
n3 = transformers.Node('ml', LinearRegression())
# send zipcode data only to OHE and don't pass target through first branch
model.add_edge_flow(n0, n11, passTarget=False, selectValue=['zipcode'], invert=False)
# send output of ohe to Linear Regression 
model.add_edge_flow(n11, n3)

# send all other design data and the target through second branch
model.add_edge_flow(n0, n21, selectValue=['zipcode'], invert=True)
model.add_edge_flow(n21, n22)
model.add_edge_flow(n22, n3)
sculpture = load_sculpture(df.select_dtypes(pd.np.number), model=model)
sculpture.fit()
print('{} fold validation'.format(sculpture.dataset.fold.folder.get_n_splits()))
for name, scores in sculpture.measurements.crossValidation.items():
    if name != 'feature_weights':
        display(view_sk_metric(scores).rename(columns={'score': name}))
    else:
        display(pd.concat([scores.mean.rename(columns={i: '{}_mean'.format(i) for i in scores.mean}), 
                           scores.std.rename(columns={i: '{}_std'.format(i) for i in scores.std})],
                           axis=1).sort_values('coefficients_mean')
               )<jupyter_output>5 fold validation
{'fraction_non_alphanumeric': 0.11174523456226458, 'fraction_numerical': 0.008218239926948978, 'mean_word_length': 4.641983258209916, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '482639', 'license': 'MIT', 'n_tokens_mistral': 2722, 'n_tokens_neox': 2625, 'n_words': 711, 'path': 'notebooks/housing_prices.ipynb', 'repo_name': 'marksweissma/donatello', 'size': '8761'}