<jupyter_start><jupyter_text>经验证：* 1、test的时间顺序并不是全部都是时序的，排行榜是按时序排列后取前0.78为public，后0.22为private，所以以后的工作重心将落到train的后44%* 2、测试集中私有部分从2018-07-24 09:00:00开始，有9173472条数据，包含全部的building_id* 3、测试集中公共部分与私有部分对应的数据从2017-07-24 09:00:00开始，有9174900条数据，包含全部的building_id* 4、重点为训练集中2017-07-24 09:00:00后的数据建模，有9003109条数据，占训练集0.445* 5、为防止测试集中不评分的那部分不知道算不算在0.22里面，也为防止一定的过拟合，可以考虑重点关注2017-07月后的训练数据，占0.51* 6、验证EDA1后，发现0-104号建筑5月20日前的大量为0的数据是有误的，训练集不应该对这部分数据进行过拟合，删除这部分数据训练对提高public评分很有帮助，遗憾的是，private并不包含这类数据* 7、猜想某些大范围为0的数据都同0-104号一样是有误的，重点验证7月后的该种数据，测试是否能提高public<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
# Suppress warnings 
import warnings
warnings.filterwarnings('ignore')
import gc


# matplotlib and seaborn for plotting
import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns
import matplotlib.patches as patches

from plotly import tools, subplots
import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.express as px
pd.set_option('max_columns', 100)


py.init_notebook_mode(connected=True)
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
import plotly.graph_objs as go

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('../input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
# 导入数据, 只有train 和 test
train = pd.read_csv('../input/ashrae-energy-prediction/train.csv')
test = pd.read_csv('../input/ashrae-energy-prediction/test.csv')
# 时间戳转换日期
train["timestamp"] = pd.to_datetime(train["timestamp"])
test["timestamp"] = pd.to_datetime(test["timestamp"])
train_exception = pd.read_pickle('../output/fork-of-ashrae-eda-exception-label2/train_exception.pkl')
train['exception1'] = train_exception.exception.values
del train_exception
gc.collect()

train_exception = pd.read_pickle('../output/ashrae-eda-exception-label5/train_exception.pkl')
train['exception'] = train_exception.exception.values
del train_exception
gc.collect()
test_exception = pd.read_pickle('../output/fork-of-ashrae-eda-exception-label2/test_exception.pkl')
test['exception1'] = test_exception.exception.values
del test_exception
gc.collect()

test_exception = pd.read_pickle('../output/ashrae-eda-exception-label5/test_exception.pkl')
test['exception'] = test_exception.exception.values
del test_exception
gc.collect()
gc.collect()
train.exception.value_counts(dropna=False)
train.exception1.value_counts(dropna=False)
test.exception.value_counts(dropna=False)
test.exception1.value_counts(dropna=False)
test[(test.building_id == 235) & (test.meter == 1)].exception.value_counts(dropna=False)
test[(test.building_id == 235) & (test.meter == 1)].exception1.value_counts(dropna=False)
updata_id = test[(test.building_id == 235) & (test.meter == 1) & (test.exception1 == 1)].row_id.values
len(updata_id)
test.loc[updata_id, 'exception'] = 1
test[(test.building_id == 235) & (test.meter == 1)].exception.value_counts(dropna=False)
test.exception.value_counts(dropna=False)
# ## 存疑异常标记

# # 30前期有大量0值
# train.loc[(train.building_id == 30) & (train.meter == 1) & (train.meter_reading < 212), 'exception'] = 2

# # 43大量0值
# train.loc[(train.building_id == 43) & (train.meter == 1) & (train.meter_reading == 0), 'exception'] = 2

# # 28大量0值
# train.loc[(train.building_id == 28) & (train.meter == 1) & (train.meter_reading == 0), 'exception'] = 2
gc.collect()
train[['exception']].to_pickle('../output/fork-of-ashrae-eda-exception-label5/train_exception.pkl')
test[['exception']].to_pickle('../output/fork-of-ashrae-eda-exception-label5/test_exception.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1412193850964044, 'fraction_numerical': 0.04377279833246483, 'mean_word_length': 6.693386773547094, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '66290', 'license': 'MIT', 'n_tokens_mistral': 1684, 'n_tokens_neox': 1611, 'n_words': 225, 'path': 'solutions/rank-4/exception_label/fork-of-ashrae-eda-exception-label5.ipynb', 'repo_name': 'madesgriya/ashrae-great-energy-predictor-3-solution-analysis', 'size': '3838'}