<jupyter_start><jupyter_text>CRISP-DM process Business UnderstandingWhen living on crowded cities or big urban centers, traffic won't pass unnoticed. All day long cars, trucks, buses and motorcycles interact, competing for street space, and in motion. Is there any way a to proof assumptions about traffic using data science? For sure is! In this notebook will be ansewered the following questions:* **Which lane on a road have the highest speed?*** **How does the features correlate?*** **How good can a new car be classified?** Data UnderstandingOn the code below will be explored data from traffic radars of São Paulo, on March, 2018.The features here have the especifications about lenght, type and register of vehycles and the place, lane where it was capture. All is organized by date.  The next steps of CRISP-DM process are done for each analysis * Prepare Data* Data Modeling* Evaluate the Results  Which lane on a road have the highest speed? Importing first databasesRadars.csv contains all cars, trucks, motorcycles and buses that comes thru São Paulo's radar system<jupyter_code># to download the data, access https://drive.google.com/file/d/1VG-t8t1mq_HZV-Ue6zkth_3Ne4IfCOrq/view?usp=sharing 

df_base = pd.read_csv(r"Radar_Data_2.03.2018.csv", index_col= "Data")

df_base.head()<jupyter_output><empty_output><jupyter_text>Reading the columns* **Radar** is the number of identification of a street section* **Lane** goes from 1 to 6 in most radars, low lane number are closer to the center of the freeway, high lane numbers are "local" lanes, to the right* **Register** represents each vehicle* **Types** are: motorcycle = 0, car = 1, bus = 2 ou truck = 3* **Classes** are: *light* (motorcycle and car) = 0 ou *heavy* (bus and truck)  = 1* **Speeds** are in kilometer per hour* **Radar_Lane** comes to identify each lane on a single radar (will be usefull to merge dataframes)  Prepare Data<jupyter_code># Preprocessing

df = df_base[["Numero Agrupado", "Faixa", "Registro", "Especie", "Classe", "Velocidade"]]
# turns speed from dm/s to km/h
df.loc[:,"Velocidade"] = df.loc[:,"Velocidade"] * 0.36

df.index.names = ["Date"]

df["Radar_Lane"] = df["Numero Agrupado"].astype(str) + df["Faixa"].astype(str)

# renaming columns to english
df.columns = ["Radar", "Lane", "Register", "Type", "Class", "Speed [km/h]", "Radar_Lane"]

df.head()<jupyter_output><empty_output><jupyter_text>Lane types databaseHelps to tell the **use of each lane** . "Tipo" contains the information of lanes where all types of vehicycles can use ( *mix_use* ) and other that are for buses only ( *exclusive_bus* )<jupyter_code>lane_types = pd.read_excel(r"Lanes_type.xlsx", usecols = ["Num_agrupado"," faixa", "Num_fx", "tipo"],engine='openpyxl')

lane_types.head()<jupyter_output><empty_output><jupyter_text>Merge dataframesTo identify the type of the lane, if it is exclusive for buses, or multipurpose<jupyter_code>df_merged = lane_types[["Num_fx", "tipo"]].merge(df, left_on = "Num_fx", right_on = "Radar_Lane", how="right")

df_merged["Lane_use"] = df_merged["tipo"].map({"mista":"mix_use", "onibus": "exclusive_bus"})

df_merged = df_merged[["Radar", "Lane", "Register", "Type", "Class", "Speed [km/h]", "Lane_use"]]

df_merged.head()<jupyter_output><empty_output><jupyter_text>Looking for NaNsAs shown below, NaNs are less than 1% (actually, less than 0,2%)With this information, there will be low loss in dropping NaNs<jupyter_code>print(df_merged.isna().mean() * 100)

df_merged.dropna(inplace=True)<jupyter_output>Radar           0.000000
Lane            0.000000
Register        0.000000
Type            0.185455
Class           0.185455
Speed [km/h]    0.185455
Lane_use        0.000000
dtype: float64<jupyter_text>Selection of LanesUsing only the data from mix_use lanes, select for each lane to create comparisonThe max number of lanes is 6, but only few roads have all 6, so it can be excluded from the analysis<jupyter_code>lanes = df_merged.loc[df_merged["Lane_use"] == "mix_use"]

lane_1 = lanes.loc[lanes["Lane"] == 1]

lane_2 = lanes.loc[lanes["Lane"] == 2]

lane_3 = lanes.loc[lanes["Lane"] == 3]

lane_4 = lanes.loc[lanes["Lane"] == 4]

lane_5 = lanes.loc[lanes["Lane"] == 5]

lane_6 = lanes.loc[lanes["Lane"] == 6]

print(lane_1.shape, lane_2.shape, lane_3.shape, lane_4.shape, lane_5.shape, lane_6.shape)<jupyter_output>(364148, 7) (586260, 7) (436164, 7) (167320, 7) (22417, 7) (6075, 7)<jupyter_text>Plotting the means Data Modeling<jupyter_code>means = [ ]

for lane in [lane_1,lane_2,lane_3,lane_4,lane_5]:
    means.append(lane["Speed [km/h]"].mean())

means = [ round(elem, 2) for elem in means ]


fig, ax = plt.subplots()

rects = ax.bar([1,2,3,4,5],means, width= 0.5)

ax.set_ylabel("Speed [km/h]")
ax.set_xlabel("Lanes")
ax.set_title('Speeds per lane')


def autolabel(rects):
    """Attach a text label above each bar in *rects*, displaying its height."""
    for rect in rects:
        height = rect.get_height()
        ax.annotate('{}'.format(height),
                    xy=(rect.get_x() + rect.get_width() / 2, height),
                    xytext=(0, 3),  # 3 points vertical offset
                    textcoords="offset points",
                    ha='center', va='bottom')

autolabel(rects)


plt.show()<jupyter_output><empty_output><jupyter_text>Evaluating the ResultsCenter lanes move faster than "local" lanes  How does the features correlate?The new column is **Length** that represents the size of a vehicycle. Must be turn into meters  Prepare Data<jupyter_code>df_corr = df_base[["Numero Agrupado", "Faixa", "Registro", "Especie", "Classe", "Velocidade", "Comprimento"]]

df_corr.loc[:,"Comprimento"] = df_corr.loc[:,"Comprimento"] /10

df_corr.loc[:,"Velocidade"] = df_corr.loc[:,"Velocidade"] * 0.36

df_corr.columns = ["Radar", "Lane", "Register", "Type", "Class", "Speed [km/h]", "Length"]<jupyter_output><empty_output><jupyter_text>Data Modeling<jupyter_code>corrMatrix = df_corr[["Lane", "Type", "Speed [km/h]","Length"]].corr()
sns.heatmap(corrMatrix, annot=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluating the ResultsIs easy to see that *speed/lane* and *length/type* have the biggest correlationsHigh speeds correlates to low lane numbers, that means that center lanes move faster, as seen in other analysis (remembering the lane number grow from left to right)Big lengths correlates with greater types that are heavier vehicycles like buses and trucks  How good can a new car be classified? Prepare Data<jupyter_code>df_regression = df_base[["Numero Agrupado", "Faixa", "Registro", "Especie", "Classe", "Velocidade", "Comprimento"]]

df_regression.loc[:,"Comprimento"] = df_regression.loc[:,"Comprimento"] /10

df_regression.loc[:,"Velocidade"] = df_regression.loc[:,"Velocidade"] * 0.36

df_regression.columns = ["Radar", "Lane", "Register", "Type", "Class", "Speed [km/h]", "Length"]


Validation = df_regression.loc[df_regression["Speed [km/h]"].isna()]
X = df_regression[["Lane", "Type", "Class", "Length"]].dropna()
X = pd.concat([pd.get_dummies(X[["Lane", "Type", "Class"]].astype("object")),X["Length"]], axis=1)

y = df_regression["Speed [km/h]"].dropna()


X.head()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Data Modeling<jupyter_code>regr = RandomForestRegressor(n_estimators=50, max_depth=10, random_state=42)
regr.fit(X_train, y_train)

print("R2 score with n_estimators={} and max_depth={} is {} ".format(50,10,regr.score(X_test,y_test)))<jupyter_output>R2 score with n_estimators=50 and  max_depth=10 is 0.13035592245707572
{'fraction_non_alphanumeric': 0.14003685180310607, 'fraction_numerical': 0.027638852329560412, 'mean_word_length': 5.478260869565218, 'pattern_counts': {'":': 2, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '426961', 'license': 'MIT', 'n_tokens_mistral': 2830, 'n_tokens_neox': 2623, 'n_words': 758, 'path': 'Radar_analysis.ipynb', 'repo_name': 'guilistocco/Udacity-Car_Radar_Lanes', 'size': '7598'}