<jupyter_start><jupyter_text>MSDS692 Project: Data Preprocessing - Sunrise and Sunset Times for Denver, CO Natalia Weakly     Original data source: https://www.timeanddate.com/<jupyter_code># Imports
import pandas as pd
import numpy as np
import os
import datetime<jupyter_output><empty_output><jupyter_text>Data load and preprocessing<jupyter_code># Load data
timeOfDay=pd.read_csv('MSDS692_Denver_SunriseSunset_5.csv', infer_datetime_format=True)
# Check the data
timeOfDay.head()
timeOfDay.tail()
# Structure of the data frame
timeOfDay.info()
# Check if there are any missing values
timeOfDay.isnull().values.any()
# Create a 'fullDate' column by concatenating 'Year', 'Month' and 'Day'
timeOfDay['fullDate'] = timeOfDay['Year'].astype(str) + '/' + timeOfDay['Month'] + '/' + timeOfDay['Day'].astype(str)
timeOfDay.head()
#Check results
timeOfDay.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1916 entries, 0 to 1915
Data columns (total 12 columns):
Year                          1916 non-null int64
Month                         1916 non-null object
Day                           1916 non-null int64
AstronomicalTwilight_Start    1916 non-null object
AstronomicalTwilight_End      1916 non-null object
NauticalTwilight_Start        1916 non-null object
NauticalTwilight_End          1916 non-null object
CivilTwilight_Start           1916 non-null object
CivilTwilight_End             1916 non-null object
Sunrise                       1916 non-null object
Sunset                        1916 non-null object
fullDate                      1916 non-null object
dtypes: int64(2), object(10)
memory usage: 179.7+ KB<jupyter_text>Add additional columns for different twighlight time In order to investigate whether there is a link between car accidents and natural light conditions,  let's add columns showing when twilight conditions began and ended on each particular date (later to be compared with accident times). Twilight is the time between day and night when there is light outside, but the sun is below the horizon. There are three types of twilight:  civil, nautical, and astronomical. Twilight occurs because of the Earth's upper atmosphere reflects sunlight and illuminates the lower atmosphere. So, its three stages are defined depending on the Sun's elevation (angle of its geometric center with the horizon) as shown below:Image credit: timeanddate.com<jupyter_code># Create a column for astronomical Twilihgt start as a full date/time  (as a string)
timeOfDay['AstroT_Start']=timeOfDay['fullDate'] + ' ' + timeOfDay['AstronomicalTwilight_Start'].astype(str)
# Check results
timeOfDay.head()
# Convert 'AstroT_Start' to the proper date/time format
timeOfDay['AstroT_Start'] = pd.to_datetime(timeOfDay['AstroT_Start'])
# Preview results
timeOfDay.head()
# Similarly, create properly formated column for the AstronomicalTwilight_End
# AstronomicalTwilight_End
# concatenate
timeOfDay['AstroT_End']=timeOfDay['fullDate'] + ' ' + timeOfDay['AstronomicalTwilight_End'].astype(str)
# convert to date and time format
timeOfDay['AstroT_End'] = pd.to_datetime(timeOfDay['AstroT_End'])
# check results
timeOfDay.head()
# NauticalTwilight_Start
# concatenate
timeOfDay['NauticalT_Start']=timeOfDay['fullDate'] + ' ' + timeOfDay['NauticalTwilight_Start'].astype(str)
# convert to date and time format
timeOfDay['NauticalT_Start'] = pd.to_datetime(timeOfDay['NauticalT_Start'])
# check results
timeOfDay.head()
# NauticalTwilight_End
# concatenate
timeOfDay['NauticalT_End']=timeOfDay['fullDate'] + ' ' + timeOfDay['NauticalTwilight_End'].astype(str)
# convert to date and time format
timeOfDay['NauticalT_End'] = pd.to_datetime(timeOfDay['NauticalT_End'])
# check results
timeOfDay.head()
# CivilTwilight_Start
# concatenate
timeOfDay['CivilT_Start']=timeOfDay['fullDate'] + ' ' + timeOfDay['CivilTwilight_Start'].astype(str)
# convert to date and time format
timeOfDay['CivilT_Start'] = pd.to_datetime(timeOfDay['CivilT_Start'])
# check results
timeOfDay.head()
# CivilTwilight_End
# concatenate
timeOfDay['CivilT_End']=timeOfDay['fullDate'] + ' ' + timeOfDay['CivilTwilight_End'].astype(str)
# convert to date and time format
timeOfDay['CivilT_End'] = pd.to_datetime(timeOfDay['CivilT_End'])
# check results
timeOfDay.head()
# Sunrise
# concatenate
timeOfDay['Sunrise']=timeOfDay['fullDate'] + ' ' + timeOfDay['Sunrise'].astype(str)
# convert to date and time format
timeOfDay['Sunrise'] = pd.to_datetime(timeOfDay['Sunrise'])
# check results
timeOfDay.head()
# Sunset
# concatenate
timeOfDay['Sunset']=timeOfDay['fullDate'] + ' ' + timeOfDay['Sunset'].astype(str)
# convert to date and time format
timeOfDay['Sunset'] = pd.to_datetime(timeOfDay['Sunset'])
# check results
timeOfDay.head()
# Convert the 'FullDate' to DateTime format
timeOfDay['fullDate'] =pd.to_datetime(timeOfDay['fullDate'])
# Check results
timeOfDay.head()
timeOfDay.info()
# Copy newly created columns in datetime format to a new data frame
naturalLightConditions = timeOfDay[['fullDate', 'AstroT_Start', 'AstroT_End', 'NauticalT_Start', 'NauticalT_End', 'CivilT_Start', 'CivilT_End', 'Sunrise', 'Sunset']]
# check results
naturalLightConditions.head()
naturalLightConditions.info()
# output naturalLightConditions to a file for future use
naturalLightConditions.to_csv('naturalLightConditions.csv', date_format='%Y-%m-%d %H:%M:%S')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10661696178937559, 'fraction_numerical': 0.01472506989748369, 'mean_word_length': 5.649318463444858, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '557392', 'license': 'MIT', 'n_tokens_mistral': 1790, 'n_tokens_neox': 1677, 'n_words': 446, 'path': 'Code/MSDS692_SunriseSunsetData.ipynb', 'repo_name': 'nweakly/MSDSProjectGitHub', 'size': '5365'}