<jupyter_start><jupyter_text>MCIS6273 Data Mining (Prof. Maull) / Fall 2021 / HW0**This assignment is worth up to 20 POINTS to your grade total if you complete it on time.**| Points Possible | Due Date | Time Commitment (estimated) ||:---------------:|:--------:|:---------------:|| 20 | Wednesday, Sep 1 @ Midnight | _up to_ 20 hours |* **GRADING:** Grading will be aligned with the completeness of the objectives.* **INDEPENDENT WORK:** Copying, cheating, plagiarism  and academic dishonesty _are not tolerated_ by University or course policy.  Please see the syllabus for the full departmental and University statement on the academic code of honor. OBJECTIVES* Familiarize yourself with the JupyterLab environment, Markdown and Python* Familiarize yourself with Github and basic git* Explore JupyterHub Linux console integrating what you learned in the prior parts of this homework* Listen to the Talk Python['Podcast'] from June 25, 2021: A Path to Data Science Interview with <NAME>* Explore Python for data munging and analysis, with an introduction to CSV and Pandas WHAT TO TURN INYou are being encouraged to turn the assignment in using the providedJupyter Notebook.  To do so, make a directory in your Lab environment called`homework/hw0`.   Put all of your files in that directory.  Then zip that directory,rename it with your name as the first part of the filename (e.g. `maull_hw0_files.zip`), thendownload it to your local machine, then upload the `.zip` to Blackboard.If you do not know how to do this, please ask, or visit one of the many tutorials out thereon the basics of using zip in Linux.If you choose not to use the provided notebook, you will still need to turn in a`.ipynb` Jupyter Notebook and corresponding files according to the instructions inthis homework. ASSIGNMENT TASKS (0%) Familiarize yourself with the JupyterLab environment, Markdown and Python As stated in the course announcement [Jupyter (https://jupyter.org)](https://jupyter.org) is thecore platform we will be using in this course andis a popular platform for data scientists around the world.  We have a JupyterLabsetup for this course so that we can operate in a cloud-hosted environment, free fromsome of the resource constraints of running Jupyter on your local machine (though you are free to setit up on your own and seek my advice if you desire).You have been given the information about the  Jupyter environment we have setup for our course, andthe underlying Python environment will be using is the [Anaconda (https://anaconda.com)](https://anaconda.com)distribution.  It is not necessary for this assignment, but you are free to look at the multitudeof packages installed with Anaconda, though we will not use the majority of them explicitly.As you will soon find out, Notebooks are an incredibly effective way to mix code with narrativeand you can create cells that are entirely code or entirely Markdown.  Markdown (MD or `md`) isa highly readable text format that allows for easy documentation of text files, while allowingfor HTML-based rendering of the text in a way that is style-independent.We will be using Markdown frequently in this course, and you will learn that there are many different"flavors" or Markdown.  We will only be using the basic flavor, but you will benefit from exploringthe "Github flavored" Markdown, though you will not be responsible for using it in this course -- only the"basic" flavor.  Please refer to the original course announcement about Markdown.&167;  **THERE IS NOTHING TO TURN IN FOR THIS PART.** Play with and become familiar with the basic functions ofthe Lab environment given to you online in the course Blackboard.&167;  **PLEASE _CREATE A MARKDOWN DOCUMENT_ CALLED `semester_goals.md` WITH 3 SENTENCES/FRAGMENTS THATANSWER THE FOLLOWING QUESTION:*** **What do you wish to accomplish this semester in Data Mining?**Read the documentation for basic Markdown [here](https://www.markdownguide.org/basic-syntax). Turn in the text `.md` file *not* the processed `.html`.  In whatever you turn in, you must show the use of *ALL* the following:* headings (one level is fine),* bullets,* bold and italicsAgain, the content of your document needs to address the question above and it should livein the top level directory of your assignment submission.  This part will be graded but nopoints are awarded for your answer. (0%) Familiarize yourself with Github and basic git [Github (https://github.com)](https://github.com) is the _de facto_ platform for open source software in the world basedon the very popular [git (https://git-scm.org)](https://git-scm.org) version control system. Git has a sophisticated setof tools for version control based on the concept of local repositories for fast commits and remoterepositories only when collaboration and remote synchronization is necessary.  Github enhances git by providingtools and online hosting of public and private repositories to encourage and promote sharing and collaboration.Github hosts some of the world's most widely used open source software.**If you are already familiar with git and Github, then this part will be very easy!**&167;  **CREATE A PUBLIC GITHUB REPO NAMED `"mcis6273-F21-datamining"` AND PLACE A README.MD FILE IN IT.**Create your first file called`README.md` at the top level of the repository.  You can put whatever text you like in the file (If you like, use something like [lorem ipsum](https://lipsum.com/)to generate random sentences to place in the file.).Please include the link to **your** Github repository that now includes the minimal `README.md`. You don't have to have anything elaborate in that file or the repo.  (0%) Explore JupyterHub Linux console integrating what you learned in the prior parts of this homework The Linux console in JupyterLab is a great way to perform command-line tasks and is an essential toolfor basic scripting that is part of a data scientist's toolkit.  Open a console in the lab environmentand familiarize yourself with your files and basic commands using git as indicated below.1. In a new JupyterLab command line console, run the `git clone` command to clone the new  repository you created in the prior part.  You will want to read the documentation on this   command (try here [https://www.git-scm.com/docs/git-clone](https://www.git-scm.com/docs/git-clone) to get a good  start).2. Within the same console, modify your `README.md` file, check it in and push it back to your repository, using  `git push`.  Read the [documentation about `git push`](https://git-scm.com/docs/git-push).3. The commands `wget` and `curl` are useful for grabbing data and files from remote resources off the web.  Read the documentation on each of these commands by typing `man wget` or `man curl` in the terminal.  Make sure you pipe the output to a file or use the proper flags to do so.&167;  **THERE IS NOTHING TO TURN IN FOR THIS PART.**  (30%) Listen to the Talk Python['Podcast'] from June 25, 2021: A Path to Data Science Interview with <NAME> Data science is one of the most important and "hot" disciplines todayand there is a lot going on from data engineering to modeling andanalysis.Bhutani is one of the top [Kaggle]() leaders and in this interviewshares his experience from computer science to data science, documenting some of the lessons he learned along the way.Please listen to this one hour podcast and answer some of the questions below.You can listen to it from one of the two links below:* [Talk Python['Podcast'] landing page](https://talkpython.fm/episodes/transcript/322/a-path-into-data-science)* [direct link to mp3 file](https://downloads.talkpython.fm/podcasts/talkpython/322-starting-in-data-sci.mp3)&167;  **PLEASE ANSWER THE FOLLOWING QUESTIONS AFTER LISTENING TO THE PODCAST:**1. List 3 things that you learned from this podcast?2. What is your reaction to the podcast?  Pick at least one point Sanyam brought up in the interview that you agree with and list your reason why.3. After listening to the podcast, do you think you are more interested or less interested in a career in Data Science? ***ANSWERS*** **1.List 3 things that you learned from this podcast?**As everyone says coding is tough though its all about the interest we build in ourselves.- Basic learning of algorithms to implement in real time projects, assignments and discover new things.- Self education and advancement to improve the skillset and ranking high on kaggle.- Top down versus bottom up learning and setting up goals.    - Deep learning on how you can get to the top of a leaderboard.**2.What is your reaction to the podcast? Pick at least one point Sanyam brought up in the interview that you agree with and list your reason why.**-  The TalkPython Podcast gives the real-time expereience of a person <NAME>. The path he travelled to learn Python language though in midway he was deviated but fought back to accomplish his goals.- To build the ability of programming he joined competitions, which gained him good experience and follow his path on coding.    - Keep self motivated    - have competitions    - got intro to fast ai    - knowledge on SENTRY**3.After listening to the podcast, do you think you are more interested or less interested in a career in Data Science?**- The podcast was very useful for me, I thought coding is only for super smart people. The life experience of Bhutni gives me more interested to pursue my career in data science. - As **Data Science** is all about the complete study of analysing data which can be finally explored in many ways, generating ideas and demonstrating.  (70%) Explore Python for data munging and analysis, with an introduction to CSV and Pandas Python's strengths shine when tasked with data munging and analysis.  As we will learn throughoutthe course, there are a number of excellent data sources for open data of all kinds nowavailable for the public.  These open data sources are heralding the new era of transparencyfrom all levels from small municipal data to big government data, from transportation, to science,to education.To warm up to such datasets, we will be working with an interestingdataset from the US Fish and Wildlife Service (FWS).  This is a water quality data set taken from a managed national refuge inVirginia called Back Bay National Wildlife Refuge, which was established in 1938.  As a function of being managed by the FWS, water quality samples are taken regularly from the marshes within the refuge.You can (and should) learn a little more about Back Bay fromthis link, since it has an interesting history, features and wildlife.* [https://www.fws.gov/refuge/Back_Bay/about.html](https://www.fws.gov/refuge/Back_Bay/about.html)The data we will be looking at can be found as a direct download from data.gov, the US data repository where many datasets froma variety of sources can be found -- mostly related to the multitude of US government agencies.The dataset is a small water quality dataset with several decadesof water quality data from Back Bay.  We will be warming upto this dataset with a basic investigation into the shape, contentand context of the data contained therein.In this part of the assignment, we will make use of Python libraries to pull the data from theendpoint and use [Pandas](https://pandas.pydata.org) to plot the data.  The raw CSV data isreadily imported into Pandas from the following URL:* [FWS Water Quality Data 12/20/2020](https://catalog.data.gov/dataset/water-quality-data/resource/f4d736fd-ade9-4e3f-b8e0-ae7fd98b2f87)Please take a look at the page, on it you will notice a link to the raw CSV file:* [https://ecos.fws.gov/ServCat/DownloadFile/173741?Reference=117348](https://ecos.fws.gov/ServCat/DownloadFile/173741?Reference=117348)We are going to explore this dataset to learn a bit more about the water quality characteristics of Bay Bay over the past couple decadesor so.&167;  **WRITE THE CODE IN YOUR NOTEBOOK TO LOAD AND RESHAPE THE COMPLETE CSV WATER QUALITY DATASET**:You will need to perform the following steps:1. **use [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) method to load the dataset** into a Pandas DataFrame;2. **clean the data so that the range of years is restricted to the 20 year period from 1999 to 2018**5. **store the entire dataset back into a new CSV** file called `back_bay_1998-2018_clean.csv`.**HINTS:** _Here are some a code hints you might like to study and use to craft a solution:_* study  [`pandas.DataFrame.query()]`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html?highlight=querypandas.DataFrame.query) to learn how to filter and query year ranges* study  [`pandas.DataFrame.groupby()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html?highlight=groupbypandas.DataFrame.groupby) to understand how to group data&167;  **USE PANDAS TO LOAD THE CSV DATA TO A DATAFRAME AND ANSWER THE FOLLOWING QUESTIONS:**1. How many and what are the names of the columns in this dataset?2. What is the mean `Dissolved Oxygen (mg/L)` over the entire dataset?3. Which year were the highest number of `AirTemp (C)` data points collected?4. Which year were the least number of `AirTemp (C)` data points collected?To answer these questions, you'll need to dive further into Pandas, which isthe standard tool in the Python data science stack for loading, manipulating,transforming, analyzing and preparing data as input to other tools such as[Numpy (http://www.numpy.org/)](http://www.numpy.org/), [SciKitLearn (http://scikit-learn.org/stable/index.html)](http://scikit-learn.org/stable/index.html), [NLTK (http://www.nltk.org/)](http://www.nltk.org/) and others.For this assignment, you will only need to learn how to load and select data using Pandas.* **LOADING DATA**The core data structure in Pandas is the `DataFrame`.  You will need to visitthe Pandas documentation [(https://pandas.pydata.org/pandas-docs/stable/reference/)](https://pandas.pydata.org/pandas-docs/stable/reference/)to learn more about the library, but to help you along with a hint, read thedocumentation on the [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) method.* **SELECTING DATA**The [tutorial here on indexing and selecting](http://pandas.pydata.org/pandas-docs/stable/indexing.html)should be of great use in understanding how to index and select subsets ofthe data to answer the questions.* **GROUPING DATA** You may use [`DataFrame.value_counts()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.value_counts.html?highlight=value_countspandas.DataFrame.value_counts) or [`DataFrame.groupby()`](https://pandas.pydata.org/pandas-docs/stable/reference/groupby.html) to groupthe data you need for these questons.  You will also find [`DataFrame.groupby()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html?highlight=groupbypandas.DataFrame.groupby) and [`DataFrame.describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html?highlight=describepandas.DataFrame.describe) very useful.**CODE HINTS**Here is example code that should give you clues about the structureof your code for this part.```python  import pandas as pd  df = pd.read_csv('your_json_file.csv')   code for question 1 ... and so on```&167;  **EXPLORING WATER SALINITY IN THE DATA**The Back Bay refuge is on the eastern coast of Virginia and tothe east is the Atlantic Ocean.  Salinity is a measure ofthe salt concentration of water, and you can learn a little moreabout salinity in water [here](https://www.usgs.gov/special-topic/water-science-school/science/saline-water-and-salinity?qt-science_center_objects=0qt-science_center_objects).You will notice that there is a `Site_Id` variable in the data, which we will find refers to the five sampling locations (see the [documentation here](https://ecos.fws.gov/ServCat/Reference/Profile/117348))of (1) the Bay, (2) D-Pool (fishing pond), (3) C-Pool, (4) B-Pool and (5) A-Pool.The ppt in Salinity is the percent salinity, and so 1 ppt is equivalent to 10000 ppm salinity.  Use this information to answer the following questions.1. Which sampling location has the highest mean ppt?  What is the equivalent ppm?2. When looking at the mean ppt, which location would you infer is furthest from the influence of ocean water inflows?    (Assume that higher salinity correlates to closer proximity to the ocean.)3. Dig a little deeper into 2, and write why there may be some uncertainty in your answer? (hint: certainty is improved by consistency in data)4. Use the data to determine the correlation between `Salinity (ppt)` and `pH (standard units)`.  Use the [DataFrame.corr()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.corr.html?highlight=correlate).  You just need to report the correlation value.   ***The following steps are:***<jupyter_code>import pandas as pd # importing the modules

mn = pd.read_csv('BKB_WaterQualityData_2020084.csv') #loading Dataset
mn.head() #data sheet
mn[mn.Year.eq(2018)] #Checking for end year 2018
mn1 = mn.query('Year >= 1999 & Year <= 2018') #displays the years between 1999 to 2018
mn1.head() #head part of the file
mn1.tail() #tail part of the file
mn1.to_csv(r'back_bay_1998-2018_clean.csv', index = False, header=True) #storing new csv file with name given<jupyter_output><empty_output><jupyter_text>***USE PANDAS TO LOAD THE CSV DATA TO A DATAFRAME AND ANSWER THE FOLLOWING QUESTIONS:***1. How many and what are the names of the columns in this dataset?***ANSWERS***<jupyter_code>mn1.info() #get the information

len(mn1.columns)# numbere of columns
mn1.columns # Names of the dataset
mn1.describe()<jupyter_output><empty_output><jupyter_text>2. What is the mean Dissolved Oxygen (mg/L) over the entire dataset?<jupyter_code>mn1["Dissolved Oxygen (mg/L)"].mean()<jupyter_output><empty_output><jupyter_text>***Answer:*** The Mean of Dissolved Oxygen (mg/L) is ***5.867704*** and the value is extracted from **mn1.describe()** 3. Which year were the highest number of AirTemp (C) data points collected?**Steps**To extract the data, replacing all the special characters.<jupyter_code>mn1.columns = [c.replace(' ', '_') for c in mn1.columns] #to replace ' 'with _
mn1.columns = [c.replace('(', '') for c in mn1.columns] #to replace ( with empty''

mn1.columns = [c.replace(')', '') for c in mn1.columns] #to replace ) with empty''
mn1.columns = [c.replace('/', '') for c in mn1.columns] #to replace / with empty''
mn1.info() #info of DataFrame<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1956 entries, 345 to 2370
Data columns (total 17 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Site_Id               1956 non-null   object 
 1   Unit_Id               32 non-null     object 
 2   Read_Date             1956 non-null   object 
 3   Salinity_ppt          1867 non-null   float64
 4   Dissolved_Oxygen_mgL  1150 non-null   float64
 5   pH_standard_units     1872 non-null   float64
 6   Secchi_Depth_m        1891 non-null   float64
 7   Water_Depth_m         1890 non-null   float64
 8   Water_Temp_?C         1842 non-null   float64
 9   Air_Temp-Celsius      46 non-null     float64
 10  Air_Temp_?F           1904 non-null   float64
 11  Time_24:00            1905 non-null   object 
 12  Field_Tech            1956 non-null   object 
 13  DateVerified          386 non-null    object 
 14  WhoVerified           386 non-null    object 
 15  AirTemp_C          [...]<jupyter_text>***Answer:*** From the above query **mn1.info** we extract the highest number of AirTemp (C) is ***33*** data points.<jupyter_code>mn1.query('AirTemp_C == 33') #query to display the row<jupyter_output><empty_output><jupyter_text>***Answer:*** From the above query the highest number of AirTemp (C) data points collected in the year **1999** 4. Which year were the least number of AirTemp (C) data points collected?<jupyter_code>mn1.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1956 entries, 345 to 2370
Data columns (total 17 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Site_Id               1956 non-null   object 
 1   Unit_Id               32 non-null     object 
 2   Read_Date             1956 non-null   object 
 3   Salinity_ppt          1867 non-null   float64
 4   Dissolved_Oxygen_mgL  1150 non-null   float64
 5   pH_standard_units     1872 non-null   float64
 6   Secchi_Depth_m        1891 non-null   float64
 7   Water_Depth_m         1890 non-null   float64
 8   Water_Temp_?C         1842 non-null   float64
 9   Air_Temp-Celsius      46 non-null     float64
 10  Air_Temp_?F           1904 non-null   float64
 11  Time_24:00            1905 non-null   object 
 12  Field_Tech            1956 non-null   object 
 13  DateVerified          386 non-null    object 
 14  WhoVerified           386 non-null    object 
 15  AirTemp_C          [...]<jupyter_text>***Answer:*** From the above given info the least number of AirTemp (C) data points collected is ***-17.777778.***<jupyter_code>mn1.query('AirTemp_C <= -17') #checking with the least data point<jupyter_output><empty_output><jupyter_text>Answer: The following years ***1999 2000 2001 2002 2004 2005 2006 2008 2010 2017*** are noticed holding the least number **-17.777778** of AirTemp (C) data points. ***§ EXPLORING WATER SALINITY IN THE DATA*** 1. Which sampling location has the highest mean ppt? What is the equivalent ppm?<jupyter_code>mn1.groupby("Site_Id").mean()
h_mean = float("-inf")
req_site = None
for site in set(mn1["Site_Id"]):
    mean = mn1[mn1["Site_Id"] == site]["Salinity_ppt"].mean()
    if mean > h_mean:
        h_mean = mean
        req_site = site

print("The sampling location is {} which has highest salinity. Mean in PPM = {}".format(req_site, h_mean * 10000))<jupyter_output>The sampling location is Bay which has highest salinity. Mean in PPM = 6360.671462829736<jupyter_text>***ANSWER***: Therefore the Sampling location is **BAY** and the equivalent ppm is **6360.671462829736** 2. When looking at the mean ppt, which location would you infer is furthest from the influence of ocean waterinflows? (Assume that higher salinity correlates to closer proximity to the ocean.)<jupyter_code>mn1.groupby("Site_Id").mean() .min()
mn1.query('Salinity_ppt == 0.000000') #row to check equals 0<jupyter_output><empty_output><jupyter_text>***ANSWER:*** Site_ID's ***A,C,D & Bay*** have the value **0.0** with lowest salinity, and considered as farthest city from the ocean  3. Dig a little deeper into 2, and write why there may be some uncertainty in your answer? (hint: certaintyis improved by consistency in data)<jupyter_code>mn1.query('Salinity_ppt == 0 & Dissolved_Oxygen_mgL == 0')<jupyter_output><empty_output><jupyter_text>***ANSWER:*** The dissolved oxygen with minimum is 0 and salinity 0 has Site ID with **A & B**. As salinity increases the solubility of dissolved oxygen decreases. Thus there is uncertainity in the dataset, due to fewer information or no values of salinity values, the caluculation is not done completely. 4. Use the data to determine the correlation between Salinity (ppt) and pH (standard units). Use theDataFrame.corr(). You just need to report the correlation value.<jupyter_code>mn1.corr() #query on correlation
salinity_ph = mn1[["Salinity_ppt", "pH_standard_units"]]
salinity_ph.corr()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0820373744650197, 'fraction_numerical': 0.025975676935463366, 'mean_word_length': 4.271387089568908, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 34, 'lorem ipsum': 1, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '550155', 'license': 'CC0-1.0', 'n_tokens_mistral': 7080, 'n_tokens_neox': 6469, 'n_words': 3028, 'path': 'homework/hw0/hw0.ipynb', 'repo_name': 'MatildaNesaraja/mcis6273_f21_datamining', 'size': '23623'}