<jupyter_start><jupyter_text>Project Programming for Data Analysis Student: <NAME> Student ID number: G00387884 For this project you must create a data set by simulating a real-world phenomenon ofyour choosing. You may pick any phenomenon you wish – you might pick one that isof interest to you in your personal or professional life. Then, rather than collect datarelated to the phenomenon, you should model and synthesise such data using Python.We suggest you use the numpy.random package for this purpose.Specifically, in this project you should:• Choose a real-world phenomenon that can be measured and for which you couldcollect at least one-hundred data points across at least four different variables.• Investigate the types of variables involved, their likely distributions, and theirrelationships with each other.• Synthesise/simulate a data set as closely matching their properties as possible.• Detail your research and implement the simulation in a Jupyter notebook – thedata set itself can simply be displayed in an output cell within the notebook.Note that this project is about simulation – you must synthesise a data set. Somestudents may already have some real-world data sets in their own files. It is okay tobase your synthesised data set on these should you wish (please reference it if you do),but the main task in this project is to create a synthesised data set. The next sectiongives an example project idea.  1.- Project plan.This project refers to a current problem that is to analyze health centers where a series of patients have been admitted as a result of the coronavirus. For this project we will use the Jupyter notebook. 2.- The problem to solve.First, we will identify a series of variables that we have collected from a health.csv file. Once the variables have been identified, we will start reading the data. For later data preprocessing. As a third step we will develop the modeling. And as a last step the visualization of the results for analysis. As you can see it is hoped to break this project down into several smaller tasks that are easier to solve and connect once they are completed. 3.- Methodology to followThe methodology for the development of this project is developed in 6 stages that make up a complete data analysis process.a) Identify the problem.b) Data collection.c) The cleaning of the data.d) Exploring the data.e) Data analysis.This project has been developed meeting the following objectives: 3.1. Research on the use of data to fight the coronavirus. 3.2. A series of data has been downloaded and our repository added. 3.3. Work begins on the project through a jupyter notebook:  I. Process for programming for data analysis. Technology* Anaconda* Python 3.8.2* Visual Studio code: March 2020 (version 1.44)* Git 2.26.1* Gihub* GDB Online is an online compilation and debugging tool for Anaconda, Python, compile, run and debug online from anywhere in the world.* Operating system: Ubuntu 19.10 - Windows 10 Pro - 64 bits Hardware requirements- Number of cores: 4- Release Date: Intel (R) Core (TM) i7-4790 CPU- RAM 8.00 GB- Clock frequency: 3.6 GHz- CPU benchmark score: 1833 Objective of the project.The main objective of this project is to work with the data set of health centers where data has been taken from possible coronavirus infections, for this we will use a jupyter and Python notebook.We will previously install [Anaconda](https://www.anaconda.com/products/individual) which is the most complete Suite for Data Science with Python.  II. Introduction.Currently the pandemic caused by covid 19 (coronavirus) has caused a high risk for all humanity, for this reason it is important to have information stored and prepared to help us fight the pandemic.However, despite the fact that we have information in those developing countries such as South America, they have had to fight not only against the pandemic but also with access to information, access to health, access to better living conditions, to decision making.Corruption in the middle of the pandemic has become more evident as is the case in Peru, Bolivia.The article "Use data to accelerate your business strategy", published in Harvard Business Review, states "The management team of the center understood that a better use of data must become a fundamental health practice" for all this, governments are giving consider the benefits of having a data-driven decision-making culture such as: 1. Identification of needs when implementing hospital facilities and thus contribute to reducing unnecessary expenses at the time of implementation. 2. Optimization of processes, which contributes to the reduction of operational costs. 3. Proactive monitoring to detect potential “unusual” situations and, in this way, avoid economic losses and identify improvements for the control environment. 4. Identification of opportunities for automation, cloud computing, machine learning and predictive analytics that could contribute to the simulation of scenarios to fight the pandemic.National and local governments must make their decisions based on data and with this they have the possibility of constantly identifying coronavirus cases and thus optimizing operations contributing to a better fight against the pandemic.Another important aspect that has increased as a result of the pandemic has been remote work and the slowdown in the economy as a result of the COVID-19 pandemic. All this aspect has meant for the companies the flexibility of their controls, the pressure to optimize operating costs and the prevention of situations that could cause economic losses.For all this, we have seen fit to work with a current problem. We have collected information from 30 health centers in Lima. In order to work on a current problem, we have collected information from **30 health centers in Metropolitan Lima** (you can view the data at [this link](https://github.com/javalpe/datasets/blob/master/health.csv)).![imagen](https://s.france24.com/media/display/46e4e94e-9c87-11ea-95ad-005056a964fe/482a8094d241cd30f07562ec0825827ec397dee4.webp) The variables that we have collected, that is, the columns, contained in the link table are described below: **ID:** identifier that distinguishes a health center from the rest**CENTRO_SALUD:** name as the health center in your district is known**PUNTAJE:** average score (1 to 5) as rated by the people who were treated at this health center. Source: Google Maps**INGRESO:** minimum income (S /.) Of the area where the health center is located, according to its socio-economic level. Source: INEI**POBLACION:** number of inhabitants of the area where the health center is located. Source: INEI**HOGARES:** number of dwellings in the area where the health center is located. Source: INEI**CONTAGIADOS_MIL:** number of people who tested positive for COVID (as of August 14, 2020) per thousand inhabitants of the district where the health center is located. Source: Matlab**POSITIVIDAD:** percentage that tested positive for COVID (as of August 14, 2020) of the total population of the district where the health center is located. Source: Matlab**DISTRITO:** district where the health center is located. Source: MatlabOur objective will be to **identify those critical health centers** based on the variables described above. However, due to the diversity of factors such as quality of care, available resources, and demand (population and positivity, for example) we resort to a **clustering algorithm** that allows us to group health centers considering all variables.Once we obtain the groups, we can **extract indicators from each group** that will allow us to identify the group that contains the most critical health centers.  III. Data ReadingTo begin, it is necessary to **transmit all the information** to the model that we are going to build and we must do it in a structured way so that it can be read without errors.![imagen](https://www.analiticaweb.es/wp-content/uploads/2017/06/Social-Map.jpg) [Pandas](https://pandas.pydata.org/) is a Python library that provides extensive means for data analysis. Data scientists often work with data stored in formats such as: .csv, .tsvo, .xlsx.All these formats are easy to process with pandas by getting a **table** (i.e. rows and columns).<jupyter_code>#Reading data with pandas
# Import: It is used to call the pandas library
# as: It is used to give a nickname to pandas, in this case the nickname that is assigned is pd
# pd.read_csv ('Location of csv file') This function reads the csv file, to
# In this case, the file is on the network.
import pandas as pd
datos = pd.read_csv("https://raw.githubusercontent.com/javalpe/datasets/master/salud.csv")
# We assign our dataframe in the data variable
#Verify the first records in data
# data = It is the name that we assigned to our imported dataframe a moment ago
# head () = It is what allows us to visualize, we can place any number
# within the relationship that will be the number of rows that will show us
# By default head () shows 5 rows
datos.head()
#shape is a python property that shows us the size, in this case of
#our dataframe, the result is (rows, columns)
datos.shape
#Verify data content
#With info we can see the type of variables.
#The type of variables that exist in python we defined previously
datos.info()
# describe (): It gives us a descriptive statistics report:
# count = number of records filled
# mean = average
# std = standard deviation
# min = minimum value
# max = maximum value
# x% = percentiles 
datos.describe()<jupyter_output><empty_output><jupyter_text>IV. Data preprocessingFor our model to work optimally it will be necessary to make some changes to the data that we have already read. At this stage we make modifications that allow us to evaluate the data in the best way. **Data scaling:** The goal of scaling is to change the values of the numeric columns of the data set to use a common scale, without distorting the differences in the value ranges, or losing information.For example, suppose that the input data set contains a column with values between 0 and 1, and another column with values between 10,000 and 100,000. Scaling avoids these problems by creating new values that maintain the distribution general within a scale that gives the same weight to all numeric columns used in the model.<jupyter_code>#From my dataframe data that I loaded previously I will take only the columns
#which I think are relevant to the analysis and I assign it in data_2
datos_2 = datos.loc[:, ['PUNTAJE', 'INGRESO', 'POBLACION', 'HOGARES','CONTAGIADOS_MIL', 'POSITIVIDAD']]
datos_2.head()
#Import from the sklearn library to MaxAbsScaler
#MaxAbsScaler: Scale each characteristic by its maximum absolute value.
# This estimator scales and translates each characteristic individually
# so that the maximum absolute value of each characteristic in the
# training set is 1.0. Does not shift / center data
# and therefore does not destroy any shortages.
from sklearn.preprocessing import MaxAbsScaler
abs_scaler = MaxAbsScaler() #Create the abs_scaler variables 
#abs_scaler = It is the variables that we created a moment ago, this variable contains
# the MaxAbsScaler () function, which will allow us to scale our data
# .fit () = Calculate the maximum absolute value to be used for subsequent scaling.
# .transform () = Fit to data, then transform.
abs_scaler.fit(datos_2)
scaled_data = abs_scaler.transform(datos_2) #The transformed (scaled) data is saved in scaled_data
#Create a new dataframe with the same name data_2, using the transformed data
#and the columns of the old dataframe which is also called data_2
datos_2 = pd.DataFrame(scaled_data, columns=datos_2.columns)
datos_2.head() #Finally we have our dataframe ready to process it in the algorithm.<jupyter_output><empty_output><jupyter_text>V. Modeling a) Reduction of dimensions: PCAPCA is a technique used to reduce the number of columns or predictor variables. For this example we are going to draw in a plane with two axes (X-Y) so we are going to reduce all the columns to only two.![imagen](https://i.imgur.com/R6qLbRS.png)<jupyter_code>#Import the PCA library from sklearn
from sklearn.decomposition import PCA
#Create the component variable to save the PCA there
componentes = PCA(n_components=2) # We specify within the PCA that we are going to reduce it to 2
# We train the PCA with the data stored in data_2 and reduce data_2 to 2 columns.
# We save the result in data_en_2D
datos_en_2D = componentes.fit_transform(datos_2)
#Import the seaborn library and rename it as sns so as not to write the word
#complete every time we use it
import seaborn as sns
#We are going to create two new columns PCA1 and PCA2 in data and store the values there
# of columns 0 and 1
datos['PCA1'] = datos_en_2D[:, 0]
datos['PCA2'] = datos_en_2D[:, 1]
# We are going to use the dot plot (scatterplot) from the seaborn library
sns.scatterplot(x='PCA1', y='PCA2',data=datos)
# We specify that the data is found in data that now already has the columns PCA1 and PCA2<jupyter_output><empty_output><jupyter_text>Thanks to the graph obtained with the two columns of the PCA, it is possible to visualize up to three **different areas** where all the data is concentrated. To check this we are going to use a clustering model(grouping) called **KMeans**.![imagen](https://i.imgur.com/Zkyeu70.png)  b) Grouping: KMEANSKMEANS is a grouping technique based on different observations recorded during a period of time, since the cluster analysis -or also called clustering- is carried out after taking a "photo" at a previously determined moment.We can visualize the grouping based on, for example, two variables as shown in the following image:![imagen](https://i.imgur.com/xRlapgn.jpg)<jupyter_code>#We import the KMeans library from sklearn
from sklearn.cluster import KMeans
#Create the grouping variable to store the KMeans there
agrupamiento = KMeans(n_clusters=3) # we specify within KMeans that we want to obtain three groups
#We train with the data stored in data_2 and we obtain the predictions of the groups.
grupos = agrupamiento.fit_predict(datos_2)<jupyter_output><empty_output><jupyter_text>Now we have obtained three groups, but remember that in Python you start counting from zero. Therefore, the first group is called 0 (zero), the second group is called 1 (one), and the third group is called 2 (two).<jupyter_code>#We are going to create a GROUPS column to store the groups generated by KMeans
datos['GRUPOS'] = grupos
#We are going to use the point plot (scatterplot) from the seaborn library
sns.scatterplot(x='PCA1', y='PCA2',data=datos, hue='GRUPOS')
# We specify in hue a filter or legend using the groups<jupyter_output><empty_output><jupyter_text>To better visualize the graph we are going to specify the colors assigned to each group, so we use an object called dictionary. In Python a **dictionary** is called through the variable **dict**<jupyter_code>#Create a dictionary called colors and specify a color for each group created: 0, 1 and 2
colores = dict({0:'red', 1:'blue', 2:'green'})
#We update the graph by entering our dictionary in palette inside the scatterplot
sns.scatterplot(x='PCA1', y='PCA2',data=datos, hue='GRUPOS', palette=colores)<jupyter_output><empty_output><jupyter_text>VI. Results visualizationNow that we have grouped our data we can obtain metrics from each group, for example CONTAGIADOS_MIL.<jupyter_code># We use seaborn barplot to measure the CONTAGIATED per thousand inhabitants of the district according to each group
sns.barplot(x="GRUPOS", y="CONTAGIADOS_MIL", data=datos, palette=colores)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.055228465378215796, 'fraction_numerical': 0.013119160373736082, 'mean_word_length': 5.019645608628659, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '468134', 'license': 'MIT', 'n_tokens_mistral': 4124, 'n_tokens_neox': 3888, 'n_words': 2280, 'path': 'Project 52465.ipynb', 'repo_name': 'hansperu/project_52465', 'size': '15645'}