<jupyter_start><jupyter_text>Using numpy for processing lists of number data---  numpy is a high performance array processing library for PythonPython lists can contain any type of data, including objects.  numpy arrays are specialised and can only contain numbers.You can customise the memory usage so that arrays can use less memory and items are always stored contiguously, which isn't always the case for Python lists.  This means that it is more efficient at storing, and faster at processing, large data sets.Use numpy arrays to **store and manipulate large lists of numbers** (for other data types use plain Python lists or a panda series)Use numpy arrays to **process panda series (columns) where these contain numerical data and a large number of records**.Use numpy arrays to **create new sets of data** to add to a dataframe.For this course we are going to focus on using numpy arrays as a means of holding and working with a list of data from a pandas dataframe column. --- Creating a new numpy array (maybe for a new series or an extended series)To use numpy, you will need to import it.  The conventional way to import numpy is to import the whole library and use an *alias*`import numpy as np`Every time you want to use a function from the numpy library you use the syntax:  `np.function_name()`  Create a new numpy array from a Python list of numbers   `arr = np.array([1,2,3])`Or a new numpy matrix from a Python 2 dimensional list of numbers`matrix = np.array([1,2,3],[4,5,6])`Create a numpy array from a dataframe column (series):`arr = df['column name'].to_numpy(datatype)`where the datatype matches the data type of the column (`df.info()` will give this information if you are unsure).  Exercise 1 - create a new numpy array to hold the numbers from 1 to 100---Write a function called `make_array()` which will:*   create a list of the numbers 1 to 10  *   create a new numpy array called **new_array** from that list*   print `new_array`Expected output:  `[1 2 3 4 5 6 7 8 9 10] `<jupyter_code>import numpy as np

def make_array():
  # add your code below here
  nums = []
  #for i in range(1, 11):
  #  nums.append(i)
  nums = [i for i in range(1, 11)]
  new_array = np.array(nums, np.int8)
  print(new_array)


# run and test the function against the expected output
make_array()<jupyter_output>[ 1  2  3  4  5  6  7  8  9 10]<jupyter_text>Setting the number type in memorynumpy allows you to set the type of number in memory (e.g. `int8`, `int32`) when you create the array.  This allows memory allocation to be as small as possible.  `new_matrix = np.array([1, 2, 3], np.int8)`    This creates a list of whole numbers which are all small enough to fit in 1 byte of memory storage.--- Exercise 2 - create list of smallish numbersWrite a function which will:*   accept a parameter **num_list**    *   create a new numpy array called **new_array** from `num_list`, with data size `int16`*   print `new_array`  Test input:  `[31112, 32321, 24567,456,324,789]`Expected output:   ```[31112 32321 24567 456 324 789]```<jupyter_code>import numpy as np

def create_new_array(num_list):
  # add your code below here to turn num_list into a numpy array
  new_array = np.array(num_list, np.int16)
  print(new_array)


# run and test the code against the expected output
create_new_array([31112, 32321, 24567,456,324,789])<jupyter_output>[31112 32321 24567   456   324   789]<jupyter_text>--- Exercise 3 - create a numpy array from a pandas dataframe columnWrite a function which  will first create a dataframe from the titanic data set, and then will create a numpy array from the Fare column.*Recap*:  *the Fare column is df['Fare'] (assuming your dataframe is called df)**  create a dataframe from the data set in the file at this url:  https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/data/titanic.csv  *  create a new numpy array called **fare** from the `Fare` column (*remember to specify the data type e.g. np.int32, float64 when using* `df.to_numpy()`)  *  print the `fare` arrayExpected output:  ```array([  7.25  ,  71.2833,   7.925 , ......  23.45  ,  30.    ,   7.75  ])```<jupyter_code>import pandas as pd
import numpy as np

def create_fare_series():
  # add your code below here to read the data set into a dataframe and then create a numpy array from the Fare column and print it
  url = 'https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/data/titanic.csv'
  df = pd.read_csv(url)
  fare = df['Fare'].to_numpy(dtype=np.float16)
  return fare

# run the function and test against the expected output.
create_fare_series()<jupyter_output><empty_output><jupyter_text>--- Exercise 4 - get some statistics from a numpy array created from a data seriesThis exercise will use data on income in certain US states.  The link is: https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true  This spreadsheet just has one sheet.  Write a function which will create a numpy array from the `Age` column in the income dataset and will print the following:*  the average (mean) age of those surveyed  *  the age of the oldest person*  the age of the youngest personTO HELP with this, refer to this helpsheet: http://datacamp-community-prod.s3.amazonaws.com/da466534-51fe-4c6d-b0cb-154f4782eb54 Expected output:  ```29.888888888888894222```<jupyter_code>from pandas.core.arrays.sparse import dtype
import pandas as pd
import numpy as np

def get_age_stats():
  # add your code below here to read the data set into a dataframe and then create a numpy array from the Fare column and print it
  url = 'https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true'
  df = pd.read_excel(url)
  age_array = df['Age'].to_numpy(dtype=np.int16)
  print(age_array.mean())
  print(age_array.max())
  print(age_array.min())


# run the function and test against the expected output.
get_age_stats()<jupyter_output>29.88888888888889
42
22<jupyter_text>--- Exercise 6 - find the mean and standard deviation of wagesThis exercise will again use data on income in certain US states.  The link is: https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true  This spreadsheet just has one sheet.Write a function which will create a numpy array from the `Income` column in the income dataset and will print the following:*  the mean income of those surveyed  *  the standard deviation of income*  the highest income*  the lowest income as a percentage of the mean (lowest / mean * 100) TO HELP with this, refer to this helpsheet: http://datacamp-community-prod.s3.amazonaws.com/da466534-51fe-4c6d-b0cb-154f4782eb54 Expected output:  ```63.38888888888888613.9369169589614638134.70639789658195```<jupyter_code>import pandas as pd
import numpy as np

def get_income_stats():
  # add your code below to calculate the stats
  url = 'https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true'
  df = pd.read_excel(url, sheet_name='county-level')
  #print(df.info())
  income_arr = df['Income'].to_numpy(dtype=np.float64)
  mean = income_arr.mean()
  print(mean)
  print(income_arr.std())
  print(int(income_arr.max()))
  print(income_arr.min()/mean * 100)  


# run the function and test against expected output
get_income_stats()<jupyter_output>63.388888888888886
13.936916958961463
81
34.70639789658195<jupyter_text>--- Exercise 7 - finding the correlation between two seriesLet's find out if there is a strong correlation between Age and Income in the income data set.*  read the Income data into a pandas dataframe*  create a numpy array from the Age column  *  create a numpy array from the Income column  *  use the np.corrcoef(nparray1, nparray2) function to get the Pearson's Correlation Coefficient (the measure of linear correlation between the two data sets) and store it in a variable called **coef***  print the correlation coefficient output (see below, it will be a 2x2 matrix)*  print the correlation coefficient (which is at position [0][1] (coef[0][1]))Expected output:  ```[[ 1.         -0.14787412] [-0.14787412  1.        ]] -0.1478741157606825```The matrix gives 4 values showing the correlation between:```   |    (Age/Age)        (Age/Income)     |   |    (Income/Age)     (Income/Income)  |```This suggests that income decreases with age (the correlation is negative so as one increases the other decreases) but that the correlation is quite weak (an absolute correlation would be 1 and no correlation would be 0)<jupyter_code>import pandas as pd
import numpy as np

def get_correlation():
  # add your code below to get the correlation figure for age and salary
  url = 'https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true'
  df = pd.read_excel(url, sheet_name='county-level')
  #print(df.info())
  age_arr = df['Age'].to_numpy(dtype=np.int8)
  income_arr = df['Income'].to_numpy(dtype=np.float64)
  coef = np.corrcoef(age_arr, income_arr)
  print(coef)
  print(coef[0][1])
  
# run the function and test against expected output
get_correlation()<jupyter_output>[[ 1.         -0.14787412]
 [-0.14787412  1.        ]]
-0.1478741157606825<jupyter_text>--- Broadcasting an operation across an arrayBecause a numpy array is created from a related set of data, it is useful to be able to operate on every item in the array in the same way.  For instance, the array might hold a set of scores out of 30 and you might want to convert all scores into percentages.We can do this in a number of ways:  1.  Create a new array to store the result after the operation in the new array```scores = np.array([29,25,15,22,30])percentages = scores / 30 * 100print(percentages)```Expected output:```[ 96.66666667  83.33333333  50.          73.33333333 100.        ]```2.  Store the result in the original array```scores = np.array([29,25,15,22,30])scores = scores / 30 * 100print(scores)```Expected output:```[ 96.66666667  83.33333333  50.  73.33333333 100. ]```Give it a try:<jupyter_code>scores = np.array([29,25,15,22,30])
percentages = scores / 30 * 100
print(percentages)

scores = scores / 30 * 100
print(scores)<jupyter_output>[ 96.66666667  83.33333333  50.          73.33333333 100.        ]
[ 96.66666667  83.33333333  50.          73.33333333 100.        ]<jupyter_text>--- Exercise 8 - increase whole array by 20Write a function which will:*  create a numpy array of 12 numbers*  create a new array adding 20 to each of the items in the first array  *  print the new arrayTest input:  `[1,2,3,4,5,6,7,8,1,2,3,4]`Expected output:  `[21 22 23 24 25 26 27 28 21 22 23 24]`<jupyter_code># add your code to define the function to increase all values in an array by 20 and then to call the function

def increase_by_twenty(num_list):
  num_arr = np.array(num_list, np.int64)
  num_arr = num_arr + 20
  return num_arr

print(increase_by_twenty([1,2,3,4,5,6,7,8,1,2,3,4]))<jupyter_output>[21 22 23 24 25 26 27 28 21 22 23 24]<jupyter_text>--- Conversion of values using broadcasting--- Exercise 9 - convert Titanic fares into 21st century values Write a function which will:  *  create a dataframe from the titanic data set (https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/data/titanic.csv)   *  create a numpy array from the Fare column*  convert the fares into current value (multiply by a factor of 120.7045 - source https://www.in2013dollars.com/uk/inflation/1912?amount=32 *accessed 22/1/2022*)*  print the average fare, the maximum fare and the minimum fareExpected output:  ```3887.192820742817361840.43992140.0```<jupyter_code>import pandas as pd
import numpy as np

def get_current_fares():
  # add your code below to get the fares data into a numpy array, convert to today's prices and print stats
  url = 'https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/data/titanic.csv'
  df = pd.read_csv(url)
  #print(df.info())
  fare_arr = df['Fare'].to_numpy(dtype=np.float64)
  FACTOR = 120.7045
  current_fare_value_arr = fare_arr * FACTOR
  print(current_fare_value_arr.mean())
  print(current_fare_value_arr.max())
  print(current_fare_value_arr.min())

# run the function and test against expected output
get_current_fares()<jupyter_output>3887.1928207428173
61840.4399214
0.0<jupyter_text>--- Exercise 10 - create a new column in the dataframe from a numpy array**Challenging**Write a function which will calculate expected salaries for all in the income data set after an inflation rate of 3.5% (with results in a new numpy array).Just to show the result, calculate and print the Pearson Correlation Coefficient between the salaries series and the inflated salaries series.  We would expect this to be 1 (ie the inflated salary is always 3.5% higher than the current salary) and the exercise is just meant to show that - the statistic has no relevance.  Create a new column in the dataframe from the new numpy array (so that the dataframe now contains the original salaries and the inflated salaries.  (**Recap**:  *to add a new column, just use* `df['new column name']`)  To assign a numpy array to a pandas column use  `df['new column name'] = numpyarrayname.tolist()`Display the new dataframe and print the correlation coefficient.<jupyter_code>import pandas as pd
import numpy as np

def new_column():
  url = 'https://github.com/futureCodersSE/working-with-data/blob/main/Data%20sets/Income-Data.xlsx?raw=true'
  df = pd.read_excel(url, sheet_name='county-level')
  #print(df.info())
  income_arr = df['Income'].to_numpy(dtype=np.float64)
  INFLATION_RATE = 3.5
  inflated_income_arr = income_arr + income_arr * INFLATION_RATE / 100
  coef = np.corrcoef(income_arr, inflated_income_arr)
  inflated_income_arr = inflated_income_arr.round(0)
  inflated_income_arr = np.array(inflated_income_arr, dtype=np.int64)

  df['Inflated_Income'] = inflated_income_arr.tolist()
  print(df)
  print('')
  print('Correlation Coefficient: ', coef[0][1])

new_column()<jupyter_output>State  County  Population   Age  Income  Inflated_Income
0     TX     1.0        72.0  34.0    65.0               67
1     TX     2.0        33.0  42.0    45.0               47
2     TX     5.0        25.0  23.0    46.0               48
3     TX     6.0        54.0  36.0    65.0               67
4     TX     7.0        11.0  42.0    53.0               55
5     TX     8.0        28.0  25.0    62.0               64
6     TX     9.0        82.0  35.0    66.0               68
7     TX    10.0         5.0  40.0    75.0               78
8     MD    11.0        61.0  27.0    22.0               23
9     MD     2.0         5.0  23.0    69.0               71
10    MD     4.0        98.0  25.0    73.0               76
11    MD     3.0        64.0  29.0    75.0               78
12    MD     2.0        36.0  24.0    65.0               67
13    MD     1.0        24.0  25.0    66.0               68
14    MD     5.0        34.0  31.0    78.0               81
15    MD     6.0        89.0  22.0    81[...]
{'fraction_non_alphanumeric': 0.09162653825575173, 'fraction_numerical': 0.07417067950775816, 'mean_word_length': 3.547749391727494, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 16, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '558711', 'license': 'MIT', 'n_tokens_mistral': 5457, 'n_tokens_neox': 4697, 'n_words': 1805, 'path': 'Worksheets/5_numpy.ipynb', 'repo_name': 'Imran-Github22/Programming-For-Data', 'size': '14952'}