<jupyter_start><jupyter_text>Statistics Data, consists of a collection of pieces of information about things you want to study. Statistics is based on data. Statistics studies methodologies to gather, review, analyze and draw conclusions from data. Now this information can take the form of descriptions, quantities, measurements, and other observations. Data is said to be new oil. specifically in the field of Digital technologies working with data is one of the primart things you will do. Suppose for example, we collected some information about students taking this course, such as their student ID, name, age, gender, their level of experience with the subject, the number of previous courses they've taken, the time span and activities in this course, and their grade for the course. Some of these data are categorical, or qualitative. In other words, they describe the characteristics of the students. Now often qualitative data fields are text based, but numeric fields can also be categorical. For example, student ID is a numeric identifier for each student. Mean is the average of a data set. Median is the value that lies at the center. In case there even number of elements in a data set, the median is the average of the two values that lie in the center. The mode is the value with the highest frequency in a data set. Outliers are abnormal elements of a data set that lie very far from the rest of the elements in the same set.Lets watch this video to understand more.<jupyter_code>from IPython.display import YouTubeVideo
YouTubeVideo('1JYGdI58xuU', width=860, height=460)
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.datasets import load_iris 
iris = load_iris() 

print(iris.data.shape)

iris = pd.DataFrame(iris.data)
iris.columns = ['Sepal_Length', "Sepal_Width", "Petal_Length", "Petal_Width"]
print(iris.head())
print(iris.describe())<jupyter_output>Sepal_Length  Sepal_Width  Petal_Length  Petal_Width
count    150.000000   150.000000    150.000000   150.000000
mean       5.843333     3.057333      3.758000     1.199333
std        0.828066     0.435866      1.765298     0.762238
min        4.300000     2.000000      1.000000     0.100000
25%        5.100000     2.800000      1.600000     0.300000
50%        5.800000     3.000000      4.350000     1.300000
75%        6.400000     3.300000      5.100000     1.800000
max        7.900000     4.400000      6.900000     2.500000<jupyter_text>Visualizing DataThere are lot of python libraries that you can use to visualize the data. We had used Mathplotlib to extensively generate the plots in our previous exercices. Following are few more of them :- Matplotlib: low level, provides lots of freedom- Pandas Visualization: easy to use interface, built on Matplotlib- Seaborn: high-level interface, great default styles- ggplot: based on R’s ggplot2, uses Grammar of Graphics- Plotly: can create interactive plotsWe can plot the dataset used before to plot it using matplot lib.<jupyter_code>YouTubeVideo('1etMjhxH_e8', width=860, height=460)
plt.figure(figsize = (10, 7)) 
x = iris.Sepal_Width 
  
plt.hist(x, bins = 20, color = "blue") 
plt.title("Sepal Width in cm") 
plt.xlabel("Sepal_Width") 
plt.ylabel("Count") 
  
plt.show() 
plt.figure(figsize = (10, 7)) 
x = iris.Sepal_Width 
y = iris.Petal_Width

plt.scatter(x, y) 
plt.title("Width in cm") 
plt.xlabel("Sepal_Width") 
plt.ylabel("Petal_Width") 
  
plt.show() 
import seaborn as sns
sns.distplot(iris['Sepal_Length'])
plt.show()
sns.distplot(iris['Petal_Width'])
plt.show()
sns.pairplot(iris, palette="husl")
plt.show()<jupyter_output><empty_output><jupyter_text>Introduction to ProbabilityProbability is numerical descriptions of how likely an event is to occur or how likely it is that a proposition is true. The probability of an event is a number between 0 and 1, where, roughly speaking, 0 indicates impossibility of the event and 1 indicates certainty. For example, rolling a die. Now assuming it's a regular six-faced die, the possible outcomes from a single roll are one, two, three, four, five, or six. This is known as the sample space for the experiment. An event is an instance of the experiment with a specific outcome, for example, rolling a six.     **Probability Basics**Let's start with some basic definitions and principles.,- An ***experiment*** or ***trial*** is an action with an uncertain outcome, such as tossing a coin.- A ***sample space*** is the set of all possible outcomes of an experiment. In a coin toss, there's a set of two possible oucomes (*heads* and *tails*).- A ***sample point*** is a single possible outcome - for example, *heads*).- An ***event*** is a specific outome of single instance of an experiment - for example, tossing a coin and getting *tails*.- ***Probability*** is a value between 0 and 1 that indicates the likelihood of a particular event, with 0 meaning that the event is impossible, and 1 meaning that the event is inevitable. There is exactly one sample point in our event space that satisfies that outcome. So assuming the die is fair, there's a one in six chance that the event will occur in any given trial roll of the die.<jupyter_code>YouTubeVideo('TLGYsch6YFA', width=860, height=460)
import random
from matplotlib import pyplot as plt

heads_tails = [0,0]

# loop through trials
trials = 1000
trial = 0
while trial < trials:
    trial = trial + 1
    # Get a random 0 or 1
    toss = random.randint(0,1)
    # Increment the list element corresponding to the toss result
    heads_tails[toss] = heads_tails[toss] + 1

print (heads_tails)

# Show a pie chart of the results
plt.figure(figsize=(5,5))
plt.pie(heads_tails, labels=['heads', 'tails'])
plt.legend()
plt.show()<jupyter_output>[510, 490]
{'fraction_non_alphanumeric': 0.07536282566882323, 'fraction_numerical': 0.05245672320335723, 'mean_word_length': 4.130044843049327, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '444315', 'license': 'Apache-2.0', 'n_tokens_mistral': 1867, 'n_tokens_neox': 1610, 'n_words': 764, 'path': 'Statistics_and_Probability/Statistics_and_Probability.ipynb', 'repo_name': 'yogeshmpandey/M4DT', 'size': '5719'}