<jupyter_start><jupyter_text>**Confidence Interval**  Confidence interval is a measure to quantify the uncertainty in an estimated statistic (like mean of a certain quantity) when the true population parameter is unknown.  The reason I specifically mention the term ‘population parameter’ is because, usually when you deal with data, you will have data of a smaller sample from the population. While you can easily compute a sample statistic (example: average weight of sample of 10 adult mice), the true population parameter (example: average weight of ALL adult mice that exists) is usually unknown (not always).In simpler terms, confidence interval provides the upper and lower bounds between which a given estimated statistic can vary. This range between which the statistic can vary is usually referred to as the ‘margin of error’.  The CI is often referred to as the margin of error and may be used to graphically depict the uncertainty of an estimate on graphs through the use of error bars.  **Example:**Let’s suppose you want to know the expected annual yield for a given variety of Palm tree in a given region. To know this, you start collecting annual yield data from a small sample of trees. Based on this sample data you want to know the expected yield of that variety ‘in general’ and what could be the range of values the annual yield can take assuming a 95% certainty (confidence).  **Confidence Interval Formula**The formula and method of estimating confidence interval depends on whether the population’s standard deviation is known on not.    **Types of Confidence Intervals problems**When you speak of confidence intervals, there are largely two types of problems where you would compute confidence intervals. The formula to compute confidence interval changes depending on the type.- **Confidence interval of a sample**.    - Example: Find the confidence interval for mean weight of adult white mice.-  **Confidence interval of a proportion**    - Example: Find the confidence interval of the percentage of voters who  voted for candidate A in an election (based only on exit polls data).Depending on the type of problem, you need to apply the appropriate formula to calculate confidence intervals.Secondly, the approach you take to compute the confidence intervals depends on what information you know about the population.For example, in most real world cases, you would only be working with a small sample and not know anything about the population, particularly it’s standard deviation. If that is the case, you will use the T-distribution approach. However, sometimes, you may know the population standard deviation, in which case you will use the Standard normal distribution based approach.  **For Classification Accuracy In Machine Learning**A machine learning algorithm is well understood by the data scientists and the engineers who develop them but when the product needs to be pitched, the only parameter that counts is its performance. So, a metric to gauge the performance of a model is necessary.Classification accuracy is used to assess the efficacy of a classification algorithm. To report the classification accuracy of the model alone is not best of practices.<jupyter_code>Classification Accuracy = correct predictions/ total predictions<jupyter_output><empty_output><jupyter_text>It is common to use classification accuracy or classification error (the inverse of accuracy) to describe the skill of a classification predictive model. For example, a model that makes correct predictions of the class outcome variable 75% of the time has a classification accuracy of 75%, calculated as:<jupyter_code>accuracy = total correct predictions / total predictions made * 100<jupyter_output><empty_output><jupyter_text>Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that could be correct or incorrect. Technically, this is called a Bernoulli trial, named for <NAME>. The proportions in a Bernoulli trial have a specific distribution called a binomial distribution.    We can use the assumption of a Gaussian distribution of the proportion (i.e. the classification accuracy or error) to easily calculate the confidence interval.<jupyter_code>#In the case of classification error, the radius of the interval can be calculated as:
interval = z * sqrt( (error * (1 - error)) / n)
#In the case of classification accuracy, the radius of the interval can be calculated as:
interval = z * sqrt( (accuracy * (1 - accuracy)) / n)<jupyter_output><empty_output><jupyter_text>Where interval is the radius of the confidence interval, error and accuracy are classification error and classification accuracy respectively, n is the size of the sample, sqrt is the square root function, and z is a critical value from the Gaussian distribution. Technically, this is called the Binomial proportion confidence interval.  **CODE FOR CALCULATING ACCURACY SCORES**<jupyter_code># split the data into a train and validation sets
X1, X2, y1, y2 = train_test_split(X_train, y_train, test_size=0.5)
base_prediction = base_model.predict(X2)
error = mean_squared_error(base_prediction, y2) ** 0.5
mean = base_model.predict(X_test)
st_dev = error
X1, X2, y1, y2 = train_test_split(X, y, test_size=0.5)
base_model.fit(X1, y1)
base_prediction = base_model.predict(X2)
validation_error = (base_prediction - y2) ** 2
error_model.fit(X2, validation_error)
mean = base_model.predict(X_test)
st_dev = error_model.predict(X_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.047126026419136026, 'fraction_numerical': 0.006247768654052124, 'mean_word_length': 5.295505617977528, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '543455', 'license': 'MIT', 'n_tokens_mistral': 1327, 'n_tokens_neox': 1274, 'n_words': 783, 'path': 'Machine_Learning/Statistics_for_Machine_Learning/ML_1_13_Confidence_Intervals(D).ipynb', 'repo_name': 'SathiyaEthi/winter-of-contributing', 'size': '5611'}