<jupyter_start><jupyter_text>1. Programming And Data ScienceData science involves processing, analyzing, and visualizing data. While some tools like Microsoft Excel allow us to perform basic data science tasks, they're limited to the functionality built in to the user interface. If you want to work with datasets that aren't structured like a spreadsheet or create entire new data visualizations from scratch, you'll need to become proficient in **programming**. Instead of using a program written by others that can solve a narrow set of tasks, you can create your own programs that can solve your specific problems.Programming involves organizing a collection of instructions into a program for a computer to carry out. To express these instructions, we use a **programming language**. In this track, we focus on a language called Python. **Python** is a general-purpose programming language that is becoming more and more popular for doing **data science**. Companies worldwide are using Python to harvest insights from their data and get a competitive edge.**Python** is a popular choice for working with data because it has good support for:- handling large datasets- working with common mathematical functions- creating powerful data visualizationsWhat you will learn- Python versions- Basic data types- List- Files and Loops- If statements- Dictionaries- Functions and Modules 2. Python versionsThere are currently two different supported versions of Python, 2.7 and 3.6. Python 3.x introduced many backwards-incompatible changes to the language, so code written for 2.7 may not work under 3.x and vice versa. For this class all code will use Python 3.6. https://wiki.python.org/moin/Python2orPython3"Python 2.x is legacy, Python 3.x is the present and future of the language"" 3. Basic data typesLike most languages, Python has a number of basic types including integers, floats, booleans, and strings. These data types behave in ways that are familiar from other programming languages.<jupyter_code>x = 3
print(type(x)) # Prints "<class 'int'>"
print(x)       # Prints "3"
print(x + 1)   # Addition; prints "4"
print(x - 1)   # Subtraction; prints "2"
print(x * 2)   # Multiplication; prints "6"
print(x ** 2)  # Exponentiation; prints "9"
x += 1
print(x)       # Prints "4"
x *= 2
print(x)       # Prints "8"
y = 2.5
print(type(y)) # Prints "<class 'float'>"
print(y, y + 1, y * 2, y ** 2) # Prints "2.5 3.5 5.0 6.25"<jupyter_output><class 'int'>
3
4
2
6
9
4
8
<class 'float'>
2.5 3.5 5.0 6.25<jupyter_text>Note that unlike many languages, **Python** does not have unary increment **(x++)** or decrement **(x--)** operators. Python also has built-in type for complex numbers; you can find all of the details in the [documentation](https://docs.python.org/3/library/stdtypes.htmlnumeric-types-int-float-long-complex).**Booleans**: Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols (&&, ||, etc.):<jupyter_code>t = True
f = False
print(type(t)) # Prints "<class 'bool'>"
print(t and f) # Logical AND; prints "False"
print(t or f)  # Logical OR; prints "True"
print(not t)   # Logical NOT; prints "False"
print(t != f)  # Logical XOR; prints "True"<jupyter_output><class 'bool'>
False
True
False
True<jupyter_text>**Strings**: Python has great support for strings:<jupyter_code>hello = 'hello'           # String literals can use single quotes
world = "world"           # or double quotes; it does not matter.
print (hello)             # Prints "hello"
print (len(hello))        # String length; prints "5"
hw = hello + ' ' + world  # String concatenation
print(hw)                 # prints "hello world"
hw12 = '%s %s %d' % (hello, world, 12)  # sprintf style string formatting
print(hw12)  # prints "hello world 12"<jupyter_output>hello
5
hello world
hello world 12<jupyter_text>String objects have a bunch of useful methods, you can find a list of all string methods in the [documentation](https://docs.python.org/3/library/stdtypes.htmlstring-methods).<jupyter_code>s = "hello"
print (s.capitalize())  # Capitalize a string; prints "Hello"
print (s.upper())       # Convert a string to uppercase; prints "HELLO"
print (s.rjust(7))      # Right-justify a string, padding with spaces; prints "  hello"
print (s.center(7))     # Center a string, padding with spaces; prints " hello "
print (s.replace('l', '(ell)'))  # Replace all instances of one substring with another;
                                 # prints "he(ell)(ell)o"
print ('  world '.strip())       # Strip leading and trailing whitespace; prints "world"<jupyter_output>Hello
HELLO
  hello
 hello 
he(ell)(ell)o
world<jupyter_text>4. Using a list to store multiple valuesSo far, we've been storing individual values in variables. Often in data science, we're working with thousands of data points that are grouped together in a certain way and have an order to them. We need a container that can hold multiple values that we can use to perform operations on. We can use a **list**, which is an object that represents a sequence of values. For example, we can represent the cities in our dataset as a list as a sequence of strings ("Natal", "São Paulo", and so on).A list is the Python equivalent of an array, but is resizeable and can contain elements of different types. You can find all about list in [documentation](https://docs.python.org/3/tutorial/datastructures.htmlmore-on-lists).<jupyter_code># Create the areas list
areas = ["hallway", 11.25, "kitchen", 18.0, "living room", 20.0, "bedroom", 10.75, "bathroom", 9.50]

# Print out areas
print(areas)

# Print out the type of areas
print(type(areas))

# Print out second element from areas
print(areas[1])

# Print out last element from areas
print(areas[-1])

# Print out the area of the living room
print(areas[-5])

# Add two new elements to the end of the list
areas.append("laundry")
areas.append(8.75)

# Print out the new list
print(areas)

# Lists have a feature called slicing that allows you to return all of the values between
# a starting index and an ending index. When you slice a list, you return a new list containing
# just the values you're interested in.

# Remove the first two elements of list
del(areas[0:2])
print(areas)

# Remove the last two elements of list
del(areas[-1:])
print(areas)<jupyter_output>['hallway', 11.25, 'kitchen', 18.0, 'living room', 20.0, 'bedroom', 10.75, 'bathroom', 9.5]
<class 'list'>
11.25
9.5
20.0
['hallway', 11.25, 'kitchen', 18.0, 'living room', 20.0, 'bedroom', 10.75, 'bathroom', 9.5, 'laundry', 8.75]
['kitchen', 18.0, 'living room', 20.0, 'bedroom', 10.75, 'bathroom', 9.5, 'laundry', 8.75]
['kitchen', 18.0, 'living room', 20.0, 'bedroom', 10.75, 'bathroom', 9.5, 'laundry']<jupyter_text>5. Files and LoopsWe'll learn how to work with files and use loops to iterate through lists. We'll be working with crime rate data for 73 cities in the United States. Datasets are often represented in files that you can download and manipulate. Before we get started, we'll first need to learn how to work with files in Python.To open a file in Python, we use the open() function. This function accepts two different arguments (inputs) in the parentheses, always in the following order:- the name of the file (as a string)- the mode of working with the file (as a string)We'll learn about the various modes in a later mission. For now, we'll just use "r", the mode for reading in files.When entering multiple inputs, separate them with commas (,). For example, to open a file named story.txt in read mode, we write the following:>```pythonopen("story.txt", "r")```The open() function returns a *File* object. This object stores the information we passed in, and allows us to call methods specific to the File class. We can assign the File object to a variable so we can refer to it later:>```pythona = open("story.txt", "r")```Note that the File object, a, won't contain the actual contents of the file. It's instead an object that acts as an interface to the file and contains methods for reading in and modifying the file's contents (which we'll cover in the next screen).<jupyter_code>#open the file
f = open("crime_rates.csv", "r")

#read the file
data = f.read()

#print data
print(type(data))
print(data)
#split the crime_rates.csv based on '\n' filter
rows = data.split('\n')

#print the first five rows
print(rows[0:5])
#create an empty list
int_crime_rates=[]

#print the rate of crimes for each city using a list(int)
for i in rows:
    int_crime_rates.append(int(i.split(",")[1]))
type(int_crime_rates[0])<jupyter_output><empty_output><jupyter_text>6. BooleansWe'll learn how to express conditional logic. We can use **conditional logic** to add criteria to the code we write. Some examples of operations that use criteria include:- Finding all the integers in a list that are greater than "5".- Identifying which elements in a list are strings, and printing only those values.We can break down both of these examples into logic we can code:- For each integer in a list, if the integer is greater than "5", add to the list greater_than_five.- For each element in a list, if the value has a string data type, use the print() function to display it; if it's not a string, ignore it.Python has a class called **Boolean** that helps express conditional logic. There are only two Boolean values: True and False. Because they're words, Boolean values may look like strings, but they're an entirely separate class. For example, string operations like concatenation won't work with Booleans.The following code example assigns True to t and False to f:>```pythont = Truef = False```If we display the data type for either t or f, we'll see class 'bool', shorthand for Boolean. 6.1 Boolean OperatorsPython has comparison operators that allow us to compare variables:== returns True if both variables are equivalent, and False if they're different!= returns True if both variables are different, and False if they're equivalent> returns True if the first variable is greater than the second variable, and False otherwise returns True if the first variable is less than the second variable, and False otherwise>= returns True if the first variable is greater than or equal to the second variable, and False otherwise returns True if the first variable is less than or equal to the second variable, and False otherwise<jupyter_code>print(8 == 8) # True
print(8 != 8) # False
print(8 == 10) # False
print(8 != 10) # True
rates = [10, 15, 20]
print(rates[0] > rates[1]) # False
print(rates[0] >= rates[0]) # True<jupyter_output><empty_output><jupyter_text>7. If StatementsNow that we know how to work with Boolean values, let's dive more into how we use Booleans to express conditional logic. To complement Booleans, Python contains the if operator. We can use this operator to write a statement that tests whether certain conditions exist. Our if statement will evaluate to either True or False, and only run the specified code when True.>```pythonsample_rate = 749greater = (sample_rate > 5)if greater:                    This is the conditional statement.    print(sample_rate)```We can nest if statements to specify multiple criteria. >```pythonvalue = 1500if value > 500:    if value > 1000:        print("This number is HUGE!")```We can also nest if statements within for loops, and vice versa. For example, we can search a list for the existence of a specific value by combining a for loop with an if statement. The if statement determines whether the current element is equivalent to the value we're interested in:>```pythonfound = Falsefor city in cities:    if city == 'João Pessoa':        found = True``` Exercise Start.**Description**: For this challenge, you'll be working with the data set behind this [FiveThirtyEight](http://fivethirtyeight.com/features/there-are-922-unisex-names-in-america-is-yours-one-of-them/) article on common unisex (gender-neutral) names in the United States. You'll start by reading in the file and iteratively converting the data to more useful representations. At the end of this challenge, you'll filter the data so that it only includes the names that at least 1,000 people share.The staff at [FiveThirtyEight](http://fivethirtyeight.com/) compiled this data set from information at the [Social Security Adminstration's website](https://www.ssa.gov/oact/babynames/limits.html). You'll work with a shortened version of the full data set to complete this challenge.Here's a preview of the shortened data set, which is in a CSV file named [unisex_names_table.csv](https://github.com/fivethirtyeight/data/blob/master/unisex-names/unisex_names_table.csv) (tip: raw,save as):>1) Use the open() function with the following parameters to return a File object: >>+ unisex_names_table.csv for the file name.+ r for read mode+ Then, use the read() method of the File object to read the file into a string. Assign that string to a variable named names.>```pythondata = open("unisex_names_table.csv","r")names = data.read()```<jupyter_code>data = open("unisex_names_table.csv","r")
names = data.read()
print(names)<jupyter_output>"","name","total","male_share","female_share","gap"
"1","Casey",176544.328149,0.584286566204162,0.415713433795838,0.168573132408324
"2","Riley",154860.665173,0.507639071226889,0.492360928773111,0.0152781424537786
"3","Jessie",136381.830656,0.47783426831522,0.52216573168478,0.04433146336956
"4","Jackie",132928.78874,0.421132601798505,0.578867398201495,0.15773479640299
"5","Avery",121797.419516,0.335213073103216,0.664786926896784,0.329573853793568
"6","Jaime",109870.18729,0.561792900862907,0.438207099137093,0.123585801725814
"7","Peyton",94896.395216,0.433719376329487,0.566280623670513,0.132561247341027
"8","Kerry",88963.92625,0.483948843602212,0.516051156397788,0.0321023127955753
"9","Jody",80400.519199,0.352068031749129,0.647931968250871,0.295863936501742
"10","Kendall",79210.873961,0.372366742002144,0.627633257997856,0.255266515995713
"11","Payton",64151.630388,0.334357683090347,0.665642316909653,0.331284633819305
"12","Skyler",53486.390419,0.646053060924541,0.353946939075459,0.292106[...]<jupyter_text>>2) Convert the string to a list>>+ Use the split() method that *strings* have to split on the new-line delimiter ("\n"), and assign the resulting list to names_list.+ Select the first five elements in names_list, and assign them to first_five.+ Display first_five using the print() function.>```pythonnames_list = names.split("\n")skip the header and tailnames_list = names_list[1:-1]first_five = names_list[0:5]print(first_five)```>3) Convert the list of strings to a list of lists. Split each element in names_list on the comma delimiter (,) and append the resulting list to nested_list. To accomplish this:>>+ Create an empty list and assign it to nested_list.+ Write a for loop that iterates over names_list.+ Within the loop body, run the split() method on each element to return a list (assign that list to comma_list).+ Within the loop body, run the append() method to add each list (comma_list) to nested_list.+ Use the print() function to display the first five elements in nested_list>```pythonnested_list = []for i in names_list:    comma_list = i.split(',')    nested_list.append(comma_list)```>4) Create a new list of strings called thousand_or_greater that only contains the names shared by 1,000 people or more. To accomplish this:>>+ Create an empty list and assign it to thousand_or_greater.+ Write a for loop that iterates over numerical_list.>```pythonnumerical_list = []for i in nested_list:    a = i[1]    b = float(i[2])    numerical_list.append([a,b])```+ In the loop body, use an if statement to determine if the value at index 2 for that element (which is a list) is greater than or equal to 1000.+ If the value is greater than or equal to 1000, use the append() method to add its name to thousand_or_greater.+ Finally, display the first 10 elements in thousand_or_greater.  8. DictionariesA dictionary is like a list in that it has indexes, but the indexes aren't necessarily sequential numbers. We can create our own indexes with values of any data type, including strings.A dictionary stores (key, value) pairs, similar to a Map in Java or an object in Javascript.<jupyter_code># Definition of countries and capital
countries = ['spain', 'france', 'germany', 'norway']
capitals = ['madrid', 'paris', 'berlin', 'oslo']

# Get index of 'germany': ind_ger
ind_ger = countries.index('germany')

# Use ind_ger to print out capital of Germany
print(capitals[ind_ger])
# From string in countries and capitals, create dictionary europe
europe = {'spain':'madrid','france':'paris','germany':'berlin','norway':'oslo'}

# Print europe
print(europe)

# Print out the keys in europe
print(europe.keys())

# Print out value that belongs to key 'norway'
print(europe['norway'])
# Add italy to europe
europe['italy'] = 'rome'

# Print out italy in europe
print('italy' in europe)

# Add poland to europe
europe['poland'] = 'warsaw'

print(europe)

del(europe['france'])

# Print europe
print(europe)
# Dictionary of dictionaries
europe = { 'spain': { 'capital':'madrid', 'population':46.77 },
           'france': { 'capital':'paris', 'population':66.03 },
           'germany': { 'capital':'berlin', 'population':80.62 },
           'norway': { 'capital':'oslo', 'population':5.084 } }


# Print out the capital of France
print(europe['france']['capital'])

# Create sub-dictionary data
data = {'capital':'rome', 'population':59.83}

# Add data to europe under key 'italy'
europe['italy'] = data

# Print europe
print(europe)<jupyter_output>paris
{'spain': {'capital': 'madrid', 'population': 46.77}, 'france': {'capital': 'paris', 'population': 66.03}, 'germany': {'capital': 'berlin', 'population': 80.62}, 'norway': {'capital': 'oslo', 'population': 5.084}, 'italy': {'capital': 'rome', 'population': 59.83}}<jupyter_text>8.1 Loop over a dictionaryIn Python 3, you need the **items()** method to loop over a dictionary:```python  world = { "afghanistan":30.55,              "albania":2.77,             "algeria":39.21 }  for key, value in world.items() :      print(key + " -- " + str(value))```<jupyter_code># Definition of dictionary
europe = {'spain':'madrid', 'france':'paris', 'germany':'bonn', 
          'norway':'oslo', 'italy':'rome', 'poland':'warsaw', 'australia':'vienna' }
          
# Iterate over europe
for key, value in europe.items():
    print("the capital of " + key + " is " + value)<jupyter_output>the capital of spain is madrid
the capital of france is paris
the capital of germany is bonn
the capital of norway is oslo
the capital of italy is rome
the capital of poland is warsaw
the capital of australia is vienna<jupyter_text>Exercise Start.**Description**: In this mission, we'll look at daily weather data for Los Angeles (L.A.) during 2014. Here's a look at the beginning of la_weather.csv, the data set we'll be working:>```csvDay,Type of Weather1,Sunny2,Sunny3,Sunny4,Sunny5,Sunny6,Rain7,Sunny8,Sunny9,Fog10,Rain```The first row in our data is the header row, which contains labels for the values beneath them. As the header row indicates, each row has two values:- Day - A number from 1 to 365 indicating the day of the year. January 1st is 1, and December 31st is 365.- Type of Weather - The type of weather experienced on that day. The values that may appear here include Rain, Sunny, Fog, Fog-Rain, or Thunderstorm.>```pythonweather_data = []f = open("la_weather.csv", 'r')data = f.read()rows = data.split('\n')for row in rows:    split_row = row.split(",")    weather_data.append(split_row[1])weather_data = weather_data[1:]```**Instructions**: 1. Count how many times each type of weather occurs in the weather list, and store the results in a new dictionary called - weather_counts.>```pythonweather_counts = {}for item in weather_data:    if item in weather_counts:    ...    else:    ...```<jupyter_code>x = ['sun', 'fri', 'sat', 'sat', 'sat', 'sat', 'sun', 'sun']
counts = {}
for val in x:
        if val in counts.keys():<jupyter_output>sun
fri
sat
sat
sat
sat
sun
sun<jupyter_text>9. FunctionsTo leverage the code that brilliant Python developers have written, you'll learn about using functions, methods and packages. This will help you to reduce the amount of code you need to solve challenging problems!<jupyter_code># Maybe you already know the name of a Python function, but you still have to figure out how to use it. 
# Ironically, you have to ask for information about a function with another function: help(). 
# In IPython specifically, you can also use ? before the function name.
# To get help on the max() function, for example, you can use one of these calls:

?max #or # help(max)
# Create list areas
areas = [11.25, 18.0, 20.0, 10.75, 9.50]

# Print out the index of the element 20.0
print(areas.index(20.0))

# Print out how often 14.5 appears in areas
print(areas.count(14.5))

# Reverse the orders of the elements in areas
areas.reverse()

# Print out areas
print(areas)
# Python functions are defined using the "def" keyword. For example:

def sign(x):
    if x > 0:
        return 'positive'
    elif x < 0:
        return 'negative'
    else:
        return 'zero'

for x in [-1, 0, 1]:
    print(sign(x))
# Prints "negative", "zero", "positive"
# We will often define functions to take optional keyword arguments, like this:

def hello(name, loud=False):
    if loud:
        print('HELLO, %s!' % name.upper())
    else:
        print('Hello, %s' % name)

hello('Ivanovitch') # Prints "Hello, Ivanovitch"
hello('Silva', loud=True)  # Prints "HELLO, SILVA!"<jupyter_output>Hello, Ivanovitch
HELLO, SILVA!<jupyter_text>Exercise Start.**Description**: Implement the previous exercise as a function. The function must pass as parameter a list of weather data and return a dictionary having how many times each type of weather occurs.  10. ModulesIn the previous sections, we explored the basics of programming in the Python language. In that sections, we only worked with the functions and objects that are available by default. Most of the useful functions and objects live in separate modules that we need to specifically import into the environment to use. In addition, the Python ecosystem has a wealth of other modules that have been developed by others and are available for free. In this mission, we'll learn how to work with modules and explore the problems some specific modules help us solve.A module is a collection of functions and variables that have been bundled together in a single file. module help us:- Organize our code by separating related functions and objects into their own module.- Gain new functionality by using code written by others without spending a large amount of time diving into how exactly it's implemented.Module are usually organized around a theme. We'll start by working with the math module, which contains many useful mathematical operations to make analyzing data easier. Later, we'll learn about the csv module, which makes working with CSV files easier.To access a module, we import it using Python's import statement. If we had a module named my_module that we wanted to use, we'd import it using the following code:>```pythonimport my_module```By importing my_module, we're indicating that we want to use some of the functions or variables from a pre-existing file called my_module.py. (The Python interpreter already knows where to look for module files on your machine.) Note that the name of the module is the same as the name of the Python file, but without the extension. Once we import a module, we can access its functions and variables using **dot notation**.<jupyter_code>#Import the math module.
import math

#Use the sqrt() function from the math module to compute the square root of 99. Assign the result to root.
root = math.sqrt(99)

#Use the floor() function within the math module to assign the floor of 89.9 to flr.
flr = math.floor(89.9)

print(flr)<jupyter_output>89<jupyter_text>10.1 Importing using aliasIn the last step, we learned that we can import an entire module using the  import statement. When we import a module this way, we add all of the objects and functions within that module to the global **namespace**. A namespace is a dictionary that contains all of the names we can refer to in our code. Before running our code, the Python interpreter adds all of the objects and functions that are available by default (print(),  list(), etc.) to the global namespace. When we create variables or define our own functions, these are also added to the same namespace.When we import an entire module, we're adding everything within that module into the global namespace, prefixed by the module name.Some modules have long names, however, and this means that we need to use the full module name each time we want to use any of the objects within that module. Instead, we can assign an alias when we import the module:>```pythonimport my_module as m```Now, we can access all of the objects within my_module using the prefix m:>```pythonm.function1()m.function2()```<jupyter_code>#Import the math module as m
import math as m

#Use the sqrt() function from the math module to compute the square root of 33. Assign the result to root.
root = m.sqrt(33)

print(root)<jupyter_output>5.744562646538029<jupyter_text>10.2 Importing A Specific ObjectIf a module contains hundreds of functions and we only need a few, specific functions, we can import just the ones we need:>```pythonfrom math import function1from math import function1, function2```The downside with this approach is that when we import by cherry picking, the objects are added to the top level namespace and they aren't prefixed.If we import function1() from my_module and then define our own function1(), then the reference to the original function1() we imported will get overrwriten in the namespace.Lastly, we can import all of the objects (this includes both objects and functions) from a module into the global namespace using the * symbol:>```pythonfrom my_module import *```This is generally considered not good practice, because your namespace becomes polluted with all of these named references to objects you may not be using. In addition, if you're working with multiple modules, you run into the danger of overwriting some of the names in the namespace.<jupyter_code>#Import all of the functions from math.
from math import *

#Use the sqrt() function from the math module to compute the square root of 1001. Assign the result to root.
root = sqrt(1001)

print(root)<jupyter_output>31.63858403911275
{'fraction_non_alphanumeric': 0.09580161302479837, 'fraction_numerical': 0.04379286952589131, 'mean_word_length': 4.72986396026776, 'pattern_counts': {'":': 3, '<': 56, '<?xml version=': 0, '>': 93, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '448295', 'license': 'Apache-2.0', 'n_tokens_mistral': 8438, 'n_tokens_neox': 7510, 'n_words': 3522, 'path': 'Aula 02 - Plataformas/Warming_up.ipynb', 'repo_name': 'Andrelourencogit/Probabilidade', 'size': '26534'}