<jupyter_start><jupyter_text>Python session - 3.1 Functions and modules`Pandas` cheat sheet: https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdfSoftware Carpentry reference files: http://tobyhodges.github.io/python-novice-gapminder/  FunctionsFunctions are reusable blocks of code that you can name and execute any number of times from different parts of your script(s). This reuse is known as "calling" the function. Functions are important building blocks of a software.There are several built-in functions of Python, which can be called anywhere (and any number of times) in your current program. You have been using built-in functions already, for example, `len()`, `range()`, `sorted()`, `max()`, `min()`, `sum()` etc.  Structure of writing a function:- `def` (keyword) + function name (you choose) + `()`.- newline with 4 spaces or a tab +  block of code  Note: Codes at the 0 position are always read- Call your function using its name<jupyter_code># Examples of the functions that you have alreasy know, i.e. print(), len(), max(), min()

myvar = 'something'
print(myvar)
print(len(myvar))
print(list(myvar))
print(max([12,23,45,67]))
## Non parametric function
# Define a function that prints a sum of number1 and number2 defined inside the function

def get_sum():
    a = 98
    b = 56
    sum_this = a+b
    print(sum_this)
get_sum()

def get_sum2():
    c = 90
    d = 5690
    sum_this2 = c+d
    print(sum_this2)
get_sum2()
get_sum() #calling the first function again
# Parametric function
# Define a function that prints a sum of number1 and number2 provided by the user
# Hint: get_sum_param(number1, number2)

def get_sum_param(number1, number2):
    sum_this = number1 + number2
    print(sum_this)
    
a = 98
b = 56
get_sum_param(a, b)

c = 90
d = 5690
get_sum_param(c, d)
# Returning values
# Define a function that 'returns' a sum of number1 and number2 provided by the user
# Hint: print(get_sum_param(number1, number2))

def get_sum_param(number1, number2):
    sum_this = number1 + number2
    return sum_this

a = 98
b = 56
val_ab = get_sum_param(a, b)

c = 90
d = 5690
val_cd = get_sum_param(c, d)

print(val_ab, val_cd)
print(val_ab * val_cd/val_ab) # do more operations on the variables
# Local Vs. global variable

# Define a function that returns a sum of number1 and number2 to a variable
# and print it after calling the function
# Hint: returned_value = get_sum_param(number1, number2)

def get_sum_param(number1, number2):
    sum_this = number1 + number2
    number3 = 6789
    sum_this_too = sum_this + number3
    return sum_this, sum_this_too
a = 98
b = 56
returned_values = get_sum_param(a, b)
print(returned_values)
print(a, b, number3) # number3 is a local variable for the function, so it will not be recognised outside the function<jupyter_output>(154, 6943)<jupyter_text>Exercises: write old code into a function<jupyter_code># Optional exercise
# Let’s take one of our older code blocks and write it in a function

def check_temp_range():
    temperature = 20 # define temperature inside the function
    if temperature > 25 and temperature < 50:
        print('Warm weather')
    elif temperature <= 25:
        print('Not very Warm')
    elif temperature == 25:
        print('Pleasant weather')
    else:
        print('I do not care about the weather')
        
check_temp_range()
def check_temp_range_with_param(temperature):
    if temperature > 25 and temperature < 50:
        print('Warm weather')
    elif temperature <= 25:
        print('Not very Warm')
    elif temperature == 25:
        print('Pleasant weather')
    else:
        print('I do not care about the weather')
        
temperature_var = 20 # define temperature inside the function
temperature_var2 = 35 # define temperature inside the function
check_temp_range_with_param(temperature_var)
check_temp_range_with_param(temperature_var2)
def check_temp_range_with_more_param(temperature, upper_limit, lower_limit):
    if temperature > lower_limit and temperature < upper_limit:
        print('Warm weather')
    elif temperature <= lower_limit:
        print('Not very Warm')
    elif temperature == lower_limit:
        print('Pleasant weather')
    elif temperature >= upper_limit:
        print("It's too hot")
    else:
        print('I do not care about the weather')
        
temperature_var = 20 # define temperature inside the function
lower_limit = 15 # define temperature inside the function
upper_limit = 45 # define temperature inside the function
check_temp_range_with_more_param(temperature_var, upper_limit, lower_limit)<jupyter_output>Warm weather<jupyter_text>Libraries and ModulesOne of the great things about Python is the free availability of a _huge_ number of libraries (also called package) that can be imported into your code and (re)used. Modules contain functions for use by other programs and are developed with the aim of solving some particular problem or providing particular, often domain-specific, capabilities. A library is a collection of modules, but the terms are often used interchangeably, especially since many libraries only consist of a single module (so don’t worry if you mix them). In order to import a library, it must available on your system or should be installed.  A large number of libraries are already available for import in the standard distribution of Python: this is known as the standard library. If you installed the Anaconda distribution of Python, you have even more libraries already installed - mostly aimed at data science.Importing a library is easy:- Import (keyword) + library name, for example:     - `import os     contains functions for interacting with the operating system`    - `import sys    contains utilities to process command line arguments`More at: https://pypi.python.org/pypi<jupyter_code>import os

# Get current directory
cwd = os.getcwd()
print(cwd)
# Make new directory
os.mkdir('test_dir')
help(os)            # manual page created from the module's docstrings<jupyter_output><empty_output><jupyter_text>Using loops to iterate through files in a directory<jupyter_code># define a function that lists all the files in the folder called data

import os

def read_each_filename(pathname):
    for files in os.listdir(pathname):
        print(files)
   
pathname = 'data' # name of path with multiple files
read_each_filename(pathname)
# define a function that reads and prints each line of each file in the folder called data

import os

def read_each_line_of_each_file(pathname): # name of path with multiple files
    for files in os.listdir(pathname):
        with open(pathname+'/'+files, 'r') as in_fh:
            for lines in in_fh:
                print(lines)
                
pathname = 'data' # name of path with multiple files
read_each_line_of_each_file(pathname)

# Hints:
# Options for opening files
# option-1: with open("{}/{}".format(pathname, filename)) as in_fh:
# option-2: with open('%s/%s' % (pathname, filename)) as in_fh:
# option-3: with open(pathname + '/' + filename) as in_fh:
# option-4: with open(os.path.join(pathname, filename)) as in_fh:
# Exercise: Go through each filename in the directory 'data'
# Print the names of the files that contains the keyword 'asia'

def country_files(pathname, country):
    for files in os.listdir(pathname):
        if 'asia' in files:
            print(files)

pathname = 'data'
country = 'asia'
country_files(pathname, country)
# Open each file containing the keyword 'asia' and print all the entries

def read_filename_with_asia(pathname, country):
    for files in os.listdir(pathname):
        if country in files:
            # Print the names of the files that contains the keyword 'asia'
            print(files)
            # Open each file containing the keyword 'asia' and print all the entries
            with open(os.path.join(pathname, files), 'r') as in_fh:
                for lines in in_fh:
                    # Print entries that starts with gdp information on 'Japan', 'Korea', 'China' and 'Taiwan'
                    if lines.startswith('Japan')or lines.startswith('Korea')\
                    or lines.startswith('China')or lines.startswith('Taiwan'):
                        print(lines)
read_filename_with_asia('data', 'asia')<jupyter_output><empty_output><jupyter_text>Examples of importing basic modules. Questions- How can I read tabular data? Objectives- Import the Pandas library.- Use Pandas to load a simple CSV data set.- Get some basic information about a Pandas DataFrame.<jupyter_code>import pandas
# Use Oceania data here

df = pandas.read_csv("data/gapminder_gdp_asia.csv") #put data from a file in the dataframe

#read header
df.head()

#read header with 3 lines
df.head(n=3)

#read tail
df.tail()

#get the column with the data of 'gdpPercap_1952'
df['gdpPercap_1952']

#get multiple columns (Hint: pass list of the columns)
df[['gdpPercap_1952', 'gdpPercap_1957', 'gdpPercap_1962']]

#this can be done by writing the column number as well
df[[0]] #one column
df[[1,2,3]] #multiple columns

#this can be done by generating the column number by range too
df[list(range(1,3))] #single square

#datatypes in the dataframe
df.dtypes

#dimention of the dataframe
df.shape<jupyter_output><empty_output><jupyter_text>Aside: NamespacesPython uses namespaces a lot, to ensure appropriate separation of functions, attributes, methods etc between modules and objects. When you import an entire module, the functions and classes available within that module are loaded in under the modules namespace - `pandas` in the example above.  It is possible to customise the namespace at the point of import, allowing you to e.g. shorten/abbreviate the module name to save some typing:<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Also, as in the examples above, if you need only a single function from a module, you can import that directly into your main namespace (where you don't need to specify the module before the name of the function):<jupyter_code>from pandas import read_csv<jupyter_output><empty_output><jupyter_text>Conventions- You should perform all of your imports at the beginning of your program. This ensures that  - users can easily identify the dependencies of a program, and   - that any lacking dependencies (causing fatal `ImportError` exceptions) are caught early in execution- the shortening of `numpy` to `np` and `pandas` to `pd` are very common, and there are others too - watch out for this when e.g. reading docs and guides/SO answers online.  Execises - ImportingUse this link to follow further exercises: http://tobyhodges.github.io/python-novice-gapminder/37-reading-tabular/<jupyter_code># Use index_col to specify that a column’s values should be used as row headings.

df1 = pandas.read_csv("data/gapminder_gdp_asia.csv", index_col = 'country')
# Use DataFrame.info to find out more about a dataframe.
df1.info()
# The DataFrame.columns variable stores information about the dataframe’s columns.

df1.columns
# Use DataFrame.T to transpose a dataframe.
df1.T
# Use DataFrame.describe to get summary statistics about data.
df1.describe()<jupyter_output><empty_output><jupyter_text>Reading other dataRead the data in `gapminder_gdp_americas.csv` (which should be in the same directory as `gapminder_gdp_oceania.csv`) into a variable called `americas` and display its summary statistics.<jupyter_code>americas = pandas.read_csv("data/gapminder_gdp_americas.csv", index_col = 'country')
americas.describe()<jupyter_output><empty_output><jupyter_text>Inspecting DataAfter reading the data for the Americas, use `help(americas.head)` and `help(americas.tail)` to find out what `DataFrame.head` and `DataFrame.tail` do.What method call will display the first three rows of this data?What method call will display the last three columns of this data? (Hint: you may need to change your view of the data.)<jupyter_code>help(americas.head)
help(americas.tail)

americas.head(n=3)
americas.tail(n=3)<jupyter_output><empty_output><jupyter_text>Reading Files in Other DirectoriesThe data for your current project is stored in a file called `microbes.csv`, which is located in a folder called `field_data`. You are doing analysis in a notebook called `analysis.ipynb` in a sibling folder called `thesis`:```your_home_directory+-- field_data/|   +-- microbes.csv+-- thesis/    +-- analysis.ipynb```What value(s) should you pass to `read_csv` to read `microbes.csv` in `analysis.ipynb`?<jupyter_code>"../field_data/microbes.csv"<jupyter_output><empty_output><jupyter_text>Writing dataAs well as the `read_csv` function for reading data from a file, Pandas provides a `to_csv` function to write dataframes to files. Applying what you’ve learned about reading from files, write one of your dataframes to a file called `processed.csv`. You can use help to get information on how to use `to_csv`.<jupyter_code>americas.T.describe().to_csv("americas_data.csv")
%%bash
ls
cat americas_data.csv<jupyter_output>1-variables-and-data-structures.ipynb
2-flow-control.ipynb
3_1-functions-and-modules.ipynb
3_2-pandas-dataframe.ipynb
3_3-plotting.ipynb
4-styling-and-flowchart-exercises.ipynb
americas_data.csv
data
images
,Argentina,Bolivia,Brazil,Canada,Chile,Colombia,Costa Rica,Cuba,Dominican Republic,Ecuador,El Salvador,Guatemala,Haiti,Honduras,Jamaica,Mexico,Nicaragua,Panama,Paraguay,Peru,Puerto Rico,Trinidad and Tobago,United States,Uruguay,Venezuela
count,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0
unique,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0,13.0
top,10079.026740000001,3326.143191,7807.095818000001,36319.235010000004,4315.6227229999995,7006.580419,5118.146939,5180.75591,3614.101285,7103.702595000001,3421.523218,4031.4082710000002,1823.015995,3548.3308460000003,5246.107524,3478.125529,2749.320965,7356.031934000001,3998.875695,4446.380[...]
{'fraction_non_alphanumeric': 0.08943380040034316, 'fraction_numerical': 0.04074921361166714, 'mean_word_length': 4.5511904761904765, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '226335', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4716, 'n_tokens_neox': 4313, 'n_words': 1527, 'path': 'Day2/3_1-functions-and-modules.ipynb', 'repo_name': 'MatheLi/software-carpentry-embl-2017', 'size': '13988'}