<jupyter_start><jupyter_text>Coding Basics for Researchers - Day 1*Notebook by [<NAME>](https://github.com/pedrohserrano)* --- 1. Python Building Blocks* [1.1. Python Basic Commands](1.1)* [1.2. Strings](1.2)* [1.3. Lists](1.3)--- <NAME> | Monty python- | - ![](https://gvanrossum.github.io/images/guido-portrait-dan-stroud.jpg) | ![](https://upload.wikimedia.org/wikipedia/en/c/cd/Monty_Python%27s_Flying_Circus_Title_Card.png)  A bit of history Python starts with ABC.- ABC is a general-purpose programming language and programming environment, which had been developed in the Netherlands, Amsterdam, at the CWI (Centrum Wiskunde & Informatica).- The greatest achievement of ABC was to influence the design of Python.  He emphasizes on the DRY (Don’t Repeat Yourself) principle and readability.- Python was conceptualized in the late 1980s. <NAME> worked that time in a project at the CWI, called Amoeba, a distributed operating system.- Python was designed as a simple scripting language that possessed some of ABC's better properties, but without its problems.-  So, what about the name "Python": Most people think about snakes, but the name has something to do with excellent British humour. A show called Monty Python's Flying Circus was the culprit.  Tutorials for Learning Python    - [Codecademy](https://www.codecademy.com/tracks/python) is great for beginner levels.- There is also the [Official Beginners Guide](https://wiki.python.org/moin/BeginnersGuide).- [Learn Python the Hard Way](https://learnpythonthehardway.org/book/) is a great tutorial for a more in-depth overview.    - It isn't actually particularly hard, although note that the currently available version is in Python2. \n",- [Whirlwind Tour of Python](https://github.com/jakevdp/WhirlwindTourOfPython) is a free collection of Jupyter notebooks that takes you through Python.  - [Leet Code](https://leetcode.com/) is a place for more intense technical coding questions and challenges (geared towards industry interviews).  Getting Un-StuckAt some point, you will get stuck. It happens. The internet is your friend.    If you get an error, or aren't sure how to proceed, use {your favourite search engine} with specific search terms relating to what you are trying to do. Sometimes this just means searching the error that you got.   Your are likely to find responses on [StackOverflow](https://stackoverflow.com) - which is basically a forum for programming questions, and a good place to find answers.  Managing Cells in the Notebooks__Add__ a new cell to the notebook by: - click the + button on the toolbar - `Insert -> Insert Cell Above` or `ESC-A` - `Insert -> Insert Cell Below` or `ESC-B` __Delete__ a cell by selecting it and: - click the scissors button on the toolbar - `Edit -> Delete cells` or `ESC-DD`__Undelete__ the last deleted cell:- `Edit -> Undo Delete cells` or `ESC-Z`Each cell has a __cell history__ associated with it. Use `CMD-Z` to step back through previous cell contents. __Reorder__ cells by:- moving them up and down the notebook using the up and down arrows on the toolbar- `Edit -> Move Cell Up` or `Edit -> Move Cell Down` - cutting and pasting them: - `Edit - >Cut` or `Edit->Paste Cells Above` or `Edit->Paste Cells Below` - on the toolbar, `Cut selected cells` then `Paste selected cells`Copy and cut selected cells from the toolbar:- `Edit -> Copy Cells` or `ESC-C`.- `Edit -> Cut Cells` or `ESC-X`.  PackagesPackages are basically just collections of code. The anaconda distribution comes with all the core packages you will need for this class.   For getting other packages, anaconda comes with    conda    a package manager, with support for downloading and installing other packages. --- 1.1. Python basic commands Many of the things I used to use a calculator for, I now use Python for:<jupyter_code>2+2
(50-5*6)/4<jupyter_output><empty_output><jupyter_text>There are some gotchas compared to using a normal calculator.<jupyter_code>7/3<jupyter_output><empty_output><jupyter_text>Alternatively, you can convert one of the integers to a floating point number, in which case the division function returns another floating point number.<jupyter_code>7/3.0
7/float(3)<jupyter_output><empty_output><jupyter_text>Checking the datatype<jupyter_code>type(7/3)<jupyter_output><empty_output><jupyter_text>In the last few lines, we have sped by a lot of things that we should stop for a moment and explore a little more fully. We've seen, however briefly, two different data types: - **integers**, also known as *whole numbers* to the non-programming world, and - **floating point numbers**, also known as *decimal numbers* to the rest of the world. But also important is not only to do calculations but assign values - **Variables** are names for values.- In Python the `=` symbol assigns the value on the right to the name on the left. (Similar to `<-` in R)- The variables is created when a value is assigned to it.<jupyter_code>width = 20
length = 30
area = length*width
print(area)<jupyter_output>600<jupyter_text>But if you try to access a variable that you haven't yet defined, you get an error: ```Python> volume``````Python---------------------------------------------------------------------------NameError                                 Traceback (most recent call last) in ----> 1 volumeNameError: name 'volume' is not defined``` Variables must be created before they are used<jupyter_code>depth = 10
volume = area*depth
print(volume)<jupyter_output>6000<jupyter_text>You can name a variable *almost* anything you want. It needs to start with an alphabetical character or "\_", can contain alphanumeric charcters plus underscores ("\_"). Certain words, however, are reserved for the language:    and, as, assert, break, class, continue, def, del, elif, else, except,     exec, finally, for, from, global, if, import, in, is, lambda, not, or,    pass, print, raise, return, try, while, with, yieldTrying to define a variable using one of these will result in a syntax error: ```Pythonreturn = 0File "", line 1    return = 0           ^SyntaxError: invalid syntax``` The [Python Tutorial](http://docs.python.org/2/tutorial/introduction.htmlusing-python-as-a-calculator) has more on using Python as an interactive shell. The [IPython tutorial](http://ipython.org/ipython-doc/dev/interactive/tutorial.html) makes a nice complement to this, since IPython has a much more sophisticated iteractive shell. --- 1.2. Strings Strings are lists of printable characters, and can be defined using either single quotes<jupyter_code>'Hello, Maastricht!'<jupyter_output><empty_output><jupyter_text>or double quotes<jupyter_code>"Hello, Maastricht!"<jupyter_output><empty_output><jupyter_text>But not both at the same time, unless you want one of the symbols to be part of the string.<jupyter_code>"She's a Researcher"
'She asked, "How are you today?"'<jupyter_output><empty_output><jupyter_text>Just like the other two data objects we're familiar with (ints and floats), you can assign a string to a variable<jupyter_code>greeting = "Hello, Maastricht! "
subject = "She's a Researcher"<jupyter_output><empty_output><jupyter_text>The **print** statement is often used for printing character strings:<jupyter_code>example_text = greeting + subject

print(example_text)
type(example_text)<jupyter_output><empty_output><jupyter_text>Use an index to get a single character from a string.* The characters (individual letters, numbers) in a string are order. We can then treat the string as a list of characters.* Each position in the string is given a number called **index**.* Indices are numbered from 0.* Use the position's index in sqaure brackets to get the character at that position.![](https://swcarpentry.github.io/python-novice-gapminder/fig/2_indexing.svg)<jupyter_code># assign variable 
atom_name = "helium"

#print index 0 position
print(atom_name[0])<jupyter_output>h<jupyter_text>Use a slice to get a substring* A part of a string is called a substring. A substring can be as short as a single character.* An item in a list is called an element. Whenever we treat a string as if it were a list, the string’s elements are its individual characters.* A slice is a part of a string.* We take a slice by using `[start:stop]`, where `start` is replaced with the index of the first element we want and `stop` is replaced with the index of the element just after the last element we want.<jupyter_code># print name and substring first 3 characters
print(example_text[0:5])<jupyter_output>Hello<jupyter_text>But it can also print data types, separating by commas:<jupyter_code>print ("The area is ",area, volume, 10, 5*4, example_text)<jupyter_output>The area is  600 6000 10 20 Hello, Maastricht! She's a Researcher<jupyter_text>Also possible with the format method<jupyter_code>print ("The area is {} and volume is {}".format(area, volume))<jupyter_output>The area is 600 and volume is 6000<jupyter_text>In the above snipped, the number 600 (stored in the variable "area") is converted into a string before being printed out. If you have a lot of words to concatenate together, there are other, more efficient ways to do this. But this is fine for linking a few strings together.<jupyter_code># Number of characters in the text
len(example_text)<jupyter_output><empty_output><jupyter_text>Use `split` method to get the individual words<jupyter_code>split_text = example_text.split(' ') # Return a list of the words in text2, separating by ' '.
print(split_text)
len(split_text)<jupyter_output><empty_output><jupyter_text>More advanced functionalities allow us to find different type of words<jupyter_code>[w for w in split_text if len(w) < 6] # Words that are greater than 3 characteres long in text2
[w for w in split_text if w.istitle()] # Capitalized words in text2
[w for w in split_text if w.endswith('!')] # Words in text2 that end in 's'<jupyter_output><empty_output><jupyter_text>All the string tricks could be used for:- Data cleaning of clinical records- Text analysis on policy documents- Data analysis of gene sequence- ...etc. --- 1.3. Lists Very often in a programming language, one wants to keep a group of similar items together. The object we used in the above example is a Python data type called **lists**.<jupyter_code>days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"]
type(days_of_the_week)<jupyter_output><empty_output><jupyter_text>You can access members of the list using the **index** of that item:<jupyter_code># Index the 3rd element of the list, then
# index the 2nd element of the word
days_of_the_week[2][1]<jupyter_output><empty_output><jupyter_text>Python lists, like C, but unlike Fortran, use 0 as the index of the first element of a list. Thus, in this example, the 0 element is "Sunday", 1 is "Monday", and so on. If you need to access the *n*th element from the end of the list, you can use a negative index. For example, the -1 element of a list is the last element:<jupyter_code>print(days_of_the_week[-2] == days_of_the_week[5])<jupyter_output>True<jupyter_text>You can add additional items to the list using the .append() command:<jupyter_code># set a list of elements
languages = ["Java","R","C++"]

# append a new element
languages.append("Python")

# print the object 
print(languages)<jupyter_output>['Java', 'R', 'C++', 'Python']<jupyter_text>We could remove an element<jupyter_code>languages.remove('Java')
languages
del languages[-2]
languages<jupyter_output><empty_output><jupyter_text>The **range()** command is a convenient way to make sequential lists of numbers:<jupyter_code>range(10)<jupyter_output><empty_output><jupyter_text>Note that range(n) starts at 0 and gives the sequential list of integers less than n. If you want to start at a different number, use range(start,stop)<jupyter_code>list(range(2,8))<jupyter_output><empty_output><jupyter_text>Lists do not have to hold the same data type. For example,<jupyter_code>["Today",7,99.3,"", languages, days_of_the_week]<jupyter_output><empty_output><jupyter_text>However, it's good (but not essential) to use lists for similar objects that are somehow logically connected. If you want to group different data types together into a composite data object, it's best to use **tuples**, which we will learn about below.You can find out how long a list is using the **len()** command:<jupyter_code>help(len)<jupyter_output>Help on built-in function len in module builtins:

len(obj, /)
    Return the number of items in a container.<jupyter_text>- Iteration in Python  One of the most useful things you can do with lists is to *iterate* through them, i.e. to go through each element one at a time. To do this in Python, we use the **for** statement:<jupyter_code>for day in days_of_the_week:
    print (day)<jupyter_output>Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday<jupyter_text>This code snippet goes through each element of the list called **days_of_the_week** and assigns it to the variable **day**. It then executes everything in the indented block (in this case only one line of code, the print statement) using those variable assignments. When the program has gone through every element of the list, it exists the block.(Almost) every programming language defines blocks of code in some way. In Fortran, one uses END statements (ENDDO, ENDIF, etc.) to define code blocks. In C, C++, and Perl, one uses curly braces {} to define these blocks.Python uses a colon (":"), followed by indentation level to define code blocks. Everything at a higher level of indentation is taken to be in the same block. In the above example the block was only a single line, but we could have had longer blocks as well: ---  **EXERCISES - ** + _1. If you assign `a = 123`, what happens if you try to get the second digit of `a` via index `a[1]`?____ + _2. Which is a better variable name, m, min, or minutes? Why?   Hint: think about which code you would rather inherit from someone who is leaving the lab:_```python1. ts = m * 60 + s2. tot_sec = min * 60 + sec3. total_seconds = minutes * 60 + seconds```___ + _3. What do you think is the error in the code and how would you fix it```pythonatom_name = 'carbon'print('atom_name[1:3] is', atom_name[1:3])```___ + _3. What type of value (integer, floating point number, or character string) would you use to represent each of the following? Try to come up with more than one good answer for each problem. For example, in  1, when would counting days with a floating point variable make more sense than using an integer?_1. Number of days since the start of the year.2. Serial number of a piece of lab equipment.3. A lab specimen’s age4. Current population of a city.5. Average population of a city over time.--- + _4. Which of the following will return the floating point number 2.0? Note: there may be more than one right answer._```python1. first + float(second)2. float(second) + float(third)3. first + int(third)4. first + int(float(third))5. int(first) + int(float(third))6. 2.0 * second```--- + _5. You want to select a random character from a string_```pythonbases = 'ACTTGCTTGAC'```    + _2.1. Which [standard library](https://docs.python.org/3/library/) module could hep you?_    + _2.2. Which function would you select from that module?     + _2.3. Try to write a program that uses the function.____ + _6. When a colleague of yours types help(math), Python reports an error:_```pythonNameError: name `math` is not defined```    How would you help her?   --- + _7. Given the following:_```pythonprint('string to list:', list('silver'))print('list to string:', ''.join(['g', 'o', 'l', 'd']))```+ What does `list('some string') do?_+ What does `'-'.join(['x', 'y', 'z'])` generate?____ + _8. How many words are in the following text? Use Python to find out:_```Python The future is in Maastricht:UM multidisciplinary collaborations contribute to solving major societal issues within our primary research themes. We develop new methods to make plastics from organic materials, but we also conduct research into migration, and look into methods to get more people interested taking the necessary financial preparations for their retirement. Whenever possible, UM research is translated into economic, financial, or social value. UM participates in centres of excellence, both technological and social, to allow scientific discoveries to be swiftly converted into practical applications. What is more, research is integrated into education at every level. Our educational method, Problem-Based Learning, lays the groundwork for students to embrace research and the scientific method from the very first day of their studies.``` --- 2. Pandas Fundamentals* [2.1. Dealing with different data sources](2.1)* [2.2. Data structures](2.2)--- ![](http://warscapes.com/sites/default/files/assam_1.png) A research use case Insurgency-Civilian Relations & EU Policy- International Relations, Peace and Policy Studies- PhD proposal, <NAME> 2020- Understand patterns in international relations and to make effective policies and strategic decisions.- Normally:     - is studied the social order, and    - lived experiences of civilians living under rebel rule- Proposed:    - how and when such factors are perceived, discussed and considered in third party policy decisions    - what impact third party policy decisions have on rebel-civilian relations. - It contributes to a critical perspective and discussion on the motivations, interests, effectiveness, of third party policymakers --- 2.1. Dealing with different data sources * Pandas is widely-used Python library for handling data, particularly on tabular data.* Borrows many features from R's dataframes:  - Two dimensional table whose columns have names and potentially have different type of data types/* Load it with `import pandas as pd`. The alias pd is commonly used for Pandas.* Pandas is able to handle virtually any kind of formats![](https://pandas.pydata.org/docs/_images/02_io_readwrite.svg)<jupyter_code>import pandas as pd
path = '../data/'<jupyter_output><empty_output><jupyter_text>* The columns in a dataframe are the observed variables, and the rows are the observations.* Pandas uses backslash `\` to show wrapped lines when output is too wide to fit the screen.**File Not Found:**> Our lessons store their data files in a `data` sub-directory, which is why the path to the file is `data/...csv`. If you formet to include `data/` or if you include it but your cope of the file is somewhere else, you will bet a runtime error that ends with a line like this:`ERROR`: _OSError: File b'gapminder_gdp_oceania.csv' does not exist_  Reputation of Terror Groups (RTG) DatasetDescription: The dataset contains all domestic terrorist groups, which defined in Enders et al. (2011) and based on Global Terrorism Database, with more than 5 terrorist attacks from 1980 to 2011. The data is in group name - year format. The data codes terrorist groups' actions which can build reputation among constituency and out-group. Researchers can found originally coded variables in regard to building positive and negative reputation among the audience as well as existing group level variables.[Link to data](http://www.efetokdemir.com/data.html)<jupyter_code># READING A STATA FILE
rtg_table = pd.read_stata(path+'replicationdatajpr-oldstata.dta')
rtg_table.head()<jupyter_output><empty_output><jupyter_text>The Foundations of Rebel Group Emergence (FORGE) DatasetIt provides information on the origins of violent non-state actors engaged in armed conflict against their government resulting in 25+ yearly battle deaths, active between 1946 and 2011. The unit of observation in this dataset is the rebel group organization. We also include information on the dyad and conflict in which these groups are participants for ease of integration with various Uppsala Conflict Data Program (UCDP) datasets. We draw upon the population of groups included in the Non-State Actor database described in greater detail here:    [Link to data](http://ksgleditsch.com/eacd.html)<jupyter_code># READING AN ASCII FILE ON TABULAR FORMAT WITH CSV FUNCTION
forge_table = pd.read_csv(path+'nsa_v3.4_21November2013.asc', delimiter='\t')
forge_table.head()<jupyter_output><empty_output><jupyter_text>The CEPS EurLex dataset: EU laws from 1952-2019 with full text and 22 variablesThe dataset contains 142.036 EU laws - almost the entire corpus of the EU's digitally available legal acts passed between 1952 - 2019. It encompasses the three types of legally binding acts passed by the EU institutions: 102.304 regulations, 4.070 directives, 35.798 decisions in English language. The dataset was scraped from the official EU legal database (Eur-lex.eu) and transformed in machine-readable CSV format with the programming languages R and Python. The dataset was collected by the Centre for European Policy Studies (CEPS) for the TRIGGER project (https://trigger-project.eu/). We hope that it will facilitate future quantitative and computational research on the EU. [Link to data](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/0EGYWY)<jupyter_code># READING AN EXCEL FILE
eurlex_table = pd.read_excel(path+'EurLex_all_no_text.xlsx')
eurlex_table.head()<jupyter_output><empty_output><jupyter_text>--- 2.2. Data structures A DataFrame is a collection of Series; The DataFrame is the way Pandas represents a table, and Series is the data-structure Pandas use to represent a column.Pandas is built on top of the Numpy library, which in practice means that most of the methods defined for Numpy Arrays apply to Pandas Series/DataFrames.What makes Pandas so attractive is the powerful interface to access individual records of the table, proper handling of missing values, and relational-databases operations between DataFrames.<jupyter_code># Basic representation of a dataframe
dictionary = {'age': [20, 30], 'height': [1.80, 1.60], 'course':['Python', None]}

# Define a dataframe
df = pd.DataFrame(data=dictionary)

# Print the dataframe
df<jupyter_output><empty_output><jupyter_text>Use the `DataFrame.info()` method to find out more about a dataframe<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   age     2 non-null      int64  
 1   height  2 non-null      float64
 2   course  1 non-null      object 
dtypes: float64(1), int64(1), object(1)
memory usage: 176.0+ bytes<jupyter_text>* This is a DataFrame* Columns named `age`, `height` and `course`* Two actual 64-bit int values and one is floating point* The columns are not null* Uses 176 bytes of memory The DataFrame.columns variable stores information about the dataframe's columns* Note that this is data, _not_ a method. (it doesn't have parentheses)  - Like `math.pi`  - So do not use `()` to call it<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Use `DataFrame.T` to transpose a dataframe:* Sometimes want to treat columns as rows and vice versa.* Transpose (written `.T`) doesn't copy the data, just changes the program's view of it.* Like `columns`, it is a member variable<jupyter_code>df.T<jupyter_output><empty_output><jupyter_text>Use `DataFrame.describe()` to get summary statistics about data`DataFrame.describe()` gets the summary statistics of only the columns that have numerical data. All other columns are ignored, unless you use the argument `include = 'all'`<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Make use of a list to select desired columns and create a subset<jupyter_code># Desired columns
my_columns = ['age','course']

# Selection of subset
df[my_columns]<jupyter_output><empty_output><jupyter_text>Use `DataFrame.to_csv()` to generate a CSV file as result of the analysed dataframe<jupyter_code>df[my_columns].to_csv('example.csv', index=False)<jupyter_output><empty_output><jupyter_text>**EXERCISES - ** + _1. Reproduce the examples given in the introduction of this notebook    - Download the datasets    - Import Pandas and define a path to the files    - Read the files and save them in Python variables___ + _2.  Use the `help()` function to find out what `DataFrame.head` and `DataFrame.tail` do--- + _3.  Use the `DataFrame.info()` and `DataFrame.describe()`functions to learn general information about the datasets--- + _4.  Usinng the `CEPS EurLex dataset`.- Get the name of the columns with `DataFrame.columns`- Select the columns `Act_name` and `Eurlex_link` to create a subset of the original dataset- Select the frist 50 entries of the data using `head()` function- Save this resulted subset in a `CSV` file--- --- 3. Python and Automation* [3.1. Creating basic functions](3.1)* [3.2. Sharing is caring](3.2) --- 3.1. Creating basic functions A function is a block of organized, reusable code that can make your scripts more effective, easier to read, and simple to manage. You can think functions as little self-contained programs that can perform a specific task which you can use repeatedly in your code.We have already used some functions such as the `print()` command which is actually a built-in function in Python.  Steps: - Begin the definition of a new function with def.- Followed by the name of the function.    - Must obey the same rules as variable names.- Then parameters in parentheses.    - Empty parentheses if the function doesn’t take any inputs.- Then a colon.- Then an indented block of code.<jupyter_code>def print_greeting():
    print('Hello!')<jupyter_output><empty_output><jupyter_text>- Defining a function does not run it.    - Like assigning a value to a variable.- Must call the function to execute the code it contains.<jupyter_code>print_greeting<jupyter_output><empty_output><jupyter_text>- More useful when we can specify parameters when defining a function.    - These become variables when the function is executed.    - Are assigned the arguments in the call (i.e., the values passed to the function).    - If you don’t name the arguments when using them in the call, the arguments will be matched to parameters in the order the parameters are defined in the function.<jupyter_code>def print_date(year, month, day):
    joined = str(year) + '/' + str(month) + '/' + str(day)
    return print(joined)
print_date(1871, 3, 19)
print_date(month=3, day=19, year=1871)<jupyter_output>1871/3/19<jupyter_text>- Let's create a temperature converter<jupyter_code>def celsiusToFahr(tempCelsius):
    '''This function converts celsius to fahrenheit'''
    celsius_value = 9/5 * tempCelsius + 32
    return celsius_value
freezingPoint =  celsiusToFahr(0)

print('The freezing point of water in Fahrenheit is:', freezingPoint)
print('The boiling point of water in Fahrenheit is:', celsiusToFahr(100))<jupyter_output>The freezing point of water in Fahrenheit is: 32.0
The boiling point of water in Fahrenheit is: 212.0<jupyter_text>Having a **docstring** in the function it helps to know what the function is about through the python command line:<jupyter_code>help(celsiusToFahr)<jupyter_output>Help on function celsiusToFahr in module __main__:

celsiusToFahr(tempCelsius)
    This function converts celsius to fahrenheit
{'fraction_non_alphanumeric': 0.08797589298838748, 'fraction_numerical': 0.01315596060561517, 'mean_word_length': 4.8800777873811585, 'pattern_counts': {'":': 2, '<': 170, '<?xml version=': 0, '>': 182, 'https://': 14, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '44', 'dir': 'jupyter-structured-clean-dedup', 'id': '290776', 'license': 'MIT', 'n_tokens_mistral': 7980, 'n_tokens_neox': 7479, 'n_words': 3494, 'path': 'notebooks/all-notebooks.ipynb', 'repo_name': 'DeniseSl22/coding-for-researchers', 'size': '27255'}