<jupyter_start><jupyter_text>Reading/writing files BIOINF 575 - Fall 2020 Functions recap - important functions continued   RECAP & RESOURCES RECAP```python FUNCTIONS DEFINITION - creating a functiondef function_name(arg1, arg2, darg=None):     instructions to compute result    return result CALL - running a functionfunction_result = function_name(val1, val2, dval)```* A function doesn't need to have arguments to work* `return` statments exit the function while passing on the data* if there is no `return` statement None is returned* Defining a function does not run a function. * To run a function, it must be called using `([args])` after the function name```pythonfunction_name(val1, val2, [dval])``` RESOURCES FUNCTIONS   https://docs.python.org/3/tutorial/introduction.html https://docs.python.org/3/library/functions.htmlhttps://www.python.org/dev/peps/pep-0257/https://www.tutorialspoint.com/python3/python_functions.htmhttps://www.geeksforgeeks.org/functions-in-python/https://github.com/Pierian-Data/Complete-Python-3-Bootcamp/tree/master/03-Methods%20and%20FunctionsREADING & WRITING FILES   https://docs.python.org/3/tutorial/inputoutput.htmlreading-and-writing-fileshttps://www.tutorialspoint.com/python/python_files_io.htmhttps://www.geeksforgeeks.org/reading-writing-text-files-python/https://www.w3schools.com/python/python_file_write.asphttps://www.python-course.eu/python3_file_management.phphttps://colab.research.google.com/github/computationalcore/introduction-to-python/blob/master/notebooks/4-files/PY0101EN-4-1-ReadFile.ipynbhttps://github.com/aushafy/Python-Write-and-Read-Filehttps://eeob-biodata.github.io/BCB546X-Fall2017/Week_09/additional-lesson/ --- Function Examples ___ `*args` - unkown no. of arguments - unpack collection of argument values `**kargs` - unkown no. of arguments - unpack mapping of names and values<jupyter_code>x = 20
print(*("EGFR", "TP53", "AAACGTTA", 30, [x, 60]))
print("EGFR", "TP53", "AAACGTTA", 30, [x, 60])
def test_karg(**keys_args_dict):
    for name,value in keys_args_dict.items():
        print("name = ", name)
        print("value = ", value)
test_karg(**{"gene":"EGFR", "expression": 20,"transcript_no": 4})
test_karg(gene = "EGFR", expression = 20, transcript_no = 4)<jupyter_output><empty_output><jupyter_text>___ `Recursive` function - function that calls itself<jupyter_code>test_list = ["CGTA", "CCCT", "GGGA", "ACGG"]
def display_elements(seq_list, pos):
    if pos <= len(seq_list)-1:
        print(seq_list[pos])
        display_elements(seq_list, pos + 1)

display_elements(test_list, 0)<jupyter_output><empty_output><jupyter_text>____ `lambda` function - anonymous function - it has no nameShould be used only with simple expressions https://docs.python.org/3/reference/expressions.htmllambdahttps://www.geeksforgeeks.org/python-lambda-anonymous-functions-filter-map-reduce/https://realpython.com/python-lambda/ `lambda arguments : expression` A lambda function can take any number of arguments, but must always have only one expression.<jupyter_code>compute_expression = lambda x, y: x + y + x*y
compute_expression(2, 3)<jupyter_output><empty_output><jupyter_text>____ Useful functions Built-in functionshttps://docs.python.org/3/library/functions.html  `zip(*iterables)` - make an iterator that aggregates respective elements from each of the iterables.   https://docs.python.org/3/library/functions.htmlzip `map(function, iterable, ...)` - apply function to every element of an iterable - return iterable with resultshttps://docs.python.org/3/library/functions.htmlmap `filter(function, iterable)` - apply function (bool result) to every element of an iterable - return the elements from the input iterable for which the function returns Truehttps://docs.python.org/3/library/functions.htmlfilter `functools.reduce(function, iterable[, initializer])` - apply function to every element of an iterable to reduce the iterable to a single valuehttps://docs.python.org/3/library/functools.htmlfunctools.reduce<jupyter_code>combined_res = zip([10,20,30],["ACT","GGT","AACT"],[True,False,True])
combined_res
for element in combined_res:
    print(element)
list(combined_res)
# unzip list
x, y, z = zip(*[(3,4,7), (12,15,19), (30,60,90)])
print(x, y, z)<jupyter_output><empty_output><jupyter_text>_____<jupyter_code>map(abs,[-2,0,-5,6,-7])
list(map(abs,[-2,0,-5,6,-7]))<jupyter_output><empty_output><jupyter_text>https://www.geeksforgeeks.org/python-map-function/<jupyter_code>numbers1 = [1, 2, 3] 
numbers2 = [4, 5, 6] 
  
result = map(lambda x, y: x + y, numbers1, numbers2) 
list(result)<jupyter_output><empty_output><jupyter_text>____Use a lambda function and the map function to compute a result from the followimg 3 lists.If the element in the third list is divisible by 3 return 3*x, otherwise return 2*y.<jupyter_code>numbers1 = [1, 2, 3, 4, 5, 6] 
numbers2 = [7, 8, 9, 10, 11, 12] 
numbers3 = [13, 14, 15, 16, 17, 18] 

result = map(lambda x, y, z: 3*x if z%3 ==0 else 2*y, \
             numbers1, numbers2, numbers3) 
list(result)


def compute_res(x,y,z):
    res = None
    if z%3 == 0:
        res = 3*x
    else:
        res = 2*y
    return res


result = map(compute_res, numbers1, numbers2, numbers3) 
list(result)<jupyter_output><empty_output><jupyter_text>____<jupyter_code>test_list = [3,4,5,6,7]
result = filter(lambda x: x > 4, test_list)
result
list(result)
# Filter to remove empty structures or 0
test_list = [3, 0, 5, None, 7, "", "AACG", []]
result = filter(bool, test_list)
list(result)<jupyter_output><empty_output><jupyter_text>____<jupyter_code>from functools import reduce
reduce(lambda x,y: x+y, [47,11,42,13])<jupyter_output><empty_output><jupyter_text>https://www.python-course.eu/lambda.php https://www.geeksforgeeks.org/reduce-in-python/https://www.tutorialsteacher.com/python/python-reduce-function<jupyter_code>test_list = [1,2,3,4,5,6]
# compute factorial of n
n=5
reduce(lambda x, y: x*y, range(1, n+1))<jupyter_output><empty_output><jupyter_text>_____ Input/Output and File Handling  Reading user input: the `input()` function<jupyter_code># reads text, evaluates expression
res = input()
while res != "STOP":
    print("input", res)
    res = input()<jupyter_output><empty_output><jupyter_text>File is a named location on disk to store related information    It is used to permanently store data in a non-volatile memory (e.g. hard disk)https://www.programiz.com/python-programming/file-operation     Open a file for reading or writing    https://docs.python.org/3/library/functions.htmlopen```pythonopen(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) fileObj = open(fileName, ‘r’)  open file for reading, r+fileObj = open(fileName, ‘w’)  open file for writing, w+fileObj = open(fileName, ‘a’)  open file for appending, a+```(Note: fileName must be a string or reference to one)The file object is iterable by line<jupyter_code>#help(open)<jupyter_output><empty_output><jupyter_text>Write<jupyter_code># open file and write lines into a file
test_file = open("test.txt", mode = "w")
test_file.write("Writing some text.\n")
test_file.write("Writing another line.")
test_file.close()
for elem in dir(test_file):
    if "_" not in elem:
        print(elem)<jupyter_output><empty_output><jupyter_text>Close file<jupyter_code># close()
help(test_file.close)<jupyter_output><empty_output><jupyter_text>Read<jupyter_code># open file and read file contents
test_file = open("test.txt", "r")
res = test_file.read()
print(res)
test_file.close()<jupyter_output><empty_output><jupyter_text>Read line<jupyter_code>test_file = open("test.txt", "r")
res = test_file.readlines()
print(res)
test_file.close()<jupyter_output><empty_output><jupyter_text>Go at position<jupyter_code># seek
test_file = open("test.txt", "r")
test_file.seek(10)
res = test_file.readline()
print(res)
print(test_file.tell())
test_file.close()<jupyter_output><empty_output><jupyter_text>Return current position<jupyter_code># tell<jupyter_output><empty_output><jupyter_text>Context manager with: give code context The special part about with is when it is paired with file handling or database access functionality<jupyter_code># Without with
test_file = open("test.txt",'r')
print(test_file.read())
test_file.close()
# With with :)
with open("test.txt",'r') as test_file:
    print(test_file.read())<jupyter_output><empty_output><jupyter_text>The file is opened and processed. As soon as you exit the with statement, the file is closed automatically.<jupyter_code># parsing files
def parse_line(line):
    return line.strip().split(" ")

with open("test.txt",'r') as test_file:
    line = test_file.readline()
    while (line != ""):
        print(parse_line(line))
        line = test_file.readline()<jupyter_output><empty_output><jupyter_text>Exercise:   Open the file test.txt and add 10 lines in a for loop.   The line should contain: Line index    Line 0  Line 1   https://www.tutorialspoint.com/python/python_files_io.htmhttps://www.tutorialspoint.com/python/file_methods.htm _____ NumPy - Numeric python  NumPy (np) is the premier Python package for scientific computinghttps://numpy.orgIts powerful comes from the N-dimensional array objectnp is a *lower*-level numerical computing library. This means that, while you can use it directly, most of its power comes from the packages built on top of np:* Pandas (*Pan*els *Da*tas)* Scikit-learn (machine learning)* Scikit-image (image processing)* OpenCV (computer vision)* more... Importing NumPyConvention: use np alias<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>NumPy basics Arrays are designed to:* handle vectorized operations lists are not    * if you apply a function it is performed on every item in the array, rather than on the whole array object* store multiple items of the same data type* have 0-based indexing* Missing values can be represented using `np.nan` object    * the object `np.inf` represents infinite* Array size cannot be changed, should create a new array* An equivalent numpy array occupies much less space than a python list of lists Create Arrayhttps://docs.scipy.org/doc/numpy-1.13.0/user/basics.creation.html<jupyter_code># Build array from Python list
vector = np.array([1,2,3])
vector
# matrix with zeros 
np.zeros((3,4), dtype = int)
# matrix with 1s
np.ones((3,4), dtype=int)
# matrix with a constant value
value = 20
np.full((3,4), value)
# Create a 4x4 identity matrix
np.eye(4)        
# arange - numpy range
np.arange(10, 30, 2)
# evenly spaced numbers over a specified interval
ev_array = np.linspace(1, 10, 20)
print(ev_array)
ev_array.shape<jupyter_output><empty_output><jupyter_text>Random datahttps://docs.scipy.org/doc/numpy-1.13.0/reference/routines.random.html<jupyter_code># Create an array filled with random values
np.random.random((3,4))        
# Create an array filled with random values from the standard normal distribution
np.random.randn(3,4)    
# Generate the same random numbers every time
# Set seed
np.random.seed(10)
np.random.randn(3,4)
np.random.seed(10)
print(np.random.randn(3,4))

print(np.random.randn(3,4))

np.random.seed(100)
print(np.random.randn(3,4))<jupyter_output><empty_output><jupyter_text>```python Create the random staters = np.random.RandomState(100)``` Basic array attributes:* shape: array dimension* size: Number of elements in array* ndim: Number of array dimension (len(arr.size))* dtype: Data-type of the array* T: The transpose of the array<jupyter_code>matrix = np.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12]])
matrix
# let's check them out 
matrix.shape
matrix.size
matrix = np.array([[[1,2],[2,3],[3,4]],[[4,5],[4,6],[6,7]]])
matrix.ndim
matrix.dtype
matrix.T
matrix<jupyter_output><empty_output><jupyter_text>Reshaping<jupyter_code>matrix
# Reshaping
matrix_reshaped = matrix.reshape(2,6)
matrix_reshaped<jupyter_output><empty_output><jupyter_text>Slicing/Indexing<jupyter_code># List-like
matrix_reshaped[1][1]
matrix_reshaped[1,3]
matrix_reshaped[1,:3]
matrix_reshaped[:2,:3]
# iterrating ... let's print the elements of matrix_reshaped
nrows = matrix_reshaped.shape[0]
ncols = matrix_reshaped.shape[1]

for i in range(nrows):
    for j in range(ncols):
        print(matrix_reshaped[i,j])


# Fun arrays
checkers_board = np.zeros((8,8),dtype=int)
checkers_board[1::2,::2] = 1
checkers_board[::2,1::2] = 1
print(checkers_board)<jupyter_output><empty_output><jupyter_text>Create a 2d array with 1 on the border and 0 inside<jupyter_code>boarder_array = np.zeros((8,8),dtype=int)
boarder_array[0,:] = 1

boarder_array
boarder_array = np.ones((8,8),dtype=int)
boarder_array[1:-1,1:-1] = 0
boarder_array
boarder_array[:,-1]<jupyter_output><empty_output><jupyter_text>Performance test_list = list(range(int(1e6)))test_vector = np.array(test_list)<jupyter_code>test_list = list(range(int(1e6)))
test_vector = np.array(test_list)
%%timeit
sum(test_list)
%%timeit
np.sum(test_vector)<jupyter_output><empty_output><jupyter_text>https://numpy.org/devdocs/user/quickstart.htmluniversal-functions Matrix operationshttps://www.tutorialspoint.com/matrix-manipulation-in-pythonArithmetic operators on arrays apply elementwise.  A new array is created and filled with the result.Array broadcastinghttps://docs.scipy.org/doc/numpy/user/basics.broadcasting.htmlThe term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.  https://www.tutorialspoint.com/numpy/numpy_broadcasting.htm<jupyter_code>matrix
np.array([1,2,3,4]).reshape(4,1)
matrix + np.array([1,2,3,4]).reshape(4,1)
matrix + np.array([1,2,3])
matrix * np.array([1,2,3,4]).reshape(4,1)
matrix2 = np.array([[1,2,3],[5,6,7],[1,1,1],[2,2,2]])
matrix2
matrix * matrix2
# matrix multiplication
matrix.dot(np.array([1,2,3]).reshape(3,1))
# matrix multiplication - more recently
matrix@(np.array([1,2,3]).reshape(3,1))
# stacking arrays together
np.vstack((matrix,matrix2))
np.hstack((matrix,matrix2))
# splitting arrays 
np.vsplit(matrix,2)
np.hsplit(matrix,(2,3))<jupyter_output><empty_output><jupyter_text>Copy<jupyter_code>matrix
# shallow copy - looks at the same data
matrix_copy = matrix
matrix_copy1 = matrix.view()
print(matrix_copy)
print(matrix_copy1)
print(matrix)

print(matrix_copy)

print(matrix_copy1)
matrix_copy1[0,0] = 5
# deep copy
matrix_copy2 = matrix.copy()
print(matrix_copy2)
matrix_copy2[0,0] = 7
print(matrix)

print(matrix_copy)

print(matrix_copy1)

print(matrix_copy2)<jupyter_output><empty_output><jupyter_text>More matrix computation<jupyter_code># conditional subsetting
matrix[(6 < matrix[:,0])]
matrix[(4 <= matrix[:,0]) & (matrix[:,0] <= 7)
       & (2 <= matrix[:,1]) & (matrix[:,1] <= 7),]
matrix
# col mean
matrix.mean(axis = 0)
# row mean
matrix.mean(axis = 1)
# unique values and counts
matrix = np.random.random((3,4), )
matrix = np.array([[ 5,  2,  3],
       [ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12]])
uvals, counts = np.unique(matrix, return_counts=True)
print(uvals,counts)<jupyter_output><empty_output><jupyter_text>https://www.w3resource.com/python-exercises/numpy/index.php Create a matrix of 5 rows and 6 columns with numbers from 1 to 30.Add 2 to the odd values of the array.<jupyter_code>matrix = np.arange(1,31).reshape(5,6)
matrix[matrix%2==1 ] +=  2 
matrix<jupyter_output><empty_output><jupyter_text>Normalize the values in the matrix. Substract the mean and divide by the standard deviation.<jupyter_code>mat_mean = np.mean(matrix)
mat_std = np.std(matrix)
matrix_norm = (matrix - mat_mean)/mat_std
matrix_norm
matrix<jupyter_output><empty_output><jupyter_text>Create a random array (5 by 3) and compute:    * the sum of all elements    * the sum of the rows     * the sum of the columns<jupyter_code>matrix = np.random.rand(5,3)
print(matrix)
matrix.sum()
matrix.sum(1)
matrix.sum(0)
#Given a set of Gene Ontology (GO) terms and the genes that are associated with these terms find the gene 
#that is associated with the most GO terms

go_terms=np.array(["cellular response to nicotine",
                   "cellular response to hypoxia",
                   "cellular response to lipid"])
genes=np.array(["BAD","KCNJ11","MSX1","CASR","ZFP36L1"])

assoc_matrix = np.array([[1,1,0,1,0],[1,0,0,1,1],[1,0,0,0,0]])

print(assoc_matrix)
max(assoc_matrix.sum(0))
genes[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13516812107158113, 'fraction_numerical': 0.028376151827668273, 'mean_word_length': 5.808475280432073, 'pattern_counts': {'":': 4, '<': 143, '<?xml version=': 0, '>': 138, 'https://': 38, 'lorem ipsum': 0, 'www.': 18, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '24628', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5935, 'n_tokens_neox': 5632, 'n_words': 1357, 'path': 'class_notebooks/files_rw.ipynb', 'repo_name': 'mitreac/b57520', 'size': '16387'}