<jupyter_start><jupyter_text>Carnegie Python BootcampWelcome to the python bootcamp. This thing you're reading is called an _ipython notebook_ and will be your first introduction to the Python programming language. Notebooks are a combination of text markup and code that you can run in real time. Importing what you needIt is often said that python comes with the _batteries included_, which means it comes with almost everything you need, bundled up in seperate _modules_. But not everything is loaded into memory automatically. You need to import the modules you need. Try running the following to import the antigravity module. Just hit `CTRL-ENTER` in the cell to execute the code<jupyter_code>import antigravity<jupyter_output><empty_output><jupyter_text>You will find that python is full of nerdy humour like this. By the way, python is named after Monty Python's Flying Circus, *not* the reptile. Let's go ahead and import some useful modules you **will** need. Use this next command cell to import the the modules named ``os``, ``sys``, and ``numpy``.<jupyter_code>import os,sys,numpy<jupyter_output><empty_output><jupyter_text>The `os` module gives you functions relating to the operating system, `sys` gives you information on the python interpreter and your script, and `numpy` is the mathematical powerhorse of python, allowing you to manipulate arrays of numbers.Once you import a module, all the functions and data for that module live in its _namespace_. So if I wanted to use the `getcwd` function from the `os` module, I would have to refer to it as ``os.getcwd``. Try running the next command:<jupyter_code>os.getcwd()<jupyter_output><empty_output><jupyter_text>As you can probably guess, `os.getcwd` gets the current working directory. Try some commands for yourself. The ``numpy`` module has many mathematical functions. Try computing the square root (``sqrt``) of 2. You can also try computing $\sin(\pi)$ (numpy has a built-in value ``numpy.pi``) and even $e^{i\pi}$ (python uses the engineering convention of ``1j`` as $\sqrt{-1}$).<jupyter_code>numpy.sqrt(2)
numpy.sin(numpy.pi)
numpy.exp(numpy.pi*1j)<jupyter_output><empty_output><jupyter_text>There can also be namespaces inside namespaces. For example, to test if a file exists, you would do:<jupyter_code>print (os.path.isfile('.bash_profile'))<jupyter_output><empty_output><jupyter_text>Here is a non-comprehensive list of modules you may find useful. Documentation for all of them can be found with a quick google search.- `os`, `sys`: As mentioned above, these give you access to the operating system, files, and environment.- `numpy`: Gives you arrays (vectors, matrices) and the ability to do math on them.- `scipy`: Think of this as "Numerical Recipes for Python". Root-finding, fitting functions, integration, special                mathematical functions, etc.- `pandas`: Primarily used for reading/writing data. Useful for [data wrangling](https://en.wikipedia.org/wiki/Data_wrangling).- `astropy`: Astronomy-related functions, including FITS image reading, astrometry, and cosmological calculations. Getting HelpAlmost everything in python has documentation. Just use the `help()` function on anything in python to get information. Here is an example help output on `numpy.sqrt`. Try others.<jupyter_code>help(numpy.sqrt)<jupyter_output><empty_output><jupyter_text>Variables and TypesA variable is like a box that you put stuff (values) in for later use. In Python, there are lots of different ``types`` of variables corresponding to what you put in. Unlike other languages, you don't have to tell ``python`` what type each variable is:  ``python`` figures it out on its own. To put the value 1 into a box called x, just use the equals sign, like you would when solving a math problem.<jupyter_code>x=1<jupyter_output><empty_output><jupyter_text>Now you can later use this value. Try printing it out using the ``print()`` function. You can also modify the variable. Try adding or subtracting from x and print it out again.<jupyter_code>print(x)
x=x+1
print(x)<jupyter_output><empty_output><jupyter_text>But be careful about the type. Try assigning the value of 1 to ``y``. Now divide it by two and print out the result.<jupyter_code>y = 1
y = y/2
print(y)<jupyter_output><empty_output><jupyter_text>Repeat, but this time assign the value of 1.0 to y.<jupyter_code># make sure you include a decimal in the number one! (fyi, this is a comment)
y = 1.0
y = y/2
print (y)<jupyter_output><empty_output><jupyter_text>You get different results because in the first instance, ``y`` was interpreted as an integer, so the result of dividing by 2 must also be an integer. In the second case, ``y`` is interpreted as a real number (aka floating point or float), so the result of dividing by zero is what you expect. There are other types of variables. The most commonly used are strings, lists, and arrays. But literally **anything** can be assigned to an array. You can even assign the ``numpy`` module to the variable ``np``.<jupyter_code>np = numpy<jupyter_output><empty_output><jupyter_text>Now you can use ``np.sqrt()`` rather than ``numpy.sqrt()``. Most python programmers do this with commonly-used modules. In fact, we usually just use the special form:  ``import numpy as np``.  StringsStrings are collections of alpha-numeric characters and are the primary way data are represented outside your code. So you will find yourself manipulating strings in Python ALL THE TIME. You might need to convert strings from a text file into python data ojects to work with or you might need to do the opposite and generate an output file with final results that can be read by humans. Below are the most common things we need.Strings are enclosed in matching singe (`'`) or double (`"`) quotes. Python doesn't distinguish as long as you match them consistent.<jupyter_code># Strings are enclosed by single or double quotes
s='this is a string'
print (s)<jupyter_output><empty_output><jupyter_text>We can use the `len()` function to determine the length of the string. Try this out below. Strings are very similar to lists in `python` (we'll see that below). Each alpha-numeric character is an element in the string, and you can refer to individual elements (characters) using an index enclosed in square brackets (`[]`). You can also specify a range of indices by separating them with a colon (`:`). `Python` indexes from 0 (not 1), so the first element is index `[0]`, the second `[1]` and so on.<jupyter_code>print(s[2])
print(s[2:6])<jupyter_output><empty_output><jupyter_text>Specifying a range of indices (as well as more complicated indexing we'll see later) is called *slicing*. Formatting stringsSometimes you'll need your integers and floats to be converted into strings and written out into a table with specific formats (e.g., number of significant figures). This involves a syntax that's almost a separate language itself (though if you've used `C` or `C++` it will be very familiar). Here is a good reference: https://pyformat.info/We'll cover the most important. First, if you just print out a regular floating point number, you get some arbitrary number of significant figures. The same is true if you just try to convert the float to a string using `str()`:<jupyter_code>x=3.141592653589
print(x)

s=str(x)
print(s)<jupyter_output><empty_output><jupyter_text>If you only want two significant figures, or you want the whole number of span 15 spaces (to make nicely lined-up colums for a table), you need to use a format string. A format string is just like a regular string, but has special place-holders for your numbers. These start with a percent ('%'), followed by an optional width, an optional precision after a period, and the type (`f` for float, `d` for integer, `s` for string). For example, a float with two decimals:<jupyter_code>fmt = "This is a float with two decimals:  %.2f"
print (fmt)<jupyter_output><empty_output><jupyter_text>By itself, the format string didn't do much. In order to inject the number, we use another `%`, but as an operator:<jupyter_code># two decimal places
print(fmt % x)<jupyter_output><empty_output><jupyter_text>You may want to make a format string with more than one number or string. You can do this by grouping the arguments into a list with round brackets (called a *tuple*):<jupyter_code>fmt = "Here is a float: '%.2f', and another '%8.4f', an integer %d, and a string %s"
print (fmt % (x, np.pi, 1000, 'look ma, no quotes!'))<jupyter_output><empty_output><jupyter_text>We will not cover the *new* string formatting now available in `python 2.7` and `python 3.x`. It has some advantages over the old style, but both are fully supported by both versions, so we've stuck with the old as it is by far more common. The reference above describes both methods. You can mix and match as you see fit.  Lists and numpy arraysLists contain a sequence of objects, typically, a sequence of numbers.  The are enclosed by square brackets, (e.g., ``[]``), and each item separated by a comma. Here are some examples.<jupyter_code># A list of floats
x1=[1.,2.,7.,2500.]
print(x1)

# A list of strings
x2=['a','b','c','x','y','z']
print(x2)

# Lists can also contain a mix of object types
x3=[1.,'a',6,'z']
print(x3)<jupyter_output><empty_output><jupyter_text>Numpy arrays allow for more functionality than lists.  While they may also contain a mix of object types, you will primarily be working with numpy arrays that are comprised of numbers: either integers or floats.  For example, you will at some point read in a table of data into a numpy array and do things to it, like add, multiply, etc.Above, we imported the `numpy` module as `np`.  We will use the latter to create arrays.<jupyter_code>x=np.array([1.,2.,3.,4.])
print(x)
# We can add one number to every element:
y1=x+2
print(y1)

# Or raise every element to some power:
y2=x**2
print(y2)<jupyter_output><empty_output><jupyter_text>Here is where the real power of numpy arrays comes into play.  We can use `np` to carry out all kinds of mathematical tasks that in other programming languages (like ``C``, ``FORTRAN``, etc) would require some kind of loop. Here are some of the most common tasks we'll use. Array creationThere are many functions in ``numpy`` that allow you to make arrays from scratch. We can create an array of zeros:<jupyter_code>x1=np.zeros(5)
print(x1)<jupyter_output><empty_output><jupyter_text>Or an array of ones:<jupyter_code>x2=np.ones(5)
print(x2)<jupyter_output><empty_output><jupyter_text>Or an array of pi's:<jupyter_code>x3=np.ones(5)*np.pi
print(x3)<jupyter_output><empty_output><jupyter_text>We can create a sequence of numbers using np.arange:<jupyter_code>x1=np.arange(5.)
print(x1)<jupyter_output><empty_output><jupyter_text>Or spaced by a specified amount between two numbers:<jupyter_code>x2=np.arange(0,10.,2)
print(x2)<jupyter_output><empty_output><jupyter_text>Or _N_ elements equally spaced between two specified numbers:<jupyter_code>x3=np.linspace(0,8.,5)
print(x3)<jupyter_output><empty_output><jupyter_text>One can also create N-dimensional numpy arrays.  For example, images that you read into Python will typically be stored into 2D numpy arrays.<jupyter_code>x=np.ones((4,2))
print(x)
print(x.shape)<jupyter_output><empty_output><jupyter_text>Array MathJust as with spreadsheets, you can do math on entire arrays using operators (``+``,``-``,``*``, and ``/``) and ``numpy`` functions.We can add, multiply, etc., numpy arrays:<jupyter_code>x4=0.5+x1+x2*x3/2.
print(x4)<jupyter_output><empty_output><jupyter_text>We also have access to most mathematical functions you'll need:<jupyter_code>x5 = np.power(x4,2)
print(x5)
x6 = np.log10(x5)
print (x6)<jupyter_output><empty_output><jupyter_text>Matrix Math``numpy`` can treat 2D arrays as matrices and leverage special compiled libraries so that you can do linear algebra routines easily (and very fast). Here, we construct a rotation matrix and use ``dot`` do do the dot-product. Try changing theta to different angles (in radians!).<jupyter_code># CCW rotation by 180 degrees
theta=np.pi/4
# Rotation matrix
x=np.array([[np.cos(theta),-np.sin(theta)],
            [np.sin(theta),np.cos(theta)]])
print(x)

# Lets rotate (1,1) about the origin
y=np.array([1.,1.])
z=np.dot(x,y)

print(y)
print(z)<jupyter_output><empty_output><jupyter_text>The other common tasks matrix tasks can also be done with either ``numpy`` functions, ``numpy.linalg``, or member functions of the object itself<jupyter_code># Taking the transpose
x_t = x.T
print(x_t)

# Computing the inverse
x_i = np.linalg.inv(x)

# Matrix Multiplication
I = np.dot(x_i,x)
print (I)<jupyter_output><empty_output><jupyter_text>Array SlicingOften, you need to access elements or sub-arrays within your array. This is referred to as *slicing*. We can select individual elements in an array using indices (note that 0 is the first element and negative indices count backwards from the end):<jupyter_code>x=np.arange(5)
print(x)
print(x[0])
print(x[4])
print(x[2:])   # 3rd element onward
print(x[:-1])  # all elements up until the last one
print(x[-1:])  # last element
print(x[::-1]) # reverse array<jupyter_output><empty_output><jupyter_text>We can also "slice" N-dimensional arrays:<jupyter_code>x=np.arange(8)
print(x)
x=x.reshape((4,2))
print(x)

print(x[0,:])
print(x[:,0])<jupyter_output><empty_output><jupyter_text>Control blocks So far, we've been running individual sets of commands and looking at their results immediately. Later, we will write a complete *program*, which is really just bundling up instructions into a recipe for solving a given task. But as the tasks we want to perform become more complicated, we need *control blocks*. These allow us to:* Repeat tasks again and again (loops)* Perform tasks only if certain conditions are met (if-else blocks)* Group instructions into a single logical task (user-defined functions)`python` is rather unique in that it uses indenting to indicate the beginning and end of a logical block. This actually forces you to write readable code, which is a really good thing! For loops`for` loops are useful for repeating a series of operations a given number of times. In `python`, you loop over elements of an list or array. So if you want to loop over a sequence of integers (say, they indices of an array), then you would use the `range()` function to generate the list of integers. You might also use the `len()` function if you need the length of the array.<jupyter_code># range(n) creates a list of n elements
print(range(5))

# We can use it to iterate over a for loop
for ii in range(5):
    print(ii)<jupyter_output><empty_output><jupyter_text>Notice that after the line containing the for statement, the subsequent lines are indented to indicate that they are all part of the same block You can use a `for` loop to build up a list of elements by appending to an existing list (using its `append()` member function. For example, to create the first N elements of the Fibonacci sequence:<jupyter_code>fib = [1,1]
N = 100
for i in range(N-2):
    fib.append(fib[-2]+fib[-1])
print (fib)<jupyter_output><empty_output><jupyter_text>You may notice (if N is large enough) that you get numbers that have an `L` at the end. `python` has a special type called a *long integer* which allows for abitrarily large numbers.Here is an example of what **not** to do with for loops (if you can help it). `For` loops are more computationally expensive in python than using the `numpy` functions do do the math. *Always* try to cast the problem in terms of `numpy` math and functions. It will make your code faster. Try making N in the following example larger and larger and you'll see the difference.<jupyter_code># A slightly more complex example of a for loop:
import time
N = 100
x=np.arange(N)
y=np.zeros(N)

t1 = time.time()   # start time
for ii in range(x.size):
    y[ii]=x[ii]**2
t2 = time.time()   # end time
    
print ("for loop took "+str(t2-t1)+" seconds")

# Note that the above could have been acheived more easily with straight array operations:
t1 = time.time()
y2=x**2
t2 = time.time()
print ("numpy square took "+str(t2-t1)+" seconds")<jupyter_output><empty_output><jupyter_text>If blocks`if` statements act as a way to carry out a task only if a certain _condition_ is met. As with the `for` loops above, everything that is indented after the `if` statement is part of the block of code that is executed if the condition is met. Likewise for the `else` block.<jupyter_code>x=5
if x==5:
    print('Yes! x is 5')
    
# The two equal signs evaluate whether x is equal to 5.  One can also use >, >=, <, <=, != (not equal to)
x=5
if x==3:
    print('Yes! x is 3')<jupyter_output><empty_output><jupyter_text>`if-else` statements execute the code in the `if` block if the condition is `true`, otherwise it executes the code in the `else` block:<jupyter_code>x=5
if x==3:
    print('Yes! x is 3')
else:
    print('x is not 3')
    print('x is '+str(x))<jupyter_output><empty_output><jupyter_text>One can also have a series of conditions in the form of an `elif` block:<jupyter_code>x=5

if x==2:
    print('Yes! x is 2')
elif x==3:
    print('Yes! x is 3')
elif x==4:
    print('Yes! x is 4')
else:
    print('x is '+str(x))<jupyter_output><empty_output><jupyter_text>You can also have multiple conditions that are evaluated:<jupyter_code>x=5

# '&' below stands for AND: both conditions have to be met to execute the if block
if (x>2) & (x*2==10):
    print('x is 5')
    
    
# '|' below stands for OR: either condition can be met to execute the if block
if (x>7) | (x*2==10):
    print('x is 5')<jupyter_output><empty_output><jupyter_text>FunctionsFunctions allow you to make a bundle of python statements that are executed whenever the function is called. Each function has arguments you pass in and value(s) that are returned. For example, you've been using the `print` function.  There are also some functions above that you have been using. Now we will make our own.<jupyter_code># the function 'myfunc' takes two numbers, x and y, adds them together and returns the results
def myfunc(x,y):
    z=x+y
    return z

# to call the function, we simply invoke the name and feed it the requisite inputs:
g=myfunc(2,3.)
print(g)
# you can set input parameters to have a default values
def myfunc2(x,y=5.):
    z=x+y
    return z

g=myfunc2(2.)
print(g)

g=myfunc2(2.,4.)
print(g)<jupyter_output><empty_output><jupyter_text>Let's put the `if`, `for` and `def` together into one example. Here we will make a Fibonacci list generator. It takes a single argument: the length of the list to make.<jupyter_code>def Fibonacci(N):
    # check the input value
    if N < 2:
        print("Sorry, N must be at least 2")
        return None
    elif N > 1000:
        print ("Are you nuts?! That will take too long!")
        return None
    fib = [1,1]
    for i in range(N-2):
        fib.append(fib[-1]+fib[-2])
    return fib<jupyter_output><empty_output><jupyter_text>Note that in cases where the function fails, we return a special python object called `None`. It's a good thing to use when you want to indicate that somethign didn't go right. Now go ahead and try running the Fibonacci function with different input values.<jupyter_code>Fibonacci(1001)<jupyter_output><empty_output><jupyter_text>Importing Python packages: examplesOne of the major advantages of `python` is a wealth of specialized packages for doing common scientific tasks. Sure, you could write your own least-squares fitter using what we've shown you so far, but before you attempt anything like that, take a little time to "google that" and see if a solution exists already.You will have to `import` Python modules/packages to carry out many of the tasks you will need for your research.  As already discussed, `numpy` is probably the most useful.  `scipy` and `astropy` are other popular packages.  Lets play around with a few of these to give you an idea of how useful they can be. Random Numbers<jupyter_code># I like to declare all of my imported packages at the top of my script so that I know what is available.
# Also note that there are many ways to import packages.

import numpy.random as npr
from scipy import stats
import scipy.interpolate as si
from astropy.cosmology import FlatLambdaCDM
# Random numbers are useful for many tasks.

# draw random numbers from a uniform distribution:
x1=npr.rand(5)
print(x1)

# draw random numbers from a normal distribution:
x2=npr.randn(5)
print(x2)

# draw 10 random integers between 0 and 5(exclusive)
x3=npr.randint(0,5,10)
print(x3)<jupyter_output><empty_output><jupyter_text>Here is a practical example of using random numbers. Often in statistics, we have to compute the mean of some population based on a limited sample. For example, a survey may ask car drivers their age and make/model of car. A marketing team may want to know the average age of drivers of <NAME> so they can target their audience. Calculating a mean is easy, but what about the uncertainty in that mean? You could compute the population standard deviation, but that pre-supposes the underlying distribution is Gaussian. Another method, that does not make any assumptions about the distribution is *bootstrapping*. Randomly remove values from the data and replace them with copies of other values. Compute a new mean. Do that N times and compute the standard deviation of these *bootstrapped* mean values.Below is an example of bootstrapping a sample to determine the uncertainty on a measurement.  In this case, we will compute the mean of a sample of ages, and the uncertainty on the mean.<jupyter_code># x below represents a measurement of the ages of N people, where N=x.size
x=np.array([19.,20.,22.,19.,21.,24.,35.,22.,21.])
# This is the mean age:
print(np.mean(x))

# Now we "bootstrap" to determine the error on this measurement:
ntrials=10000 # number of times we will draw a random sample of N ages
x_arr=np.zeros(ntrials) # store the mean of each random sample in this array

for ii in range(ntrials):
    # draw N random integers, where N equals the number of samples in x
    ix=npr.randint(0,x.size,x.size)
    # subscript the original array with these random indices to get a new sample and compute the mean
    x_arr[ii]=np.mean(x[ix])

# Finally, compute the standard deviation of the array of mean values to get the uncertainty on the *mean* age
print(np.std(x_arr))<jupyter_output><empty_output><jupyter_text>Binning DataAnother common statistical task is *binning* data. This is often done on noisy spectra to make "real" features more visible. In the following example, we are going to bin galaxies based on their redshift and compute the mean stellar mass for each bin.<jupyter_code># This is an example of binning data and computing a particular value for the data in each bin.
# The scipy package is used to carry out this task.

# Lets make some fake data of galaxies spanning random redshifts between 0<z<3:
z=npr.rand(10000)*3.
# And these galaxies have random stellar masses between 9<log(M/Msun)<12:
m=npr.rand(10000)*3.+9.

# Now we want to compute the median stellar mass for galaxies at 0<z<1, 1<z<2, and 2<z<3:
# So lets declalre the bin edges
bins=[0.,1.,2.,3.]

m_med,xbins,btemp = stats.binned_statistic(z,m,statistic='median',bins=bins)
print(bins)
print(m_med)<jupyter_output><empty_output><jupyter_text>InterpolationIn science, we measure *discrete* values of data. Sometimes you need to interpolate between two (or more) points. A common example is drawing a smooth line through the data when making a graph. You could do this by hand using `numpy`, but the module `scipy` has an entire interpolation package that offers an easy solution.<jupyter_code># Interpolating between data points is another common task.  We'll again use scipy to do some interpolating:

x=np.arange(5.)
y=x**2

print(x)
print(y)

# Linear interpolation
f=si.interp1d(x,y,kind='linear')
# si.interp1d returns a function, f, which can be used to feed values to.
# For example, lets evaluate f(x)
print(f(x))
# And now a different value
print(f(0.5))

# We can employ a higher order interpolation scheme to get more precise results (assuming a smoothly varying function)
f=si.interp1d(x,y,kind='quadratic')
print(f(x))
print(f(0.5))<jupyter_output><empty_output><jupyter_text>Astronomy-specific ExampleNow we get into a really specific case. `Astropy` is a collection of several packages that are very useful for astronomers. It is actively developed and has new stuff all the time. Here, we show how you can use the cosmology calculator to compute the age of the Universe given a redshift.<jupyter_code># The astropy package has all kinds of astronomy related routines.
# Here, we define a cosmology that allows to compute things like 
# the age of the universe or Hubble constant at different redshifts

cosmo = FlatLambdaCDM(H0=70., Om0=0.3)
redshift=0.
print(cosmo.age(redshift))

redshift=[0,1,2,3]
print(cosmo.age(redshift))<jupyter_output><empty_output><jupyter_text>Running codeThere are many environments in which one can run Python code:- iPython notebooks like this one are good for running quick snippets of code.- Spyder (provided with Anaconda) provides a space for writing scripts, executing them, and also for easily looking up definitions of different functions.- One can also write code in a text editor, like Emacs/Aquamacs.  Then execute the code in a terminal running Python or iPython. Running code from the command-lineThis is the most common and *agnostic* way to run your code. If you send your code to someone else, assume they will run it from the command line. If you are running your code on an HPC cluster, it needs to be run from the command-line. Lastly, writing code that runs with minimal user-interaction makes it more repeatable.There are two aspects of writing command-line code that you should be familiar with: 1) getting arguments from the command line; and 2) working with files. The first is done through the `sys.argv` variable, the second is done with `os.path` package. Here we look at each briefly. `sys.argv`Quite simply, this is a list of the command-line arguments. You've seen several unix commands. Suppose we wanted to write the equivalent of the `cp` command, but using a python script. Usually, you run the command like this from the command-line:    cp file1 file2That would copy file1 to file2. So our python script will need to get both the source and destination file name. Here is how I would write a simple script to do the same thing as `cp`:    import sys    f1 = sys.argv[1]    f2 = sys.argv[2]    print ("copying %s to %s" % (f1,f2))         Here would be the code to actually copy one file to the other    Note: you can try to print out sys.argv on the next command-block. It will show you how this ipython notebook was actually run. But it is not very useful for doing anything practical.With more complicated code, your command-line arguments may also get rather complicated (you may have optional arguments, switches, etc). There is a really good module in `python` for dealing with such complicated arguments so that your script isn't filled with code just to deal with parsing `sys.argv`. Have a look at the [argparse](https://docs.python.org/2/library/argparse.html) module.<jupyter_code>print sys.argv<jupyter_output><empty_output><jupyter_text>`os.path`Up until now, we have been printing output to the screen. You can still do that with command-line  scripts, but once you close down the terminal, that output is lost. Your code will need to write to output files, but likely will also have to read from files. `os.path` gives you some functions that are useful when dealing with files.You may have to check to see if a file exists, if a folder exists, etc.<jupyter_code># Check to see if a file exists
if os.path.isfile('/bin/ls'):
    print("Oh good, you can list files")

if not os.path.isfile('test_output.dat'):
    print ("It is safe to use this")
    
# Check if a folder exists
if os.path.isdir('/tmp'):
    print ("you have a tmp folder")
    
# construct the path to a file using the correct separator
of = os.path.join('tmp','some','file')
print (of)<jupyter_output><empty_output><jupyter_text>File access with `open`In order to access the contents of an existing file, or write contents into a new file, you use the `open` function. This will return a file object that you can use to read or write data. Here are some common tasks. First, let's write data to a file:<jupyter_code># Open a file for writing (note the 'w')
f = open('a_test_file', 'w')
# write a header, always a good idea!
f.write("# This is a test data file. It contains 3 columns\n")
for i in range(10):
    f.write("%d %d %d\n" % (i, 2*i, 3*i))
# We need to "close" the file to make sure it is written to disk
f.close()<jupyter_output><empty_output><jupyter_text>Now, depending what your current working directory is (see beginning of tutorial), there will be a file called `a_test_file` in that folder. If you like, have a look at it using an editor or file viewer. It will have 10 rows and 3 columns. Note that we needed to have a "newline" (`\n`) at the end of the string we used in the `write()` function, otherwise the output would have been one long line.Now, let's read the file back into python. You can either read the whole thing in at once as a single string, read it in line-by-line or read all lines in at once as a list.<jupyter_code># This time we use 'r' to indicate we only want to read the file
f = open('a_test_file', 'r')
everything = f.read()

# This brings us back to the beginning of the file
f.seek(0)
one_row = f.readline()

f.seek(0)
list_of_rows = f.readlines()
f.close()
print(list_of_rows)<jupyter_output><empty_output><jupyter_text>Use `print()` to have a look at the data we read in. Note that the rows will include the newline character (`\n`). You can use the `string.strip()` function to get rid of it.In the next tutorial (visualization), we'll show you a better way to read in standard format data files like this. But sometimes you'll be faced with data that these more automatic functions can't handle, so it's good to know how to read it in by hand.  DebuggingYou almost never write a script without introducing bugs (the term comes from when computers were mechanical machines and insects literallly interfered with the running of the code). Luckily, python gives a very nice "traceback" report when it encounters a problem with what you've written. let's just generate a mistake on purpose and see what happens.<jupyter_code>x1=np.arange(5)
x2=np.arange(3)
print(x1)
print(x2)
print(x1+x2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08586732860828951, 'fraction_numerical': 0.011006956919358527, 'mean_word_length': 4.8892094633583385, 'pattern_counts': {'":': 0, '<': 223, '<?xml version=': 0, '>': 215, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '52', 'dir': 'jupyter-structured-clean-dedup', 'id': '529314', 'license': 'MIT', 'n_tokens_mistral': 9258, 'n_tokens_neox': 8907, 'n_words': 4307, 'path': 'PythonBootcamp.ipynb', 'repo_name': 'aliafzal8899/Python-Tutorials', 'size': '30624'}