<jupyter_start><jupyter_text>BLU04 - Learning Notebook - Part 1 of 3 - Notebook magics 1. Introduction Welcome to data wrangling!Up to this point of the academy, you already handled several datasets. Some were big, some were small. Some were very clean, some were a little messier (maybe with some missing values).But in all the cases, they were pretty easy to handle (thanks pd.read_csv!), and conveniently accessible.Well, in real life, that never happens. ![title](../media/all_the_data.png) When dealing with a data science problem, you'll find that the data you need is probably scattered around multiple sources, stored in severeal different formats, and in need of deep cleaning!But worry not, that's what we're here for :)In this specialization you'll learn many tools that will turn you into a professional data wrangler.Let's start with some handy Jupyter notebook's tools.  2. ! (system shell access)In a Jupyter notebook, any statement that you start with an exclamation mark (!), will be sent to the underlying operating system.In practice, this means that you can run shell commands in the notebooks, in the same way as you do in your computer terminal.Let's see some examples. The first is to list the files in the current directory.<jupyter_code># list the current directory
# in Windows: ! dir
! ls<jupyter_output>BLU04 - Learning Notebook - Part 1 of 3 - Notebook magics.ipynb
BLU04 - Learning Notebook - Part 2 of 3 - Reading files.ipynb
BLU04 - Learning Notebook - Part 3 of 3 - Data cleaning.ipynb<jupyter_text>In order to see the contents of a file, we just have to use command **cat** (Unix) or **type** (Windows), followed by the file path.<jupyter_code># print the contents of a file
# in Windows: ! type ../data/lorem/lorem_ipsum_short.txt
! cat ../data/lorem/lorem_ipsum_short.txt<jupyter_output>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis et tristique enim. Sed eget venenatis eros, quis suscipit lorem. Ut malesuada, erat a scelerisque cursus, odio mi sodales ligula, at elementum ipsum dui condimentum ipsum. Nam imperdiet viverra dictum. Aenean commodo accumsan iaculis. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Maecenas sed suscipit odio, ornare ullamcorper felis. Donec sit amet egestas sem, ac semper sem. Integer mattis purus sed orci volutpat porta. Donec cursus dapibus interdum. Donec ornare mattis dolor. Sed quis scelerisque nisl, in efficitur odio. Phasellus posuere libero eget orci feugiat scelerisque. Phasellus ullamcorper tortor nec facilisis blandit.
Vestibulum tempus, purus id ultrices eleifend, lorem nunc malesuada odio, vitae aliquam elit ipsum a nulla. Vivamus sed neque arcu. Vivamus commodo nunc a est hendrerit tempor. Nullam viverra sit amet augue in sagittis. Pellentesque molestie porttitor volutpa[...]<jupyter_text>Sometimes files are very big. In that case, printing all the content of a file can be too expensive and not very usefull.Thus, it's important to know how big is a file before opening it.A good way to do it is by counting the number of lines that the file has.<jupyter_code># counting the number of lines in a file
# in Windows: ! type ../data/lorem/lorem_ipsum_long.txt | find /c /v ""

! wc -l ../data/lorem/lorem_ipsum_long.txt<jupyter_output>150 ../data/lorem/lorem_ipsum_long.txt<jupyter_text>When a file is in fact very big, we can still preview it. But instead of printing all its content, w'll just want to print its first lines.For this we can use the command **head** (Unix) or **more** (Windows).<jupyter_code># print the content of the first two lines of the file
# in Windows: ! more /e ../data/lorem/lorem_ipsum_long.txt P 2
! head -2 ../data/lorem/lorem_ipsum_long.txt<jupyter_output>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed pharetra pulvinar nulla, nec ultricies velit posuere nec. Nulla rhoncus convallis lectus in dignissim. Nulla aliquet justo risus, ac dapibus urna faucibus aliquet. Cras in pretium leo. Etiam id neque a erat feugiat vehicula. Curabitur accumsan volutpat ante, vitae vestibulum lorem congue et. Vivamus fringilla massa id dictum bibendum. Maecenas iaculis arcu ut tellus varius, at imperdiet metus lacinia. Maecenas eget turpis metus. Maecenas est mauris, venenatis cursus nulla at, gravida posuere sapien. Fusce a ex purus. Pellentesque eleifend, lorem sed pulvinar scelerisque, orci eros maximus purus, id sagittis dolor est non nulla. Vestibulum ex metus, porttitor a leo ornare, pharetra pharetra libero. Suspendisse cursus ligula in ante tincidunt rhoncus malesuada eu justo.
Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Nam nec sem nisl. Aliquam at velit neque. Cras nec sem fringilla, h[...]<jupyter_text>Similarly, we can also preview the last lines of the file, with the **tail** command on Unix.Unfortunately, Windows doesn't have a built-in command equivalent to tail. But there are some packages that can be installed to fill this purpose (check this [stackoverflow post](https://stackoverflow.com/questions/187587/a-windows-equivalent-of-the-unix-tail-command)). Or you can always **type** the whole file (if not too big), and read the last lines...<jupyter_code># print the content of the last three lines of the file
# in Windows :(
! tail -3 ../data/lorem/lorem_ipsum_long.txt<jupyter_output>Morbi sodales et felis in bibendum. Nullam sollicitudin dapibus tellus, at molestie dui sagittis sed. Proin a tellus ac mi pharetra ullamcorper et in diam. Donec at posuere massa. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Quisque faucibus justo sollicitudin tincidunt tincidunt. Mauris magna massa, convallis quis erat sit amet, varius consequat nibh. In ac elit eu purus tempor lobortis quis at ante. Aliquam erat volutpat. Etiam sed arcu ut ex venenatis suscipit. Quisque ac porta diam. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce tristique malesuada diam, at hendrerit neque euismod sit amet.
Aenean commodo posuere elit, et ornare mi bibendum eget. Mauris scelerisque elit neque, vel eleifend leo pellentesque et. Aliquam erat volutpat. Vestibulum porttitor, neque in eleifend tincidunt, dolor tortor sagittis velit, a vehicula odio lectus et neque. Etiam odio diam, congue nec neque quis, hendrerit convallis nisi. Nullam leo lig[...]<jupyter_text>3. % (magic commands)In IPython, there are some commands which are called magic commands. Quoting IPython's [documentation](https://ipython.org/ipython-doc/3/interactive/reference.htmlmagic-command-system): "These [commands] allow you to control the behavior of IPython itself, plus a lot of system-type features. They are all prefixed with a % character, but parameters are given without parentheses or quotes."There are two types of magic commands: line magics and cell magics.Line magics are invoked like:```% command_name the_rest_of_the_line_is_interpreted_as_command_arguments```While cell magics are invoked like:```%% command_name the_rest_of_the_line and_all_the_other_linesin_the_cell are_interpreted_as_command_arguments```[Here](http://ipython.readthedocs.io/en/stable/interactive/magics.html) you have a list with all the existing IPython's magic commands, but here we'll just see a couple of examples.The command **magic** is a magic itself. By invoking it, without any additional arguments, the notebook will show you the magic commands' documentation<jupyter_code># print info about magic system
% magic<jupyter_output><empty_output><jupyter_text>A very usefull magic, that you've probably already seen, is the **matplotlib inline** magic.It enables the matplotlib inline backend for usage with the IPython Notebook. This means that in each cell that you write matplotlib's plotting commands, you will see the plots as output of the cell without the need to explicitly call the show method.You can invoke it as follows, and it will be applied to all the cells in the notebook.<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Another super usefull magic is the **timeit**, which allows you to measure the execution time of a line or a cell.Let's see an example of timeit as a line magic.<jupyter_code># the -n argument specifies the number of times the statement runs each time, which is called a loop
# the -r argument specifies the number of times the loop runs
%timeit -n3 -r5 [i for i in range(1000)]<jupyter_output>37.6 µs ± 2.69 µs per loop (mean ± std. dev. of 5 runs, 3 loops each)<jupyter_text>We can also use timeit as a cell magic to inspect the execution time of a whole cell.<jupyter_code>%%timeit x = 3
result = x**100
[i for i in range(1000)]<jupyter_output>34.9 µs ± 1.16 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)<jupyter_text>Another important magic is the **load_ext**, which allows us to load IPython extensions.In this example, we're loading the **line_profiler** extension, which allows us to profile functions by measuring how much time is being spent in each line of the function.<jupyter_code>%load_ext line_profiler<jupyter_output><empty_output><jupyter_text>Let's see an example on how to use the line profiler. First, we define a (very dummy) function.<jupyter_code>def very_dummy_function(r=1000):
    slow_execution_statement = [i for i in range(r)]
    fast_execution_statement = 2
    
    return 'hello!'<jupyter_output><empty_output><jupyter_text>Then, we invoke the line profiler using the **lprun** magic like this.<jupyter_code>%lprun -f very_dummy_function very_dummy_function(r=2000)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06478197371166285, 'fraction_numerical': 0.007823909868558314, 'mean_word_length': 5.003130870381966, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '420833', 'license': 'MIT', 'n_tokens_mistral': 3119, 'n_tokens_neox': 2967, 'n_words': 1303, 'path': 'Learning Notebooks/BLU04 - Learning Notebook - Part 1 of 3 - Notebook magics.ipynb', 'repo_name': 'LDSSA/batch2-BLU04', 'size': '9586'}