<jupyter_start><jupyter_text>01. BASH Unix Shell  [Mathematical Statistical and Computational Foundations for Data Scientists](https://lamastex.github.io/scalable-data-science/360-in-525/2018/04/)&copy;2018 <NAME>. [Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)1. Dropping into BASH (Unix Shell) and using basic Shell commands    * `pwd` --- print working directory    * `ls` --- list files in current working directory    * `mkdir` --- make directory    * `cd` --- change directory    * `man ls` --- manual pages for any command2. Grabbing files from the internet using `curl`<jupyter_code>def showURL(url, ht=500):
    """Return an IFrame of the url to show in notebook with height ht"""
    from IPython.display import IFrame
    return IFrame(url, width='95%', height=ht) 
showURL('https://en.wikipedia.org/wiki/Bash_(Unix_shell)',400)<jupyter_output><empty_output><jupyter_text>1. Dropping into BASH (Unix Shell)Using `%%sh` in a code cell we can access the BASH (Unix Shell) command prompt.Let us `pwd` or print working directory.<jupyter_code>%%sh
pwd
%%sh
# this is a comment in BASH shell as it is preceeded by '#'
ls # list the contents of this working directory
%%sh
mkdir mydir
%%sh
cd mydir
pwd
ls -al
%%sh
pwd
%%sh
man ls<jupyter_output>LS(1)                            User Commands                           LS(1)

NAME
       ls - list directory contents

SYNOPSIS
       ls [OPTION]... [FILE]...

DESCRIPTION
       List  information  about  the FILEs (the current directory by default).
       Sort entries alphabetically if none of -cftuvSUX nor --sort  is  speci‐
       fied.

       Mandatory  arguments  to  long  options are mandatory for short options
       too.

       -a, --all
              do not ignore entries starting with .

       -A, --almost-all
              do not list implied . and ..

       --author
              with -l, print the author of each file

       -b, --escape
              print C-style escapes for nongraphic characters

       --block-size=SIZE
              scale sizes by SIZE before printing them; e.g., '--block-size=M'
              prints sizes in units of 1,048,576 bytes; see SIZE format below

       -B, --ignore-backups
              do not list implied entries ending with ~

 [...]<jupyter_text>2. Grabbing files from internet using curl<jupyter_code>%%sh
cd mydir
curl -O http://lamastex.org/datasets/public/SOU/sou/20170228.txt
%%sh
ls mydir/
%%sh
cd mydir/
head 20170228.txt<jupyter_output><NAME> 

February 28, 2017 
Thank you very much. Mr. Speaker, Mr. Vice President, members of Congress, the first lady of the United States ... 
... and citizens of America, tonight, as we mark the conclusion of our celebration of Black History Month, we are reminded of our nation's path toward civil rights and the work that still remains to be done. 
Recent threats ... 
Recent threats targeting Jewish community centers and vandalism of Jewish cemeteries, as well as last week's shooting in Kansas City, remind us that while we may be a nation divided on policies, we are a country that stands united in condemning hate and evil in all of its very ugly forms. 
Each American generation passes the torch of truth, liberty and justice, in an unbroken chain all the way down to the present. That torch is now in our hands. And we will use it to light up the world. 
I am here tonight to deliver a message of unity and strength, and it is a message deeply delivered from my heart. A new chapt[...]
{'fraction_non_alphanumeric': 0.0855431993156544, 'fraction_numerical': 0.019104647847162818, 'mean_word_length': 2.884828349944629, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '416156', 'license': 'Unlicense', 'n_tokens_mistral': 1075, 'n_tokens_neox': 992, 'n_words': 448, 'path': '_360-in-525/2018/04/jp/360-in-525-04_01.ipynb', 'repo_name': 'r-e-x-a-g-o-n/scalable-data-science', 'size': '3527'}