<jupyter_start><jupyter_text>UNIX Bash for Data Sciencesome sources on frequent UNIX commands:* http://faculty.tru.ca/nmora/Frequently%20used%20UNIX%20commands.pdf* https://www.tjhsst.edu/~dhyatt/superap/unixcmd.html  UNIX Manual<jupyter_code>command = input('Which UNIX command manual do you want?: ')
!echo ' ' && echo Manual for: $command && echo '' && man $command | head -20<jupyter_output><empty_output><jupyter_text>Basic file and directory commands mkdir: make directory  rmdir: remove directory  cd: change directory  cp: copy file  mv: mofe file  rm: remove file  cmp: compare 2 filesmore: output per windowchmod: change file permissions    Symbols **.**  - working directory   **..** - parent directory to working directory  **~**  - home directory  **/**  - root directory  *****  - string of characters wildcard  **?**  - one character wildcard    Directing and piping commands command > file         - redirects the output of 'command' to 'file' instead of to standard output (screen)  command >> file        - appends the output of 'command' to 'file' instead of to standard output (screen)  command < file         - takes input for 'command' from file  command1 | command2    - pipe standard output of command1 to standard input of command2    Getting around the filesystem File permissions in numeric format and their meaning :0 – no permissions  1 – execute only  2 – write only  3 – write and execute  4 – read only  5 – read and execute  6 – read and write  7 – read, write and execute  e.g.: chmod 400 filename ~ read-only<jupyter_code>!pwd
!ls
!ls -ltr # sorted by modification date
!ls ..
!ls -FGlAhp | grep ''
!file utils_.ipynb
!find .<jupyter_output><empty_output><jupyter_text>Print to terminal<jupyter_code>filename = '../_data/shakespeare.txt'
!echo $filename
!head -n 5 $filename && tail $filename
!cat $filename | head
!cat $filename | more # output per window
!less $filename  # output per window
# CTRL+F – forward one window
# CTRL+B – backward one window<jupyter_output><empty_output><jupyter_text>Create new file<jupyter_code>!echo this is a file made in terminal > ../_data/new_file2.txt 
!echo appended text on next line >> ../_data/new_file2.txt 
!cat ../_data/new_file2.txt
!touch ../_data/new_file.txt 
date  = !echo date
!echo this is a file made by touch at && echo $(date) > ../_data/new_file.txt 
!cat ../_data/new_file.txt
%%writefile new_file.txt
%%writefile -a append_file.txt<jupyter_output><empty_output><jupyter_text>Sort<jupyter_code>!head $filename | sort | head
# !head -n 5 $filename | sort  # by the first letter in each line
!head $filename | sort -t' ' -k2  # by -k: 2nd column, delimitter -t: ' '<jupyter_output><empty_output><jupyter_text>Wordcount (wc)-l: lines  -c: characters<jupyter_code>!wc $filename && echo chars: && wc -c $filename && echo lines: && wc -l $filename
!sort $filename | uniq -u | wc -l<jupyter_output><empty_output><jupyter_text>Split to words  MacOS version, need to use: `\'$', $'`   - Convert DOS file to Unix (\r\n in the end of each line): $sed 's/.$//' filename - replace spaces by returns: 's/ /\'$'\n/g'   - replace carriage return by nothing: $'s/\r//g'<jupyter_code>!sed 's/.$//' $filename > ../_data/shakespeare_unix.txt
!head ../_data/shakespeare_unix.txt
!head ../_data/shakespeare.txt<jupyter_output><empty_output><jupyter_text>Reverse lines of content<jupyter_code>!sed -n '1!G;h;$p' ../_data/shakespeare_unix.txt | head
!sed -e 's/ /\'$'\n/g' -e $'s/\r//g' < $filename | head<jupyter_output><empty_output><jupyter_text>Most frequent words- remove empty lines: sed `'/^$/d'`, **`^: start of line, $: end of line, d: delete`**- sort words: sort- count consecutive words: uniq -c- sort numerically and reverse order: sort -nr<jupyter_code>!sed -e 's/ /\'$'\n/g' -e $'s/\r//g' $filename  | sed '/^$/d'| sort | uniq -c | sort -nr | head -15
!sed -e 's/ /\'$'\n/g' -e $'s/\r//g' $filename  | sed '/^$/d'| sort | uniq -c | sort -nr > ../_data/count_vs_words
!head ../_data/count_vs_words<jupyter_output><empty_output><jupyter_text>xargs<jupyter_code>!head ../_data/count_vs_words | xargs echo<jupyter_output><empty_output><jupyter_text>grep- **g**lobal **r**egular **e**xpression **p**rint<jupyter_code>!grep -i 'CDROMS' ../_data/shakespeare.txt
!grep -A 10 -i 'CDROMS' ../_data/shakespeare.txt
# Recursive search
!grep -r 'shakespeare' *
!grep 'Liberty' $filename             # add -i to make case insensitive
!grep -i 'liberty' $filename | wc -l  # location of first word<jupyter_output><empty_output><jupyter_text>sed - **s**tream **ed**itor   - like grep + replacement - **`s/from/to/g`**, **s**: substitute, **g**: general (all)<jupyter_code>!sed -e 's/parchment/manuscript/g' $filename > ../_data/temp_shakespeare.txt
!grep -i 'manuscript' ../_data/temp_shakespeare.txt
!grep -i 'parchment' ../_data/shakespeare.txt<jupyter_output><empty_output><jupyter_text>find<jupyter_code>!find .. | grep -i shakespeare<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15139845966761248, 'fraction_numerical': 0.007296311309282529, 'mean_word_length': 4.211193241816262, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '46599', 'license': 'MIT', 'n_tokens_mistral': 1799, 'n_tokens_neox': 1733, 'n_words': 515, 'path': '_productivity/bash_basics.ipynb', 'repo_name': 'aixpact/data-science', 'size': '4934'}