<jupyter_start><jupyter_text>Unix Command for Data Scientists  Declare Filename<jupyter_code>!ls ./unix
filename = './unix/shakespeare.txt'
!echo $filename
print(filename)<jupyter_output>./unix/shakespeare.txt
./unix/shakespeare.txt<jupyter_text>head<jupyter_code>!head -n 3 $filename<jupyter_output>This is the 100th Etext file presented by Project Gutenberg, and

is presented in cooperation with World Library, Inc., from their

Library of the Future and Shakespeare CDROMS.  Project Gutenberg<jupyter_text>tail<jupyter_code>!tail -n 10 $filename<jupyter_output>PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED

COMMERCIALLY.  PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY

SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>>







End of this Etext of The Complete Works of <NAME><jupyter_text>wc<jupyter_code>!wc $filename
!wc -l $filename<jupyter_output>124505 ./unix/shakespeare.txt<jupyter_text>cat<jupyter_code>!cat $filename | wc -l<jupyter_output>124505<jupyter_text>grep<jupyter_code>!grep -i 'parchment' $filename
## matching pattern one per line and counting the number of lines
!cat $filename | grep -o 'liberty' | wc -l<jupyter_output>71<jupyter_text>sed<jupyter_code>!sed -e 's/parchment/manuscript/g' $filename > temp.txt
!grep -i 'manuscript' temp.txt<jupyter_output>If the skin were manuscript, and the blows you gave were ink,

  Ham. Is not manuscript made of sheepskins?

    of the skin of an innocent lamb should be made manuscript? That

    manuscript, being scribbl'd o'er, should undo a man? Some say the

    Upon a manuscript, and against this fire

    But here's a manuscript with the seal of Caesar;  

    With inky blots and rotten manuscript bonds;

    Nor brass, nor stone, nor manuscript, bears not one,<jupyter_text>sort<jupyter_code>!head -n 5 $filename
!head -n 5 $filename | sort
# column seprated by ' ', sort on column 2 (-k2), Case insenstiive (-f)
!head -n 5 $filename | sort -f -t' ' -k2<jupyter_output>This is the 100th Etext file presented by Project Gutenberg, and

Library of the Future and Shakespeare CDROMS.  Project Gutenberg

is presented in cooperation with World Library, Inc., from their

often releases Etexts that are NOT placed in the Public Domain!!<jupyter_text>uniq<jupyter_code>!sort $filename | wc -l
!uniq $filename | wc -l<jupyter_output>121532<jupyter_text>Count most frequent word in the file using Unix<jupyter_code>!sed -e 's/ /\n/g' -e 's/\r//g' < $filename | sed '/^$/d' | sort | uniq -c | sort -nr | head -13<jupyter_output>23244 the
  19542 I
  18302 and
  15623 to
  15551 of
  12532 a
  10824 my
   9576 in
   9081 you
   7851 is
   7531 that
   7068 And
   6948 not
sort: write failed: 'standard output': Broken pipe
sort: write error<jupyter_text>Writing output to the file<jupyter_code>!sed -e 's/ /\'$'\n/g' < $filename | sort | uniq -c | sort -nr | head -13 > count_words.txt
!cat count_words.txt<jupyter_output>502289 $
  22678 the$
  19163 I$
  17868 and$
  15324 to$
  15216 of$
  14779 

  12152 a$
  10614 my$
   9347 in$
   8709 you$
   7662 is$
   7332 that$<jupyter_text>Plot by import word_counts into python<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import csv

xTicks = []
y = []

with open('count_words.txt') as csvfile:
    plots = csv.reader(csvfile, delimiter=' ')
    
    for row in plots:
        y.append(int(row[-2]))
        xTicks.append(str(row[-1]))
    
# Remove the spaces from the  first elemnt
xTicks = xTicks[1:]
y = y[1:]
# print(y)
# print(xTicks)
x = range(len(y))
plt.figure(figsize=(10, 10))

plt.xticks(x, xTicks, rotation=90) # rotating the label 90 degrees

plt.plot(x,y,'*')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10123855681206247, 'fraction_numerical': 0.04658050619278406, 'mean_word_length': 4.013495276653171, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '278877', 'license': 'MIT', 'n_tokens_mistral': 1519, 'n_tokens_neox': 1354, 'n_words': 420, 'path': 'python_data_science/Week_3/Unix Command.ipynb', 'repo_name': 'bunnydev26/micromasters-edx', 'size': '3727'}