<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unix Basics\n",
    "\n",
    "The command line allows you to input commands, such as creating folders, deleting and copying files and extracting information from files.\n",
    "\n",
    "By the end of this notebook, you will...\n",
    "\n",
    "* Have (maybe installed) and used the command line on your personal laptop\n",
    "* Created files and folders, and looked around a directory, using just the keyboard \n",
    "* Used command line tools like `sed`, `awk`, and `grep` to find replace text, fetch columns, and find words in files\n",
    "\n",
    "* * *\n",
    "\n",
    "## Opening the command line.\n",
    "\n",
    "* **From Mac OS**\n",
    "    * Applications folder, open Utilities and launch Terminal  \n",
    "* **From Linux machine**\n",
    "    * Applications, Accessories and launch Terminal  \n",
    "* **From Windows**\n",
    "    * Determine if you have a 32 or 64 bit version of Windows. \n",
    "        * https://support.microsoft.com/en-us/kb/827218. \n",
    "        * http://cygwin.com/install.html.\n",
    "    * Run setup-x86.exe if you have 32 bit windows. If not Run setup-x86_64.exe\n",
    "    * Install Cygwin and double-click on Cygwin Terminal\n",
    "\n",
    "\n",
    "## Getting Started: Navigating your folders and files\n",
    "You start any terminal session in your \"home area\".  View your \"present working directory\"\n",
    "```\n",
    "$ pwd\n",
    "```\n",
    "\n",
    "Your default home folder (also called `$HOME`) is represented by the character alias `~` (tilde)\n",
    "```\n",
    "$ echo ~\n",
    "```\n",
    "\n",
    "Change directory  \n",
    "```\n",
    "$ cd ~/Desktop\n",
    "```\n",
    "\n",
    "List all the files in the present working directory using  \n",
    "```\n",
    "$ ls\n",
    "$ ls .\n",
    "```\n",
    "    \n",
    "Arguments for unix commands  \n",
    "```\n",
    "$ man ls\n",
    "```\n",
    "    \n",
    "Creating a folder  \n",
    "```\n",
    "$ mkdir data\n",
    "$ mkdir software\n",
    "```\n",
    "    \n",
    "Change directory into data or software (tab complete or use Up and Down). `[TAB]` means to press the tab key on your keyboard, not to write out the characters.\n",
    "\n",
    "\n",
    "```\n",
    "$ cd da[TAB]\n",
    "```\n",
    "\n",
    "Change back to the parent directory from any subdirectory:\n",
    "\n",
    "```\n",
    "$ cd ..\n",
    "```\n",
    "\n",
    "Create an empty file\n",
    "\n",
    "```\n",
    "$ touch emptyfile.txt\n",
    "```\n",
    "\n",
    "Write some text in it\n",
    "```\n",
    "$ echo \"hello world\" > emptyfile.txt\n",
    "```\n",
    "\n",
    "Look at the contents of the file with `cat`\n",
    "\n",
    "```\n",
    "$ cat emptyfile.txt\n",
    "```\n",
    "\n",
    "Append to your file with `>>`\n",
    "\n",
    "```\n",
    "$ echo \"I love bioinformatics\" >> emptyfile.txt\n",
    "```\n",
    "\n",
    "### Exercise: look at the file\n",
    "\n",
    "Count the number of lines with `wc -l`\n",
    "\n",
    "```\n",
    "$ wc -l emptyfile.txt\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "Move or rename a file\n",
    "```\n",
    "$ mv emptyfile.txt notempty.txt\n",
    "```\n",
    "\n",
    "Copy a file\n",
    "\n",
    "```\n",
    "$ cp notempty.txt deleteme.txt\n",
    "```\n",
    "\n",
    "Delete a file\n",
    "```\n",
    "$ rm deleteme.txt\n",
    "```\n",
    "\n",
    "Create a shortcut aka a pointer (\"symbolic link\" or \"symlink\") to a file\n",
    "```\n",
    "$ ln -s notempty.txt pointer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Manipulation: Getting some data from UCSC's Table Browser\n",
    "\n",
    "Go to the [UCSC Table browser](http://genome.ucsc.edu/cgi-bin/hgTables) and choose \"position\" to pick a single chromosome (chr10+) to save the knownGene table with \"all fields from selected table\" (should be the default) as `knowngene.txt`.\n",
    "\n",
    "### Exercise 1\n",
    "Move `knownGene.txt` to Desktop using the command line. What is the command?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "    \n",
    "### Exercise 2\n",
    "\n",
    "`less` and `more` are other commands (besides `cat`) you can use to look at the contents of files. How are they different?\n",
    "    \n",
    "See what's in the first n lines (in this case 10)\n",
    "\n",
    "```\n",
    "$ head -n 10 knownGene.txt\n",
    "```\n",
    "\n",
    "\n",
    "How many lines are in the file?\n",
    "```\n",
    "$ wc -l knownGene.txt\n",
    "```\n",
    "    \n",
    "Check if it's indeed n lines \n",
    "```\n",
    "$ wc -l knownGene.txt\n",
    "```\n",
    "    \n",
    "What's in the last n lines?\n",
    "```\n",
    "$ tail -n 10 knownGene.txt\n",
    "```\n",
    "    \n",
    "Extract specific columns\n",
    "```\n",
    "$ cut -f\n",
    "$ paste column1.txt column2.txt > 2columns.txt\n",
    "```\n",
    "\n",
    "How many genes have 3 exons?\n",
    "```\n",
    "$ grep -c 'REGEXSEARCHTERM' target.txt\n",
    "```\n",
    "    \n",
    "How many genes have 1...max # exons? We can use the \"pipe\" (`|`) to send the output of one command into the next\n",
    "```\n",
    "$ sort | uniq -c\n",
    "```\n",
    "    \n",
    "\n",
    "## File permissions\n",
    "Which user are you logged in as?\n",
    "```\n",
    "$ whoami\n",
    "```\n",
    "\n",
    "What groups is that user associated with?\n",
    "```\n",
    "$ groups\n",
    "```\n",
    "\n",
    "What is the ownership status of all files in my current directory? Here's some help for [interpreting output](http://linuxcommand.org/lts0070.php).\n",
    "```\n",
    "$ ls -lrt\n",
    "```\n",
    "\n",
    "Changing permissions\n",
    "```\n",
    "$ chmod 775\n",
    "```\n",
    "\n",
    "The three digits indicate the affected user subset:\n",
    "* Front = Owner  \n",
    "* Middle = Group  \n",
    "* Rear = All Users  \n",
    "\n",
    "The value indicates visibility encoded as a sum of [octal numbers](https://en.wikipedia.org/wiki/Chmod#Octal_modes). For example, read + execute = 2 + 3 = 5. 775 or 755 are the most common permissions setups because then you the owner can do everything to your files, and maybe the rest of the group can, but the \"all\" or \"world\" can only read and execute your programs, but not overwrite them.\n",
    "\n",
    "|#|\tPermission\t|`rwx`|\n",
    "|--|--|--|\n",
    "|7|\tread, write and execute|\t`rwx`\n",
    "|6|\tread and write|\t`rw-`\n",
    "|5|\tread and execute|\t`r-x`\n",
    "|4|\tread only|\t`r--`\n",
    "|3|\twrite and execute|\t`-wx`\n",
    "|2|\twrite only|\t`-w-`\n",
    "|1|\texecute only|\t`--x`\n",
    "|0|\tnone|\t`---`\n",
    "\n",
    "Changing Files Recursively\n",
    "```\n",
    "$ chmod -R 777 Directory/\n",
    "$ chmod -R o-rwx ~/\n",
    "```\n",
    "\n",
    "Changing executable nature of files\n",
    "```\n",
    "$ chmod +x\n",
    "```\n",
    "\n",
    "## Introduction to `awk`\n",
    "\n",
    "`awk` is a command-line tool to \n",
    "\n",
    "Another way to extract all lines\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{print;}' knownGene.txt\n",
    "```\n",
    " \n",
    "What if we only wanted one column\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{print $8;}' knownGene.txt  | head\n",
    "```\n",
    "\n",
    "What if we wanted the length of genes?\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{ len = $5-$4;} {print len;}' knownGene.txt | head\n",
    "```\n",
    "\n",
    "Length of all genes summed?\n",
    "```\n",
    "$ awk -F \"\\t\" '{ len = $5-$4;} {tot = tot + len;} END {print tot;}' knownGene.txt | head\n",
    "```\n",
    "\n",
    "Don't process the header line (introduction to conditionals)\n",
    "```\n",
    "$ awk -F \"\\t\" '{\n",
    "if (FNR==1){\n",
    "    next\n",
    "};\n",
    "tot = tot + $5-$4};\n",
    "END {print tot;}' knownGene.txt | head\n",
    "```\n",
    " \n",
    "What if you only want the total length of genes in chromosome 1?\n",
    "```\n",
    "$ awk -F \"\\t\" '{\n",
    "    if (FNR==1){\n",
    "        next;\n",
    "        };\n",
    "    chr =$2;\n",
    "    if (chr == \"chr1\") {\n",
    "        tot = tot + $5-$4;\n",
    "    }\n",
    "};\n",
    "END {print tot;}' knownGene.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Getting comfortable with unix commands\n",
    "\n",
    "The purpose of this homework is to familiarize yourself with the different commands available on all UNIX machines. This is not meant to teach you how to use each command at an expert level, but rather to show you that these commands exist.\n",
    "\n",
    "For each of the commands below, do the following:\n",
    "\n",
    "1. Figure out how to run the command by getting its usage help. The four ways you can get help are below:\n",
    "    1. `$ programname`\n",
    "        2. Without any arguments, this will call the program. Sometimes this will give you usage information, sometimes it will result in an error.\n",
    "    2. `$ programname -h`\n",
    "        3. The `-h` is (generally) a universal short version flag for \"help me!!! I don't know what's going on!!\" Though sometimes that doesn't work either ...\n",
    "    3. `$ programname --help`\n",
    "        4. The expanded version of `-h`\n",
    "    4. `$ man programname`\n",
    "        5. `man` is for \"manual\" and will bring up the manual pages for that command\n",
    "3. Get it to run with no errors. This may mean you need to give it a file name or a command name\n",
    "4. Run it again with no errors, with at least one option flag\n",
    "5. (Optional) Search google or [StackOverflow](http://stackoverflow.com/) (programmer \"Yahoo Answers\") for more information about the command, like \"when would you actually use programname?\" You may need to add \"unix\" to your search terms \n",
    "\n",
    "Tips:\n",
    "\n",
    "* If you're in a program and want to exit, try these steps:\n",
    "    1. Press `q` (for \"quit\")\n",
    "    2. Press Control-C or Control-Z or Control-X\n",
    "\n",
    "### List of commands\n",
    "\n",
    "2. `echo`\n",
    "17. `uptime`\n",
    "8. Help with commands and programs\n",
    "    12. `man`\n",
    "    12. `history`\n",
    "    18. `whatis`\n",
    "    20. `which`\n",
    "1. Working with files and directories\n",
    "    1. `cd`\n",
    "        1. How can you go UP a directory? Hint: \"`..`\"\n",
    "    4. `pwd`\n",
    "    2. `ls`\n",
    "    2. `mkdir`\n",
    "    3. `touch`\n",
    "        1. If you `touch` a file that's already there, what changes about it? Try using `ls -l`\n",
    "    2. `cp`\n",
    "    3. `mv`\n",
    "    18. `rm`\n",
    "        1. How can you remove a directory?\n",
    "        1. `rm -rf` will force remove directories recursively - use with caution!! There is no \"trash bin\" so once it's gone - it's gone.\n",
    "5. Looking at files\n",
    "    21. `cat`\n",
    "    22. `head`\n",
    "    23. `tail`\n",
    "    19. `less`\n",
    "    20. `more`\n",
    "        1. How are `less` and `more` different?\n",
    "3. Getting information about files\n",
    "    20. `wc`\n",
    "6. Looking at which programs are running and turning them off\n",
    "    9. `ps`\n",
    "    17. `top`\n",
    "    10. `kill`\n",
    "    11. `killall`\n",
    "        1. How are `kill` and `killall` different?\n",
    "2. Working with compressed files\n",
    "    11. `gzip`\n",
    "    12. `gunzip`\n",
    "    13. `zcat`\n",
    "7. Getting information about available or occupied disk space\n",
    "    14. `df`\n",
    "    15. `du`\n",
    "        1. How are `df` and `du` different? Try using `-h` (human readable) with them\n",
    "8. Making shortcuts for files and programs\n",
    "    16. `alias`\n",
    "    3. `ln`\n",
    "        1. How are `alias` and `ln` different?\n",
    "7. Manipulating and searching files\n",
    "    1. `sed`\n",
    "    5. `grep`\n",
    "    6. `awk`\n",
    "    24. `expand`\n",
    "    25. `cut`\n",
    "    26. `paste`\n",
    "    26. `sort`\n",
    "    26. `uniq`\n",
    "6. Dealing with users and permissions\n",
    "    8. `chmod`\n",
    "    13. `whoami`\n",
    "    13. `groups`\n",
    "\n",
    "Plus 10 commands of your choice. [Here](http://www.math.harvard.edu/computing/unix/unixcommands.html) is a nice list of all possible commands."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
<commit_msg>add more explicit explanations of cut/paste<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unix Basics\n",
    "\n",
    "The command line allows you to input commands, such as creating folders, deleting and copying files and extracting information from files.\n",
    "\n",
    "By the end of this notebook, you will...\n",
    "\n",
    "* Have (maybe installed) and used the command line on your personal laptop\n",
    "* Created files and folders, and looked around a directory, using just the keyboard \n",
    "* Used command line tools like `sed`, `awk`, and `grep` to find replace text, fetch columns, and find words in files\n",
    "\n",
    "* * *\n",
    "\n",
    "## Opening the command line.\n",
    "\n",
    "* **From Mac OS**\n",
    "    * Applications folder, open Utilities and launch Terminal  \n",
    "* **From Linux machine**\n",
    "    * Applications, Accessories and launch Terminal  \n",
    "* **From Windows**\n",
    "    * Determine if you have a 32 or 64 bit version of Windows. \n",
    "        * https://support.microsoft.com/en-us/kb/827218. \n",
    "        * http://cygwin.com/install.html.\n",
    "    * Run setup-x86.exe if you have 32 bit windows. If not Run setup-x86_64.exe\n",
    "    * Install Cygwin and double-click on Cygwin Terminal\n",
    "\n",
    "\n",
    "## Getting Started: Navigating your folders and files\n",
    "You start any terminal session in your \"home area\".  View your \"present working directory\"\n",
    "```\n",
    "$ pwd\n",
    "```\n",
    "\n",
    "Your default home folder (also called `$HOME`) is represented by the character alias `~` (tilde)\n",
    "```\n",
    "$ echo ~\n",
    "```\n",
    "\n",
    "Change directory  \n",
    "```\n",
    "$ cd ~/Desktop\n",
    "```\n",
    "\n",
    "List all the files in the present working directory using  \n",
    "```\n",
    "$ ls\n",
    "$ ls .\n",
    "```\n",
    "    \n",
    "Arguments for unix commands  \n",
    "```\n",
    "$ man ls\n",
    "```\n",
    "    \n",
    "Creating a folder  \n",
    "```\n",
    "$ mkdir data\n",
    "$ mkdir software\n",
    "```\n",
    "    \n",
    "Change directory into data or software (tab complete or use Up and Down). `[TAB]` means to press the tab key on your keyboard, not to write out the characters.\n",
    "\n",
    "\n",
    "```\n",
    "$ cd da[TAB]\n",
    "```\n",
    "\n",
    "Change back to the parent directory from any subdirectory:\n",
    "\n",
    "```\n",
    "$ cd ..\n",
    "```\n",
    "\n",
    "Create an empty file\n",
    "\n",
    "```\n",
    "$ touch emptyfile.txt\n",
    "```\n",
    "\n",
    "Write some text in it\n",
    "```\n",
    "$ echo \"hello world\" > emptyfile.txt\n",
    "```\n",
    "\n",
    "Look at the contents of the file with `cat`\n",
    "\n",
    "```\n",
    "$ cat emptyfile.txt\n",
    "```\n",
    "\n",
    "Append to your file with `>>`\n",
    "\n",
    "```\n",
    "$ echo \"I love bioinformatics\" >> emptyfile.txt\n",
    "```\n",
    "\n",
    "### Exercise: look at the file\n",
    "\n",
    "Count the number of lines with `wc -l`\n",
    "\n",
    "```\n",
    "$ wc -l emptyfile.txt\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "Move or rename a file\n",
    "```\n",
    "$ mv emptyfile.txt notempty.txt\n",
    "```\n",
    "\n",
    "Copy a file\n",
    "\n",
    "```\n",
    "$ cp notempty.txt deleteme.txt\n",
    "```\n",
    "\n",
    "Delete a file\n",
    "```\n",
    "$ rm deleteme.txt\n",
    "```\n",
    "\n",
    "Create a shortcut aka a pointer (\"symbolic link\" or \"symlink\") to a file\n",
    "```\n",
    "$ ln -s notempty.txt pointer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Manipulation: Getting some data from UCSC's Table Browser\n",
    "\n",
    "Go to the [UCSC Table browser](http://genome.ucsc.edu/cgi-bin/hgTables) and choose \"position\" to pick a single chromosome (chr10+) to save the knownGene table with \"all fields from selected table\" (should be the default) as `knownGene.txt`.\n",
    "\n",
    "### Exercise 1\n",
    "Move `knownGene.txt` to Desktop using the command line. What is the command?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "    \n",
    "### Exercise 2\n",
    "\n",
    "`less` and `more` are other commands (besides `cat`) you can use to look at the contents of files. How are they different?\n",
    "    \n",
    "See what's in the first n lines (in this case 10)\n",
    "\n",
    "```\n",
    "$ head -n 10 knownGene.txt\n",
    "```\n",
    "\n",
    "\n",
    "How many lines are in the file?\n",
    "```\n",
    "$ wc -l knownGene.txt\n",
    "```\n",
    "\n",
    "What if we tried the `wc` command with no arguments? What does that output? Check `man wc` to read about it.\n",
    "\n",
    "```\n",
    "$ wc knownGene.txt\n",
    "```\n",
    "    \n",
    "What's in the last n lines?\n",
    "```\n",
    "$ tail -n 10 knownGene.txt\n",
    "```\n",
    "\n",
    "Extract specific columns. In this first line, we're extracting column 1 \n",
    "```\n",
    "$ cut -f 1 knowngene.txt \n",
    "```\n",
    "Whoa that's a lot of output. Let's save it to a file\n",
    "\n",
    "```\n",
    "$ cut -f 1 knownGene.txt > column1.txt\n",
    "```\n",
    "\n",
    "### Exercise 3: `cut` *and* `paste`\n",
    "\n",
    "1. Cut the second column of `knownGene.txt`.\n",
    "2. Use the \"`paste`\" command below to glue the two columns together into a new file\n",
    "\n",
    "```\n",
    "$ paste column1.txt column2.txt > 2columns.txt\n",
    "```\n",
    "\n",
    "### Exercise 4: How many genes start on a nucleotide count that ends in 222?\n",
    "\n",
    "Let's build this command stepwise. Let's say the fourth column has the gene start nucleotide of `knownGene.txt`. \n",
    "```\n",
    "$ cut -f 4 knownGene.txt\n",
    "```\n",
    "Whoa that's a lot of output. Let's shorten it by using the \"`|`\" character (pronounced \"pipe\") to send the output to `head`\n",
    "\n",
    "```\n",
    "$ cut -f 4 knownGene.txt | head\n",
    "```\n",
    "\n",
    "We'll use Regular Expressions \"regex\" ([cheatsheet](https://www.cheatography.com/davechild/cheat-sheets/regular-expressions/)) to search for three 1's in a row in this cut up column.\n",
    "\n",
    "```\n",
    "$ cut -f 4 knownGene.txt | grep 111 | head\n",
    "```\n",
    "\n",
    "We can do the same thing using the `{}` notation.\n",
    "\n",
    "\n",
    "```\n",
    "$ cut -f 4 knownGene.txt | grep '1{3}' | head\n",
    "```\n",
    "If that didn't give any output, you may need to add the `-E` flag for \"extended regular expressions because `grep` is a really old program and it was pretty good when it first started, so vanilla `grep` is okay. But then Perl came along and made regular expressions really awesome, but, since all the programs had to be backwards compatible, they made it so you can access Perl regular expressions with `-E`.\n",
    "\n",
    "```\n",
    "$ cut -f 4 knownGene.txt | grep -E '1{3}' | head\n",
    "```\n",
    "\n",
    "We can then pipe this to `wc -l` to count line numbers!\n",
    "\n",
    "```\n",
    "$ cut -f 4 knownGene.txt | grep -E '1{3}' | wc -l\n",
    "```\n",
    "\n",
    "This shows us how many genes have a \"`111`\" in the transcript start.\n",
    "\n",
    "**How many genes have \"`222`\" in the transcript END?**\n",
    "\n",
    "### Exercise 5: How many genes have 3 exons? \n",
    "\n",
    "Let's build this command stepwise. Let's pretend the fourth column has the exon count of `knownGene.txt`.\n",
    "```\n",
    "$ grep -c 'REGEXSEARCHTERM' target.txt\n",
    "```\n",
    "    \n",
    "How many genes have 1...max # exons? We can use the \"pipe\" (`|`) to send the output of one command into the next. Let's build this command stepwise.\n",
    "```\n",
    "$ cut -f 1 filename.txt\n",
    "$ cut -f 1 filename.txt | sort -n\n",
    "$ cut -f 1 filename.txt | sort -n | uniq -c\n",
    "```\n",
    "    \n",
    "\n",
    "## File permissions\n",
    "Which user are you logged in as?\n",
    "```\n",
    "$ whoami\n",
    "```\n",
    "\n",
    "What groups is that user associated with?\n",
    "```\n",
    "$ groups\n",
    "```\n",
    "\n",
    "What is the ownership status of all files in my current directory? Here's some help for [interpreting output](http://linuxcommand.org/lts0070.php).\n",
    "```\n",
    "$ ls -lrt\n",
    "```\n",
    "\n",
    "Changing permissions\n",
    "```\n",
    "$ chmod 775\n",
    "```\n",
    "\n",
    "The three digits indicate the affected user subset:\n",
    "* Front = Owner  \n",
    "* Middle = Group  \n",
    "* Rear = All Users  \n",
    "\n",
    "The value indicates visibility encoded as a sum of [octal numbers](https://en.wikipedia.org/wiki/Chmod#Octal_modes). For example, read + execute = 2 + 3 = 5. 775 or 755 are the most common permissions setups because then you the owner can do everything to your files, and maybe the rest of the group can, but the \"all\" or \"world\" can only read and execute your programs, but not overwrite them.\n",
    "\n",
    "|#|\tPermission\t|`rwx`|\n",
    "|--|--|--|\n",
    "|7|\tread, write and execute|\t`rwx`\n",
    "|6|\tread and write|\t`rw-`\n",
    "|5|\tread and execute|\t`r-x`\n",
    "|4|\tread only|\t`r--`\n",
    "|3|\twrite and execute|\t`-wx`\n",
    "|2|\twrite only|\t`-w-`\n",
    "|1|\texecute only|\t`--x`\n",
    "|0|\tnone|\t`---`\n",
    "\n",
    "Changing Files Recursively\n",
    "```\n",
    "$ chmod -R 777 Directory/\n",
    "$ chmod -R o-rwx ~/\n",
    "```\n",
    "\n",
    "Changing executable nature of files\n",
    "```\n",
    "$ chmod +x\n",
    "```\n",
    "\n",
    "## Introduction to `awk`\n",
    "\n",
    "`awk` is a command-line tool to \n",
    "\n",
    "Another way to extract all lines\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{print;}' knownGene.txt\n",
    "```\n",
    " \n",
    "What if we only wanted one column\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{print $8;}' knownGene.txt  | head\n",
    "```\n",
    "\n",
    "What if we wanted the length of genes?\n",
    "\n",
    "```\n",
    "$ awk -F \"\\t\" '{ len = $5-$4;} {print len;}' knownGene.txt | head\n",
    "```\n",
    "\n",
    "Length of all genes summed?\n",
    "```\n",
    "$ awk -F \"\\t\" '{ len = $5-$4;} {tot = tot + len;} END {print tot;}' knownGene.txt | head\n",
    "```\n",
    "\n",
    "Don't process the header line (introduction to conditionals)\n",
    "```\n",
    "$ awk -F \"\\t\" '{\n",
    "if (FNR==1){\n",
    "    next\n",
    "};\n",
    "tot = tot + $5-$4};\n",
    "END {print tot;}' knownGene.txt | head\n",
    "```\n",
    " \n",
    "What if you only want the total length of genes in chromosome 1?\n",
    "```\n",
    "$ awk -F \"\\t\" '{\n",
    "    if (FNR==1){\n",
    "        next;\n",
    "        };\n",
    "    chr =$2;\n",
    "    if (chr == \"chr1\") {\n",
    "        tot = tot + $5-$4;\n",
    "    }\n",
    "};\n",
    "END {print tot;}' knownGene.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Getting comfortable with unix commands\n",
    "\n",
    "The purpose of this homework is to familiarize yourself with the different commands available on all UNIX machines. This is not meant to teach you how to use each command at an expert level, but rather to show you that these commands exist.\n",
    "\n",
    "For each of the commands below, do the following:\n",
    "\n",
    "1. Figure out how to run the command by getting its usage help. The four ways you can get help are below:\n",
    "    1. `$ programname`\n",
    "        2. Without any arguments, this will call the program. Sometimes this will give you usage information, sometimes it will result in an error.\n",
    "    2. `$ programname -h`\n",
    "        3. The `-h` is (generally) a universal short version flag for \"help me!!! I don't know what's going on!!\" Though sometimes that doesn't work either ...\n",
    "    3. `$ programname --help`\n",
    "        4. The expanded version of `-h`\n",
    "    4. `$ man programname`\n",
    "        5. `man` is for \"manual\" and will bring up the manual pages for that command\n",
    "3. Get it to run with no errors. This may mean you need to give it a file name or a command name\n",
    "4. Run it again with no errors, with at least one option flag\n",
    "5. (Optional) Search google or [StackOverflow](http://stackoverflow.com/) (programmer \"Yahoo Answers\") for more information about the command, like \"when would you actually use programname?\" You may need to add \"unix\" to your search terms \n",
    "\n",
    "Tips:\n",
    "\n",
    "* If you're in a program and want to exit, try these steps:\n",
    "    1. Press `q` (for \"quit\")\n",
    "    2. Press Control-C or Control-Z or Control-X\n",
    "\n",
    "### List of commands\n",
    "\n",
    "2. `echo`\n",
    "17. `uptime`\n",
    "8. Help with commands and programs\n",
    "    12. `man`\n",
    "    12. `history`\n",
    "    18. `whatis`\n",
    "    20. `which`\n",
    "1. Working with files and directories\n",
    "    1. `cd`\n",
    "        1. How can you go UP a directory? Hint: \"`..`\"\n",
    "    4. `pwd`\n",
    "    2. `ls`\n",
    "    2. `mkdir`\n",
    "    3. `touch`\n",
    "        1. If you `touch` a file that's already there, what changes about it? Try using `ls -l`\n",
    "    2. `cp`\n",
    "    3. `mv`\n",
    "    18. `rm`\n",
    "        1. How can you remove a directory?\n",
    "        1. `rm -rf` will force remove directories recursively - use with caution!! There is no \"trash bin\" so once it's gone - it's gone.\n",
    "5. Looking at files\n",
    "    21. `cat`\n",
    "    22. `head`\n",
    "    23. `tail`\n",
    "    19. `less`\n",
    "    20. `more`\n",
    "        1. How are `less` and `more` different?\n",
    "3. Getting information about files\n",
    "    20. `wc`\n",
    "6. Looking at which programs are running and turning them off\n",
    "    9. `ps`\n",
    "    17. `top`\n",
    "    10. `kill`\n",
    "    11. `killall`\n",
    "        1. How are `kill` and `killall` different?\n",
    "2. Working with compressed files\n",
    "    11. `gzip`\n",
    "    12. `gunzip`\n",
    "    13. `zcat`\n",
    "7. Getting information about available or occupied disk space\n",
    "    14. `df`\n",
    "    15. `du`\n",
    "        1. How are `df` and `du` different? Try using `-h` (human readable) with them\n",
    "8. Making shortcuts for files and programs\n",
    "    16. `alias`\n",
    "    3. `ln`\n",
    "        1. How are `alias` and `ln` different?\n",
    "7. Manipulating and searching files\n",
    "    1. `sed`\n",
    "    5. `grep`\n",
    "    6. `awk`\n",
    "    24. `expand`\n",
    "    25. `cut`\n",
    "    26. `paste`\n",
    "    26. `sort`\n",
    "    26. `uniq`\n",
    "6. Dealing with users and permissions\n",
    "    8. `chmod`\n",
    "    13. `whoami`\n",
    "    13. `groups`\n",
    "\n",
    "Plus 10 commands of your choice. [Here](http://www.math.harvard.edu/computing/unix/unixcommands.html) is a nice list of all possible commands."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.2112973609475653, 'fraction_numerical': 0.014303525663226431, 'mean_word_length': 2.0558789289871946, 'pattern_counts': {'":': 62, '<': 3, '<?xml version=': 0, '>': 16, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3709549', 'n_tokens_mistral': 11180, 'n_tokens_neox': 10654, 'n_words': 3398}