<jupyter_start><jupyter_text>Local Version Control with GitWhile the outline of this lesson is based on the book "Effective Computation in Physics" by <NAME> and <NAME>, this notebook draws most of it's text, examples and several figures from the [Software Carpetry](https://software-carpentry.org/) lesson on [Git](http://swcarpentry.github.io/git-novice). Other sources are referenced in place.**Note:** In this IPython notebook we are using git commands that should be entered on the (bash) shell. We do this within a notebook by adding an exclamation mark (`!`) before each command. This `!` is not a part pf the command and should not be entered on an actuall (bash) shell.  Installing GitTo determine if git is already installed, open a terminal and try executing `git`. WindowsGit is part of the `swc-conda` installer. Apple macOSGit is installed when installing the Apple Developer tools and Xcode. LinuxGit is available from the package repositories of all modern Linux distributions.```shell for Ubuntu, Debian and related distributions:$ sudo apt-get install git for Fedora, RedHat, openSuSE and related distributions:$ sudo yum install git```You can always go to the Git website  and follow the instructions for the platform. [![PhD Comic: notFinal.doc](http://phdcomics.com/comics/archive/phd101212s.gif "PhD Comic: notFinal.doc")](http://phdcomics.com/comics/archive.php?comicid=1531)Source: “Piled Higher and Deeper” by <NAME>,   What is Version Control?* A way to backup changing files* Keep a history of old versions with annotations (Meta data)* Manage merging of changes between different versions/streams of the files/documents.Conceptually stop thinking of documents existing in different version, but rather having a **base document** and **sets of changes** that are applied to the document and can be *un-done* or applied to a different document that is based on the same base-document. Version control systems start with a base version of the document and then save just the changes you made at each step of the way. You can think of it as a tape: if you rewind the tape and start at the base document, then you can play back each change and end up with your latest version.![play-changes](http://swcarpentry.github.io/git-novice/fig/play-changes.svg)Once you think of changes as separate from the document itself, you can then think about “playing back” different sets of changes onto the base document and getting different versions of the document. For example, two users can make independent sets of changes based on the same document.![versions](http://swcarpentry.github.io/git-novice/fig/versions.svg)If there aren’t conflicts, you can even play two sets of changes onto the same base document.![merge](http://swcarpentry.github.io/git-novice/fig/merge.svg)  Getting Started with Git  Good resources for use of Git:* Book **Pro Git** by <NAME> and <NAME>; * Chapters 15 and 16 in **Effective Computation in Physics** by <NAME> and <NAME>* Learn the basics of Git in 15 minutes: * Interactive Git Cheatsheet: * Software Carpentry's Git lesson: * BitBucket Git tutorials: * GitHub Videos: * Git for Ages 4 And Up [Video (99 minutes)](https://www.youtube.com/watch?v=1ffBJ4sVUb4)  Getting help (git help)A very good book about Git is **Pro Git** by *<NAME>* and *<NAME>*. It is available to [read online for free](https://git-scm.com/book).  *Dead tree* versions are available from many (online) bookstores.As with most unix commands we can have a look into the man-pages:```shell$ man git```Also most unix commands have the command line argument `--help`:<jupyter_code>! git --help<jupyter_output>usage: git [--version] [--help] [-C <path>] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]

The most commonly used git commands are:
   add        Add file contents to the index
   bisect     Find by binary search the change that introduced a bug
   branch     List, create, or delete branches
   checkout   Checkout a branch or paths to the working tree
   clone      Clone a repository into a new directory
   commit     Record changes to the repository
   diff       Show changes between commits, commit and working tree, etc
   fetch      Download objects and refs from another repository
   grep       Print lines matching a pattern
   init       Create an empty Git repository or reinitialize an existing one
   log        Show commit logs
   merge      Join t[...]<jupyter_text>Git has a many sub-commands. Each of this sub-commands has it's own help page that cann be accessed with: `git help `.  Control the Behavior of Git (git config)First we need to complete the setup process by telling Git our name, email address and favorite text editor, as git will need this when we want to record our changes.**Run the following commands using your own name and email address:**```shellgit config --global user.name   "<NAME>"git config --global user.email  "<EMAIL>"git config --global core.editor "nano"git config --global alias.slog  "log --pretty=format:'%h - %an, %ar : %s'"```**Look at you current configuration:**```shellgit config --list``````user.name=<NAME>user.email=<EMAIL>core.editor=nanoalias.unstage=reset HEAD --alias.slog=log --pretty=format:'%h - %an, %ar : %s'color.ui=autopush.default=currentdiff.tool=melddiff.guitool=meldmerge.tool=meld```  Local Version Control With GitWe will:* Creating a repository.* Adding files to that repository, so that they can be tracked.* Taking snapshots of incremental versions, so that they can be logged.* Undoing changes.* Redoing them.* Trying new ideas in separate sandboxes. Create a Local Repository (git init)<jupyter_code># remove existing directory:
! test -d git_planets && rm -Rf git_planets
# create new directory and change into it
! mkdir git_planets
%cd git_planets
! git init
%ls
%ls -A<jupyter_output>[0m[01;34m.git[0m/<jupyter_text>The command `git init` has created an hidden directory called `.git`.<jupyter_code>%cd .git
%ls -A<jupyter_output>/home/ostueker/Carpentry/CMSC6950-2017/lectures/git_planets/.git
[0m[01;34mbranches[0m/  config  description  HEAD  [01;34mhooks[0m/  [01;34minfo[0m/  [01;34mobjects[0m/  [01;34mrefs[0m/<jupyter_text>This `.git` directory contains the local git repository. Navigating into that directory and listing it's content reveals how git organizes it's data internally.  With ordinary use of git, one will never need to alter these hidden files, however it is good to know that the whole repository is containded in this hidden directory at the top level of the repository.**A repository can easily be moved to a different location, as long the `.git` directory is relocated along with it!**<jupyter_code>%cd ..
# For this exercise, I configure an alternate personality:
! git config user.name  "<NAME>"
! git config user.email "<EMAIL>>"<jupyter_output><empty_output><jupyter_text>Staging Files (git add)<jupyter_code>! echo "Cold and dry, but everything is my favorite color" > mars.txt

# To follow along, please use a text editor like nano 
# to create a file mars.txt with the content below.

! cat mars.txt
! git status
! git add mars.txt<jupyter_output><empty_output><jupyter_text>Checking the Status of Your Local Copy (git status)(!) Run `git status` often! It will show you what is going on and make suggestions on what you might want to do next.<jupyter_code>! git status<jupyter_output>On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

	[32mnew file:   mars.txt[m<jupyter_text>Saving a Snapshot (git commit)<jupyter_code>! git commit -m "Start notes on Mars as a base"<jupyter_output>[master (root-commit) aa1e8ce] Start notes on Mars as a base
 1 file changed, 1 insertion(+)
 create mode 100644 mars.txt<jupyter_text>Commit often! Evey commit is a point that you can revert to in the future.  Good commits are *atomic* (self consistent), the smallest change that remains meaningful.  They should not represent more work than you are willing to loose.<jupyter_code>! git status
! git log<jupyter_output>[33mcommit aa1e8ce00755e9a92534a0946c750b274adb5db2[m
Author: <NAME> <<EMAIL>>
Date:   Wed May 10 16:20:16 2017 -0230

    Start notes on Mars as a base<jupyter_text>Where Are My Changes?If we run ls at this point, we will still see just one file called mars.txt. That’s because Git saves information about files’ history in the special .git directory mentioned earlier so that our filesystem doesn’t become cluttered (and so that we can’t accidentally edit or delete an old version).  Viewing the Differences (git diff)Let's make a change to the file:<jupyter_code>! echo "The two moons may be a problem for Wolfman" >> mars.txt

# To follow along, please use a text editor like nano 
# to edit the file.

! cat mars.txt
! git status
! git diff<jupyter_output>[1mdiff --git a/mars.txt b/mars.txt[m
[1mindex df0654a..315bf3a 100644[m
[1m--- a/mars.txt[m
[1m+++ b/mars.txt[m
[36m@@ -1 +1,2 @@[m
 Cold and dry, but everything is my favorite color[m
[32m+[m[32mThe two moons may be a problem for Wolfman[m<jupyter_text>The output is cryptic because it is actually a series of commands for tools like editors and `patch` telling them how to reconstruct one file given the other. If we break it down into pieces:1. The first line tells us that Git is producing output similar to the Unix `diff` command comparing the old and new versions of the file.2. The second line tells exactly which versions of the file Git is comparing; `df0654a` and `315bf3a` are unique computer-generated labels for those versions.3. The third and fourth lines once again show the name of the file being changed.4. The remaining lines are the most interesting, they show us the actual differences and the lines on which they occur. In particular, the `+` marker in the first column shows where we added a line.After reviewing our change, it’s time to commit it:<jupyter_code>! git commit -m "Add concerns about effects of Mars' moons on Wolfman"<jupyter_output>On branch master
Changes not staged for commit:
	[31mmodified:   mars.txt[m

no changes added to commit<jupyter_text>Whoops: Git won’t commit because we didn’t use `git add` first. Let’s fix that:<jupyter_code>! git add mars.txt
! git commit -m "Add concerns about effects of Mars' moons on Wolfman"<jupyter_output>[master b03885e] Add concerns about effects of Mars' moons on Wolfman
 1 file changed, 1 insertion(+)<jupyter_text>Git insists that we add files to the set we want to commit before actually committing anything. This allows us to commit our changes in stages and capture changes in logical portions rather than only large batches. For example, suppose we’re adding a few citations to our supervisor’s work to our thesis. We might want to commit those additions, and the corresponding addition to the bibliography, but not commit the work we’re doing on the conclusion (which we haven’t finished yet).To allow for this, Git has a special staging area where it keeps track of things that have been added to the current change set but not yet committed. The Staging AreaIf you think of Git as taking snapshots of changes over the life of a project, `git add` specifies what will go in a snapshot (putting things in the staging area), and `git commit` then actually takes the snapshot, and makes a permanent record of it (as a commit). If you don’t have anything staged when you type `git commit`, Git will prompt you to use `git commit -a` or `git commit --all`, which is kind of like gathering everyone for the picture! However, it’s almost always better to explicitly add things to the staging area, because you might commit changes you forgot you made. (Going back to snapshots, you might get the extra with incomplete makeup walking on the stage for the snapshot because you used `-a`!)Try to stage things manually, or you might find yourself searching for "`git undo commit`" more than you would like!![Staging Area](http://swcarpentry.github.io/git-novice/fig/git-staging-area.svg)<jupyter_code>! echo "But the Mummy will appreciate the lack of humidity" >> mars.txt

# To follow along, please use a text editor like nano 
# to edit the file.

! cat mars.txt
! git diff
! git add mars.txt
! git diff<jupyter_output><empty_output><jupyter_text>There is no output: as far as Git can tell, there’s no difference between what it’s been asked to save permanently and what’s currently in the directory. However, if we do this:<jupyter_code>! git diff --staged<jupyter_output>[1mdiff --git a/mars.txt b/mars.txt[m
[1mindex 315bf3a..b36abfd 100644[m
[1m--- a/mars.txt[m
[1m+++ b/mars.txt[m
[36m@@ -1,2 +1,3 @@[m
 Cold and dry, but everything is my favorite color[m
 The two moons may be a problem for Wolfman[m
[32m+[m[32mBut the Mummy will appreciate the lack of humidity[m<jupyter_text>it shows us the difference between the last committed change and what’s in the staging area. Let’s save our changes:<jupyter_code>! git commit -m "Discuss concerns about Mars' climate for the Mummy"<jupyter_output>[master 405d321] Discuss concerns about Mars' climate for the Mummy
 1 file changed, 1 insertion(+)<jupyter_text>check the status:<jupyter_code>! git status<jupyter_output>On branch master
nothing to commit, working directory clean<jupyter_text>Viewing the History (git log)and look at the history of what we've done so far:<jupyter_code>! git log<jupyter_output>[33mcommit 405d3213cce9fe8b8dd358e4c7f14552c7149b2a[m
Author: <NAME> <<EMAIL>>
Date:   Wed May 10 16:20:18 2017 -0230

    Discuss concerns about Mars' climate for the Mummy

[33mcommit b03885e5a812377d2f1bb331754fecaa8ca8b092[m
Author: <NAME> <<EMAIL>>
Date:   Wed May 10 16:20:17 2017 -0230

    Add concerns about effects of Mars' moons on Wolfman

[33mcommit aa1e8ce00755e9a92534a0946c750b274adb5db2[m
Author: <NAME> <<EMAIL>>
Date:   Wed May 10 16:20:16 2017 -0230

    Start notes on Mars as a base<jupyter_text>Some Tips: Word-based diffingSometimes, e.g. in the case of the text documments a line-wise diff is too coarse. That is where the `--color-words` option of `git diff` comes in very useful as it highlights the changed words using colors. Paging the LogWhen the output of `git log` is too long to fit in your screen, `git` uses a program to split it into pages of the size of your screen. When this "pager" is called, you will notice that the last line in your screen is a `:`, instead of your usual prompt.* To get out of the pager, press `q`.* To move to the next page, press the space bar.* To search for some_word in all pages, type `/some_word` and navigate throught matches pressing `n`. Limit Log SizeTo avoid having git log cover your entire terminal screen, you can limit the number of commits that Git lists by using -N, where N is the number of commits that you want to view. For example, if you only want information from the last commit you can use `git log -1`:<jupyter_code>! git log -1<jupyter_output>[33mcommit 405d3213cce9fe8b8dd358e4c7f14552c7149b2a[m
Author: <NAME> <<EMAIL>>
Date:   Wed May 10 16:20:18 2017 -0230

    Discuss concerns about Mars' climate for the Mummy<jupyter_text>You can also reduce the quantity of information using the --oneline option:<jupyter_code>! git log --oneline<jupyter_output>[33m405d321[m Discuss concerns about Mars' climate for the Mummy
[33mb03885e[m Add concerns about effects of Mars' moons on Wolfman
[33maa1e8ce[m Start notes on Mars as a base<jupyter_text>You can also combine the `--oneline` options with others. One useful combination is:  Write good commit messages!Log messages are very useful if used properly.  They help other people (your colleagues, your future self), a to understand the intentions of this commit.[![xkcd: Git Commit](https://imgs.xkcd.com/comics/git_commit.png)](https://xkcd.com/1296/)Source: xkcd by *<NAME>*, A good commit message captures the intention of a commit and provides some context.It is recommended to not use the `-m` argument, as it opens a text editor that allows to write a more comprehensive message. The seven rules of a great Git commit message1. Separate subject from body with a blank line2. Limit the subject line to 50 characters3. Capitalize the subject line4. Do not end the subject line with a period5. Use the imperative mood in the subject line6. Wrap the body at 72 characters7. Use the body to explain what and why vs. howFor example:```Summarize changes in around 50 characters or lessMore detailed explanatory text, if necessary. Wrap it to about 72characters or so. In some contexts, the first line is treated as thesubject of the commit and the rest of the text as the body. Theblank line separating the summary from the body is critical (unlessyou omit the body entirely); various tools like `log`, `shortlog`and `rebase` can get confused if you run the two together.Explain the problem that this commit is solving. Focus on why youare making this change as opposed to how (the code explains that).Are there side effects or other unintuitive consequences of thischange? Here's the place to explain them.Further paragraphs come after blank lines. - Bullet points are okay, too - Typically a hyphen or asterisk is used for the bullet, preceded   by a single space, with blank lines in between, but conventions   vary hereIf you use an issue tracker, put references to them at the bottom,like this:Resolves: 123See also: 456, 789```<jupyter_code>! git log --oneline --graph --all --decorate<jupyter_output>* [33m405d321[m[33m ([1;36mHEAD[m[33m, [1;32mmaster[m[33m)[m Discuss concerns about Mars' climate for the Mummy
* [33mb03885e[m Add concerns about effects of Mars' moons on Wolfman
* [33maa1e8ce[m Start notes on Mars as a base<jupyter_text>Exploring HistoryAs we saw in the previous lesson, we can refer to commits by their identifiers. You can refer to the *most recent commit* of the working directory by using the identifier `HEAD`.We’ve been adding one line at a time to `mars.txt`, so it’s easy to track our progress by looking, so let’s do that using our `HEAD`s. Before we start, let’s make a change to `mars.txt`.<jupyter_code>! echo "An ill-considered change" >> mars.txt
# To follow along, please use a text editor like nano 
# to edit the file.
!cat mars.txt<jupyter_output>Cold and dry, but everything is my favorite color
The two moons may be a problem for Wolfman
But the Mummy will appreciate the lack of humidity
An ill-considered change<jupyter_text>Now, let’s see what we get.<jupyter_code>! git diff HEAD mars.txt<jupyter_output>[1mdiff --git a/mars.txt b/mars.txt[m
[1mindex b36abfd..93a3e13 100644[m
[1m--- a/mars.txt[m
[1m+++ b/mars.txt[m
[36m@@ -1,3 +1,4 @@[m
 Cold and dry, but everything is my favorite color[m
 The two moons may be a problem for Wolfman[m
 But the Mummy will appreciate the lack of humidity[m
[32m+[m[32mAn ill-considered change[m<jupyter_text>which is the same as what you would get if you leave out `HEAD` (try it). The real goodness in all this is when you can refer to previous commits. We do that by adding `~1` to refer to the commit one before `HEAD`.<jupyter_code>! git diff HEAD~1 mars.txt<jupyter_output>[1mdiff --git a/mars.txt b/mars.txt[m
[1mindex 315bf3a..93a3e13 100644[m
[1m--- a/mars.txt[m
[1m+++ b/mars.txt[m
[36m@@ -1,2 +1,4 @@[m
 Cold and dry, but everything is my favorite color[m
 The two moons may be a problem for Wolfman[m
[32m+[m[32mBut the Mummy will appreciate the lack of humidity[m
[32m+[m[32mAn ill-considered change[m<jupyter_text>If we want to see the differences between older commits we can use git diff again, but with the notation HEAD~1, HEAD~2, and so on, to refer to them:<jupyter_code>! git diff HEAD~2 mars.txt<jupyter_output>[1mdiff --git a/mars.txt b/mars.txt[m
[1mindex df0654a..93a3e13 100644[m
[1m--- a/mars.txt[m
[1m+++ b/mars.txt[m
[36m@@ -1 +1,4 @@[m
 Cold and dry, but everything is my favorite color[m
[32m+[m[32mThe two moons may be a problem for Wolfman[m
[32m+[m[32mBut the Mummy will appreciate the lack of humidity[m
[32m+[m[32mAn ill-considered change[m<jupyter_text>We could also use `git show` which shows us what changes we made at an older commit as well as the commit message, rather than the differences between a commit and our working directory that we see by using `git diff`.```shell$ git show HEAD~2 mars.txtcommit aa1e8ce00755e9a92534a0946c750b274adb5db2Author: <NAME> Date:   Wed May 10 15:10:31 2017 -0230    Start notes on Mars as a basediff --git a/mars.txt b/mars.txtnew file mode 100644index 0000000..d927c56--- /dev/null+++ b/mars.txt@@ -0,0 +1,2 @@+Cold and dry, but everything is my favorite color+```In this way, we can build up a chain of commits. The most recent end of the chain is referred to as `HEAD`; we can refer to previous commits using the `~` notation, so `HEAD~1` (pronounced "head minus one") means "the previous commit", while `HEAD~123` goes back 123 commits from where we are now.We can also refer to commits using those long strings of digits and letters that git log displays. These are unique IDs for the changes, and “unique” really does mean unique: every change to any set of files on any computer has a unique 40-character identifier. Our first commit was given the ID aa1e8ce00755e9a92534a0946c750b274adb5db2, so let’s try this:```shell$ git diff aa1e8ce00755e9a92534a0946c750b274adb5db2 mars.txtdiff --git a/mars.txt b/mars.txtindex d927c56..1e712f8 100644--- a/mars.txt+++ b/mars.txt@@ -1,2 +1,5 @@ Cold and dry, but everything is my favorite color+The two moons may be a problem for Wolfman+But the Mummy will appreciate the lack of humidity+An ill-considered change```That’s the right answer, but typing out random 40-character strings is annoying, so Git lets us use just the first few characters:```shell$ git diff aa1e8ce mars.txtdiff --git a/mars.txt b/mars.txtindex d927c56..1e712f8 100644--- a/mars.txt+++ b/mars.txt@@ -1,2 +1,5 @@ Cold and dry, but everything is my favorite color+The two moons may be a problem for Wolfman+But the Mummy will appreciate the lack of humidity+An ill-considered change```All right! So we can save changes to files and see what we’ve changed—now how can we restore older versions of things? Let’s suppose we accidentally overwrite our file:<jupyter_code>! echo "We will need to manufacture our own oxygen" > mars.txt
! cat mars.txt<jupyter_output>We will need to manufacture our own oxygen<jupyter_text>`git status` now tells us that the file has been changed, but those changes haven’t been staged:<jupyter_code>! git status<jupyter_output>On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	[31mmodified:   mars.txt[m

no changes added to commit (use "git add" and/or "git commit -a")<jupyter_text>We can put things back the way they were by using git checkout:<jupyter_code>! git checkout HEAD mars.txt
! cat mars.txt
! git status<jupyter_output>On branch master
nothing to commit, working directory clean<jupyter_text>As you might guess from its name, `git checkout` checks out (i.e., restores) an old version of a file. In this case, we’re telling Git that we want to recover the version of the file recorded in `HEAD`, which is the last saved commit. If we want to go back even further, we can use a commit identifier instead:```shell$ git checkout aa1e8ce mars.txt$ cat mars.txtCold and dry, but everything is my favorite color$ git status On branch masterChanges to be committed:  (use "git reset HEAD ..." to unstage)	modified:   mars.txt```Notice that the changes are on the staged area. Again, we can put things back the way they were with by using git checkout:<jupyter_code>! git checkout -f master mars.txt<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0860473799941916, 'fraction_numerical': 0.029996266024976143, 'mean_word_length': 4.541149425287356, 'pattern_counts': {'":': 0, '<': 128, '<?xml version=': 0, '>': 137, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '283174', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 7883, 'n_tokens_neox': 7064, 'n_words': 3348, 'path': 'lectures/ch15-local-version-control.ipynb', 'repo_name': 'CMSC6950/CMSC6950.github.io', 'size': '24327'}