<jupyter_start><jupyter_text>Intro to git for Data Scientists This is the first part of a 3-part `git` course for Data Scientists.**Chapter I:*** How to create a git repository (init, status)* The four main stages of a file in a repository: untracked, tracked, staged and committed (ignore, add, commit, rm)**Chapter II:*** How to track changes of a file in a repository (diff, log)* How to maintain different versions of a file without going crazy or using N suffixes (branch, checkout)* Time-travel with git (revert, checkout, reset)**Chapter III:*** How to work with friends on the same file... and remain friends (merge)* Working with remote repositories (clone, fetch, pull, push, pull requests)* Contributing flows (forks, feature branches)In this session we'll cover Chapter I!  Chapter I [30']* How to create a git repository (init, status)* The four main stages of a file in a repository: untracked, tracked, staged and committed (ignore, add, commit, rm) **Warning**: This tutorial is meant to be executed ONCE and in the given order. Since in fact each step has side-effects, running one of the cells will likely change the output of re-running one of the previous cells. If you need to re-run previous cells, you will need to re-run all cells starting from the very first one below (cf. Creating a new repository)  Creating a new repositoryWe will start by creating a git repository in a new subfolder of the folder where this notebook lives.<jupyter_code>%%bash 
rm -Rf my_first_repo
mkdir my_first_repo
git init my_first_repo<jupyter_output><empty_output><jupyter_text>From now on, we have all the necessary components for git to start tracking changes in this repository. But what has the command exactly done? Let's check out the .git folder directory structure:<jupyter_code>%%bash 
ls -lai my_first_repo/.git<jupyter_output><empty_output><jupyter_text>As you can see the `git init` command has created a bunch of files and directories inside the `.git` folder. These are the files that store all the information necessary to track and navigate through changes of the files in your repository. If you are curious (or even better when you understand the basics well), you can read more about each component's role in the [Pro git book (available for free online)](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelainch10-git-internals). We will now start introducing changes (that is, files) into our repository. Do we have anything in the repository yet?<jupyter_code>%%bash 
cd my_first_repo
git status<jupyter_output><empty_output><jupyter_text>The short answer is no, there's nothing in the repository because `git` only does what we tell it to do, and we haven't asked it to track any of the contents of the folder. That is why it lists the folder's contents as "untracked" files.As a sidenote, git is telling us that we are working in our "master" branch, which is the default name for the default branch of any project. We'll talk about branches later but for now we can already notice that the status command is telling us that there are a few files that are "untracked".  The first 2 out of 4 stages of a file in a repositoryAs we saw in the previous section, when we create a new repository, git will start noticing files that exist in the same folder but are not yet tracked bygit. This suggests that there are at least two stages in the lifetime a file living in the same folder of a git repository:* untracked* trackedFor example, suppose that we start working on a new file `test.py` containing code we want to start tracking under git:<jupyter_code>%%bash
echo "print('Complicated Python pipeline')" > my_first_repo/test.py<jupyter_output><empty_output><jupyter_text>This new file (`test.py`) exists now in the repository but is not yet being tracked by git:<jupyter_code>%%bash
cd my_first_repo
git status<jupyter_output><empty_output><jupyter_text>Git tells us that our new file `test.py` is **untracked**.If we want to start tracking this new file within our repository, we just need to `add` it for the first time:<jupyter_code>%%bash
cd my_first_repo
git add test.py
git status<jupyter_output><empty_output><jupyter_text>The `test.py` file we just created is now tracked by the repository, and git status tells us as much. Note that git will complain if we ask it to to track files that are in fact in folders other than the on where the repository was created:<jupyter_code>%%bash
cd my_first_repo
touch ../another_test.txt
# git add ../another_test.txt
# This will fail with an error:
# `fatal: ../another_test.txt: '../another_test.txt' is outside repository`<jupyter_output><empty_output><jupyter_text>IMPORTANT: Things git should not track...There are some files that will be in the same folder as the repo but that we will never want to track. Examples of these are:* data files,* artifact files (serialized models, binaries),* **credentials**,* generally anything that is not source code.Suppose that we have a `data.csv` file that we don't want to include in the repository:<jupyter_code>%%bash
touch my_first_repo/data.csv<jupyter_output><empty_output><jupyter_text>This file will now show as an untracked file everytime we run `git status` which eventually will get a bit tiresome:<jupyter_code>%%bash
cd my_first_repo
git status<jupyter_output><empty_output><jupyter_text>Since we don't want to track it, what we will do is tell `git` to **ignore** it. As a matter of fact, we will tell `git` to ignore all files ending with the`.csv` extension. To do that, we just need to create a file named `.gitignore` in the repo, containing the line `*.csv`:<jupyter_code>%%bash
cd my_first_repo
echo "*.csv" > .gitignore
git add .gitignore
git status<jupyter_output><empty_output><jupyter_text>Note that we also started tracking the `.gitignore` file. Magically, git status now doesn't show `data.csv` anymore: `git` knows files like that one can be safely ignored. **Excercise 0.0**: Why do you think we need to track (`git add`) the `.gitignore` file?**Excercise 0.1**: Can you think of (at least) two reasons why we typically don't want to track binary files in a git repository?**Exercise 0.2**: How would we ignore a whole directory? And if we want to ignore a whole directory but one of its subfolders?  The last two of the four stages of a file in a repositoryWe have seen that there are at least two stages in the lifetime a file living in the same folder of a git repository:* untracked: when a file resides in the same folder where we created the repository, but it is not under source control* tracked: when add a previously _untracked_ file into the repository.There are in fact two additional stages for files _tracked_ by a repository:* staged* commitedAs you saw, we started tracking a file `test.py` by git-adding it into the repository. As a matter of fact, the `git add` commanddid actually two things:* it told git to start tracking the contents of that file* it _staged_ the changes in the file (i.e. its creation) to prepare them to be _commited_You can think of _staging_ a file as the last step prior to making the file (or the changes introduced to the file) permanent in the repository. While a file is _staged_, you can continue to modify it:<jupyter_code>%%bash
cd my_first_repo
# we add one more line to our `test.py` script
echo "print('Now it is even more complicated!')" >> test.py
git status<jupyter_output><empty_output><jupyter_text>And the file remains in its _staged_ state, but now `git status`informs us that our latest changes have not yet been staged.If we were to commit the _staged_ `test.py` changes now, we would not be "saving" the changes we just performed, because those changes haven't yet been staged. So let's stage them so that we can commit everything in one batch!<jupyter_code>%%bash
cd my_first_repo
git add test.py
git status<jupyter_output><empty_output><jupyter_text>Commiting the staged changesIn order to make the new files (or the changes in existing files) permanently stored in git, we need to _commit_ them.Here we need to explain what we mean by "permantently storing". Permanently storing changes in git (or most version control systems) doesn't mean that we won't be able to (or that we shouldn't) change those files ever again. On the contrary, it means that we will store the state of the file in the repository in a way that will allow us to build on top of that state, coming back to it at any point in the future. You read well, *any* point in the future! Git is like a time-travel machine!So let's _commit_ both of the staged changes (the ones in `.gitignore` and the ones in `test.py`):<jupyter_code>%%bash
cd my_first_repo
git commit -m 'My first commit and my first commit message!'<jupyter_output><empty_output><jupyter_text>Now when we check the status of our repository, we notice that we no longer have stagedchanges:<jupyter_code>%%bash
cd my_first_repo
git status<jupyter_output><empty_output><jupyter_text>The opposite of `add`ing is `rm`oving!Sometimes we would like to stop tracking files we previously added to a repository, perhaps because they are no longer useful or because we added them mistakenly. There's a useful git command to do just that, which is `git rm`. To illustrate its behavior, we will first create, stage and commit a dummy file that we would later like to remove:<jupyter_code>%%bash
cd my_first_repo
touch dummy.py
git add dummy.py
git commit -m 'commiting a dummy file just to remove it later!'<jupyter_output><empty_output><jupyter_text>Let's remove this dummy!<jupyter_code>%%bash
cd my_first_repo
git rm dummy.py
git status<jupyter_output><empty_output><jupyter_text>Suprise! Removing is dual to `add` in one more way: Applying `git rm` only _stages_ the delete operation but doesn't yet make it permanent. We need to further _commit_ it for the deletion to take place:<jupyter_code>%%bash
cd my_first_repo
git commit -m 'Away with the dummy!'
git status<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06593627500251281, 'fraction_numerical': 0.0014071766006633832, 'mean_word_length': 5.153370439084725, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '562327', 'license': 'MIT', 'n_tokens_mistral': 2823, 'n_tokens_neox': 2715, 'n_words': 1481, 'path': 'Chapter_I.ipynb', 'repo_name': 'atibaup/git-for-ds', 'size': '9949'}