<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A. Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Am I ready to take this course?\n",
    "\n",
    "Yes. Probably. Some programming experience will help, but is not required. If you have no programming experience, you'll probably need to spend extra time on this class, and be sure to ask lots of questions.\n",
    "\n",
    "There are lots of online resource that you can use to supplement things we learn in class. Some examples are:\n",
    "\n",
    " * [python.org Tutorial](https://docs.python.org/3/tutorial/index.html)\n",
    " * [software carpentry](http://software-carpentry.org)\n",
    " * [Google](http://google.com) (Just type in your question and follow the first [stackoverflow](http://stackoverflow.com) link. This is surprisingly effective; do this first.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What computational resources do I need for class?\n",
    "\n",
    "I assume you have a laptop that you will bring to class and use to follow along. If you don't have access to some sort of portable device that you can use in class, please let me know as soon as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How is this for geosciences specifically?\n",
    "\n",
    "The goal of this class is to provide information for all fields in geoscience. To that end, we will try to cover topics from geology, geography, atmospheric sciences, and oceanography. Specifically, we will focus on 1D timeseries, and 2D geospatial (i.e., on a map) analysis. If you have any topics you would like to cover, please let us know, and we will do our best to accommodate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class setup\n",
    "\n",
    "### Class format\n",
    "\n",
    "We will go through course materials during class time. You should bring a laptop to class so that you can follow along and participate in exercises. Also, course materials are interactive, so you can learn by running code snippets as we go and asking questions.\n",
    "\n",
    "\n",
    "### Course materials\n",
    "\n",
    "The course materials are available in the [class repository](https://github.com/kthyng/python4geosciences). Details on git and github are available later in this notebook, but I will probably just provide the materials directly to your account on `copano`. They are in the form of [Jupyter notebooks](http://jupyter.org/). More information on notebooks in the next section.\n",
    "\n",
    "\n",
    "### Course textbook\n",
    "\n",
    "There is no textbook for the course. But if you'd like an outside resource, here are two recommendations:\n",
    "\n",
    "1. From O'Reilly, Learning Python: http://www.amazon.com/Learning-Python-Edition-Mark-Lutz/dp/1449355730 (you can find both print and e versions)\n",
    "2. <NAME> has written a number of books on Python and related scientific subjects. And as a bonus, they are free (digital versions): http://greenteapress.com/wp/. In particular you would want to check out Think Python (2nd edition)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## B. Jupyter notebooks\n",
    "\n",
    "This file format makes it easy to seamlessly combine text and code. The text can be plain or formatted with [Markdown](https://daringfireball.net/projects/markdown/). The code can be written in over 40 languages including Python, R, and Scala. Most importantly, the code can be interacted with when the notebook is opened in a local (that is, on your computer) iPython server. Alternatively, it can simply be viewed through a github repository (like [this very notebook](https://github.com/kthyng/python4geosciences/blob/master/materials/intro.ipynb)) or through [nbviewer](http://nbviewer.ipython.org/).\n",
    "\n",
    "### JupyterHub\n",
    "\n",
    "You all have a username and password on a local machine that is running a server for us to use in class. By using this, we will be able to avoid a lot of installation issues that occur from using different operating systems, etc. You'll be able to run class materials (in the form of Jupyter notebooks) through this server via your web browser as well as create and work on homework assignments. If you prefer, you are welcome to run Python on your own computer, but you will need to do that mostly on your own. If you go that route, I recommend using Python 3 (which we will be using in class) and a distribution from [Anaconda](https://www.continuum.io/downloads). We'll talk about it as a class later in the semester, too. I recommend you all use this system I have set up for you. It will integrate nicely for our homeworks, too.\n",
    "\n",
    "To access the server, open your web browser and navigate to `copano.tamu.edu:8000` and sign in with the username and password that I have provided you with. I've already set it up so that you can access materials and create notebooks within this system.\n",
    "\n",
    "\n",
    "### Create a new notebook\n",
    "\n",
    "Start up your local notebook server in your new repo and create a new Jupyter notebook from the local server page.\n",
    "\n",
    "### Choose syntax for a cell\n",
    "\n",
    "Notebooks are built of cells. Cells can have multiple built-in formats including code and Markdown for text. You can select the desired format from a dropdown menu at the top.\n",
    "\n",
    "If you want to type words, use \"Markdown\"; if you want to write code, choose \"code\".\n",
    "\n",
    "### Move between cells\n",
    "\n",
    "To run a given cell, type `[shift-enter]` which active in that cell. You can run all of the cells with Cell > Run all; other variations are available in that drop down menu.\n",
    "\n",
    "\n",
    "### Homework\n",
    "\n",
    "We'll discuss homework soon and go through details. It will be in the form of Jupyter notebooks and will probably be submitted through JupyterHub.\n",
    "\n",
    "\n",
    "### On-campus access\n",
    "\n",
    "If you want to connect to copano from off-campus, you'll need to use VPN. You can do so by following directions at `connect.tamu.edu`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "The material below is bonus for any students that are interested in using a terminal window on their own computer for running Python. We won't go through it in class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command-line interface\n",
    "\n",
    "A command-line interface is a way to interact with your computer using text instead of a Graphical User Interface (GUI), a GUI being visually based with icons etc. We will use these in this class. On a Macintosh or Linux machine, this is a terminal window. On a PC this is often called a command prompt.\n",
    "\n",
    "Here are some commonly-used commands:\n",
    "\n",
    "* `cd [path]`: change directory from current location to [path]. `cd ..` can be used to move up a single directory, and `cd ../..` moves up two directories, etc.\n",
    "* `pwd`: print working directory, as in write out the current location in the terminal window.\n",
    "* `ls`: list files in current directory. `ls -l` list files in long format to include more information, `ls -a` to list all files even those that are usually not shown because the have a `.` in front, `ls -h` to show file sizes in human readable format. Flags can always be combined to use multiple options at once, as in `ls -ah` to show all files in human readable format.\n",
    "* [tab]: Tab completion. You can always push tab in the terminal window to see available options. As you have some letters entered and push tab, the options will be limited to those that fit the pattern you have started.\n",
    "* `mkdir [dirname]`: make directory called dirname.\n",
    "* `rm [filename]`: remove a file called filename. To remove a directory called dirname, use `rm -r [dirname]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Short git and GitHub tutorial\n",
    "\n",
    "Class materials are available on a [GitHub](http://github.org) repository. GitHub is a way to share and access code online which has been version-controlled using git. Version control allows changes in code to be tracked over time; this is important for reproducibility, retrieving code in case of accidents, and working on code in groups. Git is one way to version control your code — other methods include subversion (svn), cvs, and mercurial. More information on this is provided below.\n",
    "\n",
    "Remember: you can always google to learn more! Google is an infinite resource that you can ask at any time of the day. Here we summarize a brief overview of how to use git. GitHub has a [cheatsheet](https://services.github.com/kit/downloads/github-git-cheat-sheet.pdf) available.\n",
    "\n",
    "To get changes in a file in a local version of a repository tracked, saved, and then shared with the internet (on github), do the following:\n",
    "\n",
    "* `git add` to initially tell the system to track your file and subsequently to tell the system that you want to take into account new changes to the file (you can do add more than one file in this process). Then \n",
    "* `git commit -m [commit note]` to save the changes. Then\n",
    "* `git push` to share the changes with the version of your repository on github. Now you should be able to look at your repo on github and see your updated file there.\n",
    "\n",
    "**GitHub Desktop**\n",
    "\n",
    "After you have made your repository on GitHub (the website), you should clone it to GitHub Desktop (which is on your local machine). (This should be very easy if you are properly signed into your github account in GitHub Desktop.) Then to get your file tracked and pushed to GitHub (the website):\n",
    "\n",
    "* While inspecting the relevant repository, any untracked files or changes to tracked files are shown in the middle window (it is white with horizontal lines). To do the equivalent of `git add`, you should check the box of the file.\n",
    "* To commit your changes, fill out the form at the bottom of the same window. One short window leaves space for a  \"Summary\" of your changes, and if you have more to say you can put it in the \"Description\" box.\n",
    "* To push your local changes out to GitHub online, use the Sync button on the upper right hand corner of the window. As a side note, this sync button is also how you should pull down changes to a repository you are following (the equivalent of `git pull`).\n",
    "\n",
    "\n",
    "Note that you do not want to have a directory covered by two git repositories. So for file structure for this class, for example, you might want to have one directory for the class (\"Python_x89\") which contains two version-controlled subdirectories: the course materials (python4geosciences) and your homework repository (\"homework\"). That will keep everything properly separated. \n",
    "\n",
    "![XKCD](https://imgs.xkcd.com/comics/git.png)\n",
    "\n",
    "[Git as explained by XKCD](https://xkcd.com/1597/)\n",
    "\n",
    "### `git status`\n",
    "\n",
    "Type this in a git-monitored subdirectory on your computer to see the status of files that are under git version control and which files are not being monitored.\n",
    "\n",
    "### `git add`\n",
    "\n",
    "Use this to add local files to your git repository. `git add [filename]`.\n",
    "\n",
    "### `git commit`\n",
    "\n",
    "Use this to save to your repository local file changes. First you need to add the file with `git add`, then you can `git commit -m [commit message]`, where the commit message is a concise and useful note to explain what changes you have made. You may skip the `git add` step if you would like to commit at once all changes that have been made (you can see what changes would be committed by first consulting `git status`) with `git commit -am [commit message]`. The `-a` flag stands for \"all\" as in commit all changes.\n",
    "\n",
    "### `git push`\n",
    "\n",
    "To move your local changes to github so that they are saved and also for sharing with others, you need to `git push`. After making whatever commits you want to make, you can run this command to finalize your changes on github.\n",
    "\n",
    "### `git merge`\n",
    "\n",
    "If changes have been made in multiple places (say, by another person working on the same code base, or by yourself on two different machines) between `git push`es, git will try to merge if possible — which it will do if the changes don't overlap and therefore don't require your input. If the changes do overlap, you will have to merge the two versions of the code together. You probably won't need to do this in this class.\n",
    "\n",
    "\n",
    "### Get set up with GitHub\n",
    "\n",
    "You'll need an account on github if you don't already have one, and to have git installed on your computer. We will interact with github through a terminal window in class and through the website. You may also download the [GitHub Desktop application](https://desktop.github.com/) to use if you prefer, though we won't be able to help you much with the details of it.\n",
    "\n",
    "### Create a new repo\n",
    "\n",
    "In your account on the github webpage, click on the Repositories tab and then the green \"New\" button in the upper right. You'll need to keep this repo public. After creating this new repo, follow the instructions on the next page for quick setup or to create a new repository on the command line. This makes it so that you have the repo both on github and on your local machine. \n",
    "\n",
    "### Clone a repo\n",
    "\n",
    "In a terminal window in the location you want to save the repo materials, type: `git clone [repo address, e.g. https://github.com/hetland/python4geosciences]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
<commit_msg>fixed copano address in introductory materials<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A. Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Am I ready to take this course?\n",
    "\n",
    "Yes. Probably. Some programming experience will help, but is not required. If you have no programming experience, you'll probably need to spend extra time on this class, and be sure to ask lots of questions.\n",
    "\n",
    "There are lots of online resource that you can use to supplement things we learn in class. Some examples are:\n",
    "\n",
    " * [python.org Tutorial](https://docs.python.org/3/tutorial/index.html)\n",
    " * [software carpentry](http://software-carpentry.org)\n",
    " * [Google](http://google.com) (Just type in your question and follow the first [stackoverflow](http://stackoverflow.com) link. This is surprisingly effective; do this first.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What computational resources do I need for class?\n",
    "\n",
    "I assume you have a laptop that you will bring to class and use to follow along. If you don't have access to some sort of portable device that you can use in class, please let me know as soon as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How is this for geosciences specifically?\n",
    "\n",
    "The goal of this class is to provide information for all fields in geoscience. To that end, we will try to cover topics from geology, geography, atmospheric sciences, and oceanography. Specifically, we will focus on 1D timeseries, and 2D geospatial (i.e., on a map) analysis. If you have any topics you would like to cover, please let us know, and we will do our best to accommodate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class setup\n",
    "\n",
    "### Class format\n",
    "\n",
    "We will go through course materials during class time. You should bring a laptop to class so that you can follow along and participate in exercises. Also, course materials are interactive, so you can learn by running code snippets as we go and asking questions.\n",
    "\n",
    "\n",
    "### Course materials\n",
    "\n",
    "The course materials are available in the [class repository](https://github.com/kthyng/python4geosciences). Details on git and github are available later in this notebook, but I will probably just provide the materials directly to your account on `copano`. They are in the form of [Jupyter notebooks](http://jupyter.org/). More information on notebooks in the next section.\n",
    "\n",
    "\n",
    "### Course textbook\n",
    "\n",
    "There is no textbook for the course. But if you'd like an outside resource, here are two recommendations:\n",
    "\n",
    "1. From O'Reilly, Learning Python: http://www.amazon.com/Learning-Python-Edition-Mark-Lutz/dp/1449355730 (you can find both print and e versions)\n",
    "2. <NAME> has written a number of books on Python and related scientific subjects. And as a bonus, they are free (digital versions): http://greenteapress.com/wp/. In particular you would want to check out Think Python (2nd edition)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## B. Jupyter notebooks\n",
    "\n",
    "This file format makes it easy to seamlessly combine text and code. The text can be plain or formatted with [Markdown](https://daringfireball.net/projects/markdown/). The code can be written in over 40 languages including Python, R, and Scala. Most importantly, the code can be interacted with when the notebook is opened in a local (that is, on your computer) iPython server. Alternatively, it can simply be viewed through a github repository (like [this very notebook](https://github.com/kthyng/python4geosciences/blob/master/materials/intro.ipynb)) or through [nbviewer](http://nbviewer.ipython.org/).\n",
    "\n",
    "### JupyterHub\n",
    "\n",
    "You all have a username and password on a local machine that is running a server for us to use in class. By using this, we will be able to avoid a lot of installation issues that occur from using different operating systems, etc. You'll be able to run class materials (in the form of Jupyter notebooks) through this server via your web browser as well as create and work on homework assignments. If you prefer, you are welcome to run Python on your own computer, but you will need to do that mostly on your own. If you go that route, I recommend using Python 3 (which we will be using in class) and a distribution from [Anaconda](https://www.continuum.io/downloads). We'll talk about it as a class later in the semester, too. I recommend you all use this system I have set up for you. It will integrate nicely for our homeworks, too.\n",
    "\n",
    "To access the server, open your web browser and navigate to `https://copano.tamu.edu:8000` (the 's' is required!) and sign in with the username and password that I have provided you with. I've already set it up so that you can access materials and create notebooks within this system.\n",
    "\n",
    "\n",
    "### Create a new notebook\n",
    "\n",
    "Start up your local notebook server in your new repo and create a new Jupyter notebook from the local server page.\n",
    "\n",
    "### Choose syntax for a cell\n",
    "\n",
    "Notebooks are built of cells. Cells can have multiple built-in formats including code and Markdown for text. You can select the desired format from a dropdown menu at the top.\n",
    "\n",
    "If you want to type words, use \"Markdown\"; if you want to write code, choose \"code\".\n",
    "\n",
    "### Move between cells\n",
    "\n",
    "To run a given cell, type `[shift-enter]` which active in that cell. You can run all of the cells with Cell > Run all; other variations are available in that drop down menu.\n",
    "\n",
    "\n",
    "### Homework\n",
    "\n",
    "We'll discuss homework soon and go through details. It will be in the form of Jupyter notebooks and will probably be submitted through JupyterHub.\n",
    "\n",
    "\n",
    "### On-campus access\n",
    "\n",
    "If you want to connect to copano from off-campus, you'll need to use VPN. You can do so by following directions at `connect.tamu.edu`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "The material below is bonus for any students that are interested in using a terminal window on their own computer for running Python. We won't go through it in class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command-line interface\n",
    "\n",
    "A command-line interface is a way to interact with your computer using text instead of a Graphical User Interface (GUI), a GUI being visually based with icons etc. We will use these in this class. On a Macintosh or Linux machine, this is a terminal window. On a PC this is often called a command prompt.\n",
    "\n",
    "Here are some commonly-used commands:\n",
    "\n",
    "* `cd [path]`: change directory from current location to [path]. `cd ..` can be used to move up a single directory, and `cd ../..` moves up two directories, etc.\n",
    "* `pwd`: print working directory, as in write out the current location in the terminal window.\n",
    "* `ls`: list files in current directory. `ls -l` list files in long format to include more information, `ls -a` to list all files even those that are usually not shown because the have a `.` in front, `ls -h` to show file sizes in human readable format. Flags can always be combined to use multiple options at once, as in `ls -ah` to show all files in human readable format.\n",
    "* [tab]: Tab completion. You can always push tab in the terminal window to see available options. As you have some letters entered and push tab, the options will be limited to those that fit the pattern you have started.\n",
    "* `mkdir [dirname]`: make directory called dirname.\n",
    "* `rm [filename]`: remove a file called filename. To remove a directory called dirname, use `rm -r [dirname]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Short git and GitHub tutorial\n",
    "\n",
    "Class materials are available on a [GitHub](http://github.org) repository. GitHub is a way to share and access code online which has been version-controlled using git. Version control allows changes in code to be tracked over time; this is important for reproducibility, retrieving code in case of accidents, and working on code in groups. Git is one way to version control your code — other methods include subversion (svn), cvs, and mercurial. More information on this is provided below.\n",
    "\n",
    "Remember: you can always google to learn more! Google is an infinite resource that you can ask at any time of the day. Here we summarize a brief overview of how to use git. GitHub has a [cheatsheet](https://services.github.com/kit/downloads/github-git-cheat-sheet.pdf) available.\n",
    "\n",
    "To get changes in a file in a local version of a repository tracked, saved, and then shared with the internet (on github), do the following:\n",
    "\n",
    "* `git add` to initially tell the system to track your file and subsequently to tell the system that you want to take into account new changes to the file (you can do add more than one file in this process). Then \n",
    "* `git commit -m [commit note]` to save the changes. Then\n",
    "* `git push` to share the changes with the version of your repository on github. Now you should be able to look at your repo on github and see your updated file there.\n",
    "\n",
    "**GitHub Desktop**\n",
    "\n",
    "After you have made your repository on GitHub (the website), you should clone it to GitHub Desktop (which is on your local machine). (This should be very easy if you are properly signed into your github account in GitHub Desktop.) Then to get your file tracked and pushed to GitHub (the website):\n",
    "\n",
    "* While inspecting the relevant repository, any untracked files or changes to tracked files are shown in the middle window (it is white with horizontal lines). To do the equivalent of `git add`, you should check the box of the file.\n",
    "* To commit your changes, fill out the form at the bottom of the same window. One short window leaves space for a  \"Summary\" of your changes, and if you have more to say you can put it in the \"Description\" box.\n",
    "* To push your local changes out to GitHub online, use the Sync button on the upper right hand corner of the window. As a side note, this sync button is also how you should pull down changes to a repository you are following (the equivalent of `git pull`).\n",
    "\n",
    "\n",
    "Note that you do not want to have a directory covered by two git repositories. So for file structure for this class, for example, you might want to have one directory for the class (\"Python_x89\") which contains two version-controlled subdirectories: the course materials (python4geosciences) and your homework repository (\"homework\"). That will keep everything properly separated. \n",
    "\n",
    "![XKCD](https://imgs.xkcd.com/comics/git.png)\n",
    "\n",
    "[Git as explained by XKCD](https://xkcd.com/1597/)\n",
    "\n",
    "### `git status`\n",
    "\n",
    "Type this in a git-monitored subdirectory on your computer to see the status of files that are under git version control and which files are not being monitored.\n",
    "\n",
    "### `git add`\n",
    "\n",
    "Use this to add local files to your git repository. `git add [filename]`.\n",
    "\n",
    "### `git commit`\n",
    "\n",
    "Use this to save to your repository local file changes. First you need to add the file with `git add`, then you can `git commit -m [commit message]`, where the commit message is a concise and useful note to explain what changes you have made. You may skip the `git add` step if you would like to commit at once all changes that have been made (you can see what changes would be committed by first consulting `git status`) with `git commit -am [commit message]`. The `-a` flag stands for \"all\" as in commit all changes.\n",
    "\n",
    "### `git push`\n",
    "\n",
    "To move your local changes to github so that they are saved and also for sharing with others, you need to `git push`. After making whatever commits you want to make, you can run this command to finalize your changes on github.\n",
    "\n",
    "### `git merge`\n",
    "\n",
    "If changes have been made in multiple places (say, by another person working on the same code base, or by yourself on two different machines) between `git push`es, git will try to merge if possible — which it will do if the changes don't overlap and therefore don't require your input. If the changes do overlap, you will have to merge the two versions of the code together. You probably won't need to do this in this class.\n",
    "\n",
    "\n",
    "### Get set up with GitHub\n",
    "\n",
    "You'll need an account on github if you don't already have one, and to have git installed on your computer. We will interact with github through a terminal window in class and through the website. You may also download the [GitHub Desktop application](https://desktop.github.com/) to use if you prefer, though we won't be able to help you much with the details of it.\n",
    "\n",
    "### Create a new repo\n",
    "\n",
    "In your account on the github webpage, click on the Repositories tab and then the green \"New\" button in the upper right. You'll need to keep this repo public. After creating this new repo, follow the instructions on the next page for quick setup or to create a new repository on the command line. This makes it so that you have the repo both on github and on your local machine. \n",
    "\n",
    "### Clone a repo\n",
    "\n",
    "In a terminal window in the location you want to save the repo materials, type: `git clone [repo address, e.g. https://github.com/hetland/python4geosciences]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.10936349641924904, 'fraction_numerical': 0.0028110568235057893, 'mean_word_length': 3.256837606837607, 'pattern_counts': {'":': 112, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 21, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5682216', 'n_tokens_mistral': 8591, 'n_tokens_neox': 8304, 'n_words': 4295}