<jupyter_start><jupyter_text>IntroductionThis tutorial will explain the concept of document-oriented databases using MongoDB in Python. Because document-oriented databases do not express relations among data, they are less structured. However, operations in document-oriented databases are much faster than analogous operations in a relational database such as MySQL. For this reason, document-oriented databases such as MongoDB scale better than their SQL counterparts, and they are useful for big data.This tutorial will focus on using a MongoDB database as opposed to the implementation of a MongoDB database. Therefore, topics such as file storage, replication, load balancing, and sharding will not be covered in this tutorial. See the References section at the end for further reading.  Tutorial ContentIn this tutorial, we will cover the basics of MongoDB, including CRUD (create, read, update, delete) operations, the aggregation pipeline, map_reduce, indexing, and text search. Table of Contents* [Installing the Libraries](Installing-the-Libraries)* [Document-Oriented vs. Relational DB's](Document-Oriented-vs.-Relational-DB's)* [An Introduction to MongoDB](An-Introduction-to-MongoDB)* [Operations in MongoDB](Operations-in-MongoDB)* [The Aggregation Pipeline](The-Aggregation-Pipeline)* [Map_Reduce](Map_Reduce)* [Indexing and Text Search](Indexing-and-Text-Search)* [Summary](Summary)* [References](References)  Installing the LibrariesYou'll need to install MongoDB and the Python library pymongo.If you have brew, you can brew MongoDB in the following way.```$ brew update$ brew install mongodb```If you want to install MongoDB with TLS/SSL, you can run```$ brew install mongodb --with-openssl```If not, you can install MongoDB [here](https://www.mongodb.com/download-center "MongoDB Download").Now you should install pymongo, the Python library for MongoDB.```$ pip install --upgrade pymongo```  Document-Oriented vs. Relational DB'sWe will begin by comparing and contrasting document-oriented databases and relational databases. We begin with an example. Suppose you are taking 2 classes. For both of these classes, you have an associated list of test scores, which you have stored as `classes.csv`. We can view this data as a relational database in the following way.<jupyter_code>import sqlite3
from MongoTutorialLibrary import load_sql, pretty_print, pretty_print_remove_id
conn = sqlite3.connect(":memory:")
load_sql(conn, "classes.csv")<jupyter_output>class  grade_1  grade_2  grade_3
0  15-388       90       70       80
1  15-213      100       50       70<jupyter_text>This is very straightforward. We have a list of classes and corresponding test scores. However, this is more complicated when we have unequal number of tests. For example, say you scored a 95% on your fourth test in 15-213. If you tried to add this to the table, you'd have to add a fourth column for 15-213, and the table would be uneven. (Note that we could represent classes as grades in seperate tables, with one table for classes and another table for the relations between classes and grades. We represent data in the following manner for the sake of example.)Here we introduce the *document-oriented database*. Suppose I have the same data for these classes stored in some `classes.json`.<jupyter_code>import json
classes_file = open('classes.json', 'r')
classes = json.load(classes_file)
print pretty_print(classes)
classes_file.close()<jupyter_output>[
  {
    "course": "15-388",
    "scores": {
      "midterm_1": 90,
      "midterm_2": 70,
      "midterm_3": 80
    }
  },
  {
    "course": "15-213",
    "scores": {
      "midterm_1": 100,
      "midterm_2": 50,
      "midterm_3": 70
    }
  },
  {
    "course": "15-150",
    "scores": {}
  }
]<jupyter_text>A document-oriented database allows us to represent a group of *documents* collectively known as a *collection*. In our example, each class is a *document*, and the collection of classes is a *collection*.Noteably, each document does not have to have the same information, and we are able to add our fourth exam score to our 15-213 document with no problem! But how do we add the data to our document? To do this, we will introduce MongoDB.  An Introduction to MongoDBIn this section we will learn a popular implementation of a document-oriented database, MongoDB.Let's first set up our database from our `classes.json`.Note that this will require a MongoDB instance running on your local machine. To accomplish this, simply run```$ mongod```Note that you may have to pass additional flags or arguments. Run```$ mongod --help```for more information.Notably, you should set up MongoDB to to store information in a directory `data/` and with some subdirectory `db/` (so `/data/db/`) and record logs in a file within a directory `log/`. You can achieve these with the flags `--dbpath` and `--dbpath`, respectively, as shown below.```$ mongod --dbpath data/db/ --logpath log/mongo.log```<jupyter_code>import bson, pymongo
from pymongo import MongoClient

conn = MongoClient()

# create db if it doesn't already exist
db = conn.test_database

# create collection grades
grades = db.grades

# make sure grades is empty, so we can run this function multiple times
grades.delete_many({})

# add entries, initialize list of id's of objects
id_objects = grades.insert_many(classes)
ids = id_objects.inserted_ids

# analogous to "SELECT * FROM classes;"
grades_list = grades.find()

# print all grades
for grade in grades_list:
    print grade<jupyter_output>{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc2'), u'scores': {u'midterm_1': 90, u'midterm_2': 70, u'midterm_3': 80}}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdc3'), u'scores': {u'midterm_1': 100, u'midterm_2': 50, u'midterm_3': 70}}
{u'course': u'15-150', u'_id': ObjectId('581d4251d64511b506f7cdc4'), u'scores': {}}<jupyter_text>And we have successfully created a collection with our grades. But you want to put your great test score in the collection! Let's take a look at how to modify a document.If we know the ID of the document, this process is quite simple. We can find a *single* document with `.find_one()` and update a collection with `.update_one()`.<jupyter_code># this is the id we want
courseId = ids[1]
print courseId

# print newline and the document
print
print grades.find_one({"_id": courseId})

# update the document
grades.update_one({'_id': courseId}, {"$set": {'scores.midterm_4': 95}}, upsert=True)

# print newline and the updated document
print
print grades.find_one({"_id": courseId})<jupyter_output>581d4251d64511b506f7cdc3

{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdc3'), u'scores': {u'midterm_1': 100, u'midterm_2': 50, u'midterm_3': 70}}

{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdc3'), u'scores': {u'midterm_4': 95, u'midterm_1': 100, u'midterm_2': 50, u'midterm_3': 70}}<jupyter_text>Let's make a few notes. Firstly, we use the keyword `$set` to set the key `'scores.midterm_4'` to value `95`. Also, we need to pass the parameter upsert=True to indicate that we wish to add this field if it doesn't exist already.Now you have received a great homework grade for 15-388: 100%! You are anxious to add this score, but unfortunately, we have lost our list of id's!<jupyter_code>id_objects = None
ids = []<jupyter_output><empty_output><jupyter_text>We will have to find the document using only the information that the `course` is `15-388`. Fortunately, we have an alternate way to select documents!<jupyter_code># print the document
print grades.find_one({"course": '15-388'})

# update the document
grades.update_one({"course": '15-388'}, {"$set": {'scores.homework_1': 100}}, upsert=True)

# print newline and the updated document
print
print grades.find_one({"course": '15-388'})<jupyter_output>{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc2'), u'scores': {u'midterm_1': 90, u'midterm_2': 70, u'midterm_3': 80}}

{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc2'), u'scores': {u'homework_1': 100, u'midterm_1': 90, u'midterm_2': 70, u'midterm_3': 80}}<jupyter_text>You notice there's an error! In your database, your midterm 2 score in 15-388 is a 70%, but you actually got a 90%. You're going to need to change that! To change a value, we use the same syntax as adding a value. The only difference is we don't need `upsert=True`, and so we will eliminate this parameter (`upsert=False` by default).<jupyter_code># update the document
grades.update_one({"course": '15-388'}, {"$set": {'scores.midterm_2': 90}})

# print the updated document
print grades.find_one({"course": '15-388'})<jupyter_output>{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc2'), u'scores': {u'homework_1': 100, u'midterm_1': 90, u'midterm_2': 90, u'midterm_3': 80}}<jupyter_text>Finally, we will show you how to remove a document.You don't like 15-150, and so you are electing to drop the course. You want to remove it from your schedule.To remove a document, we use the `.delete_one()` function to remove one document or `.delete_many()` to remove several documents. Since we only want to remove one document, we will simply use `.delete_one()`.<jupyter_code># remove the document
grades.delete_one({"course": '15-150'})

# print the updated document
for course in grades.find():
    print course<jupyter_output>{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc2'), u'scores': {u'homework_1': 100, u'midterm_1': 90, u'midterm_2': 90, u'midterm_3': 80}}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdc3'), u'scores': {u'midterm_4': 95, u'midterm_1': 100, u'midterm_2': 50, u'midterm_3': 70}}<jupyter_text>And we've removed the course!  Operations in MongoDBIt's midsemester and you want to calculate your midsemester grade! To do this, you'll need to understand a little about operations in MongoDB.Let's start by averaging midterm scores for each class. Of course, we can select each class as a python object and iterate through them.<jupyter_code>avgs = {}
for class_ in grades.find():
    total = 0
    count = 0
    for _, score in class_['scores'].iteritems():
        total += score
        count += 1
    avgs[class_['course']] = float(total) / count
print avgs<jupyter_output>{u'15-388': 90.0, u'15-213': 78.75}<jupyter_text>But we have to write our own code, and this is inefficient. As a better solution, we introduce the Aggregation Pipeline.  The Aggregation PipelineHere we introducte the topic of the *aggregation*. To simplify this transition, we will define a new database with the same information stored differently. This database will allow for a more intuitive explanation of the concept of the *aggregation pipeline*.<jupyter_code># create collection grades
grades_by_test = db.grades_by_test

# make sure grades is empty, so we can run this function multiple times
grades_by_test.delete_many({})

# add entries, initialize list of id's of objects
classes_by_test_file = open('classes-by-test.json', 'r')
id_objects = grades_by_test.insert_many(json.load(classes_by_test_file))
ids = id_objects.inserted_ids
classes_by_test_file.close()

# analogous to "SELECT * FROM classes;"
grades_by_test_list = grades_by_test.find()

# print all grades
for grade in grades_by_test_list:
    print grade<jupyter_output>{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc5'), u'exam_name': u'midterm_1', u'score': 90}
{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc6'), u'exam_name': u'midterm_2', u'score': 90}
{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc7'), u'exam_name': u'midterm_3', u'score': 80}
{u'course': u'15-388', u'_id': ObjectId('581d4251d64511b506f7cdc8'), u'exam_name': u'homework_1', u'score': 100}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdc9'), u'exam_name': u'midterm_1', u'score': 100}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdca'), u'exam_name': u'midterm_2', u'score': 50}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdcb'), u'exam_name': u'midterm_3', u'score': 70}
{u'course': u'15-213', u'_id': ObjectId('581d4251d64511b506f7cdcc'), u'exam_name': u'midterm_4', u'score': 95}<jupyter_text>We will use the aggregation framework. Aggregation is useful when we want to perform very general tasks on a set of data. For example, we can easily take the average of all a student's scores.The aggregation pipeline is a set of *stages*. We begin with our set of documents, and then each *stage* modifies the documents and passes resulting documents to the next stage. In our example we only have one stage, a `$group`. We group documents by `course` and compute the average of each document's `score` field. Note that this example is intentionally simple, and we go over a more in-depth example with multiple stages in the next section.Note that a full list of stages can be found [here](https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/ "MongoDB Aggregate Stages")<jupyter_code>pipeline = [{
        '$group': {
            "_id": "$course",
            "avg_score": {
                "$avg":"$score"
            }
        }
}]

print pretty_print(list(grades_by_test.aggregate(pipeline)))<jupyter_output>[
  {
    "_id": "15-213",
    "avg_score": 78.75
  },
  {
    "_id": "15-388",
    "avg_score": 90.0
  }
]<jupyter_text>Let's walk through a more difficult example. We begin by constructing a new database that is more easy to analyze. We have three students each taking several classes.<jupyter_code># create collection grades
students = db.students

# make sure grades is empty, so we can run this function multiple times
students.delete_many({})

# add entries, initialize list of id's of objects
students_file = open('students.json', 'r')
id_objects = students.insert_many(json.load(students_file))
ids = id_objects.inserted_ids
students_file.close()

# analogous to "SELECT * FROM classes;"
students_list = students.find()

# print all grades
for student in students_list:
    print student<jupyter_output>{u'courses': [{u'homeworks': [95, 100, 93], u'course': u'15-388', u'midterms': [90, 85, 95]}, {u'homeworks': [90, 80, 85], u'course': u'15-213', u'midterms': [80, 97, 85]}], u'_id': ObjectId('581d4251d64511b506f7cdcd'), u'name': u'<NAME>'}
{u'courses': [{u'homeworks': [80, 100, 90], u'course': u'15-388', u'midterms': [70, 100, 92]}, {u'homeworks': [84, 96, 80], u'course': u'15-213', u'midterms': [96, 94, 80]}], u'_id': ObjectId('581d4251d64511b506f7cdce'), u'name': u'<NAME>'}
{u'courses': [{u'homeworks': [100, 95, 97], u'course': u'15-150', u'midterms': []}], u'_id': ObjectId('581d4251d64511b506f7cdcf'), u'name': u'<NAME>'}<jupyter_text>We want to calculate each student's grade in each class. We begin with an `$unwind` stage. Unwind creates a document for each element of a list within a document. Using unwind, we pass the path to the `courses` list and create a new document for each course an individual is in. Further information on `$unwind` is available [here](https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/).Note that we are completing each stage of the aggregation pipeline separately to demonstrate the effect of each stage individually.<jupyter_code>pipeline = [{
        '$unwind': {
            "path": "$courses"
        }
}]

# special print function removes id
print pretty_print_remove_id(students, pipeline)<jupyter_output>[
  {
    "courses": {
      "course": "15-388",
      "homeworks": [
        95,
        100,
        93
      ],
      "midterms": [
        90,
        85,
        95
      ]
    },
    "name": "<NAME>"
  },
  {
    "courses": {
      "course": "15-213",
      "homeworks": [
        90,
        80,
        85
      ],
      "midterms": [
        80,
        97,
        85
      ]
    },
    "name": "<NAME>"
  },
  {
    "courses": {
      "course": "15-388",
      "homeworks": [
        80,
        100,
        90
      ],
      "midterms": [
        70,
        100,
        92
      ]
    },
    "name": "<NAME>"
  },
  {
    "courses": {
      "course": "15-213",
      "homeworks": [
        84,
        96,
        80
      ],
      "midterms": [
        96,
        94,
        80
      ]
    },
    "name": "<NAME>"
  },
  {
    "courses": {
      "course": "15-150",
      "homeworks": [
        100,
        95,
        97
      ],
      "midterms": []
    },
  [...]<jupyter_text>We then use a `$project` stage, which is able to pass along a subset of data or perform computations on data. We construct a dictionary of what we want and do not want to include. We can create a new field with key, value `: , or we can select certain information to include with `: True` (or, equivalently, `: 1`). We can also choose to remove the id element with `_id: False`. Note that this is the only element we can remove.<jupyter_code>pipeline = [{
        '$unwind': {
            "path": "$courses"
        }}, {
        '$project': {
            'homework_avg': {'$avg': '$courses.homeworks'},
            'midterm_avg': {'$avg': '$courses.midterms'},
            'course': '$courses.course',
            'name': True,
            '_id': False
        }
}]

print pretty_print(list(students.aggregate(pipeline)))<jupyter_output>[
  {
    "course": "15-388",
    "homework_avg": 96.0,
    "midterm_avg": 90.0,
    "name": "<NAME>"
  },
  {
    "course": "15-213",
    "homework_avg": 85.0,
    "midterm_avg": 87.33333333333333,
    "name": "<NAME>"
  },
  {
    "course": "15-388",
    "homework_avg": 90.0,
    "midterm_avg": 87.33333333333333,
    "name": "<NAME>"
  },
  {
    "course": "15-213",
    "homework_avg": 86.66666666666667,
    "midterm_avg": 90.0,
    "name": "<NAME>"
  },
  {
    "course": "15-150",
    "homework_avg": 97.33333333333333,
    "midterm_avg": null,
    "name": "<NAME>"
  }
]<jupyter_text>We then group the documents back together. `$group` takes a field `_id` which is what you want the elements to be grouped by. In this case, we group by `$name`. We then construct some formula to combine all documents with this `_id`. In this case, we simply want to copy all fields. We use the `$push` operation, which creates a list and pushes all values (given by `$push: `) onto the list.<jupyter_code>pipeline = [{
        '$unwind': {
            "path": "$courses"
        }}, {
        '$project': {
            'homework_avg': {'$avg': '$courses.homeworks'},
            'midterm_avg': {'$avg': '$courses.midterms'},
            'course': '$courses.course',
            'name': True,
            '_id': False
        }}, {
        '$group': {
            '_id': '$name',
            'courses': {
                '$push': {'course': '$course', 'homework_avg': '$homework_avg', 'midterm_avg': '$midterm_avg'}
            }
        }
}]

print pretty_print(list(students.aggregate(pipeline)))<jupyter_output>[
  {
    "_id": "<NAME>",
    "courses": [
      {
        "course": "15-150",
        "homework_avg": 97.33333333333333,
        "midterm_avg": null
      }
    ]
  },
  {
    "_id": "<NAME>",
    "courses": [
      {
        "course": "15-388",
        "homework_avg": 90.0,
        "midterm_avg": 87.33333333333333
      },
      {
        "course": "15-213",
        "homework_avg": 86.66666666666667,
        "midterm_avg": 90.0
      }
    ]
  },
  {
    "_id": "<NAME>",
    "courses": [
      {
        "course": "15-388",
        "homework_avg": 96.0,
        "midterm_avg": 90.0
      },
      {
        "course": "15-213",
        "homework_avg": 85.0,
        "midterm_avg": 87.33333333333333
      }
    ]
  }
]<jupyter_text>We were able to compute each students' homework average and midterm average with just the aggregation pipeline. However, more complex operations often require the more versatile `map_reduce` function.  Map_ReduceIn this section we will discuss `map_reduce` in MongoDB. This is a method to aggregate large amounts of data into some convenient representation. Note that map and reduce must be JavaScript functions, and so this section will require some knowledge of JavaScript. [Here](https://developer.mozilla.org/en-US/docs/Web/JavaScript "JavaScript Documentation") is a link to JavaScript Docs, which includes a good tutorial that can be found [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide "JavaScript Guide"). I also personally endorse [Codecademy's JavaScript tutorial](https://www.codecademy.com/learn/javascript "Codecademy JavaScript") as a brief introduction to the language.We first discuss the **map** step of this process. In general, *mapping* is a concept in functional programming where we map every element of some list L to some element in a new list L'. In MongoDB, this is achieved by *emitting* key-value pairs for each element in the list L. Note that each element of list L can emit an arbitrary number of key, value pairs (including 0). After emitting, each key `k` has a corresponding list `L` of values `[l1, ..., ln]` such that for each `li` (`k`, `li`) was emitted by map.We construct a function `mapper` that will be given each document as input. We can reference the document with the keyword `this`.After mapping, we **reduce** our key-value pairs. We use a function that processes the accumulated list `L` and returns some value `v` that will be associated with each key.Let's begin writing our aggregation as a `map_reduce`! We first want to select which key-value pair to emit. `course`, `score` makes sense because a list of scores associated with each course could simply be averaged after the map step. We can then reduce by simply computing the average for each list `L`. Don't worry too much about the JavaScript syntax.<jupyter_code>from bson.code import Code

# emits course, score
mapper = Code("""
              function() {
                emit(this.course, this.score);
              }
              """)

# computes average of each list of test scores
reducer = Code("""
               function(key, values) {
                 var total = 0;
                 var length = values.length;
                 
                 for (var i = 0; i < length; i++) { //compute average
                   total += values[i];
                 }
                 
                 return total / length;
               }
               """)

# mapreduce to find averages
averages = grades_by_test.map_reduce(mapper, reducer, "averages")

for average in averages.find():
    print average<jupyter_output>{u'_id': u'15-213', u'value': 78.75}
{u'_id': u'15-388', u'value': 90.0}<jupyter_text>And we have a list of averages, like before. What's more interesting though is that `map_reduce` can be used on our original dataset! Let's try this.<jupyter_code># emits course, score
mapper = Code("""
              function() {
                for (var key in this.scores) { //iterate through dictionary and emit course, score
                  emit(this.course, this.scores[key]);
                }
              }
              """)

# mapreduce to find averages
averages_2 = grades.map_reduce(mapper, reducer, "averages_2")

for average in averages_2.find():
    print average<jupyter_output>{u'_id': u'15-213', u'value': 78.75}
{u'_id': u'15-388', u'value': 90.0}<jupyter_text>Notice that our map function on this dataset yielded the same result as last time, so we can use the same reduce function.  Indexing and Text SearchNow you want to lookup Bob's schedule and grades. But you forgot his last name! You'll need some kind of search feature.To enable search, we first must *index* our data. We assign each document a unique index, defined by one or more fields of each document. In this case, we choose to index on `name`. Note that we must specify the type on which we index (in this case `text`).More information such as weighting text fields is available [here](https://docs.mongodb.com/v3.2/core/index-text/ "MongoDB Text Indexes").<jupyter_code>result = students.create_index([('name', 'text')])

for student in students.find():
    print student<jupyter_output>{u'courses': [{u'homeworks': [95, 100, 93], u'course': u'15-388', u'midterms': [90, 85, 95]}, {u'homeworks': [90, 80, 85], u'course': u'15-213', u'midterms': [80, 97, 85]}], u'_id': ObjectId('581d4251d64511b506f7cdcd'), u'name': u'<NAME>'}
{u'courses': [{u'homeworks': [80, 100, 90], u'course': u'15-388', u'midterms': [70, 100, 92]}, {u'homeworks': [84, 96, 80], u'course': u'15-213', u'midterms': [96, 94, 80]}], u'_id': ObjectId('581d4251d64511b506f7cdce'), u'name': u'<NAME>'}
{u'courses': [{u'homeworks': [100, 95, 97], u'course': u'15-150', u'midterms': []}], u'_id': ObjectId('581d4251d64511b506f7cdcf'), u'name': u'<NAME>'}<jupyter_text>Using our find method, we can now search the data using the following syntax.<jupyter_code>search_result = students.find({"$text": {"$search": 'Bob'}})
for student in search_result:
    print student<jupyter_output>{u'courses': [{u'homeworks': [80, 100, 90], u'course': u'15-388', u'midterms': [70, 100, 92]}, {u'homeworks': [84, 96, 80], u'course': u'15-213', u'midterms': [96, 94, 80]}], u'_id': ObjectId('581d4251d64511b506f7cdce'), u'name': u'<NAME>'}
{'fraction_non_alphanumeric': 0.12837138992759964, 'fraction_numerical': 0.054857188320471, 'mean_word_length': 3.8832556332556334, 'pattern_counts': {'":': 101, '<': 79, '<?xml version=': 0, '>': 78, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '491490', 'license': 'MIT', 'n_tokens_mistral': 8978, 'n_tokens_neox': 7904, 'n_words': 2789, 'path': '2016/tutorial_final/9/Document-Oriented Database and MongoDB Tutorial.ipynb', 'repo_name': 'zeromtmu/practicaldatascience.github.io', 'size': '25235'}