Data Analysis and Map-Reduce with mongoDB and pymongo
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Hendorf, Alexander
Alexander Hendorf - Data Analysis and Map-Reduce with mongoDB and pymongo The MongoDB aggregation framework provides a means to calculate aggregated values without having to use map-reduce. While map-reduce is powerful, it is often more difficult than necessary for many simple aggregation tasks, such as totaling or averaging field values. See how to use the build-in data-aggregation-pipelines for averages, summation, grouping, reshaping. See how to work with documents, sub- documents, grouping by year, month, day, etc. This talk will give many (live) examples how to make the most of your data with pymongo with a few lines of code.
 Thank you Alexander. I'm from Mannheim in Germany. I'm a developer of my own company. I'm an organizer as well. I'm a speaker, sometimes a MongoDB trainer for our local community and for the Python community. I've served as a program workgroup co-chair building this conference. So if you have any comments, suggestions, what we could do better or so, I'm around. Just grab talk to me. I'm very interested in your intro. My talk today is MongoDB and what's the MongoDB aggregation framework. We're going to cover the pipeline bottle, the pipeline stages and map reduce in MongoDB. And at first, who knows MongoDB? Just, okay, awesome. And who's actually working with MongoDB? Okay. And who has worked with the MongoDB aggregation framework? Okay. Cool. Okay. So let's bring everybody up to speed with orientated document, okay, orientated databases and set 50 seconds. Basically, we work with the document as a JSON-like object. We can store it in our database. We have no shamer enforcement. A collection is basically just a collection of documents actually and multiple collections make up our database. So it's a pretty simple concept. The MongoDB aggregation framework was introduced about like three years ago with MongoDB 2.2. It's a framework for data aggregation. Basically, the documents are processed through a multiple stage pipeline and it's giving aggregated results. It's basically designed to be a design to work straightforward. So no unions like in SQL. And technically, this looks like this. So we have our documents. We do a match, which is a find. We get less documents because we found some, the cell phone set, and then we do some grouping and we get even less. But actually, I thought it's a little bit too technical. So basically, I think it's more like a relay race. You know, like relay racers, they go and they pass the button to each other and basically, this is how the MongoDB pipeline works. So we have our match, which is find. So we say to please, little doggy, get the button. Please pass it on to the smart fox doing something smart, which could be like a grouping. And then we want to present our data a little bit more nicely. And so we pass it on to the projection space. And let me tell you a little bit about the data set we're going to work. I'm going to do present the things and what we're doing. And I've also prepared some live demos. And this is built with MongoDB. Yes. We're using MongoDB 3.0. The new wide-tired Tiger storage engine with compression, PyMongo is obviously the driver we are going to use. It's maintained by MongoDB themselves. And it's pretty well maintained driver. It's always up to date. It's really good. And we're working on a data set of 37 gigabytes, which gives us compressible wide-tiger about 9 gigabytes of RAM. Basically, as you might remember, IT is my second career. I used to be in the record industry with a technical house startup business in the 90s. So everything I do still in IT is very close to working with music. And so we have, from a project we're doing, it's called ChartGuys. We have some collection of a playlist from the Aikons music store. A playlist is basically all the information about the release you can find in the Aikons music store. It's a set of playlists that appeared in some charts somewhere around the world within the last three years. And basically, this is what it looks like. So pretty cool. So don't worry. I've narrowed it down to what we're going to work with today just to give you an impression about our document structure for the demos. So basically, this is a document. An info is all the release information. So like the album artist, the album name when it was released, how much is it in store. And the children, it's what we call a sub document. It's a list with objects. And that's basically the songs, each and every song we have in our playlist. And I was wondering, actually, which artists to use for my demos, because it's really hard to choose music artists making everybody happy. And I thought I found something neutral because I chose Taylor Swift. And it's not because I like her music. Actually, I don't know any songs of her, but she did this great blog post making Apple pay for the trial period for the new Apple music servers. So artists get paid more money for people using a new service by Apple. So she did a good thing. And I think that's really worth mentioning her even at EuroPython. So, okay. So let's build our first pipeline. I've commented in some notes for the SQL guys to make it easier. So basically, this is a pipeline. A pipeline is basically passed in as a list into PyMonga. And match is just basically a find, as you might remember from our document. It's an artist name. So we were looking for the artist, which is a variable I've already stored as Taylor Swift, of course. Then we're going to do a project that basically it's a select and basically all we want to do is print out the, yeah, all the releases by Taylor Swift sorted. So then we switch to this. And go here. So basically it's just an import. We import PyMonga. This is just like a simple database connection. And so let's see our database, which is live on this MacBook. I must say it's only assigned two gigabytes of RAM for this database. So it's not usually we work with a lot more RAM in MongoDB. So we have 1.3 million playlists found. And it's about like 17 million songs covered in our data set. And usually you could like with your match, it could also like just like a query and our query says, okay, we found 40, 93 releases of Taylor Swift. So with the aggregation framework, that's the same code I've shown you on the slide before. We do a match find, project. Basically, just like we're just projecting here. It's just like a renaming of the attribute actually and then resort by release ascending order. And basically that's looking like this. Okay, you see we have many releases. She's quite busy artist, famous karaoke. Okay, so we have a lot of duplicates. And what else can we do? We can extend our pipeline. We can do a grouping. So now I want to group everything by name, which is basically the album tab. As you probably see, we have a lot of duplicates. You have done some duplicates in our dataset, which is because albums are released by different companies worldwide. So at the iTunes store, they get a new ID. They're basically different products, although it's the same contents from the music. So the passing in, the name as underscore ID. Underscore ID is in the grouping operator, basically, what we want to group by. It's mandatory and it's always called underscore ID. And we want to count how many albums are there. Account, we don't have the count operator in the aggregation pipelines. So basically we're just summing one for each and every document in our group. And then we project and sort just to make it a little bit more nice. And this is what we get. We still have some different versions. Okay. So, okay, now we have this nice pipeline. We've got results. And I think it's so nice. Let's print out what we found again. And we get this. And it's so nice. I just want to print it again. And oops, what happens? I just actually just rest is where we stored our query to the aggregation framework. And I just wanted to print it again. And what's happening here? Why doesn't it give us any result back? That's like the first track, I want to show you is MongoDB aggregation framework. It returns a cursor. So basically the cursor, it points to the data in the database. You get back to from the MongoDB aggregation. So once we call list, the cursor is exhausted. So all the data in and then they're printed and then they're gone. So you can't just use them again unless of course you stored them in a new variable. All right. So these are all our aggregation stages. I've put the like their SQL brothers on the right-hand side. So basically a match is where our having operator sort. Pretty obvious is order by limit. It's also, I think, no explanation necessary project is a select. We can also use it for renaming as an SQL form. Our result group is group by unwind. And we're going to go into that very soon. It's somehow a little bit of a join, not really. And redact, we're not going to cover. And out is basically just an operator. Please send the result of the aggregation back to a new collection of MongoDB to store it. So to make things a little bit easier for you to fall, in the next we're going to do something with the artist name and name is the album title. And we're making our pipeline a little bit even bigger, like with the group operator, what I've already shown you already. This is how it looks like. Print out a little bit more nicely. And next step is how can we work with lists of soft op human. So as you see, we have a list here, just with all the songs on that album. And we want to do something with it. And of course, the natural thing would be I query the database and just iterate over it with MongoDB Python. But that's quite an extensive task. We can do it in the database and there's this unwind operator. And it's basically from my experience, it's at first a really confusing step because it's quite unusual for what I've seen. So it confuses people. And so I think just like, let's just chill you. I think it's probably the best explanation because what unwind does is basically take all the sub documents in the list and for each object in our sub document list, it creates a new document. And this sounds really like an expensive operation, but I can assure you that I did a really good job and it is not expensive at all. It's really handy and basically, let me show you, this is like what we're doing now. So, and of course, I'm really sorry. Let's go to that later. Here we are. I'm sorry. So, let's do this. So, now we have all three hundred and thirty-two songs by Taylor Swift. Found them. We can immediately work with them here in our grouping stage. And as you see, the path has not really changed. Although this used to be a list before and we don't do, there's no need to do anything about like iterating over like a list index or anything like that. And I've prepared a little bit more. This is basically what's happening. So, it's basically we just get one release, limited, we get one playlist and then we do unwind and then I'm just renaming it with the project parameters. This is basically what we're getting. We're getting all these are single documents, new documents just like rated on the fly. We can immediately work with. So, basically, it's basically like just yeah, it's an unwinding of the data. It's a little bit unusual concept, but it's basically really simple. So, but, okay. So, let's go back up. Okay, another one, which is quite obvious one. Okay, that's, we have also like a sort, which is also like an obvious pipeline stage. And I want all the releases just sorted by count, descending and release ascending. And basically, it looks really simple. And it returns us something like this. And what's going wrong? Something's wrong. Because we said I want by count descending over all our data. And then I want to have it sorted by release and ascending order. But our result is basically by release and then by count. So, something's going wrong here. And I can assure you it's not broken. It's actually like a trap because in the pipeline, we pass in a Python dictionary. And the Python dictionary, of course, is in unsorted. And of course, so we just pass in something which is not ordered. And of course, our results get a little bit unpredictable. But of course, that's like, this is a solution. And I can encourage you always to use some from the BZON collection. Or you can also use collection order and pass in all sort of parameters in an ordered fashion. Because otherwise, your sorting order won't really work. And so this, oh, wow, it works. Okay. So this was just like a really quick introduction to our stages. There's a lot of pages mentioned before. It's a skip. It's just like skipping documents out, write your results to a new collection. There's a Neogear which just gives you all the documents around geospatial point. Redactors, I don't know. Some people use it to restrict document access on a document level. I've never really seen it in production. And these are like the stages. This is like our race. And now we have some data. And basically, this is very limited from what we can do. Basically, it's just like mangling around a little bit with the data. So we have more. Of course, there's like a minimum, a maximum, first and last operator. And this is what we're going to work on. Again, we're searching for an artist. We're using release date and a release date epoch. And the subtle distance is that a release date epoch is actually a date. And the release date is string. It's no date. It's a string. And we're building a new pipeline. We're doing no grouping by what we want to find out. I want to find out what's the earliest release of Taylor Swift and what's the latest release of Taylor Swift. So we do a grouping by underscore ID. As you see, underscore ID is empty. It's our new primary key. How can that be empty? Yes, it can be empty because we want a group of our complete result set. So we can just put none in there or leave it empty. So there's no need to look for an attribute which is the same on each and every document. Just leave it empty. And we introduce two new attributes, mandate, max date. And basically it's a really simple operation. We just walk the path info to the information, min, max, and project it. Okay. So let's run that. And yay. Now Taylor Swift is around things 2006. I think she started really early. She's releasing stuff. And she's been around for a while. And so what's first and last good for? I mean, we have min, max. It also would work actually on min, max, would work actually on array. But it's just like a little bit different. And it can save you some extra calculations. What's the difference? The difference to our previous pipeline is we have a match and then we do a sort by release date. And then we do our grouping and our grouping instruction is first and last. And what does first and last do? It's really simple. Get the first document of the group and last is get the last document of the group. So there's no need to iterate over your complete set within the group to find min or max values. Basically you just can say, okay, I want this document. I want to look at this document and what's in the middle. I don't really care. So this can be really effective. And as expected, same results. So and with dates, we can even do more. We have some nice state operators. Pipeline. We do basically the same result by release date. We do a grouping and I want to have releases grouped by year. I'm a fanboy now. I've talked so much about Taylor Swift. I want to really know everything. So I want to have see which year which release. So how many releases per year. Sorry. So we actually extended our ID a little bit. And now it's an object with our dollar year operator. And we pass in the date epoch, which is the date. And we just pass it in. And the dollar year, we'll basically just grab the year from our date. And this is then our ID. We want to group by. And this works like this. It's really easy. Makes it really easy if you have some data with timestamps. And so we see okay. Count. So you see she's like a bee. She's releasing every year. A lot of releases. She's hard worker. And so. But what if I want to dig even deeper. I'm not interested in getting the releases by year. I'm also interested in getting each release count for each and every month. She has released something. And of course, I wouldn't mention it if we couldn't accomplish it. In the year, we also have a month operator. And the next thing what happened is now the ID, which is our primary key, can also be a multi key. And so we have a multi key year month. And basically, we do the same. As before, we get the year and month new attribute. Our ID key has two, it's a builder of two attributes year and month. And let's just run that. And wow. We see. I haven't checked. That's probably not a month. Hardly any month, she didn't do anything. So, well. There's a lot of more data operators. As you can guess, there's also like a second many more data operators. We're not able to cover them all in this small talk. Of course, with that. But I'm getting a little bit bored now with Taylor Swift. Because I have, it's early in the morning and we want some more tension. So actually, I thought about who could and else, who could join. So I thought, hey, I just Google Taylor Swift Nemesis and Google says it's an alien space robot called Katy Perry. And so let's bring in Katy Perry. And it's really easy. We can extend our match operator. So Katy Perry is now stored in our Nemesis variable. And basically, we can also do searches with a dollar in operator. And it's basically just the same as in Python. So I think it's not really necessary to explain to you guys. So, and of course, now we have big competition. I'm wondering who delivers more song value for my 99 cents. Is it Katy Perry? Is it Taylor Swift? So I want to see the average playtime of their songs. I'm interested. Who gives me more songs, longer songs I can enjoy for my money. It's not a good thing, but it's just like a nice example. So what are we doing? As you see, we now have three unwind stages. So basically, the first thing is we unwind the songs. And then we unwind the song offers. The song offers, and within those song offers assets is basically the playtime stored. So we want to access this information. So that's why we have a pipeline of one, two, three unwind. It's unwind, unwind, unwind. And then we can group by just going down the path by the song name, which is a child name. And then we just do an average of the path of the duration we have stored within the assets. And show you like this and something's wrong. Something's really wrong. Okay. Sorry. And just fixed that. Okay. Something's broken. I'm very sorry. Won't waste any time to fix this now live. So basically, what I can explain you, basically, it's just like the same we did before with the releases here. And counting the releases. And the next step, of course, will be getting the playtime. So I hope my notebook didn't break. Yes, it didn't break. Okay. Sorry again. Of course, we have our group, our playtime, and we just projected and as a result, we can see, okay, Taylor Swift gives us more music, like about like 10% more music than Katy Perry for 99 cents. And it's a really easy operation. Okay. Now something, something a little bit more challenging. I'm interested in, thank you, I'm interested in getting the prices of the releases of the artists. And my, it's basically scraped data. So it's not probably as clean as I would wish. Basically, we see a formative price with the currency in front and the price, but it's just like in one attribute. And I'm interested in getting the prices in US dollars. And that's easily to solve with a string operation and a compare operation. So I have to speed up a little bit. Basically, we do a project phase. So just focus on the things in bold. They're important ones here. And we have US dollar. Basically, it's a comparison of the lower strings of the first three characters in our price formative, which gives us back US dollar or some currencies or numbers or whatever. And the comparison is basically, is this US dollars? It's pretty obvious. And then we just do a new match for US dollars zero. Okay. So also feels a little bit wrong, but compare. Parameter gives us zero back when it's a match. And it gives us minus one back if the value is higher and one back if the value is lower. So it's pretty handy one. We could also do, is equal. There's also an equal operator which could give us like one or two back as we expected as a Boolean true false. Then we sort, group, and we can even do something else. We can also go and push every release we find in our group into a new list with a price and a product that's basically very similar to JavaScript. It would be like actually like an append and Python. So let's go here. Back. And here you go. And you see Katy Perry's products. And here's Taylor's next object, Taylor Swift, and the list with all the products. So there's really a lot more operators. And just, I can suggest if you fend the application framework, it's probably useful. Just go to MongoDB documentation. It has a lot of examples. It's quite easy to get into. And one more, it's a variable operator. It's a map operator. And as you can imagine, it's basically the same as a Python map. And what do we do here? We're getting the ratings count, which is actually how many users have given some stalls to the product, one we scraped the data. And we want to adjust it a little bit because our management is our back and we need to make it a little bit, look at it a little bit nicer. What we don't really want to do, but it's just like a good example. So basically, we can pass in to a dollar map and input the ratings count as value. And then we can just reuse the value on our list. And we just add 10 to each and every object in our value we find in our list. And then it's applied. Then another thing, which is not probably obvious, we cannot use some operator on a list like we can do really handy in Python. We have to unwind first. So basically, for each and every value in our list, we have mangled with, we unwinded to a new document and then we can do a simple grouping as we've done before. And yeah. And there we go. Which brings us back, of course, to the next thing. You can also do map reduce in MongoDB. And how many of you guys work with map reduce? Who knows map reduce? Yes. And who actually works a lot with map reduce? Okay. So bring everybody up to speed. Map reduce, basically, it's a really simple concept. We have all these documents. We map them. Map them is basically, we just go through and we find key value pairs, which is actually, in our example, we find to find the most popular words in our release titles. And we just emit them as tuples, as you can see, to the reduce phase, which is run by a producer. In our example, it will just like sum up the counts. It's really, really, really pretty easy operation. Basically, we just will use our name operator. And you might wonder, why would we use map reduce in MongoDB? Because we have this great aggregation framework, you see in substrings, we can do so many things. So what's the point, actually, in using map reduce? And basically, for most of the time, you can work with the aggregation framework. In most of the cases, it's faster. It's more accessible. Thank you. But, however, map reduce gives us more power, because you can actually pass in JavaScript there. You can build much more complex queries. For example, our example was splitting up the release titles in words to count them. This could be quite challenging in the aggregation framework. So let's do it. So, okay, this is map. Okay, let me show you a little bit more. Okay. Okay, this fits. Okay. Sorry, this is our map. And here's our map reduce. So for map reduce, we're just using from B some import code, which we can just pass on text. This is JavaScript function. And it basically stores the name of the info which is stored in the name attribute. It just stores it and splits it. It's a really simple operation. And then we just check for some punctuation and stuff. We move it. It's not the best way to do this. It's just like for the simple example. And if we actually find the word, we emit it. So basically, if there's something like teenage, we emit teenage one. And if the album is called teenage dreams, we can also emit dreams one. We send it to the reducer. And the reducer has really simple code here. We just take all the keys and basically just count, just do some how often the key would actually appear from our emitter. And here we get a result. And this is, and now we're going to do a little bit more because I want to remove stop words, which is not really part of the aggregation framework, just to make it a little bit nicer. That's why I've added the natural language kit, remove stop words. And this is like the most popular words in Katy Perry and Taylor Swift's albums. So you see, they probably have a younger audience with dream and teenage and one and boys and fearless and speak, kissed and stuff. So, yeah, it's really easy. Of course, unfortunately, we don't have enough time left. We could also run this operation across the complete data set and to see what's basically the most popular words in album releases being sold at the iTunes music store. So, to finish, I want to give you some more best practices and tips you can use with the aggregation framework. First of all, database, think about your indexes, especially if you do queries on them. Of course, if you have a huge data set and you don't have an index, MongoDB has a collection scan and if it's a slow computer, it's, of course, taking time and probably frustrating for you. Think about probably getting your data set, your database to your RAM. You can just touch commands in MongoDB, which actually do something like, yeah, similar to Unix touch. You touch it and then it fills up your RAM as much as possible, as much as you will ever get from the system to store data. You can work with life and memory. You have to mind that the result can be only like 60 megabytes because that's the maximum we can store in a BJ's and document, but I mean like 60 megabytes is still huge. Pipeline operation has also a limit of 100B, but you will hardly ever, sounds not much, but you hardly will ever really hit it. On your queries, you can improve your queries up front. There's this nice, sorry for the break here, that's a nice explain operation, which will basically give you information what would MongoDB do when query, doing your query. You get some results, you see how many documents scanned, if indexes were hit, if it was all index, and then you can really go and say, okay, I can really optimize all my work with just like introducing a new index. Hardware is, of course, really important, especially RAM. More is better, and it's really simple equation here. Mind the disk performance, of course SSDs and cloud computing makes it really easy. And yeah, and you can also think about working about a dedicated server in case you have something like a replica set and a right heavy database. So you can also say, okay, just do another copy and work locally and do your aggregations without having to worry about if you have a lot of traffic in your database. And the last slide is some useful resources, of course, as I mentioned, MongoDB has a very good documentation, it's pretty updated by Mongo as well. And I also want to mention Asia Kamski, she works for MongoDB, also as a trainer, and she has always like awesome tricks and tips. And here we go. Thank you. We don't quite have enough time for Q&A, so if you want to ask Alexander questions, then try to find him instead. Yeah, I'm around. Just ask anytime. No problem. Thank you.
 Thank you Alexander. I'm from Mannheim in Germany. I'm a developer of my own company.  I'm an organizer as well. I'm a speaker, sometimes a MongoDB trainer for our local community  and for the Python community. I've served as a program workgroup co-chair building  this conference. So if you have any comments, suggestions, what we could do better or so,  I'm around. Just grab talk to me. I'm very interested in your intro. My talk today is  MongoDB and what's the MongoDB aggregation framework. We're going to cover the pipeline  bottle, the pipeline stages and map reduce in MongoDB. And at first, who knows MongoDB?  Just, okay, awesome. And who's actually working with MongoDB? Okay. And who has worked with  the MongoDB aggregation framework? Okay. Cool. Okay. So let's bring everybody up to  speed with orientated document, okay, orientated databases and set 50 seconds. Basically, we  work with the document as a JSON-like object. We can store it in our database. We have no  shamer enforcement. A collection is basically just a collection of documents actually and  multiple collections make up our database. So it's a pretty simple concept. The MongoDB  aggregation framework was introduced about like three years ago with MongoDB 2.2. It's  a framework for data aggregation. Basically, the documents are processed through a multiple  stage pipeline and it's giving aggregated results. It's basically designed to be a  design to work straightforward. So no unions like in SQL. And technically, this looks like  this. So we have our documents. We do a match, which is a find. We get less documents because  we found some, the cell phone set, and then we do some grouping and we get even less.  But actually, I thought it's a little bit too technical. So basically, I think it's  more like a relay race. You know, like relay racers, they go and they pass the button to  each other and basically, this is how the MongoDB pipeline works. So we have our match,  which is find. So we say to please, little doggy, get the button. Please pass it on to  the smart fox doing something smart, which could be like a grouping. And then we want  to present our data a little bit more nicely. And so we pass it on to the projection space.  And let me tell you a little bit about the data set we're going to work. I'm going to  do present the things and what we're doing. And I've also prepared some live demos. And  this is built with MongoDB. Yes. We're using MongoDB 3.0. The new wide-tired Tiger storage  engine with compression, PyMongo is obviously the driver we are going to use. It's maintained  by MongoDB themselves. And it's pretty well maintained driver. It's always up to date.  It's really good. And we're working on a data set of 37 gigabytes, which gives us compressible  wide-tiger about 9 gigabytes of RAM. Basically, as you might remember, IT is my second career.  I used to be in the record industry with a technical house startup business in the 90s.  So everything I do still in IT is very close to working with music. And so we have, from  a project we're doing, it's called ChartGuys. We have some collection of a playlist from  the Aikons music store. A playlist is basically all the information about the release you  can find in the Aikons music store. It's a set of playlists that appeared in some charts  somewhere around the world within the last three years. And basically, this is what it  looks like. So pretty cool. So don't worry. I've narrowed it down to what we're going  to work with today just to give you an impression about our document structure for the demos.  So basically, this is a document. An info is all the release information. So like the  album artist, the album name when it was released, how much is it in store. And the children,  it's what we call a sub document. It's a list with objects. And that's basically the songs,  each and every song we have in our playlist. And I was wondering, actually, which artists  to use for my demos, because it's really hard to choose music artists making everybody happy.  And I thought I found something neutral because I chose Taylor Swift. And it's not because  I like her music. Actually, I don't know any songs of her, but she did this great blog  post making Apple pay for the trial period for the new Apple music servers. So artists  get paid more money for people using a new service by Apple. So she did a good thing.  And I think that's really worth mentioning her even at EuroPython. So, okay. So let's  build our first pipeline. I've commented in some notes for the SQL guys to make it easier.  So basically, this is a pipeline. A pipeline is basically passed in as a list into PyMonga.  And match is just basically a find, as you might remember from our document. It's an  artist name. So we were looking for the artist, which is a variable I've already stored as  Taylor Swift, of course. Then we're going to do a project that basically it's a select  and basically all we want to do is print out the, yeah, all the releases by Taylor Swift  sorted. So then we switch to this. And go here. So basically it's just an import. We  import PyMonga. This is just like a simple database connection. And so let's see our  database, which is live on this MacBook. I must say it's only assigned two gigabytes  of RAM for this database. So it's not usually we work with a lot more RAM in MongoDB. So  we have 1.3 million playlists found. And it's about like 17 million songs covered in our  data set. And usually you could like with your match, it could also like just like a  query and our query says, okay, we found 40, 93 releases of Taylor Swift. So with the aggregation  framework, that's the same code I've shown you on the slide before. We do a match find,  project. Basically, just like we're just projecting here. It's just like a renaming of the attribute  actually and then resort by release ascending order. And basically that's looking like this.  Okay, you see we have many releases. She's quite busy artist, famous karaoke. Okay, so  we have a lot of duplicates. And what else can we do? We can extend our pipeline. We can do a  grouping. So now I want to group everything by name, which is basically the album tab. As you  probably see, we have a lot of duplicates. You have done some duplicates in our dataset, which  is because albums are released by different companies worldwide. So at the iTunes store, they  get a new ID. They're basically different products, although it's the same contents from the music.  So the passing in, the name as underscore ID. Underscore ID is in the grouping operator,  basically, what we want to group by. It's mandatory and it's always called underscore ID. And we  want to count how many albums are there. Account, we don't have the count operator in the aggregation  pipelines. So basically we're just summing one for each and every document in our group. And then  we project and sort just to make it a little bit more nice. And this is what we get. We still have  some different versions. Okay. So, okay, now we have this nice pipeline. We've got results. And I  think it's so nice. Let's print out what we found again. And we get this. And it's so nice. I just  want to print it again. And oops, what happens? I just actually just rest is where we stored our  query to the aggregation framework. And I just wanted to print it again. And what's happening  here? Why doesn't it give us any result back? That's like the first track, I want to show you is  MongoDB aggregation framework. It returns a cursor. So basically the cursor, it points to the data in  the database. You get back to from the MongoDB aggregation. So once we call list, the cursor is  exhausted. So all the data in and then they're printed and then they're gone. So you can't just  use them again unless of course you stored them in a new variable. All right. So these are all our  aggregation stages. I've put the like their SQL brothers on the right-hand side. So basically a  match is where our having operator sort. Pretty obvious is order by limit. It's also, I think, no  explanation necessary project is a select. We can also use it for renaming as an SQL form. Our  result group is group by unwind. And we're going to go into that very soon. It's somehow a little bit  of a join, not really. And redact, we're not going to cover. And out is basically just an operator.  Please send the result of the aggregation back to a new collection of MongoDB to store it. So to  make things a little bit easier for you to fall, in the next we're going to do something with the  artist name and name is the album title. And we're making our pipeline a little bit even bigger,  like with the group operator, what I've already shown you already. This is how it looks like.  Print out a little bit more nicely. And next step is how can we work with lists of soft op  human. So as you see, we have a list here, just with all the songs on that album. And we want to do  something with it. And of course, the natural thing would be I query the database and just  iterate over it with MongoDB Python. But that's quite an extensive task. We can do it in the  database and there's this unwind operator. And it's basically from my experience, it's at first  a really confusing step because it's quite unusual for what I've seen. So it confuses people. And  so I think just like, let's just chill you. I think it's probably the best explanation because  what unwind does is basically take all the sub documents in the list and for each object in our  sub document list, it creates a new document. And this sounds really like an expensive operation,  but I can assure you that I did a really good job and it is not expensive at all. It's really  handy and basically, let me show you, this is like what we're doing now. So, and of course, I'm  really sorry. Let's go to that later. Here we are. I'm sorry. So, let's do this. So, now we have  all three hundred and thirty-two songs by Taylor Swift. Found them. We can immediately work with  them here in our grouping stage. And as you see, the path has not really changed. Although this  used to be a list before and we don't do, there's no need to do anything about like iterating over  like a list index or anything like that. And I've prepared a little bit more. This is basically  what's happening. So, it's basically we just get one release, limited, we get one playlist and then  we do unwind and then I'm just renaming it with the project parameters. This is basically what  we're getting. We're getting all these are single documents, new documents just like rated on the  fly. We can immediately work with. So, basically, it's basically like just yeah, it's an unwinding  of the data. It's a little bit unusual concept, but it's basically really simple. So, but, okay.  So, let's go back up. Okay, another one, which is quite obvious one. Okay, that's, we have also  like a sort, which is also like an obvious pipeline stage. And I want all the releases just sorted  by count, descending and release ascending. And basically, it looks really simple. And it  returns us something like this. And what's going wrong? Something's wrong. Because we said I want  by count descending over all our data. And then I want to have it sorted by release and ascending  order. But our result is basically by release and then by count. So, something's going wrong here.  And I can assure you it's not broken. It's actually like a trap because in the pipeline,  we pass in a Python dictionary. And the Python dictionary, of course, is in unsorted. And of  course, so we just pass in something which is not ordered. And of course, our results get a little  bit unpredictable. But of course, that's like, this is a solution. And I can encourage you always  to use some from the BZON collection. Or you can also use collection order and pass in all sort  of parameters in an ordered fashion. Because otherwise, your sorting order won't really work.  And so this, oh, wow, it works. Okay. So this was just like a really quick introduction to  our stages. There's a lot of pages mentioned before. It's a skip. It's just like skipping  documents out, write your results to a new collection. There's a Neogear which just gives  you all the documents around geospatial point. Redactors, I don't know. Some people use it to  restrict document access on a document level. I've never really seen it in production. And these  are like the stages. This is like our race. And now we have some data. And basically, this is very  limited from what we can do. Basically, it's just like mangling around a little bit with the data.  So we have more. Of course, there's like a minimum, a maximum, first and last operator. And this is  what we're going to work on. Again, we're searching for an artist. We're using release date and a  release date epoch. And the subtle distance is that a release date epoch is actually a date. And  the release date is string. It's no date. It's a string. And we're building a new pipeline. We're  doing no grouping by what we want to find out. I want to find out what's the earliest release of  Taylor Swift and what's the latest release of Taylor Swift. So we do a grouping by underscore ID.  As you see, underscore ID is empty. It's our new primary key. How can that be empty? Yes, it can  be empty because we want a group of our complete result set. So we can just put none in there or  leave it empty. So there's no need to look for an attribute which is the same on each and every  document. Just leave it empty. And we introduce two new attributes, mandate, max date. And basically  it's a really simple operation. We just walk the path info to the information, min, max, and  project it. Okay. So let's run that. And yay. Now Taylor Swift is around things 2006. I think she  started really early. She's releasing stuff. And she's been around for a while. And so what's  first and last good for? I mean, we have min, max. It also would work actually on min, max,  would work actually on array. But it's just like a little bit different. And it can save you some  extra calculations. What's the difference? The difference to our previous pipeline is we have a  match and then we do a sort by release date. And then we do our grouping and our grouping  instruction is first and last. And what does first and last do? It's really simple. Get the first  document of the group and last is get the last document of the group. So there's no need to  iterate over your complete set within the group to find min or max values. Basically you just can  say, okay, I want this document. I want to look at this document and what's in the middle. I don't  really care. So this can be really effective. And as expected, same results. So and with dates,  we can even do more. We have some nice state operators. Pipeline. We do basically the same  result by release date. We do a grouping and I want to have releases grouped by year. I'm a fanboy  now. I've talked so much about Taylor Swift. I want to really know everything. So I want to have  see which year which release. So how many releases per year. Sorry. So we actually extended our ID  a little bit. And now it's an object with our dollar year operator. And we pass in the date  epoch, which is the date. And we just pass it in. And the dollar year, we'll basically just grab the  year from our date. And this is then our ID. We want to group by. And this works like this. It's  really easy. Makes it really easy if you have some data with timestamps. And so we see okay. Count.  So you see she's like a bee. She's releasing every year. A lot of releases. She's hard worker. And  so. But what if I want to dig even deeper. I'm not interested in getting the releases by year. I'm  also interested in getting each release count for each and every month. She has released something.  And of course, I wouldn't mention it if we couldn't accomplish it. In the year, we also have a month  operator. And the next thing what happened is now the ID, which is our primary key, can also be a  multi key. And so we have a multi key year month. And basically, we do the same. As before, we get the  year and month new attribute. Our ID key has two, it's a builder of two attributes year and month. And  let's just run that. And wow. We see. I haven't checked. That's probably not a month. Hardly any  month, she didn't do anything. So, well. There's a lot of more data operators. As you can guess,  there's also like a second many more data operators. We're not able to cover them all in this small  talk. Of course, with that. But I'm getting a little bit bored now with Taylor Swift. Because  I have, it's early in the morning and we want some more tension. So actually, I thought about who  could and else, who could join. So I thought, hey, I just Google Taylor Swift Nemesis and Google says  it's an alien space robot called Katy Perry. And so let's bring in Katy Perry. And it's really  easy. We can extend our match operator. So Katy Perry is now stored in our Nemesis variable. And  basically, we can also do searches with a dollar in operator. And it's basically just the same as in  Python. So I think it's not really necessary to explain to you guys. So, and of course, now we have  big competition. I'm wondering who delivers more song value for my 99 cents. Is it Katy Perry?  Is it Taylor Swift? So I want to see the average playtime of their songs. I'm interested. Who gives  me more songs, longer songs I can enjoy for my money. It's not a good thing, but it's just like  a nice example. So what are we doing? As you see, we now have three unwind stages. So basically,  the first thing is we unwind the songs. And then we unwind the song offers. The song offers,  and within those song offers assets is basically the playtime stored. So we want to access this  information. So that's why we have a pipeline of one, two, three unwind. It's unwind,  unwind, unwind. And then we can group by just going down the path by the song name, which is  a child name. And then we just do an average of the path of the duration we have stored within the  assets. And show you like this and something's wrong. Something's really wrong. Okay. Sorry.  And just fixed that. Okay. Something's broken. I'm very sorry. Won't waste any time to fix this  now live. So basically, what I can explain you, basically, it's just like the same we did before  with the releases here. And counting the releases. And the next step, of course, will be getting  the playtime. So I hope my notebook didn't break. Yes, it didn't break. Okay. Sorry again. Of course,  we have our group, our playtime, and we just projected and as a result, we can see, okay,  Taylor Swift gives us more music, like about like 10% more music than Katy Perry for 99 cents.  And it's a really easy operation. Okay. Now something, something a little bit more  challenging. I'm interested in, thank you, I'm interested in getting the prices of the releases  of the artists. And my, it's basically scraped data. So it's not probably as clean as I would  wish. Basically, we see a formative price with the currency in front and the price, but it's just  like in one attribute. And I'm interested in getting the prices in US dollars. And that's  easily to solve with a string operation and a compare operation. So I have to speed up a little  bit. Basically, we do a project phase. So just focus on the things in bold. They're important  ones here. And we have US dollar. Basically, it's a comparison of the lower strings of the first  three characters in our price formative, which gives us back US dollar or some currencies or  numbers or whatever. And the comparison is basically, is this US dollars? It's pretty obvious. And  then we just do a new match for US dollars zero. Okay. So also feels a little bit wrong, but  compare. Parameter gives us zero back when it's a match. And it gives us minus one back if the  value is higher and one back if the value is lower. So it's pretty handy one. We could also do,  is equal. There's also an equal operator which could give us like one or two back as we expected  as a Boolean true false. Then we sort, group, and we can even do something else. We can also go  and push every release we find in our group into a new list with a price and a product  that's basically very similar to JavaScript. It would be like actually like an append and Python.  So let's go here. Back. And here you go. And you see Katy Perry's products. And here's Taylor's  next object, Taylor Swift, and the list with all the products. So there's really a lot more  operators. And just, I can suggest if you fend the application framework, it's probably useful.  Just go to MongoDB documentation. It has a lot of examples. It's quite easy to get into.  And one more, it's a variable operator. It's a map operator. And as you can imagine,  it's basically the same as a Python map. And what do we do here? We're getting the ratings count,  which is actually how many users have given some stalls to the product, one we scraped the data.  And we want to adjust it a little bit because our management is our back and we need to make it  a little bit, look at it a little bit nicer. What we don't really want to do, but it's just like a  good example. So basically, we can pass in to a dollar map and input the ratings count as value.  And then we can just reuse the value on our list. And we just add 10 to each and every object  in our value we find in our list. And then it's applied. Then another thing, which is not probably  obvious, we cannot use some operator on a list like we can do really handy in Python. We have to  unwind first. So basically, for each and every value in our list, we have mangled with, we unwinded  to a new document and then we can do a simple grouping as we've done before. And yeah. And there  we go. Which brings us back, of course, to the next thing. You can also do map reduce in MongoDB.  And how many of you guys work with map reduce? Who knows map reduce? Yes. And who actually  works a lot with map reduce? Okay. So bring everybody up to speed. Map reduce, basically,  it's a really simple concept. We have all these documents. We map them. Map them is basically,  we just go through and we find key value pairs, which is actually, in our example, we find to  find the most popular words in our release titles. And we just emit them as tuples, as you can see,  to the reduce phase, which is run by a producer. In our example, it will just like sum up the counts.  It's really, really, really pretty easy operation. Basically, we just will use our name operator.  And you might wonder, why would we use map reduce in MongoDB? Because we have this great  aggregation framework, you see in substrings, we can do so many things. So what's the point,  actually, in using map reduce? And basically, for most of the time, you can work with the  aggregation framework. In most of the cases, it's faster. It's more accessible. Thank you.  But, however, map reduce gives us more power, because you can actually pass in JavaScript there.  You can build much more complex queries. For example, our example was splitting up  the release titles in words to count them. This could be quite challenging in the aggregation  framework. So let's do it. So, okay, this is map. Okay, let me show you a little bit more. Okay.  Okay, this fits. Okay. Sorry, this is our map. And here's our map reduce. So for map reduce,  we're just using from B some import code, which we can just pass on text. This is JavaScript  function. And it basically stores the name of the info which is stored in the name  attribute. It just stores it and splits it. It's a really simple operation. And then we just check  for some punctuation and stuff. We move it. It's not the best way to do this. It's just  like for the simple example. And if we actually find the word, we emit it. So basically, if there's  something like teenage, we emit teenage one. And if the album is called teenage dreams,  we can also emit dreams one. We send it to the reducer. And the reducer has really simple code  here. We just take all the keys and basically just count, just do some how often the key would  actually appear from our emitter. And here we get a result. And this is, and now we're going to do  a little bit more because I want to remove stop words, which is not really part of the aggregation  framework, just to make it a little bit nicer. That's why I've added the natural language kit,  remove stop words. And this is like the most popular words in Katy Perry and Taylor Swift's  albums. So you see, they probably have a younger audience with dream and teenage and one and boys  and fearless and speak, kissed and stuff. So, yeah, it's really easy. Of course,  unfortunately, we don't have enough time left. We could also run this operation across the complete  data set and to see what's basically the most popular words in album releases being sold at the  iTunes music store. So, to finish, I want to give you some more best practices and tips you can  use with the aggregation framework. First of all, database, think about your indexes, especially  if you do queries on them. Of course, if you have a huge data set and you don't have an index,  MongoDB has a collection scan and if it's a slow computer, it's, of course, taking time and probably  frustrating for you. Think about probably getting your data set, your database to your RAM. You can  just touch commands in MongoDB, which actually do something like, yeah, similar to Unix touch.  You touch it and then it fills up your RAM as much as possible, as much as you will ever get from  the system to store data. You can work with life and memory. You have to mind that the result can  be only like 60 megabytes because that's the maximum we can store in a BJ's and document, but I  mean like 60 megabytes is still huge. Pipeline operation has also a limit of 100B, but you will  hardly ever, sounds not much, but you hardly will ever really hit it. On your queries, you can  improve your queries up front. There's this nice, sorry for the break here, that's a nice explain  operation, which will basically give you information what would MongoDB do when query, doing your query.  You get some results, you see how many documents scanned, if indexes were hit, if it was all index,  and then you can really go and say, okay, I can really optimize all my work with just like  introducing a new index. Hardware is, of course, really important, especially RAM. More is better,  and it's really simple equation here. Mind the disk performance, of course SSDs and cloud computing  makes it really easy. And yeah, and you can also think about working about a dedicated server in  case you have something like a replica set and a right heavy database. So you can also say, okay,  just do another copy and work locally and do your aggregations without having to worry about if  you have a lot of traffic in your database. And the last slide is some useful resources, of course,  as I mentioned, MongoDB has a very good documentation, it's pretty updated by Mongo as well. And I also  want to mention Asia Kamski, she works for MongoDB, also as a trainer, and she has always like awesome  tricks and tips. And here we go. Thank you. We don't quite have enough time for Q&A, so if you  want to ask Alexander questions, then try to find him instead. Yeah, I'm around. Just ask anytime. No  problem. Thank you.
Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_0_1487.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_1488_1631.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_1632_1715.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_1716_1787.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_1788_9527.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_9528_10619.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_10620_11423.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_11424_11555.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_11556_17951.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_17952_18419.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_18420_19739.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_19740_21671.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_21672_21959.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_21960_22115.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_22116_22163.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_22164_23843.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_23844_24527.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_24528_24587.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_24588_24779.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_24780_28211.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_28212_28295.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_28296_30347.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_30348_30407.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_30408_31523.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_31524_31715.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_31716_31763.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_31764_31859.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_31860_31931.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_31932_33551.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_33552_37247.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_37248_37295.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_37296_38459.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_38460_39263.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_39264_39383.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_39384_45611.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_45612_45707.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_45708_45827.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_45828_45911.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_45912_48359.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_48360_48563.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_48564_50255.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_50256_51287.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_51288_52211.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_52212_54779.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_54780_54983.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_54984_55079.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_55080_55463.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_55464_55655.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_55656_56459.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_56460_56627.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_56628_56759.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_56760_56819.png Alexander_Hendorf_-_Data_Analysis_and_Map-Reduce_with_mongoDB_and_pymongo_56820_57025.png