Finding the Where in Big Fuzzy Data
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Turner, Andrew
We've gone to plaid. It is now easier to store any and all information that we can because it _might_ be useful later. Like a data hoarder, we would rather keep everything than throw any of it away. As a result, we now are knee-deep in bits that we are not quite sure are useful or meaningful. Fortunately, there is now a mature, and growing, family of open-source tools that make it straight-forward to organize, process and query all this data to find useful information. Hadoop has been synonymous with, and arguably responsible for, the rise of 'The Big Data'. But it's not your grandfather's mapreduce framework anymore (ok, in internet time). There are a number of open-source frameworks, tools, and techniques that are emerging that each provide a different speciality when managing and process fast, big, voracious data streams.As a Geo-community we understand the potential for location to be the common context through which we can combine disparate information. In large amounts of data with wide variety, location enables us to discover correlations that can be amazing insights that otherwise were lost when looking through our pre-defined and overly structured databases. And by using modern big data tools, we can now rapidly process queries which means we can experiment with more ideas in less time.This talk will share open-source projects that geo-enable these big data frameworks as well as use case examples of how they have been used to solve unique and interesting problems that would have taken forever to run or may not have even been possible.
 Cool. Good. How many people is this your first FOS 4G? Wow, that's awesome. That's cool. Welcome. It's been fun. It's a, I've been all the FOS 4Gs, all the way to the map, any of the map server user conferences. I'm not that old. Who here has been here to all the other FOS 4Gs? Anybody? Of course you are. Oh, you should just give this talk. I think you've seen this before. Cool. I'm going to go ahead and start. I think I have 30 seconds to use it wisely. So, my name is Andrew Turner. I'm currently the CTO of Esri RNDDC. I like just having a fully acronym title. I'm talking about some of the stuff we're doing there, particularly a lot of big data stuff and open sourced as well as other projects that we've done that kind of relate to this. And you probably ask yourself why is Esri here? And hopefully by the end of this, you'll actually understand it. It'll make a little more sense. But first, I want to talk a little bit kind of what's interesting, what's going on now in general about the big data world. Esri is in this book by Anthony Townsend about smart cities and it's a really good story in it. It's talking about the growth of cities. I live in DC now. It's been fascinating living in an urban environment seeing this happen. But during the end of the 19th century, the vast growth of how fast people were moving into cities. Now it's crossed over 50%. But prior to the 1840s and 1860s, at least in the US, there were fewer than 2 million people living in cities. Only 10% I think of the world or less were living in cities. But by 1920, there were over 50 million people living in cities. So it was a huge, huge boom of people moving into these cities in these urban environments and living in densely populated areas. People were immigrating and moving around a lot. It's hard to track where everyone was. So in the US, we have by government constitutional mandate a decadal census. Every 10 years, we have to go and count all the people, or at least as best measured we can. In the 1881, it was so many people were moving here into the US and into these cities that it took seven years to calculate all the results. So it took seven years when they only had three years left to start the next one, to start calculating it again. And by the, they estimated in the 1890 census, there were going to be too many people and it would take too long that they wouldn't be able to publish the results from the 1890 census until after the next census had taken place. So essentially, it was an unbounded problem. They couldn't, didn't know how to solve this. So people were building cities faster than we could ever count them and people were moving into them too fast. So a young enterprising census clerk, Herman Hollerith, saw this as an awesome opportunity and took the idea from Looms, where Looms had these cool things called punch cards in which you use them to program up how to actually weave rugs and shaws and clothing and things like that. And so I can do the same thing for tabulating. So you print out these punch cards in which punch cards had in it, you know, how old you were and how many people live in your household and what race you were and where you came from. He put it on, then on these little barbara pads, you pulled the handle and had these cups of mercury underneath. And these little bars that dipped into the mercury, and if it connected through the mercury because there was a hole in the punch card, it sent electrical current which moved a dial. And so you would essentially lay these cards down, pull the lever, a dial would move and they just keep doing that, keep doing that. It went up to 9,999 and then the dial would go back to zero. So they'd peer out here, write down the numbers, reset them all to zero and go. It was so effective that they went and by the 1890 census which they thought would take 10 to 12 to 15 years to calculate, within two months they'd already started publishing out the results from certain cities. They did the entire U.S. within two years. So it's kind of that beginning of this big data where more data you thought you could handle, you have to start automating it and processing it. So it's kind of inspirational for where we're going. Does anyone know who Herman Hollerith went on to become? Anybody? The tabulating machine company? They essentially became IBM. So this is how IBM got started with this little mercury cups, electrical currents and dials in 1880. So I'm from the Esri DC office. We're based there. We came out of GOIQ and GeoCommons. The idea is being based there local to the government and building tools that are actually going to be used to help solve these important problems. We're actually technically bad for geography company. We're actually based in Virginia. But hey, close enough. We do overlook DC and we're working there to really try and help and make tools more accessible and understandable to solve these problems. And a lot of them are coming open source and I'll explain why for big data in particular. Big data, again, is what's happening now with this explosion of data and information. How do we actually begin calculating meaningful ways to discover answers to problems before it's too late? Well the common concept with big data is you have these three V's and this is the kind of simple, overly simplistic view of it. You have huge volumes or it's moving too fast or you have a lot of different heterogeneous data types. And that's all true but it still is very, so very, you know, what do you feel about? Does it feel big? Does it not fit in your spreadsheet? And what people personally feel about it? But it's also starting looking at things like Internet of Things and where it's going to be going to. We have big data problems now but what happens when every single sensor of vehicle, your car already has tons of sensors in it itself, it has thousands of sensors, what happens? It starts publishing all that data out. Every single light post, street, corner, intersection, building, window starts transmitting this data, it's huge. So what's really happened is the fact that globally we now have a ubiquitous global network in which we can publish whatever the heck we want on to anybody in the world within milliseconds and the fact that hard drives and computers became insanely cheap in commodity. So what really became was the fact that while we have this ubiquitous network I can send you lots of small packets of information, it's actually still hard to move these huge volumes of data I want to capture. Really it became the data hoarders. I'm going to capture everything and I'll figure out later if it's useful to me at all. So really what it all kinds of means now is big data is when you should just stop moving the data. It's something that you capture, keep it at rest and then the idea is you can start throwing algorithms at that data because algorithms are very small. So again the thing about it is that before, you know, in typical GIS land, anyone here who's been doing really any calculations for a long time, you download your data and you run it locally or you even pull in your own database and it was on a server and you then process it, right? The difference is now the, that was good. You wrote code a lot against the data because the data were relatively small but now the data would take too long to move over the pipeline. You'd wait days just to get it to your server instead. You said the data are big so my functions are small. I'll move my functions to wherever the data reside. So that's the one kind of principle. The idea is that stop moving your data around and push your algorithms to that data and they'll talk about the tools that do that. And the other part is then open source and I think something everyone here likes and obviously it's probably our careers now, if not even our passion and our beliefs but what's interesting here specific to things like open data is the fact that this is a new kind of domain about how do we actually analyze these things using non-traditional methods. I usually liken it to thinking like Legos is by open source it means I can go and discover and try out new ideas that people never imagined before. So generally it's open source and the UNIX philosophy is I'm going to be lots of little modules that you can glue together to do things and pipe them together and I could never imagine. So now instead of doing it in one little command line how do I do that across vast numbers of machines and pipe it and play with it and try it and idea, see results in a few seconds and say I like that now run it forever. And I'll show you some examples of how that works. So that's really the power of open source is as any kind of tool builder you can't build the one tool anymore that's going to solve everybody's problem. Everybody's problem now is unique. The data are unique. The parsing is unique. It's heterogeneous. Volumes are different. Velocities are different. We have to enable the developers and the end users themselves as much as possible to put their own intelligence against that data. So I'm going to talk about three different types of big data stuff that we're thinking about and I think is kind of a framework for different methodologies and the tools we're providing to do this. So one is your traditional batch processing. It's taking what you would have done on your desktop and now just doing across lots of machines but doing it in a way that's I'm going to run the processing analysis and then I'm done. I'm going to take the answer. I'm going to visualize it or get a number and make a decision and make an action on it. And that's usually also called like map reduce is the one is a type of batch processing. It's pretty good. The problem is it can be very slow when you're waiting for it. You could have run in it, could have taken a data run and you don't know if your answer is right or wrong till after the process was run and you have to do that every single time. Stream processing has become much more interesting where you start having tens or hundreds of thousands of features per second in which you want to know when something interesting happen as that stream goes by and then you want to do alerting based on that. I don't want to watch the stream. I just want to know when it crosses a certain threshold. So it's a different kind of big data analysis. And then last is the search discovery. This idea is I don't even know unless you know what the shape of the needle I'm looking for in my haystack. I just know the general kind of problems I have. Let me know when any of these things cross over these thresholds. So essentially being able to just ask a number of questions and then letting me know whenever the answers happen to show up. So the framework that's actually kind of emerging has become a terminology to capture all these things together is this lambda architecture. What it is in these two sides is you have the left side is essentially the stream processing engine. And I'll go through a couple of tools that do this. Where you're watching the data as it streams by, you're running various aggregate statistics on it, you're looking for moving window averages and it's capturing kind of general alerting of what's going on. And when something's crossed you say, ah, something happened. There was more crime just happened in a neighborhood than I would have expected. That then kicks off a batch processing job saying how did this happen? What's happened over the last day? That's now I need to process and understand what this is. So it's applying these two things in synchronous to where the stream processing is just getting your general alerting. Here's something happened and then the batch processing helps you understand why it happened. So and then in the end you want to visualize it. I mostly just put it in kind of a mesmerizing view. But the idea that you can start doing these things like just watching all the data stream through, seeing are my algorithms kind of doing what they need to be doing and then alerting me. Like in this case is a tornado is about to form or hurricanes are about to get violent. And being able to visualize that in an ops dashboard or someone sitting and watching it and then tell me when a certain threshold is crossed. This is actually based on the, I forget his name, the Earth null, null school I guess, visualization that we've built it now in Canvas. Anyway, my point is that how do you make all these tools available so you can apply the unique knowledge against it? So we've really open sourced a number of tools that we call GIS tools for Hadoop. But really it's a set of tools that allow you to go and do these different kinds of batch, streaming, search and discovery and alerting mechanisms. So we're applying a lot of these tools against the common frameworks as well as building some out ourselves. So this is kind of a list of some of the things that we're working with and helping build out and you might recognize some of them if you haven't. I definitely recommend diving into and checking them out. Hadoop is came out of Yahoo originally as an open source map reduce framework. It's pretty well established. It's essentially synonymous with the last five years of big data mint doing Hadoop. That's not the only answer anymore, but it's still really powerful. So we've open sourced tools on top of Hadoop. WooKong is a really nice Ruby wrapper around it to process if you don't want to go and do the Java. I'll have some more examples of that. Pigeon is another kind of query interface on top of it. So if you want to write SQL like queries, HBase, Cassandra and Chemulo are three really nice big databases that we're helping spatially enable. Kafka and Storm are both stream processing engines that were, I'll show an example, actually an application example of using those for doing stream processing and we're geospatially enabling these processing engines. Elasticsearch already is pretty good and hopefully we can help out with that. It's essentially the new big data for search. It's where Lucene and Solar have now grown up to this Elasticsearch engine, which has some really advanced spatial capabilities in it for doing everything from search and just basic queries of data and features to it and the billions of records very easily to doing even alerting when certain thresholds are crossed. Tell me what's significant about this one area. It's all just actually built into the engine. And then Apache Spark is coming up to be kind of the new in-memory stream and micro, it's called MicroBatch. It's batch but in batch in tens or hundreds of milliseconds batch. So it looks like it's streaming but it's actually acts more like MapReduce which lets you do some interesting things. And again in 20 minutes I'm just going to kind of blow by these ideas. I'll have links at the end. You can follow up on. So the core of this is something that was kind of amazing for Esri and why we're here is we actually took one of our core Jools and we open sourced it. The Esri geometry API, she'll try to get a name for it, but it's a Java engine for doing spatial processing. It's JTS but under currently it's still, I believe unless someone's going to correct me, still the most open source geometry engine out there. JTS is under a copy left license so it's a bit viral. That's changing I believe which is awesome. But right now this is under the Apache license. So take it, use it, do whatever you want with it, contribute back, yay, don't have to use it in commercial products, go forth. And the reason of us open sourcing this was again this problem is we weren't in our tools going to build the end analysis solution for everybody. People were going to go and do that and apply their unique ideas and concepts against it. We wanted to enable that to really get to the best answers possible. So under the hood is things if you've used any other geometry engine, feel very familiar, but it's really full featured. It handles all, you know, numerous native geometry types, topological operations, relational combinations between these all built in native to the library. Import, export to different formats, shape file, WKB, GeoJSON, EsriJSON, WKT. So all that kind of handling. Just to show you what this looks like and how easy it is, I went and used wrapped in Ruby because writing Java on a presentation is the... So, but you can probably get the idea is that we load in the different Java libraries, we get a JSON factor in which I can parse it, parse GeoJSON and convert it into a native geometry object which I can then dump back out in another format or put in my spatial processing. I'll show you an example of that in a few slides. The idea is this kind of gives you that low level operations for handing topologies that you then need to... We'll keep wrapping up to do more higher level operations on. The library also has different validations to make sure you have good geometries, closed loops, and then just general other operations. So boundaries, buffers, clips, all your base level operations, and then even quad trees which becomes really important then obviously for doing large scale distributed, sharded spatial processing. So what this looks like is that again I can load up and I can build a quad tree and I can start just pushing objects into it and I have this index now that I can use in memory or serialize it out and then do queries against it. So these are a little piece. This is that the base of the Lego you need to start building things around it. And again you can use it, you can go and we have Ruby wrappers around it, I think we can do some Python wrappers around it, just make it easy to use in different languages and then it's very fast. What we've done is not everyone wants to go work that low level, they want to work at higher abstractions. So we've also open sourced and built in tools above that that let you do more familiar high level processing across these data. So at this lowest level you have Hadoop and this geometry API engine at the bottom, which you can go and build your own map reduce jobs, which is great if you know how to do that, but if you don't it can be a bit of a learning curve. So on top of that the hive is a framework that gives you a SQL like query against these. You can essentially write SQL and it turns those into map reduce jobs. We've extended hive to have spatial operations and queries against it. We wrap all that up and we call GIS tools for Hadoop, which is a lot of different samples and examples and best patterns. And then for ourselves obviously we pull it into ArcGIS if you just want to click a button in desktop. But the nice thing is while someone can push a button in desktop, other people can go under the hood into the libraries and get as crazy and custom as they want to. So what this does, I mentioned about quad trees and distributed sharded processing, just in a very quick picture about how this works is we essentially build spatial indices on different servers across the cluster. Things are when objects are come in they are pushed off to that high level spatial index in terms of what shard they're going to operate against and then once it gets to the machine it then does a finer grained sharding within that for inside its quadtree. And similarly then when requests come through for doing things like joins and intersects, it does a high level kind of saying here's your feature somewhere in New Mexico, go against this server or this index and then within that we'll find out what county or police district or neighborhood you're in. And that's the concept of reduce, of mapping it out and then reducing it is pushing out to the local machine where the data reside and then I'll do that locally in that machine and sharding it out. And so the distributed quadtree essentially is the technique that does that. So what this looks like at a high level as I mentioned high is in the end people are very good at writing SQL. It's a very common language. Hive essentially gives you SQL across these big data sets. So you know it's, you can read this pretty well, it's you know in this case I'm looking at the number of earthquakes by county and want to just have a calculation against that. So it's like you would count against a relational database but now it's handling across, could be across millions and billions of records and it goes distributed across however many machines you can choose to throw at it. So just for example of kind of what this looks like and what it was before is we actually did I think FAA locations of flights and we did count by counties and we had, so we had 14 million flight locations and we wanted to count those by county. It's just kind of a little silly example. The first time we did this a few months ago using our geometry engine and these tools on top of it, it took 13 minutes to run this operation. So not horrible, you know faster than a day, you know maybe I can still go run out and get a coffee but it's 14 million records so it's not bad. But then we realized we could do better. We could optimize things, we could optimize how we shard them out, we could optimize actually some of the spatial processing on the server itself. In the end it now takes about 56 seconds to calculate 14 million aggregate points which is not too shabby. And this is still back, this is not even, imagine streaming would be real time. This is saying I have a question, what does it look like? It's not even time for me to go and get a coffee. So sorry if that impacts your productivity but it's pretty cool to start seeing and this is, we just a version actually just got released this morning I heard from the team version 1.2 which has some more optimizations in it too. So we haven't run benchmarks against that yet but it could be a little faster and you know it'll be more interesting to start doing more than just points and polygons and get some more benchmarks around that. So that's something we're working on, we'd love to hear if you have use cases like that. Another example here is kind of interesting. This is a project for a automotive company in Japan and so all you can say about that you can kind of guess. They wanted to know about how to, where should they try and promote carpooling actually. They wanted to take the idea that people tend to live next to each other and might work in the same area, how can we connect them together, they should actually carpool and actually help in smog and traffic reduction. So it took 40 million points, vehicle track locations of where people are commuting, we looked at where they were in the morning within 15 minutes of each other to where they ended up at the same place within 15 minutes of each other within a 500 meter grid cells. And from that we then derived that down to finding where there actually are common carpools. There's 100 people that all could actually be collaborating because they're all starting in the same neighborhood and going to the same workplace and starting to do that. I think that actually took about 30 minutes to run. So again something we can improve but the idea is here is you can start answering some pretty interesting questions. To actually in the end, it's a map, really the answer isn't even a map, it's about just a list of addresses where you should promote carpooling. But the idea was obviously it was underneath it was a spatial question. Similarly we're using a lot of these tools for doing, we've actually talked a little bit more about this publicly, the port of Rotterdam wanted to know from all of the ships that come in and out of the different ports, you know, where is there the most traffic and congestion, where can they, you know, put in better signaling and things like that. So we took a year of AISs on MEDICA, essentially ship tracking, taking a year of all of the data of ships coming in and out and did some quick spatial aggregations to hex bins. So check that bingo card. But wanted to see where the congestions were over time and then the idea is now monitor this in real time. Does this change over time or not? Now they've answered it for the past. So just kind of saying that again this is that Lambda architecture where they've processed it, they at least know the pattern, now they're going to set up an alert, watch if this pattern shows up again and then kick off some more processing afterwards. So actually showing another real world example, thank you, Alarm. So where this also comes in, for example, in cities that wanted to say have audio detectors on the rooftops where they can say detect gunshots. So we're listening for lots of different sounds and streaming and when something crosses a certain decibel level, it then kicks off a batch processing job to triangulate from all the microphones where that gunshot probably was. So another example of this was something we've shown in the past, you might have seen it, was looking at, for example, social media tweets during a disaster. In this case this was Hurricane Sandy a few years ago in which we did a normalized aggregation of social media mentions of power outages compared to people tweeting in general in Manhattan, compared to people talking about the hurricane globally, compared to people on Twitter talking globally. And so what we did is we asked that question, we said, okay, whenever this thresholds crossed, people are talking more about power outages than I would expect, let me know. And as a disaster responder, I'm going to go off and worry about other things. And so a visualization can run and can do whatever heck it wants, I only want to know when it's important. So we're using Storm with a geo-enabled Storm to do the processing of these tweets. We then threw up the aggregate values as a web socket to the browser and visualized it. And then in a few seconds you should start seeing this, what happened is a threshold was crossed. I care about something just happened, I can now dive into the specific features that I can now cut out of the numerous features I had to find those specific ones, what did they say? A power transformer exploded, I can actually grab photos of the explosion outside people's windows, verify that. And then over time see how people are moving in response to that power outage. So by the next day people had to move north to Grand Central Station in northern Manhattan where it saw power so when it came to sending water and blankets they knew where to send them to where people are going to be, not where they had been where they were living. So that's a kind of real-time alerting. And just what that looks like, it just kind of benchmarks because numbers matter. Just what we looked at, this was a geo-enabled Storm process running on a Mac just for benchmarks. Just streaming it through is 10,000 features per second with parsing tweets was 6,000. And if we geo-joined them by grid cells it was about 5,000 tweets per second. We could do on a single Mac. So, and that's easily Storm is essentially if you haven't used Storm it's like Hadoop for streaming. It's what Twitter bought a company, open source it's what they use for their ad engine. So it can handle the volumes and we're just spatially enabling it to answer these location-based questions. So kind of going forward I was hoping to have a demo of this and I don't yet so I will soon and I'll blog about it. We've released something called ArcGIS Open Data which essentially we're giving away to every government in the world to make their data open and accessible via GeoJSON and other formats. We have 1,000 sites created globally is more showing up. We're hoping is that now with these big data tools people can go off and take all this amazing open government data and start answering meaningful questions around climate and disaster resilience, location impact of schools and poverty and health and different aspects and answer these important questions for society through open source tools and open data. So that's personally what I'm driving for and what we'll be doing with it over the next six months. So to kind of wrap up these are the URLs to check out so esri.github.io has all of our 300 open source projects you can go and explore and dive through and a lot of the ones I've shown here. GIS tools for Hadoop is the string to look for for the specific stuff I've talked about. And then one of our engineers in particular is a fairly prolific, awesome amazing engineer. He did a lot of the analyses I've shown here, Mansour Rod. He also blogs prolifically and all of every code he does is all in his GitHub project where he's done specific applications around these tools. So Thunderhead Explorer is his blog but I definitely recommend checking it out or MROD on GitHub. It definitely has a lot of other kinds of examples you might want to check out. So that's it for my super brief big open source data talk but thank you very much. I appreciate you being here and I'm open for any questions. Questions? You guys are already all downloading the tools right now? Be nice to the Wi-Fi. Cool. Well, I'm available afterwards if anyone wants to hang out and chat or we'll also have a booth up in the section or over coffee. So again, thanks very much and have a good afternoon. Bye.
 Cool. Good. How many people is this your first FOS 4G? Wow, that's awesome. That's  cool. Welcome. It's been fun. It's a, I've been all the FOS 4Gs, all the way to the  map, any of the map server user conferences. I'm not that old. Who here has been here  to all the other FOS 4Gs? Anybody? Of course you are. Oh, you should just give this talk.  I think you've seen this before. Cool. I'm going to go ahead and start. I think I have  30 seconds to use it wisely. So, my name is Andrew Turner. I'm currently the CTO of  Esri RNDDC. I like just having a fully acronym title. I'm talking about some of the stuff  we're doing there, particularly a lot of big data stuff and open sourced as well as other  projects that we've done that kind of relate to this. And you probably ask yourself why  is Esri here? And hopefully by the end of this, you'll actually understand it. It'll  make a little more sense. But first, I want to talk a little bit kind of what's interesting,  what's going on now in general about the big data world. Esri is in this book by Anthony  Townsend about smart cities and it's a really good story in it. It's talking about the growth  of cities. I live in DC now. It's been fascinating living in an urban environment seeing this  happen. But during the end of the 19th century, the vast growth of how fast people were moving  into cities. Now it's crossed over 50%. But prior to the 1840s and 1860s, at least in  the US, there were fewer than 2 million people living in cities. Only 10% I think of the  world or less were living in cities. But by 1920, there were over 50 million people living  in cities. So it was a huge, huge boom of people moving into these cities in these urban environments  and living in densely populated areas. People were immigrating and moving around a lot.  It's hard to track where everyone was. So in the US, we have by government constitutional  mandate a decadal census. Every 10 years, we have to go and count all the people, or  at least as best measured we can. In the 1881, it was so many people were moving here into  the US and into these cities that it took seven years to calculate all the results.  So it took seven years when they only had three years left to start the next one, to  start calculating it again. And by the, they estimated in the 1890 census, there were going  to be too many people and it would take too long that they wouldn't be able to publish  the results from the 1890 census until after the next census had taken place. So essentially,  it was an unbounded problem. They couldn't, didn't know how to solve this. So people were  building cities faster than we could ever count them and people were moving into them  too fast. So a young enterprising census clerk, Herman Hollerith, saw this as an awesome opportunity  and took the idea from Looms, where Looms had these cool things called punch cards in  which you use them to program up how to actually weave rugs and shaws and clothing and things  like that. And so I can do the same thing for tabulating. So you print out these punch  cards in which punch cards had in it, you know, how old you were and how many people  live in your household and what race you were and where you came from. He put it on, then  on these little barbara pads, you pulled the handle and had these cups of mercury underneath.  And these little bars that dipped into the mercury, and if it connected through the mercury  because there was a hole in the punch card, it sent electrical current which moved a dial.  And so you would essentially lay these cards down, pull the lever, a dial would move and  they just keep doing that, keep doing that. It went up to 9,999 and then the dial would  go back to zero. So they'd peer out here, write down the numbers, reset them all to zero and  go. It was so effective that they went and by the 1890 census which they thought would  take 10 to 12 to 15 years to calculate, within two months they'd already started publishing  out the results from certain cities. They did the entire U.S. within two years. So it's  kind of that beginning of this big data where more data you thought you could handle, you  have to start automating it and processing it. So it's kind of inspirational for where  we're going. Does anyone know who Herman Hollerith went on to become? Anybody? The tabulating  machine company? They essentially became IBM. So this is how IBM got started with this little  mercury cups, electrical currents and dials in 1880. So I'm from the Esri DC office. We're  based there. We came out of GOIQ and GeoCommons. The idea is being based there local to the  government and building tools that are actually going to be used to help solve these important  problems. We're actually technically bad for geography company. We're actually based  in Virginia. But hey, close enough. We do overlook DC and we're working there to really try and  help and make tools more accessible and understandable to solve these problems. And a lot of them  are coming open source and I'll explain why for big data in particular. Big data, again,  is what's happening now with this explosion of data and information. How do we actually  begin calculating meaningful ways to discover answers to problems before it's too late?  Well the common concept with big data is you have these three V's and this is the kind  of simple, overly simplistic view of it. You have huge volumes or it's moving too fast  or you have a lot of different heterogeneous data types. And that's all true but it still  is very, so very, you know, what do you feel about? Does it feel big? Does it not fit in  your spreadsheet? And what people personally feel about it? But it's also starting looking  at things like Internet of Things and where it's going to be going to. We have big data  problems now but what happens when every single sensor of vehicle, your car already has tons  of sensors in it itself, it has thousands of sensors, what happens? It starts publishing  all that data out. Every single light post, street, corner, intersection, building, window  starts transmitting this data, it's huge. So what's really happened is the fact that  globally we now have a ubiquitous global network in which we can publish whatever the heck we  want on to anybody in the world within milliseconds and the fact that hard drives and computers  became insanely cheap in commodity. So what really became was the fact that while we have  this ubiquitous network I can send you lots of small packets of information, it's actually  still hard to move these huge volumes of data I want to capture. Really it became the data  hoarders. I'm going to capture everything and I'll figure out later if it's useful to  me at all. So really what it all kinds of means now is big data is when you should just  stop moving the data. It's something that you capture, keep it at rest and then the  idea is you can start throwing algorithms at that data because algorithms are very small.  So again the thing about it is that before, you know, in typical GIS land, anyone here  who's been doing really any calculations for a long time, you download your data and you  run it locally or you even pull in your own database and it was on a server and you then  process it, right? The difference is now the, that was good. You wrote code a lot against  the data because the data were relatively small but now the data would take too long  to move over the pipeline. You'd wait days just to get it to your server instead. You  said the data are big so my functions are small. I'll move my functions to wherever  the data reside. So that's the one kind of principle. The idea is that stop moving your  data around and push your algorithms to that data and they'll talk about the tools that  do that. And the other part is then open source and I think something everyone here likes  and obviously it's probably our careers now, if not even our passion and our beliefs but  what's interesting here specific to things like open data is the fact that this is a  new kind of domain about how do we actually analyze these things using non-traditional  methods. I usually liken it to thinking like Legos is by open source it means I can go  and discover and try out new ideas that people never imagined before. So generally it's open  source and the UNIX philosophy is I'm going to be lots of little modules that you can  glue together to do things and pipe them together and I could never imagine. So now instead  of doing it in one little command line how do I do that across vast numbers of machines  and pipe it and play with it and try it and idea, see results in a few seconds and say  I like that now run it forever. And I'll show you some examples of how that works. So that's  really the power of open source is as any kind of tool builder you can't build the one tool  anymore that's going to solve everybody's problem. Everybody's problem now is unique.  The data are unique. The parsing is unique. It's heterogeneous. Volumes are different.  Velocities are different. We have to enable the developers and the end users themselves  as much as possible to put their own intelligence against that data.  So I'm going to talk about three different types of big data stuff that we're thinking  about and I think is kind of a framework for different methodologies and the tools we're  providing to do this. So one is your traditional batch processing. It's taking what you would  have done on your desktop and now just doing across lots of machines but doing it in a  way that's I'm going to run the processing analysis and then I'm done. I'm going to take  the answer. I'm going to visualize it or get a number and make a decision and make an  action on it. And that's usually also called like map reduce is the one is a type of batch  processing. It's pretty good. The problem is it can be very slow when you're waiting  for it. You could have run in it, could have taken a data run and you don't know if your  answer is right or wrong till after the process was run and you have to do that every single  time. Stream processing has become much more interesting where you start having tens or  hundreds of thousands of features per second in which you want to know when something interesting  happen as that stream goes by and then you want to do alerting based on that. I don't  want to watch the stream. I just want to know when it crosses a certain threshold. So it's  a different kind of big data analysis. And then last is the search discovery. This idea  is I don't even know unless you know what the shape of the needle I'm looking for in  my haystack. I just know the general kind of problems I have. Let me know when any of  these things cross over these thresholds. So essentially being able to just ask a number  of questions and then letting me know whenever the answers happen to show up. So the framework  that's actually kind of emerging has become a terminology to capture all these things  together is this lambda architecture. What it is in these two sides is you have the left  side is essentially the stream processing engine. And I'll go through a couple of tools  that do this. Where you're watching the data as it streams by, you're running various aggregate  statistics on it, you're looking for moving window averages and it's capturing kind of  general alerting of what's going on. And when something's crossed you say, ah, something  happened. There was more crime just happened in a neighborhood than I would have expected.  That then kicks off a batch processing job saying how did this happen? What's happened  over the last day? That's now I need to process and understand what this is. So it's applying  these two things in synchronous to where the stream processing is just getting your general  alerting. Here's something happened and then the batch processing helps you understand  why it happened. So and then in the end you want to visualize it. I mostly just put it  in kind of a mesmerizing view. But the idea that you can start doing these things like  just watching all the data stream through, seeing are my algorithms kind of doing what  they need to be doing and then alerting me. Like in this case is a tornado is about to  form or hurricanes are about to get violent. And being able to visualize that in an ops  dashboard or someone sitting and watching it and then tell me when a certain threshold  is crossed. This is actually based on the, I forget his name, the Earth null, null school  I guess, visualization that we've built it now in Canvas. Anyway, my point is that how  do you make all these tools available so you can apply the unique knowledge against it?  So we've really open sourced a number of tools that we call GIS tools for Hadoop. But  really it's a set of tools that allow you to go and do these different kinds of batch,  streaming, search and discovery and alerting mechanisms. So we're applying a lot of these  tools against the common frameworks as well as building some out ourselves. So this is  kind of a list of some of the things that we're working with and helping build out and  you might recognize some of them if you haven't. I definitely recommend diving into and checking  them out. Hadoop is came out of Yahoo originally as an open source map reduce framework. It's  pretty well established. It's essentially synonymous with the last five years of big  data mint doing Hadoop. That's not the only answer anymore, but it's still really powerful.  So we've open sourced tools on top of Hadoop. WooKong is a really nice Ruby wrapper around  it to process if you don't want to go and do the Java. I'll have some more examples  of that. Pigeon is another kind of query interface on top of it. So if you want to write SQL  like queries, HBase, Cassandra and Chemulo are three really nice big databases that we're  helping spatially enable. Kafka and Storm are both stream processing engines that were,  I'll show an example, actually an application example of using those for doing stream processing  and we're geospatially enabling these processing engines. Elasticsearch already is pretty good  and hopefully we can help out with that. It's essentially the new big data for search. It's  where Lucene and Solar have now grown up to this Elasticsearch engine, which has some  really advanced spatial capabilities in it for doing everything from search and just  basic queries of data and features to it and the billions of records very easily to doing  even alerting when certain thresholds are crossed. Tell me what's significant about  this one area. It's all just actually built into the engine. And then Apache Spark is  coming up to be kind of the new in-memory stream and micro, it's called MicroBatch.  It's batch but in batch in tens or hundreds of milliseconds batch. So it looks like it's  streaming but it's actually acts more like MapReduce which lets you do some interesting  things. And again in 20 minutes I'm just going to kind of blow by these ideas. I'll  have links at the end. You can follow up on. So the core of this is something that was  kind of amazing for Esri and why we're here is we actually took one of our core Jools  and we open sourced it. The Esri geometry API, she'll try to get a name for it, but  it's a Java engine for doing spatial processing. It's JTS but under currently it's still, I  believe unless someone's going to correct me, still the most open source geometry engine  out there. JTS is under a copy left license so it's a bit viral. That's changing I believe  which is awesome. But right now this is under the Apache license. So take it, use it, do  whatever you want with it, contribute back, yay, don't have to use it in commercial products,  go forth. And the reason of us open sourcing this was again this problem is we weren't in  our tools going to build the end analysis solution for everybody. People were going  to go and do that and apply their unique ideas and concepts against it. We wanted to enable  that to really get to the best answers possible. So under the hood is things if you've used  any other geometry engine, feel very familiar, but it's really full featured. It handles  all, you know, numerous native geometry types, topological operations, relational combinations  between these all built in native to the library. Import, export to different formats, shape  file, WKB, GeoJSON, EsriJSON, WKT. So all that kind of handling. Just to show you what  this looks like and how easy it is, I went and used wrapped in Ruby because writing Java  on a presentation is the... So, but you can probably get the idea is that we load in the  different Java libraries, we get a JSON factor in which I can parse it, parse GeoJSON and  convert it into a native geometry object which I can then dump back out in another format  or put in my spatial processing. I'll show you an example of that in a few slides. The  idea is this kind of gives you that low level operations for handing topologies that you  then need to... We'll keep wrapping up to do more higher level operations on. The library  also has different validations to make sure you have good geometries, closed loops, and  then just general other operations. So boundaries, buffers, clips, all your base level operations,  and then even quad trees which becomes really important then obviously for doing large scale  distributed, sharded spatial processing. So what this looks like is that again I can load  up and I can build a quad tree and I can start just pushing objects into it and I have this  index now that I can use in memory or serialize it out and then do queries against it. So  these are a little piece. This is that the base of the Lego you need to start building  things around it. And again you can use it, you can go and we have Ruby wrappers around  it, I think we can do some Python wrappers around it, just make it easy to use in different  languages and then it's very fast. What we've done is not everyone wants to go work that  low level, they want to work at higher abstractions. So we've also open sourced and built in tools  above that that let you do more familiar high level processing across these data. So at  this lowest level you have Hadoop and this geometry API engine at the bottom, which you  can go and build your own map reduce jobs, which is great if you know how to do that,  but if you don't it can be a bit of a learning curve. So on top of that the hive is a framework  that gives you a SQL like query against these. You can essentially write SQL and it turns  those into map reduce jobs. We've extended hive to have spatial operations and queries  against it. We wrap all that up and we call GIS tools for Hadoop, which is a lot of different  samples and examples and best patterns. And then for ourselves obviously we pull it into  ArcGIS if you just want to click a button in desktop. But the nice thing is while someone  can push a button in desktop, other people can go under the hood into the libraries and  get as crazy and custom as they want to. So what this does, I mentioned about quad trees  and distributed sharded processing, just in a very quick picture about how this works  is we essentially build spatial indices on different servers across the cluster. Things  are when objects are come in they are pushed off to that high level spatial index in terms  of what shard they're going to operate against and then once it gets to the machine it then  does a finer grained sharding within that for inside its quadtree. And similarly then  when requests come through for doing things like joins and intersects, it does a high level  kind of saying here's your feature somewhere in New Mexico, go against this server or this  index and then within that we'll find out what county or police district or neighborhood  you're in. And that's the concept of reduce, of mapping it out and then reducing it is  pushing out to the local machine where the data reside and then I'll do that locally  in that machine and sharding it out. And so the distributed quadtree essentially is the  technique that does that. So what this looks like at a high level as I mentioned high is  in the end people are very good at writing SQL. It's a very common language. Hive essentially  gives you SQL across these big data sets. So you know it's, you can read this pretty  well, it's you know in this case I'm looking at the number of earthquakes by county and  want to just have a calculation against that. So it's like you would count against a relational  database but now it's handling across, could be across millions and billions of records  and it goes distributed across however many machines you can choose to throw at it. So  just for example of kind of what this looks like and what it was before is we actually  did I think FAA locations of flights and we did count by counties and we had, so we had  14 million flight locations and we wanted to count those by county. It's just kind of  a little silly example. The first time we did this a few months ago using our geometry  engine and these tools on top of it, it took 13 minutes to run this operation. So not horrible,  you know faster than a day, you know maybe I can still go run out and get a coffee but  it's 14 million records so it's not bad. But then we realized we could do better. We could  optimize things, we could optimize how we shard them out, we could optimize actually  some of the spatial processing on the server itself. In the end it now takes about 56 seconds  to calculate 14 million aggregate points which is not too shabby. And this is still back,  this is not even, imagine streaming would be real time. This is saying I have a question,  what does it look like? It's not even time for me to go and get a coffee. So sorry if  that impacts your productivity but it's pretty cool to start seeing and this is, we just  a version actually just got released this morning I heard from the team version 1.2 which  has some more optimizations in it too. So we haven't run benchmarks against that yet  but it could be a little faster and you know it'll be more interesting to start doing more  than just points and polygons and get some more benchmarks around that. So that's something  we're working on, we'd love to hear if you have use cases like that.  Another example here is kind of interesting. This is a project for a automotive company  in Japan and so all you can say about that you can kind of guess. They wanted to know  about how to, where should they try and promote carpooling actually. They wanted to take the  idea that people tend to live next to each other and might work in the same area, how  can we connect them together, they should actually carpool and actually help in smog  and traffic reduction. So it took 40 million points, vehicle track locations of where people  are commuting, we looked at where they were in the morning within 15 minutes of each other  to where they ended up at the same place within 15 minutes of each other within a 500 meter  grid cells. And from that we then derived that down to finding where there actually  are common carpools. There's 100 people that all could actually be collaborating because  they're all starting in the same neighborhood and going to the same workplace and starting  to do that. I think that actually took about 30 minutes to run. So again something we can  improve but the idea is here is you can start answering some pretty interesting questions.  To actually in the end, it's a map, really the answer isn't even a map, it's about just  a list of addresses where you should promote carpooling. But the idea was obviously it  was underneath it was a spatial question. Similarly we're using a lot of these tools  for doing, we've actually talked a little bit more about this publicly, the port of  Rotterdam wanted to know from all of the ships that come in and out of the different ports,  you know, where is there the most traffic and congestion, where can they, you know,  put in better signaling and things like that. So we took a year of AISs on MEDICA, essentially  ship tracking, taking a year of all of the data of ships coming in and out and did some  quick spatial aggregations to hex bins. So check that bingo card. But wanted to see where  the congestions were over time and then the idea is now monitor this in real time. Does  this change over time or not? Now they've answered it for the past. So just kind of  saying that again this is that Lambda architecture where they've processed it, they at least  know the pattern, now they're going to set up an alert, watch if this pattern shows up  again and then kick off some more processing afterwards. So actually showing another real  world example, thank you, Alarm. So where this also comes in, for example, in cities  that wanted to say have audio detectors on the rooftops where they can say detect gunshots.  So we're listening for lots of different sounds and streaming and when something crosses a  certain decibel level, it then kicks off a batch processing job to triangulate from all  the microphones where that gunshot probably was. So another example of this was something  we've shown in the past, you might have seen it, was looking at, for example, social media  tweets during a disaster. In this case this was Hurricane Sandy a few years ago in which  we did a normalized aggregation of social media mentions of power outages compared to  people tweeting in general in Manhattan, compared to people talking about the hurricane globally,  compared to people on Twitter talking globally. And so what we did is we asked that question,  we said, okay, whenever this thresholds crossed, people are talking more about power outages  than I would expect, let me know. And as a disaster responder, I'm going to go off and  worry about other things. And so a visualization can run and can do whatever heck it wants,  I only want to know when it's important. So we're using Storm with a geo-enabled Storm  to do the processing of these tweets. We then threw up the aggregate values as a web  socket to the browser and visualized it. And then in a few seconds you should start seeing  this, what happened is a threshold was crossed. I care about something just happened, I can  now dive into the specific features that I can now cut out of the numerous features  I had to find those specific ones, what did they say? A power transformer exploded, I  can actually grab photos of the explosion outside people's windows, verify that. And  then over time see how people are moving in response to that power outage. So by the next  day people had to move north to Grand Central Station in northern Manhattan where it saw  power so when it came to sending water and blankets they knew where to send them to where  people are going to be, not where they had been where they were living. So that's a kind  of real-time alerting. And just what that looks like, it just kind of benchmarks because  numbers matter. Just what we looked at, this was a geo-enabled Storm process running on  a Mac just for benchmarks. Just streaming it through is 10,000 features per second with  parsing tweets was 6,000. And if we geo-joined them by grid cells it was about 5,000 tweets  per second. We could do on a single Mac. So, and that's easily Storm is essentially if  you haven't used Storm it's like Hadoop for streaming. It's what Twitter bought a company,  open source it's what they use for their ad engine. So it can handle the volumes and  we're just spatially enabling it to answer these location-based questions. So kind of  going forward I was hoping to have a demo of this and I don't yet so I will soon and  I'll blog about it. We've released something called ArcGIS Open Data which essentially  we're giving away to every government in the world to make their data open and accessible  via GeoJSON and other formats. We have 1,000 sites created globally is more showing up.  We're hoping is that now with these big data tools people can go off and take all this  amazing open government data and start answering meaningful questions around climate and disaster  resilience, location impact of schools and poverty and health and different aspects and  answer these important questions for society through open source tools and open data. So  that's personally what I'm driving for and what we'll be doing with it over the next  six months. So to kind of wrap up these are the URLs to check out so esri.github.io has  all of our 300 open source projects you can go and explore and dive through and a lot  of the ones I've shown here. GIS tools for Hadoop is the string to look for for the specific  stuff I've talked about. And then one of our engineers in particular is a fairly prolific,  awesome amazing engineer. He did a lot of the analyses I've shown here, Mansour Rod.  He also blogs prolifically and all of every code he does is all in his GitHub project  where he's done specific applications around these tools. So Thunderhead Explorer is his  blog but I definitely recommend checking it out or MROD on GitHub. It definitely has a  lot of other kinds of examples you might want to check out. So that's it for my super brief  big open source data talk but thank you very much. I appreciate you being here and I'm  open for any questions.  Questions? You guys are already all downloading the tools right now? Be nice to the Wi-Fi.  Cool. Well, I'm available afterwards if anyone wants to hang out and chat or we'll also have  a booth up in the section or over coffee. So again, thanks very much and have a good  afternoon.  Bye.
04_Andrew_Turner_720p_0_1463.png 04_Andrew_Turner_720p_1464_2543.png 04_Andrew_Turner_720p_2544_3551.png 04_Andrew_Turner_720p_3552_5771.png 04_Andrew_Turner_720p_5772_6395.png 04_Andrew_Turner_720p_6396_6707.png 04_Andrew_Turner_720p_6708_7211.png 04_Andrew_Turner_720p_7212_7739.png 04_Andrew_Turner_720p_7740_7907.png 04_Andrew_Turner_720p_7908_8363.png 04_Andrew_Turner_720p_8364_8711.png 04_Andrew_Turner_720p_8712_9587.png 04_Andrew_Turner_720p_9588_10031.png 04_Andrew_Turner_720p_10032_11171.png 04_Andrew_Turner_720p_11172_13067.png 04_Andrew_Turner_720p_13068_14351.png 04_Andrew_Turner_720p_14352_15191.png 04_Andrew_Turner_720p_15192_15695.png 04_Andrew_Turner_720p_15696_18347.png 04_Andrew_Turner_720p_18348_19691.png 04_Andrew_Turner_720p_19692_20111.png 04_Andrew_Turner_720p_20112_20423.png 04_Andrew_Turner_720p_20424_21191.png 04_Andrew_Turner_720p_21192_21311.png 04_Andrew_Turner_720p_21312_22295.png 04_Andrew_Turner_720p_22296_23711.png 04_Andrew_Turner_720p_23712_25055.png 04_Andrew_Turner_720p_25056_25931.png 04_Andrew_Turner_720p_25932_29555.png 04_Andrew_Turner_720p_29556_30503.png 04_Andrew_Turner_720p_30504_30767.png 04_Andrew_Turner_720p_30768_31391.png 04_Andrew_Turner_720p_31392_32951.png 04_Andrew_Turner_720p_32952_32999.png 04_Andrew_Turner_720p_33000_33599.png 04_Andrew_Turner_720p_33600_34559.png 04_Andrew_Turner_720p_34560_34871.png 04_Andrew_Turner_720p_34872_35567.png 04_Andrew_Turner_720p_35568_36635.png 04_Andrew_Turner_720p_36636_37427.png 04_Andrew_Turner_720p_37428_37444.png