The Experience and lessons learned
In an effort to find interesting information in a volume of crime data provided by the Secure Police Department (a fictitious name) we applied data mining techniques and algorithms to this data. We discovered that data mining is an effort that takes a lot of planning and preparation, requires domain expertise, and is extremely resource intensive. The results we got are lackluster, for instance, correlations and patterns from the results are readily apparent, they are very rarely of the "interesting" variety. However, we think the experience and lessons learned have helped us and will help others be prepared for any future similar endeavors. The lessons can be summarized as: 1) Planning and preparation is a key to successful data mining, 2) Domain expertise is a necessity to perform successful data mining, 3) Data mining applications are extremely processor and main memory resource intensive and the appropriate hardware and software are necessary for successful data mining. InTRoduCTIon And BACKgRound In the United States today crime is a major concern. The CIA, FBI, and other federal agencies are concerned with national security; city and county law enforcement agencies keep a constant vigil on criminal activities in their own jurisdictions; and individuals are worried about their safety within the community. In the meantime, the budget for local law enforcement and intelligence agencies takes a hit due to mediocre economic growth, which leads to downsizing of both personnel and programs. To better make decisions and optimally allocate limited resources, local law enforcement and intelligence agencies need better quality information in a timely manner. It has been observed that "A major challenge facing all law enforcement and intelligence gathering organizations is accurately and efficiently analyzing the growing volume of crime data" (Chen et al. 2004) As the volume of this crime data becomes enormously large, new techniques have to be used to turn this data into usable information and knowledge; and thus, appropriate actions can be taken accordingly. Data mining holds the promise of making it easy, convenient, and practical to explore very large databases for law enforcement and intelligence agencies. Data mining is a powerful tool that uses complex algorithms to look for patterns in very large sets of data. Criminal investigators who may lack extensive training as data analysis can easily explore large databases quickly and efficiently through the implementation of data mining techniques in the form of commercial and other applications (Chen et al. 2004; Fayyad and Uthurusamy, 2002) The Shared Urban Data System (SUDS) is a web based secure portal that has been created as part of the Community Information Project by the Center for Community Criminology and Research at the Academic University (a fictitious name) located in the Mid south area. SUDS provides a host of information based services for the local community including neighborhood planning and development, public safety, and public health, etc. While SUDS is a portal to all information and services housed in a large number of databases, the CRISP (Crime Research Information Statistics Portal) system is a more specific subset of information services tied to a single database. The CRISP database is of crime incident information provided by the Secure Police Department as part of the Community Information Project. The database is used for queries and reports centered on offense types, geographic locations (through a GIS mapping interface) dates and time ranges, and suspect and victim information. This paper describes a project that applied a variety of data mining techniques to the CRISP database using a commercial data mining tool. The tool selection was based on the tool availability, usage characteristics, and architecture. The tools evaluated were IBM Intelligent Miner, SPSS Clementine, and Megaputer PolyAnalyst. Our eventual decision to use Megaputer PolyAnalyst 5.0 was due to the availability to the student for a research product, the ability to use the tool directly, via ODBC, in connection with an SQL Server 2000 server without an intermediate tier, and the incorporation of a large number of data mining algorithms available within the base program. lITERATuRE REvIEW Our summary of existing research literature on data mining of crime data is organized as follows. We first talk about the imperatives of applying data mining techniques to crime data and we then go into details on different data mining techniques that have been successfully applied to crime data. To us, the marriage between data mining techniques and crime data happened naturally. A retrospective review tells that there are roughly three major reasons why this happened. First, the volume of crime data has become unwieldy. When investigating a crime, law enforcement agencies usually record as much information as possible about the crime. After the attack on WTC in New York City on September 11, 2001, concern about national and local security has increased dramatically. At a national level, the FBI, CIA and Homeland Security are all acting to collect and analyze information to prevent further terror attacks. These efforts have had an affect on local authorities, motivating them to monitor criminal activities in their own jurisdictions using similar intelligence techniques (Chen et al. 2004) All above activities result an exponential increase in the volume of crime data. Manual processing of this data is not feasible anymore; new techniques to accurately and efficiently process crime data had to be found. Second, the budget for local law enforcement and intelligence agencies continues to see reductions due to mediocre economic growth (as of this writing in 2006) These economic factors lead to downsizing of both law enforcement personnel and programs. To better make decisions and optimally allocate limited resources, leadership in law enforcement needs quality information in a timely manner. Third, its ability to reveal patterns in extremely large data sets makes data mining a tool that fits this job perfectly (Brown, 1998) Compared to manual processing, which is time and labor intensive, data mining holds the promise of making it easy, convenient, and practical to explore very large stores of crime data for law enforcement organizations and users. There are many data mining techniques can be applied to crime data. Some examples of the specific applications and techniques of this work are given below as derived from current scholarly literature on the data mining of crime data. Lin and Brown (2006) present an outlier based data association method for linking criminal incidents. In this technique, according to them, "an outlier score function is defined to measure the extremeness of an observation, and a data association method is developed based upon the outlier score function." They applied this Managing Worldwide Operations Communications with Information Technology 431 Copyright (c) 2007, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. method to the robbery data in Richmond, Virginia, and compared the result with a similarity based association method. Their results show that the outlier based data association method is promising. Estivill Castro and Lee (2001) incorporate two knowledge discovery techniques, clustering and association rule mining, into a fruitful exploratory tool for the discovery of spatial temporal patterns. They present two methods for this exploratory analysis and the detail algorithms to effectively explore geo referenced data. They illustrate the algorithms with real crime data. They demonstrate their approach to a new type of analysis of the spatial temporal dimensions of records of criminal events. Brown (1998) describes a software framework for building and applying data mining algorithms to crime analysis problems. This framework provides specific focus on spatial data mining. The author provides several reasons to justify this focus: 1) spatial queries are more time consuming, 2) spatial analysis is harder to do than analyses based on attribute matching, 3) spatial data mining has the potential to yield important immediate benefits for crime analysis as crimes have an inherently spatial component (Brantingham and Brantingham, 1984) and 4) spatial analysis is a key to law enforcement resource allocation. Chen et al. (2004) present a general framework that shows the relationship between data mining techniques applied in criminal and intelligence analysis and the crime types. They identify and arrange eight crime types (traffic violations, sex crime, theft, fraud, arson, gang/drug offences, violent crime, and cyber crime) in increasing order of public harm on the horizontal axis. On the vertical axis, they arrange the techniques in increasing order of analysis capability. They identified four major categories of crime data mining techniques: entity extraction, association, prediction, and pattern visualization. Each category represents a set of techniques for use in certain types of crime analysis. They then identified the intersection of the techniques with the crime types denoting where each technique could be effectively used for each crime type, completing the framework. ThE ARChITECTuRE uSEd In ThIS STudy PolyAnalyst is a product of Megaputer, and Version 5.0 is the latest version of PolyAnalyst. It is a data mining tool that incorporates a large number of data mining algorithms in a single package. It can be configured to work as a traditional client server application or can be set up in a multi tier architecture with a dedicated PolyAnalyst server. The client server implementation can be configured to read the data from the database importing all of the data into the project file created by PolyAnalyst or, alternatively, a subset of the algorithms can be used,