Bad big data science
As hardware and software technologies have improved, our definition of a "manageable amount of data" has increased in its scope dramatically. The term "big data" can be applied to any of several different projects and technologies sharing the ultimate goal of supporting analysis on these large, heterogeneous, and evolving data sets. The term "data science" refers to the statistical, technical, and domain specific knowledge required to ensure that the analysis is done properly. Techniques for managing some common causes for bad data and invalid analysis have been used in other areas, such as data warehousing and distributed database. However, big data projects face special challenges when trying to combine big data and data science without producing inaccurate, misleading, or invalid results. This paper discusses potential causes for "bad big data science" focusing primarily on the data quality of the input data, and suggests methods for minimizing them based on techniques originally developed for data warehousing and distributed database projects.
2016 IEEE International Conference on Big Data (Big Data)