Forecasting with Big Data: A Review <s> Architecture of Algorithms <s> Abstract Along with the increasing availability of large databases under the purview of National Statistical Institutes, the application of data mining techniques to official statistics is now a hot topic that is far more important at present than it was ever before. Presented in this article is a thorough review of published work to date on the application of data mining in official statistics, and on identification of the techniques that have been explored. In addition, the importance of data mining to official statistics is flagged and a summary of the challenges that have hindered its development over the course of the last two decades is presented. <s> BIB001 </s> Forecasting with Big Data: A Review <s> Architecture of Algorithms <s> Services like social networks, web analytics, and intelligent e-commerce often need to manage data at a scale too big for a traditional database. As scale and demand increase, so does Complexity. Fortunately, scalability and simplicity are not mutually exclusiverather than using some trendy technology, a different approach is needed. Big data systems use many machines working in parallel to store and process data, which introduces fundamental challenges unfamiliar to most developers. Big Data shows how to build these systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data. It describes a scalable, easy to understand approach to big data systems that can be built and run by a small team. Following a realistic example, this book guides readers through the theory of big data systems, how to use them in practice, and how to deploy and operate them once they're built. Purchase of the print book comes with an offer of a free PDF, ePub, and Kindle eBook from Manning. Also available is all code from the book. <s> BIB002
Data Mining techniques are suggested as important methods which could be used for forecasting with Big Data. However, these techniques have been designed to handle data of comparatively smaller sizes as opposed to the size of Big Data. Therefore, Data Mining algorithms are often unable to work with data that is not loaded on to its main memory, and thus requires the movement of Big Data between locations which can incur increased network communication costs  . The architecture of the analytics needs to be redesigned so that it could handle both historical and real time data  , and the Lambda architecture proposed in BIB002 is a good example of research currently seeking to overcome this issue. A detailed evaluation of challenges associated with the application of Data Mining techniques to Big Data (explained in the context of official statistics) can be found in BIB001 .