Research and Implementation of Network Public Opinion Analysis System Based on Hadoop Platform
Through the network data analysis, the user can be informed of network terminal preference and the network service preference and the web browser preferences of user behavior characteristics, and then targeted to optimize network traffic, improve the user experience, to increase user activity and retention, bring more profit to the enterprise. It can be said that the Internet has become the most influential in the field of information dissemination, one of the most potential of the mainstream media, but also gradually become one of the mainstream media bearer of public opinion. In this case, the negative, negative network public opinion is very easy to spread among the majority of Internet users, the harmonious development of society have a great impact. It is necessary to use the modern Natural Language Processing and data mining technology, network data analysis and processing, it is of great significance for the relevant government departments to provide timely and accurate information of network public opinion. This system using Hadoop cloud platform advantages in data processing, divided into data acquisition, data preprocessing, data clustering, public opinion analysis, results show the five functional modules, to achieve various functions of the system analysis of network public opinion. Among them, the data acquisition module to data acquisition technology according to the characteristics of different data sources, the news website, use Nutch data collection, the micro blog website, using the API interface to provide their own data acquisition; data preprocessing module using Fu dan NLP in Chinese word processing, and establish a stoplist, no actual filter the meaning of adverbs and prepositions, auxiliary words, the establishment of the text using the TF IDF algorithm based on vector space; the data clustering module, according to the Chinese language and its own characteristics, such as synonymy, polysemy, clustering algorithm is proposed for Kmeans, Canopy and semantic similarity combination, improving the accuracy of clustering. Which provides the ability to find the network public opinion; public opinion analysis module in the topic sensitive topic detection and hot spot detection Content analysis and other public opinion indicators; in the results of the module, the use of Web presentation network public opinion information. This paper shows the whole process of system development, based on the characteristics of the network public opinion, study the implementation method of the system of network public opinion analysis, this paper describes the research background and significance of this subject, the research status at home and abroad, the research goal and the structure of the dissertation, introduces the advantages of Hadoop platform, in the data processing of data the text collection technology, vector space model, text clustering algorithm and the function of public opinion.