Ravinder Singh, Marina Levina, Nelson Jiao, Asha Saini &ndash; DELL EMC
Using open data to predict market movements
DELL EMC
market research, nlp, information retrieval
We found that The Register articles for specific vendors extracted from the common crawl data set are highly correlated with our reading of General Purpose Magic Quadrant position movements in time. [...] The Figure 11 : Common Crawl Data Processing Flow Diagram shows a broad overview of the steps involved in the analysis of common crawl data. Going from the bottom up it shows how the data is extracted, processed and visualized. The amount of data in each phase becomes more streamlined and, hence, the reduction in size of the data being worked on. We start with the crawl data, extract the pages of interest int o a private storage bucket, and then process it to remove unwanted words/tags. At the end, visualization tools are used to graphically display the results. These can be used to publish standard reports or customized by users to support their own analysis.