Prediction of the crude oil price thanks to natural language processing applied to newspapers
This study presents how commodity prices can be related to newspapers. More precisely, we will focus in this paper on the crude oil prices. We used topic modeling (Latent Dirichlet Allocation) to extract major topics from news articles. As a result, our features were the daily and montly proportions of articles for each topic. Our output was the daily and monthly stock oil prices adjusted by the usd inflation. We tested different number of topics and preprocessed our dataset so as to end up with relevant topics. Then, we implemented different models to fit our data: linear regression, Theil Sen regression, ridged least square, locally weighted regression. We also introduced polynomial features. We got promising results, especially with the locally weighted regression using 60 topics trained over 90% of our dataset: we were able to follow the dynamics of the crude oil prices including peaks or sudden moves and our explained variance was equal to R 0.70. Introduction Nowadays, oil is one of the most important commodities in the world. It is used everywhere from the plastic to the bitumen via many fertilizers and pesticides. It also has a strong influence on the way stock markets behave: it can trigger an economic crisis as well as push inflation in Europe or in the US. It has now became the center of many attention, and as a result, it is often discussed in newspapers. In this project, we focused on the correlations that could exist between the stock and commodity prices of crude oil and external factors highlighted in newspapers. Indeed, most financial studies focus on numerical data in order to predict price movements because it is easier to manage and easily accessible. However, this process cannot take into account the macroeconomics events or even simple news that nevertheless impact a lot the stock oil prices. Thanks to machine learning and NLP (natural language processing) techniques, more and more documents can be processed in a semi automated way. In this study, we use topic modeling (Latent Dirichlet Allocation) to extract the main topics from the articles in newspapers such as New York Times, Reuters and the Associated Press so as to predict the movement of the stock oil price. Dataset We used the NY Times API to get automatically all articles from 1986 to 2015 containing the words "oil price" It was not possible to get the whole text of the articles but most of the time, we were able to extract the headline, an abstract and a snippet. We also used several API, the nytimesarticle package, and the python function time.sleep( to avoid the limitations of 5 articles per second and 1000 articles per day. Eventually, we successfully downloaded around 32,000 articles. We also used the crude oil price data from the EIA website as well as the US inflation from the federal Bureau of Labor Statistics. NLP methods Several machine learning algorithms could be used to process newspapers articles. Two of them lead to good results: the word2vec approach, and the topic modeling one. A short literature review showed that topic modeling was the most appropriate, as it would allow us to learn the topics present in our documents, which is exactly what we need to know in order to predict the stock oil price. Topic Modeling: Latent Dirichlet Allocation (LDA) In this section, we will offer a quick review of an algorithm that allows topic modeling, before describing how we applied it to our problem. Following the ideas of [1] we decided to use a Latent Dirichlet Allocation technique, for its capacity to capture multiple topics within a document. As the focus of this document is more on our methodology and first results, we will only give a brief description of the LDA algorithm. A complete one can be found in [1] The intuition behind the LDA is the following: each document is a "mixture" of different topics. A document may be 90 about "oil" and 10% about "cars" (see figure 1) We now make an assumption: if a document is composed of 90 about "oil" and 10% about "cars" then it is constructed by randomly sampling 90 of its words from a distribution about "oil" and 10% from a distribution about "cars" (the ordering of the words does not matter to the algorithm) We end up with three hidden variables to explain our corpus: