<jupyter_start><jupyter_text>Forecasting Brazil’s Oil Production using an ARIMA model and ANP dataLuís <NAME> - BS in Information Technology, Federal University of Rio Grande do NorteDate: November 12, 2018 IntroductionMy project aims to predict the yearly oil production of a given petroleum reservoir over a given time period, using an ARIMA model for time series forecasting and the [ANP public data base](http://www.anp.gov.br/dados-estatisticos).I also aim to predict the top-10 petroleum reservoir in the Brazil that which will produce the largest amount of oil in the next 10 years. OBJECTIVE 1: Predicting the Oil Production of a Petroleum Reservoir across a Specified Time Period Using the ANP public data base, we will do the following steps below:Pre-Processing:Build a dataframe with the columns: dt (in DateTime format), well, latitude, longitude.Dropping irrelevant columns and removing rows with NaN valuesProcessing:ARIMA models need the data to be stationary i.e. the data must not exhibit trend and/or seasonality. To identify and remove trend and seasonality, we used the following methods:Plotting the time series to visually check for trend and seasonalityChecking if the histogram of the data fits a Gaussian Curve, and then splitting data into two parts, calculating means and variances and seeing if they varyCalculating the Augmented Dickey-Fuller Test statistic and using the p-value to determine stationarityIf the data was not stationary, we performed differencing to make it stationary.Fitting the ARIMA model:We performed a grid-search to estimate the best p, q values for the model, for the given data.We then fit the ARIMA model using the calculated p, q values.Evaluation:We calculated the Mean Squared Error (MSE) to estimate the performance of the model.<jupyter_code># import packages
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from statsmodels.tsa.stattools import adfuller
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from statsmodels.tsa.arima_model import ARIMA, ARMAResults
from sklearn.metrics import mean_squared_error
import ipywidgets as widgets
import plotly.plotly as py
import seaborn as sns
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import plotly.graph_objs as go
init_notebook_mode(connected=True)

# hide warnings
import warnings
warnings.simplefilter("ignore")
import pandas as pd

#read file with the weels
dfs = pd.read_excel('data/tabela_de_pocos_abril_2018.xlsx', sheet_name='Plan1')
dfs = dfs.drop({'CADASTRO', 'OPERADOR', 'POCO_OPERADOR', 'ESTADO', 'BACIA', 'BLOCO', 'SIG_CAMPO', 'CAMPO', 'TERRA_MAR', 
                'POCO_POS_ANP', 'INICIO', 'TERMINO', 'CONCLUSAO', 'TITULARIDADE', 'LATITUDE_BASE_4C', 
                'LONGITUDE_BASE_4C', 'DATUM_HORIZONTAL', 'TIPO_DE_COORDENADA_DE_BASE', 'DIRECAO', 'PROFUNDIDADE_VERTICAL_M',
                'PROFUNDIDADE_SONDADOR_M','PROFUNDIDADE_MEDIDA_M', 'REFERENCIA_DE_PROFUNDIDADE', 'MESA_ROTATIVA', 
                'COTA_ALTIMETRICA_M','LAMINA_D_AGUA_M', 'DATUM_VERTICAL', 'UNIDADE_ESTRATIGRAFICA', 'GEOLOGIA_GRUPO_FINAL',
                'GEOLOGIA_FORMACAO_FINAL', 'GEOLOGIA_MEMBRO_FINAL', 'CDPE', 'AGP', 'PC', 'PAG', 'PERFIS_CONVENCIONAIS',
                'DURANTE_PERFURACAO', 'PERFIS_DIGITAIS', 'PERFIS_PROCESSADOS', 'PERFIS_ESPECIAIS', 'AMOSTRA_LATERAL', 
                'SISMICA', 'TABELA_TEMPO_PROFUNDIDADE', 'DADOS_DIRECIONAIS', 'TESTE_A_CABO', 'CANHONEIO', 'TESTEMUNHO', 
                'GEOQUIMICA', 'SIG_SONDA', 'NOM_SONDA', 'DHA_ATUALIZACAO'}, 1)
def get_produtors(dfs):
    '''
    Filters producting wells in a field
    '''
    return dfs[(dfs['TIPO']==u'Explotatório') & (dfs['SITUACAO']=='PRODUZINDO') & 
               (dfs['CATEGORIA']=='Desenvolvimento') & (dfs['RECLASSIFICACAO']==u'PRODUTOR COMERCIAL DE PETRÓLEO')]
def get_injections(dfs):
    '''
    Filters the injection wells in a field
    '''
    return dfs[(dfs['TIPO']==u'Explotatório') & (dfs['SITUACAO']=='INJETANDO') & \
            (dfs['CATEGORIA']==u'Injeção') & (dfs['RECLASSIFICACAO']==u'INJEÇÃO DE ÁGUA')]
produtors = get_produtors(dfs)
injetors = get_injections(dfs)
dfs = pd.concat([produtors, injetors]).reset_index(inplace=True)
dfs<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09257950530035336, 'fraction_numerical': 0.00447585394581861, 'mean_word_length': 5.207602339181286, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '249209', 'license': 'MIT', 'n_tokens_mistral': 1516, 'n_tokens_neox': 1418, 'n_words': 428, 'path': '.ipynb_checkpoints/ForecastingBrazilOilProduction2-checkpoint.ipynb', 'repo_name': 'cruxiu/forecasting-oil-production', 'size': '4262'}