<jupyter_start><jupyter_text>colab 구글 드라이브 연동<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>DB 데이터 불러온 후 전처리<jupyter_code>import numpy as np
import pandas as pd
data = pd.read_csv("/content/drive/MyDrive/Zipbob_timeseries/pricedb예제data.csv", encoding='cp949') # 임시로 DB데이터 CSV로 저장 후 불러오기
data.head(10)
data.tail()
data.shape # 기존 데이터 335,775개
data = data.drop_duplicates()
data.shape # 중복제거 후 314,270개
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 314270 entries, 0 to 331753
Data columns (total 4 columns):
 #   Column   Non-Null Count   Dtype 
---  ------   --------------   ----- 
 0   itemnm   314270 non-null  object
 1   kindnm   314270 non-null  object
 2   regdate  314270 non-null  object
 3   price    314270 non-null  object
dtypes: object(4)
memory usage: 12.0+ MB<jupyter_text>시계열 가격예측을 위한 카테고리 생성- itemnm-kindnm조합 추출하기 83개- item마다 kindnm이 여러개 있는 경우가 있고, item마다 동일한 단위 '국산(1kg)'처럼 중첩되므로 item으로만 분류하지 않음(EDA코드확인)<jupyter_code>itemlist = list(data['itemnm'].unique())
itemkind=[] # 리스트 생성
for i in itemlist :
    dataset = data[data['itemnm']==i]
    kind_unique = list(dataset['kindnm'].unique())
    for k in range(len(kind_unique)) :
        itemkind.append([i, kind_unique[k]])
itemkind[:10], len(itemkind)
data['itemkind'] = data['itemnm']+'-'+data['kindnm']<jupyter_output><empty_output><jupyter_text>결측치 '-' 처리하기햇양파같은 제철야채 식품은 특정 기간에 price가 '-'으로 등록됨. 따라서 0 처리<jupyter_code>data1 = data.copy()
data1 = data1.replace('-', '0')<jupyter_output><empty_output><jupyter_text>price가격 object타입 변환 : , 없애고 숫자로<jupyter_code>data1['price'] = data1['price'].str.replace(',', '')
data1['price'] = data1['price'].astype('int64')<jupyter_output><empty_output><jupyter_text>데이터 전처리 완료.  저장해서 다시 불러오거나 바로 진행<jupyter_code># data1.to_csv('/경로설정/농수산DB기본전처리.csv', index=False, encoding='cp949')
# data = pd.read_csv('/content/drive/MyDrive/Zipbob_timeseries/농수산DB기본전처리.csv', encoding='cp949')
data = data1.copy()
data.head()<jupyter_output><empty_output><jupyter_text>가격 예측 진행  필요한 모듈 설치<jupyter_code># !pip install tzlocal
import tzlocal
# !pip install rpy2
import rpy2
!pip install pmdarima==1.5.3 # arima 용 설치
import pmdarima # arima 용 설치
import statsmodels
import pandas as pd
import numpy as np
import datetime
import seaborn as sns
from tqdm import tqdm
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>(arima, ets용 forecast 설치. 오래걸림)<jupyter_code># from rpy2.robjects.packages import importr
# utils = importr('utils')
# utils.install_packages('forecast')<jupyter_output><empty_output><jupyter_text>시계열용 datetime 변환<jupyter_code>traindf = data.copy()
traindf['regdate'] = pd.to_datetime(traindf['regdate'],format="%Y-%m-%d")
traindf = traindf.drop(['itemnm', 'kindnm'], axis=1)<jupyter_output><empty_output><jupyter_text>fbprophet 세팅<jupyter_code>!python --version
!pip install pystan==2.19.1.1
!pip install prophet
# 관련된 오류나면 버전 바꿔서 실행하기
# !pip install cmdstanpy==0.9.5
import fbprophet
print(fbprophet.__version__)
from fbprophet import Prophet<jupyter_output><empty_output><jupyter_text>fbprophet에 맞게 컬럼명 ds(날짜), y(예측값)로 지정<jupyter_code>train_rename = traindf.rename(columns = {'regdate' : 'ds', 'price':'y'})
train_rename<jupyter_output><empty_output><jupyter_text>시계열 예측 대상 날짜구간 생성하기DB의 최근 날짜에 맞춰서 +28일 데이터 자동 생성<jupyter_code>from datetime import datetime, timedelta
pred_dayslist = []
for k in range(1, 29) :
  a = train_rename['ds'].max()
  pred_dayslist.append(a+timedelta(days=k))
new_data = pd.DataFrame()
# 예측을 위한 ds컬럼 생성
new_data['ds'] = pred_dayslist<jupyter_output><empty_output><jupyter_text>fbprophet 모델링<jupyter_code>item_pred_list = []
itemkind_name_list = []<jupyter_output><empty_output><jupyter_text>- 야채 itemkind마다 모델 생성 -> 모델링하므로 colab에서 20분 이상 소요- 전체 django프로젝트 실행 시 메모리할당량 초과되면 colab에서 별도로 예측 데이터 추출 후 활용하기<jupyter_code>for i in tqdm(train_rename.itemkind.unique()): # 진행상황 표시하는 tqdm
  selected_train = train_rename[train_rename['itemkind']==i] # 각 야채카테고리별 데이터셋 생성

  model = Prophet(
    # trend
    changepoint_prior_scale=0.5,
    # seasonality
    weekly_seasonality=10,
    yearly_seasonality=10,
    daily_seasonality=True,
    seasonality_mode='multiplicative'
  )
  model.fit(selected_train)

  forecast = model.predict(new_data)
  y_pred = forecast['yhat'].values
  print(i, "예측값완료")
  
  itemkind_name_list.append(i)
  item_pred_list.append(y_pred)
# itemkind_name_list
len(item_pred_list)
item_pred_list[:3]
y_pred # 가장 마지막 야채컬럼 예측값(0은 마이너스)<jupyter_output><empty_output><jupyter_text>농수산별 예측값 통합 DataFrame 생성<jupyter_code>formerge = pd.DataFrame()
formerge['pred_date'] = None
formerge['itemkind'] = None
formerge['price_pred'] = None

for i in range(len(itemkind_name_list)) :
  formerge1 = pd.DataFrame()
  formerge1['pred_date'] = new_data['ds']
  formerge1['itemkind'] = itemkind_name_list[i]
  formerge1['price_pred'] = item_pred_list[i]
  formerge1['itemnm'] = itemkind_name_list[i].split("-")[0]
  formerge1['kindnm'] = itemkind_name_list[i].split("-")[1]
  formerge = pd.concat([formerge,formerge1],axis=0)
formerge = formerge.reset_index()
formerge.drop('index', axis=1, inplace=True)
formerge
# formerge = formerge.drop_duplicates()<jupyter_output><empty_output><jupyter_text>마이너스 값 0처리<jupyter_code>formerge.info()
def cleansing(x) :
  if x < 0 : x = 0
  return x
formerge['price_pred'] = formerge['price_pred'].apply(cleansing)
formerge
formerge['price_pred'].min()<jupyter_output><empty_output><jupyter_text>최종 결과 저장 저장 경로 바꿔서 진행<jupyter_code>formerge.to_csv('/content/drive/MyDrive/Zipbob_timeseries/kamis_4weeks_predict.csv', index=False, encoding='cp949')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13833157338965152, 'fraction_numerical': 0.024111228440689896, 'mean_word_length': 6.248724489795919, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '107104', 'license': 'Apache-2.0', 'n_tokens_mistral': 2692, 'n_tokens_neox': 2979, 'n_words': 425, 'path': 'ML/timeseries/KAMISprice_timeseries_predict_final.ipynb', 'repo_name': 'Songgplant/Zipbob', 'size': '5682'}