<jupyter_start><jupyter_text>Spatial Dynamics: Markov based methods Markov chain's assumption is that the observations are on the rows of the input and the different points in time on the columns.<jupyter_code>import pysal
import numpy as np

# 3 states (a,b,c) and 5 different pixels at three different points in time.
# So the first pixel was in class ‘b’ in period 1, class ‘a’ in period 2, and class ‘c’ in period 3.
c = np.array([['b','a','c'],['c','c','a'],['c','b','c'],['a','a','b'],['a','b','c']])
c.shape, c
m = pysal.Markov(c)
m.classes  # array (k, 1), all different classes (bins) of the matrix.
m.transitions  # matrix (k, k), count of transitions between each state i and j.
m.p  # matrix (k, k), transition probability matrix.
m.steady_state  # matrix (k, 1), ergodic distribution.
import geopandas as gpd
import pysal as ps
import matplotlib.pyplot as plt
import pandas as pd
import random
import numpy as np
import datetime as dt
import glob
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from pylab import figure, scatter, show
from matplotlib import colors
import csv
import requests
import zipfile, urllib, os
from urllib.request import Request,urlopen, urlretrieve
import urllib
import io

import warnings
warnings.filterwarnings('ignore')
urls = ['https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2017-{:02d}.csv'.format(a) for a in range(1,13)]

list_ = []
for url in urls:
    csv_response = requests.get(url)
    c = pd.read_csv(io.StringIO(csv_response.text)).sample(frac=0.001)
    list_.append(c)
    
df = pd.concat(list_)
df_clean = df[(df['passenger_count'] < 10) & 
                        (df['passenger_count'] > 0) & 
                        (df['extra'] >= 0) &
                        (df['extra'] <= 1) &
                        (df['RatecodeID'] < 7) &
                        (df['mta_tax'] >= 0) &
                        (df['trip_distance'] > 0) &
                        (df['tip_amount'] >= 0) &
                        (df['tolls_amount'] >= 0) &
                        (df['improvement_surcharge'] > 0) &
                        (df['total_amount'] > 0)]

df_clean = df_clean[(df_clean['payment_type'] == 1)] # only credit card payment

df_clean['tpep_pickup_datetime'] = df_clean.tpep_pickup_datetime.apply(
                                    lambda x:dt.datetime.strptime(x,"%Y-%m-%d %H:%M:%S"))


df_clean['pickup_month'] = df_clean.tpep_pickup_datetime.apply(lambda x: x.month)

df_clean = df_clean[(np.abs(stats.zscore(df_clean[['tip_amount']]))<2.5).all(axis=1)]
taxi_zone_tip_df = pd.pivot_table(df_clean, values='tip_amount', index=['PULocationID'], columns=['pickup_month'], aggfunc=np.mean)
taxi_zone_tip_df['PULocation'] = taxi_zone_tip_df.index
taxi_zone_tip_df.shape
taxi_zones = gpd.read_file('taxi_zones_tip.shp')
taxi_zone_tip = pd.merge(taxi_zones, taxi_zone_tip_df, on='PULocation', how='left')
taxi_zone_tip.shape, taxi_zones.shape
taxi_zone_tip.to_csv('taxi_zone_tip.csv', float_format='%.3f', index=False, header=True, sep=",", decimal=".")<jupyter_output><empty_output><jupyter_text>Classic Markov In this section, all the spatial units are treated as independent- The transition dynamics are assumed to hold for all units and for all time periods.- Tnteractions between the transitions are ignored.<jupyter_code>zone_tip = taxi_zone_tip[[1,2,3,4,5,6,7,8,9,10,11,12]]
#zone_tip = zone_tip.dropna()

zone_tip = np.array(zone_tip)
#zone_tip.index = taxi_zone_tip['zone']
zone_tip_q = np.array([ps.Quantiles(y).yb for y in zone_tip]).transpose()
zone_tip_q.shape
markov_tip = ps.Markov(zone_tip_q)
markov_tip.classes
markov_tip.transitions<jupyter_output><empty_output><jupyter_text>- count of transitions between each state i and j<jupyter_code>markov_tip.p<jupyter_output><empty_output><jupyter_text>- transition probability matrix<jupyter_code>markov_tip.steady_state<jupyter_output><empty_output><jupyter_text>- ergodic distribution.- Prob. of the 1st quantil state of tip amount = 72%    * Ergodicity: a state i is said to be erdodic if it is aperiodic and positive recurrent.<jupyter_code>ps.ergodic.fmpt(markov_tip.p)<jupyter_output><empty_output><jupyter_text>- calculates the matrix of first mean passage times for an ergodic transition probability matrix.  Spatial Markov The spatial Markov allows us to compare the global transition dynamics to those conditioned on regional context.단순히 전체적으로 한 상태에서 다른 상태로 변환 간으한 확률값을 보여주는 것에서 더 나아가서 각 상태값을 갖는 지역들 가까이에 어던 상태의 지역이 있느냐에 따라 다음 상태로 넘어가는 확률을 보여준다.|<jupyter_code>W = ps.queen_from_shapefile("taxi_zones_tip.shp")
W.transform = 'r'
W.sparse
f = pd.read_csv("taxi_zone_tip.csv",sep=',')
pci = np.array([f[str(y)] for y in [3,6,11]])

pci.shape
pci = pci.transpose()
pci = np.nan_to_num(pci)
rpci = pci / (pci.mean(axis = 0))
rpci.shape

sm = ps.Spatial_Markov(rpci, W, k = 5)<jupyter_output><empty_output><jupyter_text>The global transition probability matrix for relative tip amount:<jupyter_code>sm.p
for p in sm.P:
    print(p)<jupyter_output>[[0.94545455 0.01212121 0.01818182 0.02424242 0.        ]
 [0.         0.         0.         0.         0.        ]
 [0.         0.         0.         0.         0.        ]
 [0.66666667 0.         0.         0.33333333 0.        ]
 [0.         0.         0.         0.         0.        ]]
[[1.   0.   0.   0.   0.  ]
 [0.   0.   0.   0.   0.  ]
 [0.   0.   0.   0.   0.  ]
 [0.5  0.   0.25 0.25 0.  ]
 [0.   0.   0.   0.   0.  ]]
[[0.89830508 0.01694915 0.03389831 0.05084746 0.        ]
 [0.5        0.33333333 0.16666667 0.         0.        ]
 [0.33333333 0.33333333 0.11111111 0.22222222 0.        ]
 [0.5        0.1875     0.0625     0.25       0.        ]
 [0.         0.         0.         0.         0.        ]]
[[0.8        0.06666667 0.         0.13333333 0.        ]
 [0.         0.33333333 0.66666667 0.         0.        ]
 [0.02222222 0.42222222 0.48888889 0.06666667 0.        ]
 [0.03703704 0.18518519 0.40740741 0.37037037 0.        ]
 [0.         0.         0.         0.        [...]<jupyter_text>The different steady state distributions implied by these different transition probabilities: - The prob. of the 1st quantil state remaining in the 1st quantil     - if their neighbors are in the 1st quantil: 0.945    - if their neighbors are in the 5th quantil: 0.75- The prob. of the 5th quantil state reamining in the 5th quantil: 0<jupyter_code>for f in sm.F:
    print(f)<jupyter_output>[[  1.06181779  78.66949632 158.36978515  30.50032936   1.        ]
 [  2.04997403  79.70028883 159.40057766  31.27632649   1.        ]
 [  2.04997403  79.70028883 159.40057766  31.27632649   1.        ]
 [  1.22690895  79.89640527 159.5966941   25.38179064   1.        ]
 [  2.04997403  79.70028883 159.40057766  31.27632649   1.        ]]
[[1.   1.   1.   1.5  1.  ]
 [2.   1.   1.   1.5  1.  ]
 [1.   1.   1.   1.5  1.  ]
 [1.75 1.   0.75 1.   1.  ]
 [2.   1.   1.   1.5  1.  ]]
[[ 1.47250032 31.09090909 16.10029499  7.24363636  1.        ]
 [ 2.07578254 31.06818182 14.33628319  5.97818182  1.        ]
 [ 3.32454695 27.45454545 11.4252704   3.60727273  1.        ]
 [ 2.64085667 28.81818182 13.09439528  4.97090909  1.        ]
 [ 2.889058   31.31469825 16.02963523  7.42742553  1.        ]]
[[ 1.         14.93333333  5.1         3.33333333  1.        ]
 [ 6.5         1.         10.2         6.66666667  1.        ]
 [ 6.5        14.78333333  1.          2.08333333  1.        ]
 [ 6.5       [...]<jupyter_text>- The 1st quantil state with neighbors in the 1st quantil    - return to the 1st qauntil after 1.06 months after leaving the 1st quantil    - they enter the 5th quanil a month after.- The 1st quantil state within neighbors in the 5th quantil    - return to the 1st quatil after 12.5 months    - enter the 5th quantil after a month.  LISA (Local Indicators of Spatial Associations) Markov To consider the joint transitions of an observation and its spatial lag in the distribution- the staties of the chain are defined as the four quadrants in the Moran scatter plot.<jupyter_code>lm = ps.LISA_Markov(rpci, W)
lm.classes
lm.transitions<jupyter_output><empty_output><jupyter_text>The estimated transition probability matrix is:<jupyter_code>lm.p<jupyter_output><empty_output><jupyter_text>- The diagonal values indicate the staying prob. and the staying prob. of the 1st and 3rd quantil is high. The implied long run steady state distribution of the chain is:<jupyter_code>lm.steady_state<jupyter_output><empty_output><jupyter_text>- The 3rd quantil state might have a positive autocorrelation. Finally the first mean passage time for the LISAs is:<jupyter_code>ps.ergodic.fmpt(lm.p)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11742949607027277, 'fraction_numerical': 0.10725843735552473, 'mean_word_length': 2.7523850823937552, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 55, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 15, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '597853', 'license': 'MIT', 'n_tokens_mistral': 3786, 'n_tokens_neox': 3255, 'n_words': 855, 'path': 'data/Spatial_Dynamics/Markov based methods.ipynb', 'repo_name': 'pinguino-lim/Taxi_NY', 'size': '8652'}