<jupyter_start><jupyter_text>The Battle of Neighborhoods  Introduction & Business Problem Introduction New York City is the most diverse and populated city in the United States (the city has a population density of over 27,000 people per square mile). In New York, you’ll find almost every race and language you can think of under the sun, hence the common moniker, “the melting pot of the world.” The city is made up of 5 boroughs: Manhattan, Brooklyn, Queens, the Bronx and Staten Island, all of which were “grouped” together into a single city in 1898.Due to the prosperity and grandeur of New York City resulting in people from many countries want to come to New York City whether coming to work or coming to study. Many Thai people also have come to New York for traveling, working, and studying that make Thai food is one of the popular foods in New York City. Business ProblemBecause many Thai people come to live in New York City. These people usually eat Thai food in their everyday life so they need to live in the area that has many Thai restaurants so that they can easily find a good Thai food.In addition, there are also many group of tourists whether Thai tourists who miss their home country food, people from other countries who like Thai food or even people who want to try Thai food would like to know information about which area in New York has many Thai restaurants so that there are many options for them.Therefore, I will solve this problem by answer the questions as following:1. Which neighborhood in New York City have the most Thai restaurants?2. Which neighborhood in New York City have the best average rating for Thai restaurants? Target Audience- Main target audience is Thai people who living in New York City want to live in an area with a variety of Thai restaurants.- Secondary target audience is tourists who interested in Thai food.  Data Section New York City Neighborhoods DataI will use New York City neighborhoods dataset that published from New York (City). Department of City Planning on the following url:https://geo.nyu.edu/catalog/nyu_2451_34572 This dataset contains 5 boroughs and 306 neighborhoods that exist in each borough as well as the latitude and the longitude coordinates of each neighborhood.In this dataset, I will extract borough, neighborhood, latitude and longitude to visualize the data New York City map. Thai Restaurants DataI will use Foursquare API which is well-known location data provider to extract venue data for Thai restaurant in New York City using Thai Restaurant (4bf58dd8d48988d149941735) venue category.  Methodology1. Extract borough, neighborhood, latitude and longitude of New York City from https://geo.nyu.edu/catalog/nyu_2451_345722. Convert data into pandas dataframe format for easy to analyze.3. Let's see the example of the data, shape and descriptive statistics.4. Extract Thai restuarants data in each neighborhood from Foursqare API5. Convert data into pandas dataframe and csv.6. Let's see the example of the data, shape and descriptive statistics.7. Visualize the data the answer the questions. Firstly, let's import nescessary libraries.<jupyter_code>import numpy as np # library to handle data in a vectorized manner

import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors
%matplotlib inline

import folium # map rendering library

print('Libraries imported.')<jupyter_output>Libraries imported.<jupyter_text>Extract data from newyork_data.json<jupyter_code>with open('newyork_data.json') as json_data:
    newyork_data = json.load(json_data)
neighborhoods_data = newyork_data['features']
neighborhoods_data[0]<jupyter_output><empty_output><jupyter_text>Convert json to dataframe<jupyter_code># define the dataframe columns
column_names = ['Borough', 'Neighborhood', 'Latitude', 'Longitude'] 

# instantiate the dataframe
neighborhoods = pd.DataFrame(columns=column_names)
neighborhoods
for data in neighborhoods_data:
    borough = neighborhood_name = data['properties']['borough'] 
    neighborhood_name = data['properties']['name']
        
    neighborhood_latlon = data['geometry']['coordinates']
    neighborhood_lat = neighborhood_latlon[1]
    neighborhood_lon = neighborhood_latlon[0]
    
    neighborhoods = neighborhoods.append({'Borough': borough,
                                          'Neighborhood': neighborhood_name,
                                          'Latitude': neighborhood_lat,
                                          'Longitude': neighborhood_lon}, ignore_index=True)<jupyter_output><empty_output><jupyter_text>Let's see the example of the data.<jupyter_code>neighborhoods.head()
print('The dataframe has {} boroughs and {} neighborhoods.'.format(
        len(neighborhoods['Borough'].unique()),
        neighborhoods.shape[0]
    )
)<jupyter_output>The dataframe has 5 boroughs and 306 neighborhoods.<jupyter_text>Then, extract Thai restuarants data from Foursquare API.<jupyter_code>def get_venues(lat,lng):
    
    #set variables
    CLIENT_ID = '2YZ3UDVZXAD4ODFRVNPW0FJ1D31LRWJH1RTFTH5GG0YNIQJE' # your Foursquare ID
    CLIENT_SECRET = '<KEY>' # your Foursquare Secret
    VERSION = '20200424' # Foursquare API version
    LIMIT = 500
    radius = 500
    
    #url to fetch data from foursquare api
    url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng)
    
    # get all the data
    results = requests.get(url).json()
    venue_data=results["response"]['groups'][0]['items']
    venue_details=[]
    for row in venue_data:
        try:
            venue_id=row['venue']['id']
            venue_name=row['venue']['name']
            venue_category=row['venue']['categories'][0]['name']
            venue_details.append([venue_id,venue_name,venue_category])
        except KeyError:
            pass
        
    column_names=['ID','Name','Category']
    df = pd.DataFrame(venue_details,columns=column_names)
    return df
# prepare neighborhood list that contains Thai resturants
column_names=['Borough', 'Neighborhood', 'ID','Name']
thai_rest_ny=pd.DataFrame(columns=column_names)
count=1
for row in neighborhoods.values.tolist():
    Borough, Neighborhood, Latitude, Longitude=row
    venues = get_venues(Latitude,Longitude)
    thai_resturants=venues[venues['Category']=='Thai Restaurant']   
    print('(',count,'/',len(neighborhoods),')','Thai Resturants in '+Neighborhood+', '+Borough+':'+str(len(thai_resturants)))
    for resturant_detail in thai_resturants.values.tolist():
        id, name , category=resturant_detail
        thai_rest_ny = thai_rest_ny.append({'Borough': Borough,
                                                'Neighborhood': Neighborhood, 
                                                'ID': id,
                                                'Name' : name
                                               }, ignore_index=True)
    count+=1<jupyter_output>( 1 / 306 ) Thai Resturants in Wakefield, Bronx:0
( 2 / 306 ) Thai Resturants in Co-op City, Bronx:0
( 3 / 306 ) Thai Resturants in Eastchester, Bronx:0
( 4 / 306 ) Thai Resturants in Fieldston, Bronx:0
( 5 / 306 ) Thai Resturants in Riverdale, Bronx:0
( 6 / 306 ) Thai Resturants in Kingsbridge, Bronx:0
( 7 / 306 ) Thai Resturants in Marble Hill, Manhattan:0
( 8 / 306 ) Thai Resturants in Woodlawn, Bronx:1
( 9 / 306 ) Thai Resturants in Norwood, Bronx:0
( 10 / 306 ) Thai Resturants in Williamsbridge, Bronx:0
( 11 / 306 ) Thai Resturants in Baychester, Bronx:0
( 12 / 306 ) Thai Resturants in Pelham Parkway, Bronx:0
( 13 / 306 ) Thai Resturants in City Island, Bronx:0
( 14 / 306 ) Thai Resturants in Bedford Park, Bronx:0
( 15 / 306 ) Thai Resturants in University Heights, Bronx:0
( 16 / 306 ) Thai Resturants in Morris Heights, Bronx:0
( 17 / 306 ) Thai Resturants in Fordham, Bronx:0
( 18 / 306 ) Thai Resturants in East Tremont, Bronx:0
( 19 / 306 ) Thai Resturants in West Farms, Bronx:0
[...]<jupyter_text>Let's see the example of Thai restuarants.<jupyter_code>thai_rest_ny.head()<jupyter_output><empty_output><jupyter_text>We found 66 Thai restuarants in New York City.<jupyter_code>thai_rest_ny.shape<jupyter_output><empty_output><jupyter_text>Keep data into csv to prevent API limit.<jupyter_code>thai_rest_ny.to_csv('thai_rest_ny.csv', index=False)
thai_rest_ny_csv = pd.read_csv('thai_rest_ny.csv')
thai_rest_ny_csv.head()
thai_rest_ny_csv.shape<jupyter_output><empty_output><jupyter_text>Let's see which borough has the most Thai restaurants.<jupyter_code>plt.figure(figsize=(9,5), dpi = 100)
# title
plt.title('Number of Thai Resturants for each Borough in New York City')
#On x-axis
plt.xlabel('Borough', fontsize = 15)
#On y-axis
plt.ylabel('No.of Thai Resturants', fontsize=15)
#giving a bar plot
thai_rest_ny_csv.groupby('Borough')['ID'].count().plot(kind='bar')
#legend
plt.legend()
#displays the plot
plt.show()<jupyter_output><empty_output><jupyter_text>We found that Queens has the most Thai restaurants in New York City. Then let's see which neighborhood has the most Thai restaurants.<jupyter_code>plt.figure(figsize=(9,5), dpi = 100)
# title
plt.title('Number of Thai Resturants for each Neighborhood in New York City')
#On x-axis
plt.xlabel('Neighborhood', fontsize = 15)
#On y-axis
plt.ylabel('No.of Thai Resturants', fontsize=15)
#giving a bar plot
thai_rest_ny.groupby('Neighborhood')['ID'].count().nlargest(5).plot(kind='bar')
#legend
plt.legend()
#displays the plot
plt.show()<jupyter_output><empty_output><jupyter_text>So, we found that Elmhurst has 9 Thai restaurants which is the most in New York City.<jupyter_code>thai_rest_ny_csv[thai_rest_ny_csv['Neighborhood']=='Elmhurst']<jupyter_output><empty_output><jupyter_text>Let's get more insight on each restaurant (Rating, Tips, Like) from Foursquare API.<jupyter_code>def get_venue_details(venue_id):
    #set variables
    CLIENT_ID = '2YZ3UDVZXAD4ODFRVNPW0FJ1D31LRWJH1RTFTH5GG0YNIQJE' # your Foursquare ID
    CLIENT_SECRET = '<KEY>' # your Foursquare Secret
    VERSION = '20200424' # Foursquare API version
    
    #url to fetch data from foursquare api
    url = 'https://api.foursquare.com/v2/venues/{}?&client_id={}&client_secret={}&v={}'.format(
            venue_id,
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION)
    
    # get all the data
    results = requests.get(url).json()
    venue_data=results['response']['venue']
    venue_details=[]
    try:
        venue_id=venue_data['id']
        venue_name=venue_data['name']
        venue_likes=venue_data['likes']['count']
        venue_rating=venue_data['rating']
        venue_tips=venue_data['tips']['count']
        venue_details.append([venue_id,venue_name,venue_likes,venue_rating,venue_tips])
    except KeyError:
        pass
        
    column_names=['ID','Name','Likes','Rating','Tips']
    df = pd.DataFrame(venue_details,columns=column_names)
    return df
# prepare neighborhood list that contains indian resturants
column_names=['Borough', 'Neighborhood', 'ID','Name','Likes','Rating','Tips']
thai_rest_stats_ny=pd.DataFrame(columns=column_names)
count=1


for row in thai_rest_ny_csv.values.tolist():
    Borough,Neighborhood,ID,Name=row
    try:
        venue_details=get_venue_details(ID)
        print(venue_details)
        id,name,likes,rating,tips=venue_details.values.tolist()[0]
    except IndexError:
        print('No data available for id=',ID)
        # we will assign 0 value for these resturants as they may have been 
        #recently opened or details does not exist in FourSquare Database
        id,name,likes,rating,tips=[0]*5
    print('(',count,'/',len(thai_rest_ny_csv),')','processed')
    thai_rest_stats_ny = thai_rest_stats_ny.append({'Borough': Borough,
                                                'Neighborhood': Neighborhood, 
                                                'ID': id,
                                                'Name' : name,
                                                'Likes' : likes,
                                                'Rating' : rating,
                                                'Tips' : tips
                                               }, ignore_index=True)
    count+=1<jupyter_output>ID         Name  Likes  Rating  Tips
0  4dc5d0b47d8b14fb463525e9  Accent Thai     14     8.4     4
( 1 / 66 ) processed
                         ID          Name  Likes  Rating  Tips
0  5585c948498e0bd8a8a706f1  Thai Cuisine     17     7.5     1
( 2 / 66 ) processed
                         ID          Name  Likes  Rating  Tips
0  5585c948498e0bd8a8a706f1  Thai Cuisine     17     7.5     1
( 3 / 66 ) processed
                         ID         Name  Likes  Rating  Tips
0  4dc5d0b47d8b14fb463525e9  Accent Thai     14     8.4     4
( 4 / 66 ) processed
                         ID                   Name  Likes  Rating  Tips
0  4b4c90d3f964a52046b626e3  Honey's Thai Pavilion     19     7.2    28
( 5 / 66 ) processed
                         ID          Name  Likes  Rating  Tips
0  596ffbcadee77066f36be772  Ruammit Thai     24     8.5     5
( 6 / 66 ) processed
                         ID                  Name  Likes  Rating  Tips
0  54daa265498e58e310cae373  Sawa[...]<jupyter_text>Let's see the example of the data.<jupyter_code>thai_rest_stats_ny.head()
thai_rest_stats_ny.shape
thai_rest_stats_ny.to_csv('thai_rest_stats_ny.csv', index=False)
thai_rest_stats_ny_csv=pd.read_csv('thai_rest_stats_ny.csv')
thai_rest_stats_ny_csv.head()
thai_rest_stats_ny_csv.shape
thai_rest_stats_ny_csv.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 66 entries, 0 to 65
Data columns (total 7 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   Borough       66 non-null     object 
 1   Neighborhood  66 non-null     object 
 2   ID            66 non-null     object 
 3   Name          66 non-null     object 
 4   Likes         66 non-null     int64  
 5   Rating        66 non-null     float64
 6   Tips          66 non-null     int64  
dtypes: float64(1), int64(2), object(4)
memory usage: 3.7+ KB<jupyter_text>Convert tips data type from string to float for easy to analyze.<jupyter_code>thai_rest_stats_ny['Likes']=thai_rest_stats_ny['Likes'].astype('float64')
thai_rest_stats_ny['Tips']=thai_rest_stats_ny['Tips'].astype('float64')
thai_rest_stats_ny.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 66 entries, 0 to 65
Data columns (total 7 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   Borough       66 non-null     object 
 1   Neighborhood  66 non-null     object 
 2   ID            66 non-null     object 
 3   Name          66 non-null     object 
 4   Likes         66 non-null     float64
 5   Rating        66 non-null     float64
 6   Tips          66 non-null     float64
dtypes: float64(3), object(4)
memory usage: 3.7+ KB<jupyter_text>Restaurant with maximum Likes<jupyter_code># Restaurant with maximum Likes
thai_rest_stats_ny.iloc[thai_rest_stats_ny['Likes'].idxmax()]<jupyter_output><empty_output><jupyter_text>Restaurant with maximum Rating<jupyter_code># Restaurant with maximum Rating
thai_rest_stats_ny.iloc[thai_rest_stats_ny['Rating'].idxmax()]<jupyter_output><empty_output><jupyter_text>Restaurant with maximum Tips<jupyter_code># Restaurant with maximum Tips
thai_rest_stats_ny.iloc[thai_rest_stats_ny['Tips'].idxmax()]<jupyter_output><empty_output><jupyter_text>Let's summarize average rating of Thai restaurants in each neighborhood.<jupyter_code>ny_neighborhood_stats=thai_rest_stats_ny.groupby('Neighborhood',as_index=False).mean()[['Neighborhood','Rating']]
ny_neighborhood_stats.columns=['Neighborhood','Average Rating']
ny_neighborhood_stats.sort_values(['Average Rating'],ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Let's summarize average rating of Thai restuarants in each borough.<jupyter_code>ny_borough_stats=thai_rest_stats_ny.groupby('Borough',as_index=False).mean()[['Borough','Rating']]
ny_borough_stats.columns=['Borough','Average Rating']
ny_borough_stats.sort_values(['Average Rating'],ascending=False).head()<jupyter_output><empty_output><jupyter_text>Visualize average rating of Thai restuarants for each borough.<jupyter_code>plt.figure(figsize=(9,5), dpi = 100)
# title
plt.title('Average rating of Thai Resturants for each Borough')
#On x-axis
plt.xlabel('Borough', fontsize = 15)
#On y-axis
plt.ylabel('Average Rating', fontsize=15)
#giving a bar plot
thai_rest_stats_ny.groupby('Borough').mean()['Rating'].plot(kind='bar')
#legend
plt.legend()
#displays the plot
plt.show()<jupyter_output><empty_output><jupyter_text>Select only neighborhoods which average rating more than 9.0<jupyter_code>ny_neighborhood_stats=ny_neighborhood_stats[ny_neighborhood_stats['Average Rating']>=9.0]
ny_neighborhood_stats<jupyter_output><empty_output><jupyter_text>Merge high average rating neighborhoods with their latitude and longitude to visualize in map.<jupyter_code>ny_neighborhood_stats=pd.merge(ny_neighborhood_stats,neighborhoods, on='Neighborhood')
ny_neighborhood_stats=ny_neighborhood_stats[['Borough','Neighborhood','Latitude','Longitude','Average Rating']]
ny_neighborhood_stats<jupyter_output><empty_output><jupyter_text>Visualize the location of each neighborhood which highest average rating in the map.<jupyter_code>address='New York City, NY'
def geo_location(address):
    # get geo location of address
    geolocator = Nominatim(user_agent="foursquare_agent")
    location = geolocator.geocode(address)
    latitude = location.latitude
    longitude = location.longitude
    return latitude,longitude
ny_map = folium.Map(location=geo_location('New York'), zoom_start=12)
# instantiate a feature group for the incidents in the dataframe
thai_res = folium.map.FeatureGroup()

# loop through the neighborhood and add each to the feature group
for lat, lng, in ny_neighborhood_stats[['Latitude','Longitude']].values:
    thai_res.add_child(
        folium.CircleMarker(
            [lat, lng],
            radius=10, # define how big you want the circle markers to be
            color='yellow',
            fill=True,
            fill_color='blue',
            fill_opacity=0.6
        )
    )
ny_neighborhood_stats['Label']=ny_neighborhood_stats['Neighborhood']+', '+ny_neighborhood_stats['Borough']+'('+ny_neighborhood_stats['Average Rating'].map(str)+')'
# add pop-up text to each marker on the map
for lat, lng, label in ny_neighborhood_stats[['Latitude','Longitude','Label']].values:
    folium.Marker([lat, lng], popup=label).add_to(ny_map)        
# add incidents to map
ny_map.add_child(thai_res)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09391259105098855, 'fraction_numerical': 0.027055150884495317, 'mean_word_length': 3.6226551226551225, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 92, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '571621', 'license': 'MIT', 'n_tokens_mistral': 6394, 'n_tokens_neox': 5790, 'n_words': 1664, 'path': 'the-battle-of-neighborhoods.ipynb', 'repo_name': 'phattp/the-battle-of-neighborhoods', 'size': '19306'}