<jupyter_start><jupyter_text>Nighttime Lights Exploratory Data AnalysisThis notebook contains the exploratory data analysis of the nighttime lights 2016 dataset and generates CSV files required for splitting the images into training and validation datasets. After running the contents of this notebook and generating the necessary files, you may download the GSM images using the script: `src/data_download.py`.  Imports and Setup<jupyter_code>import sys
sys.path.insert(0, '../utils')
import warnings
warnings.filterwarnings('ignore')

import os
import time
import copy
import re

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import data_utils

SEED = 42
np.random.seed(SEED)

%matplotlib inline
TM_pal_categorical_3 = ('#ef4631', '#10b9ce', '#ff9138')
sns.set(style='white', font_scale=1, palette=TM_pal_categorical_3)

%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>File Locations<jupyter_code># Input data file paths
data_path = '../data/'
nightlights_file = data_path + 'nightlights.csv'
dhs_indicators_file = data_path + 'dhs_indicators.csv'

# Output data file paths 
nightlights_unstacked_file = data_path + 'nightlights_unstacked.csv'
nightlights_bins_file = data_path + 'nightlights_bins.csv'
nightlights_train_file = data_path + 'nightlights_train.csv'
nightlights_train_bal_file = data_path + 'nightlights_train_bal.csv' #balanced training set
nightlights_val_file = data_path + 'nightlights_val.csv'<jupyter_output><empty_output><jupyter_text>Night Time Lights Dataset EDA  Load Nighttime Lights Dataset<jupyter_code>nightlights = pd.read_csv(nightlights_file)
print('Number of images to download: {} \n'.format(nightlights.ID.nunique()))
nightlights.tail(3)<jupyter_output>Number of images to download: 134540<jupyter_text>Sanity check number of pixels by nighttime light intensity<jupyter_code>data_utils.plot_hist(
    nightlights[nightlights['ntl2016'] > 1]['ntl2016'],
    title='Number of pixels by nighttime light intensity values', 
    x_label='Nightlight Intensity', 
    y_label='Number of Pixels'
)<jupyter_output><empty_output><jupyter_text>Unstack Nighttime Lights DatasetUnstack dataset so that each row consists of a unique (cluster, lat, lon) tuple.<jupyter_code>nightlights = data_utils.unstack_clusters(nightlights)
nightlights.to_csv(nightlights_unstacked_file)
print("Total number of pixels (including duplicates belonging to different clusters): {}".format(len(nightlights)))<jupyter_output>100%|██████████| 134540/134540 [00:19<00:00, 6830.39it/s]<jupyter_text>Sanity Check DHS Clusters and Nighttime Lights Summary Statistics<jupyter_code>print("Number of unique clusters in night lights dataset: ", nightlights['DHSCLUST'].nunique())
nightlights['ntl2016'].describe()<jupyter_output>Number of unique clusters in night lights dataset:  1214<jupyter_text>Sanity Check Number of Pixels/Images Per Cluster<jupyter_code>data_utils.plot_hist(
    nightlights.groupby('DHSCLUST').count()['ntl2016'],
    title='Number of pixels (images) per cluster', 
    x_label='Number of Pixels', 
    y_label='Number of Clusters'
)<jupyter_output><empty_output><jupyter_text>Sanity Check Proportion of Zero Intensity Levels in the Dataset<jupyter_code>nightlights_zero = nightlights.groupby(['DHSCLUST', 'ntl2016']).size().unstack(fill_value=0)
data_utils.plot_hist(
    nightlights_zero[0.0]/nightlights_zero.sum(axis=1),
    title='Proportion of zero intensity values per cluster', 
    x_label='Proportion of zero intensity values', 
    y_label='Number of clusters'
)
print("Around 150 clusters have ~100% zero night light intensity values (total darkness).")
percent_zeros = nightlights[nightlights['ntl2016'] == 0].shape[0]/len(nightlights)
print("Percent of zero intensity pixels in dataset: {:.0f}%".format(percent_zeros*100))<jupyter_output><empty_output><jupyter_text>Sanity Check Average Nighttime Lights Per Cluster<jupyter_code>nightlights_avg = pd.DataFrame(nightlights.groupby(['DHSCLUST'])['ntl2016'].mean()).reset_index()
print("Shape of average night lights dataframe: ", nightlights_avg.shape)
data_utils.plot_hist(
    nightlights_avg['ntl2016'],
    title='Distribution of average night light intensities', 
    x_label='Average nightlight intensity', 
    y_label='Frequency'
)<jupyter_output>Shape of average night lights dataframe:  (1214, 2)<jupyter_text>Get Sum of Population Per Cluster<jupyter_code>pop_sum = pd.DataFrame(nightlights.groupby(['DHSCLUST'])['pop_sum'].sum()).reset_index()
print("Shape of population dataframe: ", pop_sum.shape)
data_utils.plot_hist(
    pop_sum['pop_sum'],
    title='Distribution of Total Population', 
    x_label='Total Population', 
    y_label='Number of Clusters'
)<jupyter_output>Shape of population dataframe:  (1214, 2)<jupyter_text>Demographic and Health Survey (DHS) Dataset EDA  Load DHS indicators Load DHS dataste and merge with average night time lights data.<jupyter_code>dhs_indicators = pd.read_csv(dhs_indicators_file)
nightlights_avg = nightlights_avg.merge(dhs_indicators, left_on='DHSCLUST', right_on='Cluster number')
pop_sum = pop_sum.merge(dhs_indicators, left_on='DHSCLUST', right_on='Cluster number')

print("Number of unique clusters in DHS data merged with NTL data: ", nightlights_avg['DHSCLUST'].nunique())
print("Number of unique clusters in DHS data merged with pop data: ", pop_sum['DHSCLUST'].nunique())<jupyter_output>Number of unique clusters in DHS data merged with NTL data:  1213
Number of unique clusters in DHS data merged with pop data:  1213<jupyter_text>Correlations bet. Average NTL Intensity and Socioeconomic Indicators<jupyter_code>data_utils.plot_regplot(pop_sum, 'Wealth Index', 'Population', 'pop_sum')
data_utils.plot_regplot(nightlights_avg, 'Wealth Index')
data_utils.plot_regplot(nightlights_avg, 'Education completed (years)')
data_utils.plot_regplot(nightlights_avg, 'Access to electricity')
data_utils.plot_regplot(nightlights_avg, 'Access to water (minutes)')<jupyter_output><empty_output><jupyter_text>Binning Nighttime Light Intensities   Load Nighttime Lights Dataset<jupyter_code>nightlights = pd.read_csv(nightlights_file)
nightlights.head(3)<jupyter_output><empty_output><jupyter_text>Gaussian Mixture Model<jupyter_code>bin_labels = ['low', 'low medium', 'medium', 'high medium', 'high']
nightlights = data_utils.gaussian_mixture_model(nightlights, n_components=5, bin_labels=bin_labels)
for label in bin_labels:
    print(
        "Number of {} intensity pixels: {}".format(
            label, nightlights[nightlights['label'] == label]['ntl2016'].count()
        )
    )<jupyter_output>{0: 0.0, 1: 1.83839, 2: 122.43231000000002, 3: 0.4707, 4: 9.16312}
{0: 'low', 3: 'low medium', 1: 'medium', 4: 'high medium', 2: 'high'}
Number of low intensity pixels: 94680
Number of low medium intensity pixels: 16169
Number of medium intensity pixels: 10785
Number of high medium intensity pixels: 7844
Number of high intensity pixels: 5062<jupyter_text>Ad Hoc Label Assignment<jupyter_code>bin_caps = [0, 2, 15, 30]
bin_labels = ['low', 'low medium', 'medium', 'high medium', 'high']
nightlights['label'] = nightlights['ntl2016'].apply(lambda x: data_utils.ad_hoc_binning(x, bin_caps, bin_labels))
print(nightlights[nightlights['label'] == 'high medium']['ntl2016'].describe())
for label in bin_labels:
    print(
        "Number of {} intensity pixels: {}".format(
            label, nightlights[nightlights['label'] == label]['ntl2016'].count()
        )
    )<jupyter_output>count    2161.000000
mean       20.726912
std         4.231244
min        15.009130
25%        17.022590
50%        19.820690
75%        23.909240
max        29.994180
Name: ntl2016, dtype: float64
Number of low intensity pixels: 94680
Number of low medium intensity pixels: 27415
Number of medium intensity pixels: 9389
Number of high medium intensity pixels: 2161
Number of high intensity pixels: 895<jupyter_text>Save Binned Dataset<jupyter_code>nightlights.to_csv(nightlights_bins_file, index=False)

print('Total number of images: ', nightlights.shape[0])
print(nightlights['label'].value_counts())
print(nightlights['label'].value_counts()/nightlights.shape[0])<jupyter_output>Total number of images:  134540
low            94680
low medium     27415
medium          9389
high medium     2161
high             895
Name: label, dtype: int64
low            0.703731
low medium     0.203768
medium         0.069786
high medium    0.016062
high           0.006652
Name: label, dtype: float64<jupyter_text>Balancing Nighttime Lights Intensity Levels  Load Nighttime Lights Dataset<jupyter_code>nightlights = pd.read_csv(nightlights_bins_file)
print('Total number images downloaded: ', len(nightlights))<jupyter_output>Total number images downloaded:  134540<jupyter_text>Shuffle NTL Dataset and Split into Train/Val Sets<jupyter_code>nightlights = nightlights.sample(frac=1, random_state=SEED).reset_index(drop=True)
train, val = data_utils.train_val_split(nightlights, train_size=0.9)
print('Size of training set: ', len(train))
print(train['label'].value_counts())
print('\nSize of validation set: ', len(val))
print(val['label'].value_counts())<jupyter_output>Size of training set:  121086
low            85150
low medium     24731
medium          8467
high medium     1945
high             793
Name: label, dtype: int64

Size of validation set:  13454
low            9530
low medium     2684
medium          922
high medium     216
high            102
Name: label, dtype: int64<jupyter_text>Upsample Minority Class of Training Set<jupyter_code>train_balanced = data_utils.balance_dataset(train, size=30000)
print('Number of images in training set (balanced): ', len(train_balanced))
train_balanced['label'].value_counts()<jupyter_output>Number of images in training set (balanced):  150000
{'fraction_non_alphanumeric': 0.10429009193054137, 'fraction_numerical': 0.04708886618998979, 'mean_word_length': 6.048956083513319, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '282735', 'license': 'MIT', 'n_tokens_mistral': 3463, 'n_tokens_neox': 3090, 'n_words': 810, 'path': 'notebooks/01_lights_eda.ipynb', 'repo_name': 'sunayana/ph-poverty-mapping', 'size': '9790'}