<jupyter_start><jupyter_text>Understanding the Amazon from Space  Objective:Label satellite image chips with atmospheric conditions and various classes of land cover/land use. Available labels are: *agriculture, artisinal_mine, bare_ground, blooming, blow_down, clear, cloudy, conventional_mine, cultivation, habitation, haze, partly_cloudy, primary, road, selective_logging, slash_burn, water*<jupyter_code># imports
import pandas as pd
import numpy as np
import os
import matplotlib.image as image
import matplotlib.pyplot as plt
import rasterio
from IPython.display import display
%matplotlib inline
PLANET_KAGGLE_ROOT = os.path.abspath("../input/amazon")
PLANET_KAGGLE_TIFF_DIR = os.path.join(PLANET_KAGGLE_ROOT, './train-tif/')
PLANET_KAGGLE_LABEL_CSV = os.path.join(PLANET_KAGGLE_ROOT, './train.csv')
assert os.path.exists(PLANET_KAGGLE_ROOT)
assert os.path.exists(PLANET_KAGGLE_LABEL_CSV)<jupyter_output><empty_output><jupyter_text>1. Training Label Analysis<jupyter_code># read our data and take a look at what we are dealing with
train_csv = pd.read_csv(PLANET_KAGGLE_LABEL_CSV)
train_csv.head()
# how much data do we have?
train_csv.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 40479 entries, 0 to 40478
Data columns (total 2 columns):
image_name    40479 non-null object
tags          40479 non-null object
dtypes: object(2)
memory usage: 632.6+ KB<jupyter_text>Observations- First objective will be to extract and and split the tag data into binary feature columns to be analyzed seperately.<jupyter_code>AVAILABLE_LABELS = [
    'agriculture', 
    'artisinal_mine', 
    'bare_ground', 
    'blooming', 
    'blow_down', 
    'clear', 
    'cloudy', 
    'conventional_mine', 
    'cultivation', 
    'habitation', 
    'haze', 
    'partly_cloudy', 
    'primary', 
    'road', 
    'selective_logging', 
    'slash_burn', 
    'water']

tags = pd.DataFrame()

for label in AVAILABLE_LABELS:
    tags[label] = train_csv.tags.apply(lambda x: np.where(label in x, 1, 0))
    
train = pd.concat([train_csv, tags], axis=1)
train.head()
plt.style.use('ggplot')

train.drop(['image_name', 'tags'], 1).sum().sort_values().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Observations- Some labels have almost no samples- The 'primary' tag has almost no negative samples.- Clear seems to have the best class balance, and might be the best place to start  Sample Data AnalysisThe training images are provided in 2 formats, jpg and GeoTiff. The end evaluation will be done on the GeoTiff Data. First lets examine the jpg data and compare to our training labels.<jupyter_code># load jpg data
# load files
train_jpg_sample = os.listdir(PLANET_KAGGLE_JPEG_DIR)
filtered_samples = [sample[0:-4] for sample in train_jpg_sample]

# get labels for these files
sample = train[train['image_name'].isin(filtered_samples)]
sample.reset_index(inplace=True)
sample.head()
sample.drop(['index', 'image_name', 'tags'], 1).sum().sort_values().plot(kind='bar')
NUM_FILES = 2

for i in range(0, NUM_FILES):
    sample_jpg = image.imread(os.path.join(PLANET_KAGGLE_JPEG_DIR, str(train_jpg_sample[i])))
    plt.imshow(sample_jpg)
    plt.title(str(train_jpg_sample[i]) + ': ' + sample.tags[i])<jupyter_output><empty_output><jupyter_text>Now lets learn to work with the GeoTiff Data. We will use some sample functions provided by PlanetLabs.<jupyter_code>def load_image(filename):
    """Return a 4D (r, g, b, nir) numpy array with the data in the specified TIFF filename."""
    for dirname in os.listdir(PLANET_KAGGLE_ROOT):
        path = os.path.abspath(os.path.join(PLANET_KAGGLE_ROOT, PLANET_KAGGLE_TIFF_DIR, filename))
        print('path: ' + str(path))
        if os.path.exists(path):
            with rasterio.open(path) as src:
                b, g, r, nir = src.read()
                return np.dstack([r, g, b, nir])
            
def sample_images(tags, value, n=None):
    """Randomly sample n images with the specified tags."""
    condition = True
    if isinstance(tags, str):
        raise ValueError("Pass a list of tags, not a single tag.")
    for tag in tags:
        condition = condition & sample[tag] == value
    if n is not None:
        return sample[condition].sample(n)
    else:
        return sample[condition]<jupyter_output><empty_output><jupyter_text>Lets target our analysis to one particular type of tag. We will start with _cloudy_ images.<jupyter_code># get labels for these files
sample = train
# cloud_image_path = sample_images(['cloudy'], n=1, value=1)['image_name'].iloc[0] # fix bug here

cloud_image_path = 'train_0'

rgbn_image = load_image(cloud_image_path + '.tif')
rgb_image = rgbn_image[:,:,:3]
r, g, b, nir = rgbn_image[:, :, 0], rgbn_image[:, :, 1], rgbn_image[:, :, 2], rgbn_image[:, :, 3]
for slice_, name, color in ((r,'r', 'red'),(g,'g', 'green'),(b,'b', 'blue'), (nir, 'nir', 'magenta')):
    plt.hist(slice_.ravel(), bins=100, 
             range=[0,rgb_image.max()], 
             label=name, color=color, histtype='step')
plt.legend()

fig = plt.figure()
fig.set_size_inches(12, 4)
for i, (x, c) in enumerate(((r, 'r'), (g, 'g'), (b, 'b'), (nir, 'near-ir'))):
    a = fig.add_subplot(1, 4, i+1)
    a.set_title(c)
    plt.imshow(x)
plt.show()<jupyter_output>path: /Users/user/kaggle/input/amazon/train-tif/train_0.tif
{'fraction_non_alphanumeric': 0.1279003961516695, 'fraction_numerical': 0.011129975476325221, 'mean_word_length': 4.5, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '622438', 'license': 'Apache-2.0', 'n_tokens_mistral': 1931, 'n_tokens_neox': 1836, 'n_words': 492, 'path': 'scrap/Initial Data Analysis.ipynb', 'repo_name': 'cseale/kaggle-amazon-rainforests', 'size': '5301'}