<jupyter_start><jupyter_text>Setup notebook<jupyter_code>#from PIL import *
%reload_ext autoreload
%autoreload 2
%matplotlib inline
#import fastai
from fastai.vision import *
from fastai.metrics import error_rate
import scipy.io
import os
from pathlib import Path
import numpy as np
from PIL import Image
import spectral<jupyter_output><empty_output><jupyter_text>Prepare PAVIA CENTRE dataset<jupyter_code>path = Path("/home/zeefan/Repo/Pavia")
data = scipy.io.loadmat(path/"Pavia.mat")
data = data["pavia"]<jupyter_output><empty_output><jupyter_text>In the case of the PAVIA dataset, each pixel can be classified.<jupyter_code>data.shape
new2d = data.transpose(2,0,1).reshape(102,-1)
new2d.shape<jupyter_output><empty_output><jupyter_text>Now that we have reshaped our data to make each voxel a vector in the new array new2d, we can label each pixel.For the sake of clarity in our transformation, the pixel data[1, 2, 0] corresponds to the following pixel in the new array: new2d[0, (1*715 + 2)].<jupyter_code>print(data[1,2,0])
new2d[0,717]<jupyter_output><empty_output><jupyter_text>Now we setup the labels using the given Ground Truth<jupyter_code>GT = scipy.io.loadmat(path/"Pavia_gt.mat")
truth = GT["pavia_gt"]
truth.shape<jupyter_output><empty_output><jupyter_text>We know what the classes are as they are given on the website we obtained the dataset from. The classes are defined by integers, and the integers are populated on a 2d matrix to reference the index of each object: 1 = Water 2 = Trees 3 = Asphalt 4 = Self-Blocking Bricks 5 = Bitumen 6 = Tiles 7 = Shadows 8 = Meadows 9 = Bare SoilNow we can collect the indices of pixels in their respective classes<jupyter_code>water = np.where(truth==1)
lim = len(water[0])
x = 0
waterIndex=['water']
while x<lim:
    waterIndex.append((water[0][x],water[1][x]))
    x=x+1
    
trees = np.where(truth==2)
lim = len(trees[0])
x=0
treesIndex=[]
while x<lim:
    treesIndex.append((trees[0][x],trees[1][x]))
    x=x+1
    
asphalt = np.where(truth==3)
lim = len(asphalt[0])
x=0
asphaltIndex=['asphalt']
while x<lim:
    asphaltIndex.append((asphalt[0][x],asphalt[1][x]))
    x=x+1
    
bricks = np.where(truth==4)
lim = len(bricks[0])
x=0
bricksIndex=['bricks']
while x<lim:
    bricksIndex.append((bricks[0][x],bricks[1][x]))
    x=x+1
    
bitumen = np.where(truth==5)
lim = len(bitumen[0])
x=0
bitumenIndex=['bitumen']
while x<lim:
    bitumenIndex.append((bitumen[0][x],bitumen[1][x]))
    x=x+1
    
tiles = np.where(truth==6)
lim = len(tiles[0])
x=0
tilesIndex=['tiles']
while x<lim:
    tilesIndex.append((tiles[0][x],tiles[1][x]))
    x=x+1
    
shadows = np.where(truth==6)
lim = len(shadows[0])
x=0
shadowsIndex=['shadows']
while x<lim:
    shadowsIndex.append((shadows[0][x],shadows[1][x]))
    x=x+1
    
meadows = np.where(truth==7)
lim = len(meadows[0])
x=0
meadowsIndex=['meadows']
while x<lim:
    meadowsIndex.append((meadows[0][x],meadows[1][x]))
    x=x+1
    
soil = np.where(truth==7)
lim = len(soil[0])
x=0
soilIndex=['soil']
while x<lim:
    soilIndex.append((soil[0][x],soil[1][x]))
    x=x+1<jupyter_output><empty_output><jupyter_text>Sort the column vectors (voxels) into the appropriate groups of classes. Remember, the ground truth index (x,y) corresponds to the ( x * data.shape[1] + y ) column of the voxel array (new2d). SETUP INDICES TO SORT THROUGH THE NEW2D ARRAY<jupyter_code>waterIndex
# rescaled = (255.0 / data.max() * (data - data.min())).astype(np.uint8)
help(Image.fromarray)
# im = Image.fromarray(rescaled)
# im.save('test.png')<jupyter_output><empty_output><jupyter_text>Break the tif dataset into pieces, and use ground truth to label each piece<jupyter_code>#from spectral import *
#from spectral import *
import gdal
from pathlib import Path
import PIL
import sys
#from osgeo import gdal_array
import numpy as np
path2tif = Path('/home/zeefan/Repo/Pavia')
path2tif
ds = gdal.Open('/home/zeefan/Repo/Pavia/Pablo.tif')
#gdal_translate -of GTiff -co 
# rasterArray = gdal_array.LoadFile(filepath)
copy
#myarray = np.array(ds.GetRasterBand(1).ReadAsArray())

myarray[2][0]<jupyter_output><empty_output><jupyter_text>Figure out the format the dataset needs to be in to input it into cnn_learner()<jupyter_code>#data = ImageDataBunch.from_name_re(path_img, fnames, pat, ds_tfms=get_transforms(), size=224, bs=bs
#                                 ).normalize(imagenet_stats)
help(DataBunch)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13884484711211778, 'fraction_numerical': 0.024688561721404302, 'mean_word_length': 5.513274336283186, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '263737', 'license': 'Apache-2.0', 'n_tokens_mistral': 1737, 'n_tokens_neox': 1669, 'n_words': 397, 'path': 'nbs/dl1/PaviaTest.ipynb', 'repo_name': 'Ziller97/course-v3', 'size': '4415'}