<jupyter_start><jupyter_text>Programming PowerupChecking wether something an element in a list can be slow<jupyter_code>from time import time
dataset=np.random.uniform(0,1,size=(200000,2))
train=np.random.choice(range(len(dataset)),int(len(dataset)*.8),replace=False)

# This loop is slow
itime=time()
develop=[]
for i in train:
    if i not in train:
        develop.append(i)
print("Takes ",time()-itime," Seconds")<jupyter_output><empty_output><jupyter_text>If you want to do something like the above it is much faster if you use a dictionary to keep track of what elements exist.<jupyter_code>train_dict={}
for i in train:
    train_dict[i]=0 #The value here dosen't matter 
    
# This loop is fast!
itime=time()
develop=[]
for i in train:
    if i not in train_dict:
        develop.append(i)
print("Takes ",time()-itime," Seconds")<jupyter_output><empty_output><jupyter_text>Why? When using a list every element is checked, however, a dictionary uses a trick called a hash function to convert the item into an location, and it makes only one checks to see if there is data in that location.    Real world Bio-Medical DatasetsThis notebook is a start on how you can use machine learning on bio-medical datasets. Tumor Identification SegmentationWe're going to start with the PatchCamelyon (PCam)https://github.com/basveeling/pcam The RAW DataThis dataset was created from whole slide images for example this one:* Whole slide image scans are gigantic!    * Really Big     (i.e. in this example 97,792 pixels by 221,184 pixels        * ~21,000 MegaPixels* Below is a low resolution view* These image don't generally fit into memory, but python packages exist to manipulate themFor this file we're relying on a package called openslide<jupyter_code>import openslide<jupyter_output><empty_output><jupyter_text>Labels in this dataset are the boundaries of each tumor.   Slide Data* The data we're using is stored in .tif files that are very large    * Too large to fit into memory    * Too large to easily visualize * We'll use https://openslide.org/ to read these files* Files have levels of magnification from 0 (full-res) to 9 (low res)<jupyter_code># open a full slide image
tiff_file='/projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/tumor_001.tif'
slide_image=openslide.OpenSlide(tiff_file)

# Slide images are divided into levels with level 0 being the highest resolution
for i,res in enumerate(slide_image.level_dimensions):
    print('The Resolution at level',i, 'is', res)<jupyter_output><empty_output><jupyter_text>We are going to use the index coordinate systemNote that for open slide we need to use the index coordinates for **level 0**<jupyter_code>#We can read each region of the image at one time at different levels

#Top left_pixel (x,y) in level 0, level,width,height
print("Full Image at level 8")
image=np.asarray(slide_image.read_region( (0,0),8,(382, 864)  ))
f=plt.figure(figsize=(10,10))
plt.imshow(image)
plt.show()


for i,v in enumerate(slide_image.level_dimensions):
    print("Level",i,"Resolution",v)
    f=plt.figure(figsize=(10,10))
    image=np.asarray(slide_image.read_region( (50000,120000),i,(500,500)  ))
    plt.imshow(image)
    plt.show()<jupyter_output><empty_output><jupyter_text>LabelsYou'll often get image labels in many different forms. The labeled information in this dataset comes from expert annotations delivered as an XML file.* Note: When working on real models these data steps often take up much of your time and are essential to get correct* How you use your data determines what you ML model learns Raw Annotations (XML)Annotations for this dataset come in an XML format. This format is self-describing text. Let's look at an example.As a reminder we can use "!" to run shell commands, and a simple shell command to print out a file is **cat**<jupyter_code>!cat /projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/tumor_001.xml<jupyter_output><empty_output><jupyter_text>Reading XML by EyeXML has data described by elements. An element looks like this`...`Everything in between belongs to this element. In this example, we have a hierarchy that looks like * ASAP_annotations      * Annotations           * Annotation             * Coordinates             * Coordinates             * ...           * Annotation             * Coordinates             * Coordinates             * ...In other words, each image has annotations that are made up of a bunch of coordinates. For this data, each Annotation is the outline of a tumor, and this image has two tumors. AttributesEach Element can also have **Attributes** in the file above they show up in the element definitionThis entry``has attributes Order,X,Y  Reading XML with codeXML is a standard format so there are many tools around to read it. We'll use python's xml library.**Goal:** Get a numpy array with the list of coordinates for each tumor<jupyter_code>import xml.etree.ElementTree as ET
#This import is different from normal but matplotlib has a useful function we'll use 
import matplotlib
import matplotlib.pyplot as plt

# Open XML File
tree = ET.parse('/projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/tumor_001.xml')
root = tree.getroot()

# Select All Annotations
tumor_annotation=[el for el in root.findall('Annotations/Annotation')]

# Make A List of Tumors
tumors=[]
for annotation in tumor_annotation:
    tumors.append(np.array([ [float(el.attrib['X']),float(el.attrib['Y'])] for el in annotation.findall('Coordinates/Coordinate')]))

#convert to np array
tumors=[np.array(t) for t in tumors]
    
#Plot the outlines of each tumor
for t in tumors:
    plt.scatter(t[:,0],t[:,1])
plt.show()<jupyter_output><empty_output><jupyter_text>Exercise We've read in our two tumors, try to plot them on top of the of the image data at Level 5 Resolution* You'll need to read the image in* You'll neet to make sure you're tumor coordinates are in the correct place for level 5 resolution**Give it a Try**<jupyter_code>"""Your Code Here"""<jupyter_output><empty_output><jupyter_text>Making a Dataset for Classification* Where going to turn our images and annotation into a **classification** problemSteps:  1. Ask a question we want an answer too:    * Q. Is a these cells part of a tumor?  2. Create our Xs   * Divide our big picture into lots of small pictures  3. Create or Ys   * Is any pixel within our region of interst part of a tumor (T/F)?    * Our region of interest could be all of our small picture, but how would are algorithm be able to succesfully guess if only the corner pixel is part of a tumor. Instead it's better to take a box in the middle of the   Q. How Big Do Our Small Pictures need to be    A. It Depends on the question you're askingIf you questions are how many dounuts are in this picture, you'll need the whole imageIf your question is where are the yellow sprinkles the smaller images will do fine  Lets Split of the DataLets start with the an interesting region and cut it up into 96x96x3 images<jupyter_code>#These coordinates are at level 0
width=6400
height=6400

x_start=60000
y_start=120000

print("Small Images Produced Required",width//32*height//32)

#Covert these for later use to level2
level0_dimension=slide_image.level_dimensions[0]
level2_dimension=slide_image.level_dimensions[2]

sfactor_x=level0_dimension[0]/level2_dimension[0]
sfactor_y=level0_dimension[1]/level2_dimension[1]

x_stop=x_start+width*4
y_stop=y_start+height*4


print('Scaling factor to level 2',sfactor_x,sfactor_y)

# It's just a factor of 4
f=plt.figure(figsize=(50,50))
plt.imshow(slide_image.read_region( (x_start,y_start),2,(height,width)))
plt.show()<jupyter_output><empty_output><jupyter_text>Is a point inside a Polygon?to get our labels we'll ask if a point is within a tumor we'll use a helpful function from matplotlib "contains points", we'll turn our tumor coords in to a path.<jupyter_code># Here is Example using our first turmor


    
     
print(tumors[0].shape)
tumor_paths=[matplotlib.path.Path(t) for t in tumors]
path = tumor_paths[0]

points=np.zeros(shape=(1000,2))
points[:,0]=np.random.uniform(69000,73000,size=1000)
points[:,1]=np.random.uniform(131000,136000,size=1000)
inside = path.contains_points(points)
not_inside=np.logical_not(inside)


plt.scatter(tumors[0][:,0],tumors[0][:,1])
plt.scatter(points[inside,0],points[inside,1])
plt.scatter(points[not_inside,0],points[not_inside,1])

plt.show()

## Exercise 

Make a good looking plot with points inside and outside of the second tumor in this image
""" Your code here"""<jupyter_output><empty_output><jupyter_text>Writing out an Image FileA lot of the code we've seen uses pillow (PIL) in the background will use it now to turn each of our numpy arrays into jpgs<jupyter_code>import PIL<jupyter_output><empty_output><jupyter_text>The below code is simple in concept, move a 96x96 box in steps of 32 pixles accross and image and save each path. However it takes a fair bit of code, and the book keeping can get complicated.* We'll Create a directory structure that can be used directly by keras    * /Folder        * /tumor            * Patches with a tumor        * /normal            * Patches that do not have a tumor<jupyter_code>out_dir='wslide_test_data/'
if not os.path.exists(out_dir):
    os.mkdir(out_dir)
    os.mkdir(out_dir+'tumor')
    os.mkdir(out_dir+'normal')

def scan_image(image_file,batch_size,x_range,y_range,tumors):
    labels=[]
    slide_image=openslide.OpenSlide(image_file)
    res_x,res_y=slide_image.level_dimensions[2]    
    coord_x,coord_y=slide_image.level_dimensions[0] 
    #This is factor we need to scale the pixels at resolution 2 to the coordinates in resolution 0

    sfactor_x=coord_x/res_x  
    sfactor_y=coord_y/res_y 
    
    index=0
    itr=0
    
    
    for x in range(x_range[0]//4,x_range[1]//4,32):
        for y in range(y_range[0]//4,y_range[1]//4,32):
            image=slide_image.read_region( (int(x*sfactor_x),int(y*sfactor_y)),2,(96,96)  )
            if itr%100==0:
                print(image)
                plt.imshow(image)
                plt.show()
                plt.close()
            
            
            
            path=matplotlib.path.Path([((x+32)*sfactor_x,(y+32)*sfactor_y),
                                       ((x+64)*sfactor_x,(y+32)*sfactor_y),
                                       ((x+64)*sfactor_x,(y+64)*sfactor_y),
                                       ((x+32)*sfactor_x,(y+64)*sfactor_y) ])
            label=any([ path.contains_points(t).any() for t in tumors])
            
            if label:
                image.save(out_dir+'normal/'+str(itr)+'.png')
            else:
                image.save(out_dir+'tumor/'+str(itr)+'.png')

                
            itr+=1
                
coord_x,coord_y=slide_image.level_dimensions[0] 
              
x_range=[0,coord_x]
y_range=[0,coord_y]


x_range=[x_start,x_stop]
y_range=[y_start,y_stop]


generator=scan_image(tiff_file,10,x_range,y_range,tumors)



!ls wslide_test_data/neg | wc
!ls wslide_test_data/pos | wc
! du -sm wslide_test_data<jupyter_output><empty_output><jupyter_text>You've made you first dataset This was just one image and you would have to run a similar function of all the data in the<jupyter_code>!ls /projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/images/*

!ls /projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/images/test/normal/*|wc
!ls /projects/bgmp/shared/2019_ML_workshop/datasets/pcamv1/images/test/tumor/*|wc<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09293809938971229, 'fraction_numerical': 0.023190932868352224, 'mean_word_length': 4.093694493783303, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '149382', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3833, 'n_tokens_neox': 3540, 'n_words': 1213, 'path': 'notebooks/6-WholeSlides.ipynb', 'repo_name': 'jsearcy1/DNNWS_2022', 'size': '11470'}