<jupyter_start><jupyter_text>K-means  Clustering and Image Sharpening In this exercise, you are asked to approximate the depth of field (DOF) effect on animage. The depth of field is the distance between the nearest and farthest objects in ascene which appear acceptably sharp in an image. When the DOF is small, the objectof interest (i.e., focused object) appears sharp and the rest appears blurred. This effectis usually employed for artistic reasons and is achieved by having a large aperture whilecapturing an image.In our case, the image has already been captured, and we are trying to simulate a similareffect. The task is to first segment the image into 3 areas (front/middle/backgroundareas), and then to sharpen the object of interest while blurring the rest. In our case in addition to the RGB image, a depth image has also been captured.The intensity of each pixel in the depth image correspond to the inverse distancebetween the camera and the objects of the scene (up to a scale). Instead of finding thecorrect segmentation of the image, which is a difficult task, we are going to perform thesegmentation on the depth image. The depth image, however, is usually a bit noisy, so asimple thresholding does not produce good results in most cases, thus you will need toimplement a k-means segmentation algorithm. In this exercise, you are asked to:* implement the k-means algorithm to segment the depth image into three disjointregions. This will produce a label image where the three objects (doll, hand,background) have a different label.* generate three binary images each focusing on a different object.* sharpen the focused object and smooth the rest, using these binary images andappropriate filters.          Figure 1: Pair of images, captured by Microsoft Kinect. The image on the left depicts the RGB imageand the one on the right shows the corresponding depth image.<jupyter_code>%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import cv2
import math
import scipy.signal as conv
# Read the image 
img = cv2.imread('images/rgb.png',0)
depth_img = cv2.imread('images/depth_noisy.png',0)

# Display the images
plt.figure(1)
plt.imshow(img,cmap='gray')
plt.figure(2)
plt.imshow(depth_img,cmap='gray') 

depth_img = np.float64(depth_img)*4; # rescaled for better results<jupyter_output><empty_output><jupyter_text>Exercise 1: Depth-based Segmentation To start with, You are asked to implement the function ```labelImage = clusterImage(depthImage,max_iters)``` by the following the steps below:* For every pixel $i, j: 0 ≤ i < H, 0 ≤ j < W$ of the depth image, compute the Euclideandistance between its depth $D[i,j]$ and each cluster mean depth $k^l_d$: $dist[i,j,l] = \sqrt{(D[i,j] - {k^{l}}_{d})^2}$ and $l = 0,1,2$. * For each pixel find the smallest distance and assign this pixel to thecorresponding cluster.* Compute the new mean of each cluster.* Iterate over the above two steps till the cluster centroids (mean of the clusters) do not move any more or a fixed number of iterations is reached.What is the disadvantage of the result? Is the segmentation correct? Inputs:- ```depthImage```: an image where each value indicates the depth of the corresponding pixel.- ```max_iters```: the maximum number of iterations  Outputs:- ```labelImage```:  output label image where each value indicates to which cluster the corresponding pixels belongs. - There are three clusters: value 0 for the background, value 1 for the hand and value 2 for the doll.<jupyter_code>def clusterImage(depthImage, max_iters):
    
    k = 3  # Number of clusters
    height, width = depthImage.shape  # Size of depth image
    count = 0  # Iteration count
    distance = [None] * 3  # distance container in iteration
    
    # initiate centroids
    # we create 3 centroids, in the format [Z X Y], where Z in the inverse depth
    mu = np.zeros((k,3)) 
    mu[0,:] = [0, 0, 0]  # background centroid
    mu[1,:] = [500, 300, 150]  # hand centroid
    mu[2,:] = [1000, 100, 200]  # doll centroid
    prev_mu = mu + 50  # initialize iteration
    ### Note: For Exercise 1, only Z (inverse) depth is used. X,Y is used in Exercise 2.
 
    # Create X and Y matrices to store coordinate information
    X, Y = np.meshgrid(np.arange(0, width), np.arange(0, height))
    # print(distance)
    
    # Iteratively calculate distance between depth and update cluster mean
    while ((count < max_iters) and np.sum((prev_mu - mu) > 0)):
        print("Iteration: {}".format(count))
        
        # calculate depth distance map for depth image of all clusters
        for k in range(3):
            distance[k] = np.sqrt((depthImage - mu[k,0]) ** 2)
        
        # assign pixels to the cluster that has min distance
        cluster_assignments = np.argmin(np.array(distance), axis=0)
        
        # store old cluster mean and update new cluster mean
        prev_mu = np.copy(mu)
        for k in range(3):
            mu[k,:] = np.array([np.mean(depthImage[cluster_assignments == k]), np.mean(X[cluster_assignments == k]), np.mean(Y[cluster_assignments == k])])
        
        count += 1
    
    return cluster_assignments
cluster_assignments = clusterImage(depth_img, 5)   
plt.imshow(cluster_assignments)<jupyter_output>Iteration: 0
Iteration: 1
Iteration: 2
Iteration: 3
Iteration: 4<jupyter_text>Exercise 2: Using Location InformationAs you saw in the previous exercise, sometimes the depth information is not enough to correctly segment the image. This can be fixed by taking into account also **the spatial proximity of the pixels** - pixels from the same cluster should be close to each other in the image. This can beaccomplished by adding 2 dimensions to the original depth image representing the i andj coordinates of each pixel.You are asked to implement a new function ```labelImage = clusterImage(depthImage,max_iters)``` following the steps:* For every pixel $i, j: 0 ≤ i < H, 0 ≤ j < W$ of the depth image, compute the Euclideandistance between its combined depth $D[i,j]$ and position $[i,j]$ and each cluster mean $[k^l_d,k^l_i,k^l_j]$: $dist[i,j,l] = \sqrt{(D[i,j] - {k^{l}}_{d})^2 + (i-k^{l}_{i})^2) + (j-k^{l}_{j})^2)}$ for $l = 0,1,2$. * For each pixel find the smallest distance and assign this pixel to thecorresponding cluster.* Compute the new mean of each cluster.* Iterate over the above two steps till the cluster centroids (mean of the clusters)do not move any more or a fixed number of iterations is reached.How does addition of spatial information improve the segmentation results? Inputs:- ```depthImage```: an image where each value indicates the depth of the corresponding pixel.- ```max_iters```: maximum number of iterations Outputs:- ```labelImage```:  output label image where each value indicates to which cluster the corresponding pixels belongs. There are three clusters: value 0 for the background, value 1 for the hand and value 2 for the doll.<jupyter_code>def clusterImage(depthImage, max_iters):
    
    k = 3 # Number of clusters
    height, width = depthImage.shape  # Size of depth image
    count = 0  # Iteration count
    distance = [None] * 3  # distance container in iteration
    
    #initiate centroids
    #we create 3 centroids, in the format [Z X Y], where Z in the inverse depth
    mu = np.zeros((k,3)) 
    mu[0,:] = [0, 0, 0] #background centroid
    mu[1,:] = [500, 300, 150] #hand centroid
    mu[2,:] = [1000, 100, 200] #doll centroid
    prev_mu = mu + 50  # initialize iteration
    
    # Create X and Y matrices to store coordinate information
    X, Y = np.meshgrid(np.arange(0, width), np.arange(0, height))
    # print(distance)
    
    # Iteratively calculate distance between depth and update cluster mean
    while ((count < max_iters) and np.sum((prev_mu - mu) > 0)):
        print("Iteration: {}".format(count))
        
        # calculate xyz distance map for depth image of all clusters
        for k in range(3):
            distance[k] = np.sqrt((depthImage - mu[k,0]) ** 2 + (X - mu[k,1]) ** 2 + (Y - mu[k,2]) ** 2)
        
        # assign pixels to the cluster that has min distance
        cluster_assignments = np.argmin(np.array(distance), axis=0)
        
        # store old cluster mean and update new cluster mean
        prev_mu = np.copy(mu)
        for k in range(3):
            mu[k,:] = np.array([np.mean(depthImage[cluster_assignments == k]), np.mean(X[cluster_assignments == k]), np.mean(Y[cluster_assignments == k])])
        
        count += 1
    
    return cluster_assignments
    
cluster_assignments = clusterImage(depth_img,5)   
plt.imshow(cluster_assignments)<jupyter_output>Iteration: 0
Iteration: 1
Iteration: 2
Iteration: 3
Iteration: 4<jupyter_text>Exercise 3: Image Sharpening We further need to implement a region-based filtering function to sharpen the object of interest and blur the rest of the image. In the previous parts you produced a label image (with each object assigned a different label) and produced three binary images corresponding to different objects. Now **the task is to sharpen the focused regions by applying a Laplacian filter and smooth the other regions using a Gaussian filter**. It should produce results as shown below.To this end we introduce a Laplacian filter parametrized by $\alpha$ defined as:\begin{align}L = \frac{\partial ^2}{\partial x^2} + \frac{\partial^2}{\partial y^2} = \frac{4}{\alpha+1} \begin{bmatrix} \frac{\alpha}{4} & \frac{1-\alpha}{4} & \frac{\alpha}{4} \\\frac{1-\alpha}{4} & -1 & \frac{1-\alpha}{4} \\\frac{\alpha}{4} & \frac{1-\alpha}{4} & \frac{\alpha}{4} \\\end{bmatrix}\end{align}Then, the sharpening filter is defined as $\delta - L$, where $\delta$ is a $3\times3$ Kronecker delta kernel with a values of 1 at the diagonal elements and zero elsewhere. The Gaussian kernel is parametrized by standard deviation $\sigma$. * Implement functions ``` s = SharpeningKernel(alpha) ``` and ``` g = GaussianKernel(sigma, sz)``` that compute sharpening filter of size $3\times3$ and gaussian kernel of size $sz\times sz$ respectively (you computed gaussian kernel already in Exercise Session 2).<jupyter_code>def SharpeningKernel(alpha):
    l = np.ones((3, 3))
    
    ### YOUR CODE HERE
    l[0,:] = np.array([alpha/4, (1-alpha)/4, alpha/4])
    l[1,:] = np.array([(1-alpha)/4, -1, (1-alpha)/4])
    l[2,:] = np.array([alpha/4, (1-alpha)/4, alpha/4])
    
    l = l * 4 / (1+alpha)
    
    delta = np.zeros((3, 3))
    delta[1,1] = 1
    
    s = delta - l
    
    return s

### We implemented gaussian already multiple times, 
### so you can feel free to use the following snippet:
def GaussianKernel(sigma,sz):
    x, y = np.mgrid[-sz//2 + 1:sz//2 + 1, -sz//2 + 1:sz//2 + 1]
    g = np.exp(-((x**2 + y**2)/(2.0*sigma**2)))
    g = g/g.sum()
    
    return g<jupyter_output><empty_output><jupyter_text>* Implement a function ``` filteredImage = FilterRegions(origImage,binaryForeground, alpha, sigma)``` that smooths the background and sharpens the objects in the foreground.(You can use `conv.convolve2d` function to do convolution) Outputs:* ```filteredImage``` - original image with sharpened foreground and smoothed background Inputs:* ```origImage``` - original image* ```binaryForeground``` - binary mask with foreground pixels value equal to 1* ```alpha``` - sharpening parameter* ```sigma``` - smoothing parameter<jupyter_code>def FilterRegions(origImage, binaryForeground, alpha, sigma, sz):
    ### YOUR CODE HERE
    s = SharpeningKernel(alpha)
    g = GaussianKernel(sigma, sz)
    
    filteredImage = conv.convolve2d(origImage, g, mode='same')
    sharpenedImage = conv.convolve2d(origImage, s, mode='same')
    
    filteredImage[binaryForeground == 1] = sharpenedImage[binaryForeground == 1]
    
    return filteredImage<jupyter_output><empty_output><jupyter_text>* Visualize 3 images by focusing on different objects detected in the image. Use sharpnening filter of size $3\times3$ and $\alpha = 0.5$ and Gaussian kernel of size $7\times7$ and $\sigma = 2.0$.<jupyter_code>for i in range(np.amax(cluster_assignments) + 1):
    binaryImg = cluster_assignments == i
    filteredImg = FilterRegions(img,binaryImg, 0.5, 2, 7)
    plt.figure()
    plt.imshow(filteredImg,cmap='gray')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10421267893660531, 'fraction_numerical': 0.017914110429447853, 'mean_word_length': 3.8786911412609735, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '427958', 'license': 'MIT', 'n_tokens_mistral': 3729, 'n_tokens_neox': 3508, 'n_words': 1475, 'path': 'Image Segmentation & Sharpening/kmeans_sharpening.ipynb', 'repo_name': 'PhilipLGQ/CS442-Computer-Vision', 'size': '12225'}