<jupyter_start><jupyter_text>Introduction to Computer VisionThis notebook will give an introduction to computer vision fundamentals grounded in the corresponding mathematical concepts from geometry and algebra. What is Computer Vision?To start off, perhaps it would be helpful to discuss what exactly is computer vision? Different people use this broad term for different things, but broadly computer vision involving designing algorithms and systems that take in visual inputs to derive some useful understanding of the 3D world.The earliest work in this field was labeled under the banner of "image processing," working with 2D images from cameras. By the late 1960's, you see research robots like [SRI's Shaky](https://en.wikipedia.org/wiki/Shakey_the_robot) using analog television cameras to do basic tasks. Many early image processing advances were made in medical applications. In particular the invention of CT scans, which use 2D x-ray images from multiple angles to reconstruct a 3D scan, earned <NAME> and <NAME> the [1979 Nobel Prize in Physiology or Medicine](https://www.nobelprize.org/prizes/medicine/1979/summary/).As cameras became better, cheaper, and more ubiquitous, computer vision has become a part of our everyday lives. From smartphones to cars, more and more products are integrating vision based algorithms. Furthermore, computer vision has found a key role in factories and industrial applications for driving robots and inspection systems.Moving forward, the emergence of new types of sensors such as LIDAR and light-field cameras capable of directly capturing 3D representations of the world are driving new algorithms and use cases. More than any other field, computer vision has embraced deep learning and neural networks as a tool for tackling challenging classification and segmentation tasks. This is an exciting time for computer vision, so let's jump right in!We will be using a Jupyter notebook for this course. Select the code block and hit Shift+Enter to run it and import all the Python libraries we will need.<jupyter_code>%matplotlib inline
import numpy as np
import scipy.signal
import skimage.color
import cv2

from mpl_toolkits import mplot3d
import matplotlib.pyplot as plt
from matplotlib import collections
from mpl_toolkits.mplot3d import art3d

print("Import done!")

! git clone https://github.com/Tino-FRC-2473/CVTraining repo_files<jupyter_output><empty_output><jupyter_text>Part 1: ImagesBefore we can start talking about image processing, let's take a look at how 2D images are stored and intepreted by computers. At the simplest level, a digital image is a grid of values. For images from a camera, these values usually represent brightness of some kind and are called *pixels*. Raw images are integer valued, as in each value is stored as a positive integer. The range of values is set by their *bit-depth*, with the number of bits per pixel setting the number of levels that can be stored. For instance, the pixels of an 8-bit image can have values in the range $[0, 2^8]$, or 0-255.  1.1 Grayscale ImagesLet's take a look at an actual image. We'll start with grayscale image, which have a single value or channel per pixel and so have no color information.<jupyter_code>! curl https://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Abraham_Lincoln_O-77_matte_collodion_print.jpg/192px-Abraham_Lincoln_O-77_matte_collodion_print.jpg -o lincoln.jpg

lincoln = cv2.imread('lincoln.jpg', cv2.IMREAD_GRAYSCALE)
print(lincoln)
plt.imshow(lincoln, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>For performing work on images, it is often convenient to convert the integer valued image into a normalized, floating point image that takes values from 0.0-1.0. You can think of this as representing a percentage brightness in each pixel, and has some advantages when combining or doing math on images.<jupyter_code>print("Integer range:", lincoln.min(), "to", lincoln.max())

lincoln_float = lincoln.astype(float) / 255
print(lincoln_float)
print("Float range:", lincoln_float.min(), "to", lincoln_float.max())<jupyter_output><empty_output><jupyter_text>One thing to note is that the origin of the image is at the top right. This means positive x is to the right, and positive y is to the bottom. In addition, the image array is addressed first with y, then with x. This convention is a byproduct of the beam scan pattern of analog CRT televisions. As a result, it may be helpful to think about indexing discrete images as `image[row, column]`.  ExerciseTo see how this works in practice, let's write some simple functions for flipping an image horizontally and vertically. Hint: remember how Numpy slicing works with a negative step count!<jupyter_code>def flip_horizontal(image):
  return image[:, ::-1]

def flip_vertical(image):
  # WRITE CODE HERE
  return image
  # END CODE
plt.imshow(flip_horizontal(lincoln), cmap='gray')
plt.title("Flip horizontal")

plt.figure()
plt.imshow(flip_vertical(lincoln), cmap='gray')
plt.title("Flip vertical")
plt.show()<jupyter_output><empty_output><jupyter_text>1.2 Point operationsNow that we have some image data, let's do something with it. The simplest operations transform an image one pixel at a time. These are commonly called point operations since they only depend on each individual point in an image.Let's take a look at what happens when we shift or scale pixel values. Of course, if a remapped value goes outside the acceptable range of values, we need to clamp it.<jupyter_code>def clamp(float_image):
  return float_image.clip(0.0, 1.0)

plt.figure()
plt.title("Original")
plt.imshow(clamp(lincoln_float), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Scale by 1.5")
plt.imshow(clamp(lincoln_float * 1.5), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Scale by 0.5")
plt.imshow(clamp(lincoln_float * 0.5), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Shift by 0.5")
plt.imshow(clamp(lincoln_float + 0.5), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Shift by -0.5")
plt.imshow(clamp(lincoln_float - 0.5), cmap='gray', vmin=0.0, vmax=1.0)

plt.show()<jupyter_output><empty_output><jupyter_text>Notice how multiplying by a constant stretches out or squeezes differences between low and high values. This is why contant multiplication is often used to implement contrast adjustment. In the same way, a constant shift is used to implement brightness adjustment.To better see this, we can create a histogram of the values using `plt.hist`. Note that we have to flatten the image first, since we want to look at all of the pixels together rather than row by row.<jupyter_code>plt.figure()
plt.title("Original histogram")
plt.hist(lincoln_float.flatten(), bins=20, range=(0, 1))

plt.figure()
plt.title("Scale by 1.5 histogram")
plt.hist(clamp(lincoln_float * 1.5).flatten(), bins=20, range=(0, 1))

plt.figure()
plt.title("Shift by 0.5 histogram")
plt.hist(clamp(lincoln_float * 0.5).flatten(), bins=20, range=(0, 1))

plt.show()<jupyter_output><empty_output><jupyter_text>ExerciseOf course, the point operation can be any function that takes in a pixel value and outputs a different pixel value. Let's try implementing a shadow boost function, that keeps pixels brighter than a threshold the same, but multiplies the brightness of pixels darker than the threshold by some constant.<jupyter_code>def shadow_boost(image, threshold, scale):
  # WRITE CODE HERE
  return image
  # END CODE

plt.figure()
plt.title("Original")
plt.imshow(clamp(lincoln_float), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Shadow Boost")
plt.imshow(clamp(shadow_boost(lincoln_float, 0.4, 2)), cmap='gray', vmin=0.0, vmax=1.0)

plt.figure()
plt.title("Shadow Boost Histogram")
plt.hist(clamp(shadow_boost(lincoln_float, 0.4, 2)).flatten(), bins=20, range=(0, 1))
plt.show()<jupyter_output><empty_output><jupyter_text>1.3 Linear Filtering and ConvolutionPoint operations are great for doing bulk adjustments across the image, but they only consider one pixel at a time. The next major class of operations operate on groups of pixels. The way this works is that we remap each pixel to some function of the pixel values in its viscinity. The most commonly used subset of these are called linear filters, where the output values is a linear combination of the pixels around a each input pixel. This is usually done using the operation of convolution. A small mask or *kernel* of scaling factors is overlaid on each pixel, and the sum of the products of this mask with the underlying pixel values is used as the output pixel value. This filter kernel is then slid across the image to get new values for each pixel.![Convolution](https://www.pyimagesearch.com/wp-content/uploads/2016/06/convolutions_kernel_sliding.jpg)Let's make this more concrete with an example. Let's blur an image by taking the average of all the pixels in a 3x3 area around each input pixel. There are nine pixels in this area, so the output value will be the sum of all the pixels in this 3x3 neighborhood divided by 9. Alternatively, we each pixel contributes 1/9 times its value to end output value.<jupyter_code>kernel = np.ones((3,3)) / 9
print(kernel)<jupyter_output><empty_output><jupyter_text>Now we use the convolve2D function from scipy to convolve an image with this filter.<jupyter_code>blurred = scipy.signal.convolve2d(lincoln_float, kernel)
plt.imshow(blurred, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>ExerciseOf course, we can get an even smoother blur by average across a larger area. Try writing a function that generates larger blur filter kernels, and see what happens. This filter is commonly known as a box blur.<jupyter_code>def box_blur_kernel(size):
  # WRITE CODE HERE
  return
  # END CODE

plt.figure()
plt.title("Original")
plt.imshow(lincoln_float, cmap='gray')
plt.show()

plt.figure()
plt.title("Size 5 blur")
plt.imshow(clamp(scipy.signal.convolve2d(lincoln_float, box_blur_kernel(5))), cmap='gray')
plt.show()

plt.figure()
plt.title("Size 7 blur")
plt.imshow(clamp(scipy.signal.convolve2d(lincoln_float, box_blur_kernel(7))), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>1.4 Edge DetectionOne major use of linear filtering is for edge detection. Fundamentally, an edge is a place where the pixel values changes sharply. In math terms, the derivative at that pixel is very high. If you remember from calculus, for a discrete function you can approximate the derivative by taking the difference over a small distance.Let's construct a filter that adds up the differences between the target pixel and all its neighbors.<jupyter_code>kernel = np.array([[-1,-1,-1],[-1,8,-1],[-1,-1,-1]])
print(kernel)
filtered = scipy.signal.convolve2d(lincoln_float, kernel)
plt.imshow(clamp(filtered), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>As a bonus, what happens when we highlight these pixels on top of the original image? We can implement this by adding an extra copy of the original pixel value, so increase the mask's value at the anchor pixel's entry by 1. In photo editing software, this kind of kernel is often used for the "sharpen" operator.<jupyter_code>kernel = np.array([[-1,-1,-1],[-1,9,-1],[-1,-1,-1]])
print(kernel)
filtered = scipy.signal.convolve2d(lincoln_float, kernel)
plt.imshow(clamp(filtered), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Exercise Check<jupyter_code>print("CHECKING 1.1")
test_image = np.array([[0.1, 0.2, 0.3],
                       [0.4, 0.5, 0.6]])
assert(np.all(flip_vertical(test_image) == np.array([[0.4, 0.5, 0.6], [0.1, 0.2, 0.3]])))
print("PASS")

print("CHECKING 1.2")
assert(np.all(shadow_boost(test_image, 0.3, 2) == np.array([[0.2, 0.4, 0.6], [0.4, 0.5, 0.6]])))
print("PASS")

print("CHECKING 1.3")
assert(np.all(box_blur_kernel(1) == np.array([[1]])))
assert(np.all(box_blur_kernel(3) == np.array([[1.0 , 1.0, 1.0], [1.0 , 1.0, 1.0], [1.0 , 1.0, 1.0]]) / 9))
print("PASS")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09514288094643006, 'fraction_numerical': 0.020828126301757895, 'mean_word_length': 5.4021333333333335, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '55466', 'license': 'MIT', 'n_tokens_mistral': 3819, 'n_tokens_neox': 3554, 'n_words': 1460, 'path': 'IntroToCV_1_Images.ipynb', 'repo_name': 'Tino-FRC-2473/CVTraining', 'size': '12028'}