<jupyter_start><jupyter_text>Self-Driving Car Engineer Nanodegree Project: **Finding Lane Lines on the Road** ***In this project, the lanes on the road are detacted using Canny Edge Dectection and Hough Transform line detection. Meanwhile, I also use HSL color space, grayscaling, color selection ,color selection and Gaussian smoothing to reduce noise in pictures and vedios. To achieve optimal performance, this detection code is with memory of lanes in previous frames so the result is smooth. The code is verified by pictures and vedios. The code has good performance in challenge vedio, which has curved lane and shadow on the ground. All picture results are in folder 'test_image_output'. Vedio outputs are in 'test_vedios_output'. Example picture output:---      Original Image           Lane Detaction Result    Python Code:  Import Packages<jupyter_code>#importing some useful packages
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import cv2
from scipy import stats
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read in an Image<jupyter_code>#reading in an image
image_sWR = mpimg.imread('test_images/solidWhiteRight.jpg')

#printing out some stats.
print('This image is:', type(image_sWR), 'with dimensions:', image_sWR.shape)<jupyter_output>This image is: <class 'numpy.ndarray'> with dimensions: (540, 960, 3)<jupyter_text>Some important functions:`find_hough_lines` Seperate left lane and right lane  `linear_regression_left/linear_regression_right` Use linear regression to extrapolate lanes  `create_lane_list` Use deque to store previous lanes  Lane finding functions<jupyter_code>import math
from collections import deque


def find_hough_lines(img,lines):
    # Seperate left/right lanes
    xl = []
    yl = []
    xr = []
    yr = []
    middel_x = img.shape[1]/2
    
    for line in lines:
        for x1,y1,x2,y2 in line:
            if ((y2-y1)/(x2-x1))<0 and ((y2-y1)/(x2-x1))>-math.inf and x1<middel_x and x2<middel_x:
                xl.append(x1)
                xl.append(x2)
                yl.append(y1)
                yl.append(y2)

            elif ((y2-y1)/(x2-x1))>0 and ((y2-y1)/(x2-x1))<math.inf and x1>middel_x and x2>middel_x:
                xr.append(x1)
                xr.append(x2)
                yr.append(y1)
                yr.append(y2)
    
    return xl, yl, xr, yr

def linear_regression_left(xl,yl):
    # Extrapolate left lane
    slope_l, intercept_l, r_value_l, p_value_l, std_err = stats.linregress(xl, yl)
    return slope_l, intercept_l

def linear_regression_right(xr,yr):
    # Extrapolate right lane
    slope_r, intercept_r, r_value_r, p_value_r, std_err = stats.linregress(xr, yr)
    return slope_r, intercept_r

def create_lane_list():
    # Use deque to store previous lanes
    return deque(maxlen = 15)

def left_lane_mean(left_lane_que):
    # Derive mean parameters of left lane based on memory
    if len(left_lane_que) == 0:
        return 0,0
    slope_l_mean , intercept_l_mean = np.mean(left_lane_que,axis=0)
    return slope_l_mean, intercept_l_mean

def right_lane_mean(right_lane_que):
    # Derive mean parameters of right lane based on memory
    if len(right_lane_que) == 0:
        return 0,0
    slope_r_mean , intercept_r_mean = np.mean(right_lane_que,axis=0)
    return slope_r_mean, intercept_r_mean

def left_lane_add(left_lane_que,slope_l, intercept_l):
    # Add left lane to memory
    left_lane_que.append([slope_l,intercept_l])
    return left_lane_que

def right_lane_add(right_lane_que,slope_r, intercept_r):
    # Add right lane to memory
    right_lane_que.append([slope_r,intercept_r])
    return right_lane_que
        

def grayscale(img):
    # Convert image to grayscale
    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    
def canny(img, low_threshold, high_threshold):
    #Applies the Canny transform
    return cv2.Canny(img, low_threshold, high_threshold)

def gaussian_blur(img, kernel_size):
    #Applies a Gaussian Noise kernel
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

def region_of_interest(img):
    # Defining a blank mask to start with
    mask = np.zeros_like(img)   
    
    # Defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255
    vertices = get_vertices_for_img(img)  
    # Filling pixels inside the polygon defined by "vertices" with the fill color    
    cv2.fillPoly(mask, vertices, ignore_mask_color)
    
    # Returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)
    return masked_image           

def draw_lines(img, intercept_l, slope_l,intercept_r, slope_r, xl, xr,color=[255, 0, 0], thickness=10):
    # Draw lines based on mean intercept and slope
    max_y = img.shape[0]
    yl_LR = []
    yr_LR = []
    for x in xl:
        yl_LR.append(intercept_l+slope_l*x)
    for x in xr:
        yr_LR.append(intercept_r+slope_r*x)
        
    x_left_bottom = (max_y - intercept_l)/slope_l
    x_right_bottom = (max_y - intercept_r)/slope_r
    
    cv2.line(img, (int(x_left_bottom), int(max_y)), (int(max(xl)), int(min(yl_LR))), color, thickness)
    cv2.line(img, (int(x_right_bottom), int(max_y)), (int(min(xr)), int(min(yr_LR))), color, thickness)
    
    return img
    
    
    
def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap):
    # Derive Hough lines of the image, this would return the points on the edge
    lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
    line_img = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
    
    return line_img, lines


def weighted_img(img, initial_img, α=0.8, β=1., γ=0.):
    # Combine images with weights
    return cv2.addWeighted(initial_img, α, img, β, γ)

def isolate_yellow_hsl(img):
    # Extract yellow color in the HSL color space. 
    # We are interested in the yellow lanes on the ground
    low_threshold = np.array([15, 38, 115], dtype=np.uint8)
    high_threshold = np.array([35, 204, 255], dtype=np.uint8)  
    
    yellow_mask = cv2.inRange(img, low_threshold, high_threshold)
    
    return yellow_mask
                            


def isolate_white_hsl(img):
    # Extract white color in the HSL color space. 
    # We are interested in the white lanes on the ground
    low_threshold = np.array([0, 200, 0], dtype=np.uint8)
    high_threshold = np.array([180, 255, 255], dtype=np.uint8)  
    
    white_mask = cv2.inRange(img, low_threshold, high_threshold)
    
    return white_mask

def get_vertices_for_img(img):
    # Get the top points of polygon based on the size of image for function 'region_of_interest'
    height = img.shape[0]
    width = img.shape[1]

    
    if (width, height) == (960, 540):
        bottom_left = (130 ,img.shape[0] - 1)
        top_left = (410, 330)
        top_right = (650, 350)
        bottom_right = (img.shape[1] - 30,img.shape[0] - 1)
        vert = np.array([[bottom_left , top_left, top_right, bottom_right]], dtype=np.int32)
    else:
        bottom_left = (200 , 680)
        top_left = (600, 450)
        top_right = (750, 450)
        bottom_right = (1100, 680)
        vert = np.array([[bottom_left , top_left, top_right, bottom_right]], dtype=np.int32)
    return vert<jupyter_output><empty_output><jupyter_text>Test ImagesFirstly, use images to test the lane detection piplane<jupyter_code>import os
# Read in a image list
test_img_dir = 'test_images/'
test_image_names = os.listdir("test_images/")
test_image_names = list(map(lambda name: test_img_dir + name, test_image_names))<jupyter_output><empty_output><jupyter_text>Build a Lane Finding Pipeline Build the pipeline and run your solution on all test_images. Make copies into the `test_images_output` directory, and you can use the images in your writeup report.Try tuning the various parameters, especially the low and high Canny thresholds as well as the Hough lines parameters.<jupyter_code># Read in images
image_wCLS = mpimg.imread('test_images/whiteCarLaneSwitch.jpg')
image_sYL = mpimg.imread('test_images/solidYellowLeft.jpg')
image_sYC2 = mpimg.imread('test_images/solidYellowCurve2.jpg')
image_sYC = mpimg.imread('test_images/solidYellowCurve.jpg')
image_sWC = mpimg.imread('test_images/solidWhiteCurve.jpg')
image_ch = mpimg.imread('test_images/challenge.jpg')

def Lane_Detect(image):
    # Lane detection pipeline
    image_hsl = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
    image_yellow = isolate_yellow_hsl(image_hsl)
    image_white = isolate_white_hsl(image_hsl)
    # Combine white parts and yellow parts in a single pic
    image_wy = cv2.bitwise_or(image_yellow,image_white)
    # Combine yellow and white masks and original picture to derive the parts we are interested.
    # This would reduce the noise and improve the performance if there is shadow on the ground.
    image_com = cv2.bitwise_and(image,image,mask=image_wy)
    image_gray = grayscale(image_com)
    # Smoothing the image
    kernal_size = 11
    blur_image = cv2.GaussianBlur(image_gray,(kernal_size,kernal_size),0)
    # Setup Canny
    low_threshold = 10
    high_threshold = 150
    edges_image = cv2.Canny(blur_image,low_threshold,high_threshold)
    # Define range of interest
    masked_image = region_of_interest(edges_image) 
    
    bland_image, houghLines= hough_lines(masked_image, 1, np.pi/180, 1, 5, 1)
    xl,yl,xr,yr = find_hough_lines(bland_image,houghLines)
    slope_l, intercept_l = linear_regression_left(xl,yl)
    slope_r, intercept_r = linear_regression_right(xr,yr)
    hough_image = draw_lines(bland_image, intercept_l, slope_l, intercept_r, slope_r, xl, xr)
    Final_image = weighted_img(hough_image,image)  
    return Final_image


# Process images and save
Final_wCLS = Lane_Detect(image_wCLS)
plt.imsave('test_images_output/whiteCarLaneSwitch.png',Final_wCLS)
Final_sWR = Lane_Detect(image_sWR)
plt.imsave('test_images_output/solidWhiteRight.png',Final_sWR)
Final_sYL = Lane_Detect(image_sYL)
plt.imsave('test_images_output/solidYellowLeft.png',Final_sYL)
Final_sYC2 = Lane_Detect(image_sYC2)
plt.imsave('test_images_output/solidYellowCurve2.png',Final_sYC2)
Final_sYC = Lane_Detect(image_sYC)
plt.imsave('test_images_output/solidYellowCurve.png',Final_sYC)
Final_sWC = Lane_Detect(image_sWC)
plt.imsave('test_images_output/solidWhiteCurve.png',Final_sWC)
Final_ch = Lane_Detect(image_ch)
plt.imsave('test_images_output/challenge.png',Final_ch)<jupyter_output><empty_output><jupyter_text>Test on Videos<jupyter_code># Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
# Set threshold to decide if the lane should be add to memory
MAXIMUM_SLOPE_DIFF = 0.1
MAXIMUM_INTERCEPT_DIFF = 50.0

class LaneDetectWithMemo:
    def __init__(self):
        self.left_lane_que = create_lane_list()
        self.right_lane_que = create_lane_list()
        
    def LanePipe(self,image):
        
        image_hsl = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
        image_yellow = isolate_yellow_hsl(image_hsl)
        image_white = isolate_white_hsl(image_hsl)
        # Combine white parts and yellow parts in a single pic
        image_wy = cv2.bitwise_or(image_yellow,image_white)
        # Combine yellow and white masks and original picture to derive the parts we are interested.
        # This would reduce the noise and improve the performance if there is shadow on the ground.
        image_com = cv2.bitwise_and(image,image,mask=image_wy)
        image_gray = grayscale(image_com)
        # Smoothing the image
        kernal_size = 11
        blur_image = cv2.GaussianBlur(image_gray,(kernal_size,kernal_size),0)
        # Setup Canny
        low_threshold = 10
        high_threshold = 150
        edges_image = cv2.Canny(blur_image,low_threshold,high_threshold)
        # Define range of interest
        masked_image = region_of_interest(edges_image) 
        bland_image, houghLines= hough_lines(masked_image, 1, np.pi/180, 1, 5, 1)
        xl,yl,xr,yr = find_hough_lines(bland_image,houghLines)
        slope_l, intercept_l = linear_regression_left(xl,yl)
        slope_r, intercept_r = linear_regression_right(xr,yr)
        # If the lane diverges too much, then use the mean value in memory to draw the lane
        # If the lane is within thershold, then add it to memory and recalculate the mean value
        if len(self.left_lane_que) == 0 and len(self.right_lane_que) == 0:
            self.left_lane_que = left_lane_add(self.left_lane_que, slope_l, intercept_l)
            self.right_lane_que = right_lane_add(self.right_lane_que, slope_r, intercept_r)
            slope_l_mean, intercept_l_mean = left_lane_mean(self.left_lane_que)
            slope_r_mean, intercept_r_mean = right_lane_mean(self.right_lane_que)
        else:
            slope_l_mean, intercept_l_mean = left_lane_mean(self.left_lane_que)
            slope_r_mean, intercept_r_mean = right_lane_mean(self.right_lane_que)
            slope_l_diff = abs(slope_l-slope_l_mean)
            intercept_l_diff = abs(intercept_l-intercept_l_mean)
            slope_r_diff = abs(slope_r-slope_r_mean)
            intercept_r_diff = abs(intercept_r-intercept_r_mean)
            if intercept_l_diff < MAXIMUM_INTERCEPT_DIFF and slope_l_diff < MAXIMUM_SLOPE_DIFF:
                self.left_lane_que = left_lane_add(self.left_lane_que, slope_l, intercept_l)
                slope_l_mean, intercept_l_mean = left_lane_mean(self.left_lane_que)
            if intercept_r_diff < MAXIMUM_INTERCEPT_DIFF and slope_r_diff < MAXIMUM_SLOPE_DIFF:
                self.right_lane_que = right_lane_add(self.right_lane_que, slope_r, intercept_r)
                slope_r_mean, intercept_r_mean = right_lane_mean(self.right_lane_que)
                
        
        hough_image = draw_lines(bland_image, intercept_l_mean, slope_l_mean,intercept_r_mean, slope_r_mean, xl, xr)
        Final_image = weighted_img(hough_image,image)
        
        return Final_image

# Test on the first vedio, with solid white lane on the right
LaneDetect_1 = LaneDetectWithMemo()
white_output = 'test_videos_output/solidWhiteRight.mp4'
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(LaneDetect_1.LanePipe) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)
HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))
# Now for the one with the solid yellow lane on the left. This one's more tricky!
LaneDetect_2 = LaneDetectWithMemo()
yellow_output = 'test_videos_output/solidYellowLeft.mp4'
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(LaneDetect_2.LanePipe)
%time yellow_clip.write_videofile(yellow_output, audio=False)
HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))<jupyter_output><empty_output><jupyter_text>Optional ChallengeThis vedio has curved lane and shadow on the ground. In the futrue I would use polynomial to represent the lane instead of a single line. The shadow is improved by extracting yellow and white in the picture and combine them with the original image, which represent the parts we are interested<jupyter_code>LaneDetect_ch = LaneDetectWithMemo()
challenge_output = 'test_videos_output/challenge.mp4'
clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(LaneDetect_ch.LanePipe)
%time challenge_clip.write_videofile(challenge_output, audio=False)
HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08795218622208241, 'fraction_numerical': 0.01774142812205096, 'mean_word_length': 3.959750390015601, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '20061', 'license': 'MIT', 'n_tokens_mistral': 5730, 'n_tokens_neox': 5384, 'n_words': 1321, 'path': 'P1.ipynb', 'repo_name': 'wpiHWzhao/Lane-Find-P1', 'size': '15895'}